First Hand Experience of Question Difficulty

This is a follow-up to the last post – I want to expand on how the different aspects of our work fit together. (These two strands are hosted quiz nights and quiz question writing for TV shows, games, iPhone apps etc.)

Those have always been the two main areas of our business – over the years the hosted quizzes have taken the lead, certainly they’ve been more consistent. The question writing side obviously depends a little more on what comes along. I mean, we’re always writing questions, but we’re not always working on a major commission – more like bits and bobs here and there.

In the last couple of years, there’s been a lot of really good question writing work, so much so that there has been less time for our main question writers to run quizzes.

Yet, the experience of hosting quizzes is vital, I think, to our writing questions successfully.

I’ve run over 400 quizzes for people all over this country and occasionally overseas, for people of all ages, in different industries, for different purposes. I’ve asked questions on every topic that makes a good quiz question and a few that don’t.

And I get to see, first hand, how those questions go down. I get to see what people know and don’t know, what they’re proud to know and what they don’t care about knowing, what’s workoutable and what’s not.

And because our quizzes are for different clients, we get to re-use questions, so we know whether a response, positive or negative, is a one-off or not.

And that’s just me – between us, as a company, we’ve run over 3000 quizzes, and we ask our clients and our quiz masters to feed back on every event. So, we know very well if a question is a big hit or not.

This gives us a vital edge when it comes to question writing for TV, we think. To us, calibration, alongside entertainment, is more than guesswork. We have evidence to back up the fact that we know how to set quizzes, to write questions that people want to participate in and puzzle over.

It’s not just the hosted quizzes, either. There’s also the Friday Quiz, which started in 2008 and now goes out to thousands of people a week. Every week, I look at how people have done, how many people have bothered trying to answer each question, how many have got it right. This is vital information to understanding what people do and don’t know.

Anyone can reasonably think they’re an expert in quizzes, anyone who writes questions, participates in a lot, watches a lot, but we think our combined experience puts us in a privileged position. You’re left with egg on your face if you think you always know exactly how a question is going to be answered, but the numbers work themselves out.

We see hundreds, if not thousands, of people answering our questions. Most question writers only ever see one or two people answering questions they write, so they get very skewed calibration feedback.

We tell our quiz masters, when they run quizzes, that the right level involves the worst team not slipping much below 50% and the best team not getting above 90% – an ideal spread is between about 60% and 85%. And that’s what happens. Almost every time.

It’s not a naturally easy thing – the first round I ever set, which I was terribly proud of, the scores ranged between 6 and 11 out of 20. It was a disaster. The questions, in and of themselves, were mainly interesting enough, but they were all at the harder end of the scale, some of them weren’t possible to work out. Despite my love for quizzes and my concern for getting it right, I didn’t yet have the first-hand experience of getting the overall level right.

So, this is what we do. We host quizzes and we write questions. They feed into each other. Every question I’ve ever written and every question I’ve ever asked and seen answered feeds into how I write now.

Questions about Ed Balls

When a blog falls silent, it’s usually either a good or bad sign. Thankfully, in this case, it’s the former. We’ve been BusyBusyBusy rather than QuietQuietQuiet (sorry, that’s terrible …).

I’ve been writing, rather than hosting, a lot – almost exclusively. in fact. This blog has had three main purposes since it began – 1. (being honest) to help bring traffic to our website 2. to provide specific information on our quiz nights for our clients and 3. to just be informative and a bit of fun while being a bit of an authority on all things quiz.

A lot of my posts over the last few years have been about the joys and pitfalls of running quiz nights, and, as I say, they’ve served as places to point a client about the way our quizzes work. Until last week, though, I hadn’t run a quiz for about 9 months, so I just didn’t feel inspired to be writing all that much about quiz nights (as well as the fact I’ve written over 100 previous posts and I’d run the risk of repeating myself).

The writing work has been good – interesting, creative, exactly the kind of work we want to be doing. For me, it’s also often quite solitary, and a world away from the quiz nights. The atmosphere at quiz nights varies, but they do very often turn into loud and raucous mass participation events, which appear to be barely on the edge of control (though in reality we are always in control!). The best ones do, anyway.

For the last year, though, I’ve more often been in my special sound-proof QuizQuizQuiz shed trying to construct quiz questions/rounds/shows as if they’re haikus hewn from the very core of language and knowledge. Who knows, maybe sometimes they are …

Anyway, what’s my point? (I’m out of practice at writing blogs with a point.) Just that it’s a big quiz world and getting bigger. Gosh, some of those quizzers are turning into rock stars, as this rather good  documentary claimed. Even our own director, Jack, has been on the radio talking about the whole quiz thing (among other things) on ‘The Museum of Curiosity‘. It’s a broad church.

For me, as a quiz writer, the essence is now boiled down to knowing what people know. I’m good at that now. Whichever people, in whatever setting, whether online, on a TV show, in a room, in a pub, that’s a skill I’ve got. It’s far from faultless, though. There’s as much joy in someone unexpectedly knowing something you thought would stump them, as there is despair in people using neither knowledge nor knowhow, and failing miserably when you least expect it.

Quizzes should always reward knowledge and knowhow – it’s a bit of a shame when people apply good reasoning to a question and still get it wrong. That applies to any quiz situation.

For some reason, this year, I’ve written a lot of questions, often in completely different contexts, about Ed Balls. Currently no man alive lends themselves better to slightly comical quiz questions. Thank you Ed Balls. And as my own little tribute to Ed Balls Day … Ed Balls.

I ran a quiz last week – a big old quiz for 200 people in a bar in London – an old routine I’d fallen out of but thankfully fell back into pretty quickly. My joy for the last year has been applying a fair bit of science and a little bit of art to question writing, initially on my own, then in close, limited collaboration. However, last week I remembered the joy of playing ‘Sound of da Police’ at high volume to a room full of tipsy but fiercely competitive business-folk, and, of course, I remembered the age-old rush of saying “And the year when they were all Number 1 is Nineteen …. ninety ……………. nine”


Creativity – perhaps as overused and meaningless a term these days as “interactive”, “passionate” or “110%”. I remember, for one of the first big question-writing projects I worked on for QuizQuizQuiz, shuffling with my colleague into the imposing offices of a large multinational firm who was our client, and being introduced to the various serious and important people there as “the creative” … I’m the creative, am I? If only I’d known …

It can vary how much creativity this job involves. If I read in the news that Leicester City have won the Premier League (I know, a ridiculous thought, but just as an example …), and then write the question “Who won the English Premier League in 2015-16?”, I accept that is not the very height of creative endeavour. Plenty of question writing is like that. You see simple facts and you package them into questions. In particular, this is the case with high-volume multiple choice, multi-level question writing, against a deadline.

We’ve had to write 20,000 Multiple Choice questions from scratch in a couple of months, with a very tight word limit on each question. There is not much room for anything but the barest form of creativity. But it’s still possible to get some satisfaction and show a little flair, usually in wrong answer options on easy questions. I think my favourite was “What follows this line in the Meredith Brooks song ‘Bitch?’ – “I’m a bitch, I’m a lover …”? to which one of the options was “My name’s Mitch, I’m your brother” … Well, you get your fun where you can.

Thus a lot of writing feels just as much reactive as creative. You take something that already exists and just reshape it. I try very hard not to use other people’s quiz questions. I’ve written before about how I get a certain bittersweet tang from seeing a really fine quiz question, knowing that it is not something that I will have the opportunity to think of myself. Indeed, I can’t use it. But I think it is acceptable to bank the facts in the question, and reshape it, a little while later, into something a bit different. If you couldn’t create quiz material from the same sources that other people create it  from, well, we’d all be done for.

We’ve been doing quite a lot of writing for TV in the last few years, and that certainly has plenty of scope for a satisfying creative process, be it trying to put together Hives for Hive Minds,  Only Connect sequences and connections (after 11 series, I sometimes think it’s amazing that we and the other writers are still able to come up with new material and, believe me, this requires digging deep into the well of resourcefulness and creativity) or, on The Code, nice sets of 3 answer/questions. We threw a few Easter eggs into The Code, little rhyming sequences or phrases, I spent a lot of time coming up with little nuggets of joy which only a few people spotted, but that’s part of the fun of it.

A huge amount of work can go into things which are still, at the end of the day, only quiz questions or quiz rounds. They’re not going to win any awards. But there is sometimes, dare I say it, a little of the rigour and discipline of poetry in writing a quiz round.

At our pub quiz, we used to have a round called Follow On (where each answer has one letter more than the previous) and another round called Blitz (30 quickfire questions, some of which were themed). For Christmas, we decided to write a Christmas-themed 30 question round where each answer was one letter longer than the previous answer, from 1 to 30. Frankly, I still consider it my finest hour … well, not hour, actually, but a week of writing … and five minutes of participation.

So, creativity, yes, I suppose this is a creative job. There have been many times down the years when we’ve had the opportunity to use a bit of imagination in our work. Anyone writing or running a quiz can mix it up, try new formats, be clever without being confusing. It should never become boring or a chore. We’re passionate about giving 110% to interactive, creative quizzery …

The Code – Trailer

Earlier this year we worked on the questions for ‘The Code’, a new BBC1 quiz show hosted by Matt Allwright, along with our own Lesley-Anne Brewis as the resident quiz expert. It starts on Monday 18th April, and is on every weekday at 2:15pm (and of course on iPlayer) until May 20th. We’ll post links to iPlayer on the blog once the series is underway. We hope you enjoy the show!