First Hand Experience of Question Difficulty

This is a follow-up to the last post – I want to expand on how the different aspects of our work fit together. (These two strands are hosted quiz nights and quiz question writing for TV shows, games, iPhone apps etc.)

Those have always been the two main areas of our business – over the years the hosted quizzes have taken the lead, certainly they’ve been more consistent. The question writing side obviously depends a little more on what comes along. I mean, we’re always writing questions, but we’re not always working on a major commission – more like bits and bobs here and there.

In the last couple of years, there’s been a lot of really good question writing work, so much so that there has been less time for our main question writers to run quizzes.

Yet, the experience of hosting quizzes is vital, I think, to our writing questions successfully.

I’ve run over 400 quizzes for people all over this country and occasionally overseas, for people of all ages, in different industries, for different purposes. I’ve asked questions on every topic that makes a good quiz question and a few that don’t.

And I get to see, first hand, how those questions go down. I get to see what people know and don’t know, what they’re proud to know and what they don’t care about knowing, what’s workoutable and what’s not.

And because our quizzes are for different clients, we get to re-use questions, so we know whether a response, positive or negative, is a one-off or not.

And that’s just me – between us, as a company, we’ve run over 3000 quizzes, and we ask our clients and our quiz masters to feed back on every event. So, we know very well if a question is a big hit or not.

This gives us a vital edge when it comes to question writing for TV, we think. To us, calibration, alongside entertainment, is more than guesswork. We have evidence to back up the fact that we know how to set quizzes, to write questions that people want to participate in and puzzle over.

It’s not just the hosted quizzes, either. There’s also the Friday Quiz, which started in 2008 and now goes out to thousands of people a week. Every week, I look at how people have done, how many people have bothered trying to answer each question, how many have got it right. This is vital information to understanding what people do and don’t know.

Anyone can reasonably think they’re an expert in quizzes, anyone who writes questions, participates in a lot, watches a lot, but we think our combined experience puts us in a privileged position. You’re left with egg on your face if you think you always know exactly how a question is going to be answered, but the numbers work themselves out.

We see hundreds, if not thousands, of people answering our questions. Most question writers only ever see one or two people answering questions they write, so they get very skewed calibration feedback.

We tell our quiz masters, when they run quizzes, that the right level involves the worst team not slipping much below 50% and the best team not getting above 90% – an ideal spread is between about 60% and 85%. And that’s what happens. Almost every time.

It’s not a naturally easy thing – the first round I ever set, which I was terribly proud of, the scores ranged between 6 and 11 out of 20. It was a disaster. The questions, in and of themselves, were mainly interesting enough, but they were all at the harder end of the scale, some of them weren’t possible to work out. Despite my love for quizzes and my concern for getting it right, I didn’t yet have the first-hand experience of getting the overall level right.

So, this is what we do. We host quizzes and we write questions. They feed into each other. Every question I’ve ever written and every question I’ve ever asked and seen answered feeds into how I write now.