Posts

First Hand Experience of Question Difficulty

This is a follow-up to the last post – I want to expand on how the different aspects of our work fit together. (These two strands are hosted quiz nights and quiz question writing for TV shows, games, iPhone apps etc.)

Those have always been the two main areas of our business – over the years the hosted quizzes have taken the lead, certainly they’ve been more consistent. The question writing side obviously depends a little more on what comes along. I mean, we’re always writing questions, but we’re not always working on a major commission – more like bits and bobs here and there.

In the last couple of years, there’s been a lot of really good question writing work, so much so that there has been less time for our main question writers to run quizzes.

Yet, the experience of hosting quizzes is vital, I think, to our writing questions successfully.

I’ve run over 400 quizzes for people all over this country and occasionally overseas, for people of all ages, in different industries, for different purposes. I’ve asked questions on every topic that makes a good quiz question and a few that don’t.

And I get to see, first hand, how those questions go down. I get to see what people know and don’t know, what they’re proud to know and what they don’t care about knowing, what’s workoutable and what’s not.

And because our quizzes are for different clients, we get to re-use questions, so we know whether a response, positive or negative, is a one-off or not.

And that’s just me – between us, as a company, we’ve run over 3000 quizzes, and we ask our clients and our quiz masters to feed back on every event. So, we know very well if a question is a big hit or not.

This gives us a vital edge when it comes to question writing for TV, we think. To us, calibration, alongside entertainment, is more than guesswork. We have evidence to back up the fact that we know how to set quizzes, to write questions that people want to participate in and puzzle over.

It’s not just the hosted quizzes, either. There’s also the Friday Quiz, which started in 2008 and now goes out to thousands of people a week. Every week, I look at how people have done, how many people have bothered trying to answer each question, how many have got it right. This is vital information to understanding what people do and don’t know.

Anyone can reasonably think they’re an expert in quizzes, anyone who writes questions, participates in a lot, watches a lot, but we think our combined experience puts us in a privileged position. You’re left with egg on your face if you think you always know exactly how a question is going to be answered, but the numbers work themselves out.

We see hundreds, if not thousands, of people answering our questions. Most question writers only ever see one or two people answering questions they write, so they get very skewed calibration feedback.

We tell our quiz masters, when they run quizzes, that the right level involves the worst team not slipping much below 50% and the best team not getting above 90% – an ideal spread is between about 60% and 85%. And that’s what happens. Almost every time.

It’s not a naturally easy thing – the first round I ever set, which I was terribly proud of, the scores ranged between 6 and 11 out of 20. It was a disaster. The questions, in and of themselves, were mainly interesting enough, but they were all at the harder end of the scale, some of them weren’t possible to work out. Despite my love for quizzes and my concern for getting it right, I didn’t yet have the first-hand experience of getting the overall level right.

So, this is what we do. We host quizzes and we write questions. They feed into each other. Every question I’ve ever written and every question I’ve ever asked and seen answered feeds into how I write now.

Water Displacement

Just a very brief blog about one insignificant question, but hopefully a little insight into the thought processes of someone trying to constantly come up with good quiz questions.

There’s a squeaky door in my house. That’s where I got the idea for this. I remembered, as the door squeaked, that I wrote a question about four years ago asking “What does the WD in WD40 stand for?” The question has, to my knowledge, never been used.

For all these years, it’s sat quite near the top of the huge excel spreadsheet I keep of potential QuizQuizQuiz Friday Quiz questions, many times for my eyes to pass over it, consider it, then go “Nah, not this week”. It was in our database for Corporate Quizzes for  a couple of years but I don’t believe it was ever used. It is a question that has truly not made the grade.

Why not? Although we’re justifiably proud of the Friday Quiz and effort and thought is put into it every week, it would be untrue, I admit, to claim that every question in its seven-year history has been a top-quality thriller. There has been the odd bit of filler, and yet WD40 has never been deemed up to the job.

What’s wrong with it? It has some hallmarks of a good question. Word origin questions are usually very well received. And questions about the everyday are often very popular, questions about things which are everywhere and nowhere. Everyone uses WD40, probably very few people know what it means – the best one can hope for in that circumstance is that response of “aah, I’m surprised I didn’t know that, you learn something new everyday”. But I just don’t think it would get that response. It would get “Meh, who cares …”. I think “displacement” is somehow too dull and disconnected a word, it’s unsatisfactorily hard to work out. It would be the dampest of damp squibs. I think … many times I’ve been tempted to think I might be wrong, and that it might be a surprise hit. That’s the thing …we don’t always know. Sometimes questions we think will be great don’t work, and sometimes seemingly dull, nondescript questions get an enthusiastic response.

If I’m wrong about this one, let me know. If you’re there, going “Wow, I use WD-40 all the time and I’d never considered what it stood for. Thanks, QuizQuizQuiz. The fact that it’s so named because it was the 40th attempt to make a formula for Water Displacement is one of the great hidden gems of the quiz world”, well, more fool me.

Of course, any life the question might have had I’ve surely killed now.