Too True?

/1 Comment/in , /by

Just a quick thought.

Can you put too much effort into making sure a question satisfies everyone in its lack of ambiguity?

People want quiz questions to be perfectly clear with perfectly unambiguous answers, and rightly so, but it can sometimes be a little muddier than that.

Just as an example, if you watch Only Connect, you’ll notice that sometimes when teams give an answer Victoria Coren Mitchell asks them to explain why they’ve given their answer, and either accepts or doesn’t. They haven’t given the expected answer, but they have found a valid sequence or connection or at leas their own way to explain it (sometimes more specific or less so than envisaged). It doesn’t happen very often, but OC is perhaps more likely to have that issue than a standard quiz, perhaps, and requires everyone involved to be very much on their toes to keep up with the quick minds of the contestants.

But even in a standard quiz, whether hosted by a quiz master or on the page of a book or magazine or on a website or in an app, a question can be technically sound but people might still find what they think is reasonable cause for alternative answers.

I’m not talking about faulty questions really, or those which are poorly phrased.

There are different cases –

1. a question where the participant has a slightly different understanding of what the terms on the question mean, and by their terms, they have a different answer … an example might be something like the use of World Cup Finals or World Cup in a question – people might misunderstand what those terms mean.

2. a question might be technically correct but actually be too complex, so that most of what is in the question points to a certain answer, but there is a little detail which means that is not the right answer. It might be deemed a little unfair or tricksy – an example might be: “What film did this actor make with this director, in which he played this superhero opposite this actress, in 2002?” … when every detail might point to one answer except the year (there being two such films made in different years).

So, in each case the quiz master can reasonably enough say “No, sorry, your answer is wrong, the question is right and you’ve misunderstood it or not listened closely enough …”

Of course it always ought to be possible to add in plenty of wording and definitions – but where do you stop? Where do you draw the line? Is it worth the effort not to cover any eventuality, not to have to deal with a query?

It would seem that the obvious answer is “Yes” in most cases, but as a question writer, brevity, clarity and the form of the question are also very important, and sometimes you can get bogged down with caveats. So sometimes you take a judgement call to go for brevity and be armed with the facts for clarification. Or sometimes you have to just bite the bullet and bin a question because you can’t narrow it down sensibly enough. The key is to be aware of what you are doing.

It is always helpful for the quiz host (whether at a pub quiz or at a corporate quiz night or on a TV show) to have the facts at hand to put anyone straight if they are a little befuddled. It often actually enhances the host’s authority (as it does, for example, that of Victoria Coren Mitchell on Only Connect) to be able to say e.g. Interesting, but in fact THIS was in THIS COUNTRY, not THAT COUNTRY, so close but no cigar …”

So, for what it’s worth, I think the answer is “be fair and be armed with the facts”.

Reads like a rather good slogan …

 

First Hand Experience of Question Difficulty

/0 Comments/in , , , /by

This is a follow-up to the last post – I want to expand on how the different aspects of our work fit together. (These two strands are hosted quiz nights and quiz question writing for TV shows, games, iPhone apps etc.)

Those have always been the two main areas of our business – over the years the hosted quizzes have taken the lead, certainly they’ve been more consistent. The question writing side obviously depends a little more on what comes along. I mean, we’re always writing questions, but we’re not always working on a major commission – more like bits and bobs here and there.

In the last couple of years, there’s been a lot of really good question writing work, so much so that there has been less time for our main question writers to run quizzes.

Yet, the experience of hosting quizzes is vital, I think, to our writing questions successfully.

I’ve run over 400 quizzes for people all over this country and occasionally overseas, for people of all ages, in different industries, for different purposes. I’ve asked questions on every topic that makes a good quiz question and a few that don’t.

And I get to see, first hand, how those questions go down. I get to see what people know and don’t know, what they’re proud to know and what they don’t care about knowing, what’s workoutable and what’s not.

And because our quizzes are for different clients, we get to re-use questions, so we know whether a response, positive or negative, is a one-off or not.

And that’s just me – between us, as a company, we’ve run over 3000 quizzes, and we ask our clients and our quiz masters to feed back on every event. So, we know very well if a question is a big hit or not.

This gives us a vital edge when it comes to question writing for TV, we think. To us, calibration, alongside entertainment, is more than guesswork. We have evidence to back up the fact that we know how to set quizzes, to write questions that people want to participate in and puzzle over.

It’s not just the hosted quizzes, either. There’s also the Friday Quiz, which started in 2008 and now goes out to thousands of people a week. Every week, I look at how people have done, how many people have bothered trying to answer each question, how many have got it right. This is vital information to understanding what people do and don’t know.

Anyone can reasonably think they’re an expert in quizzes, anyone who writes questions, participates in a lot, watches a lot, but we think our combined experience puts us in a privileged position. You’re left with egg on your face if you think you always know exactly how a question is going to be answered, but the numbers work themselves out.

We see hundreds, if not thousands, of people answering our questions. Most question writers only ever see one or two people answering questions they write, so they get very skewed calibration feedback.

We tell our quiz masters, when they run quizzes, that the right level involves the worst team not slipping much below 50% and the best team not getting above 90% – an ideal spread is between about 60% and 85%. And that’s what happens. Almost every time.

It’s not a naturally easy thing – the first round I ever set, which I was terribly proud of, the scores ranged between 6 and 11 out of 20. It was a disaster. The questions, in and of themselves, were mainly interesting enough, but they were all at the harder end of the scale, some of them weren’t possible to work out. Despite my love for quizzes and my concern for getting it right, I didn’t yet have the first-hand experience of getting the overall level right.

So, this is what we do. We host quizzes and we write questions. They feed into each other. Every question I’ve ever written and every question I’ve ever asked and seen answered feeds into how I write now.

Questions about Ed Balls

/0 Comments/in , , , /by

When a blog falls silent, it’s usually either a good or bad sign. Thankfully, in this case, it’s the former. We’ve been BusyBusyBusy rather than QuietQuietQuiet (sorry, that’s terrible …).

I’ve been writing, rather than hosting, a lot – almost exclusively. in fact. This blog has had three main purposes since it began – 1. (being honest) to help bring traffic to our website 2. to provide specific information on our quiz nights for our clients and 3. to just be informative and a bit of fun while being a bit of an authority on all things quiz.

A lot of my posts over the last few years have been about the joys and pitfalls of running quiz nights, and, as I say, they’ve served as places to point a client about the way our quizzes work. Until last week, though, I hadn’t run a quiz for about 9 months, so I just didn’t feel inspired to be writing all that much about quiz nights (as well as the fact I’ve written over 100 previous posts and I’d run the risk of repeating myself).

The writing work has been good – interesting, creative, exactly the kind of work we want to be doing. For me, it’s also often quite solitary, and a world away from the quiz nights. The atmosphere at quiz nights varies, but they do very often turn into loud and raucous mass participation events, which appear to be barely on the edge of control (though in reality we are always in control!). The best ones do, anyway.

For the last year, though, I’ve more often been in my special sound-proof QuizQuizQuiz shed trying to construct quiz questions/rounds/shows as if they’re haikus hewn from the very core of language and knowledge. Who knows, maybe sometimes they are …

Anyway, what’s my point? (I’m out of practice at writing blogs with a point.) Just that it’s a big quiz world and getting bigger. Gosh, some of those quizzers are turning into rock stars, as this rather good  documentary http://www.bbc.co.uk/programmes/b084fs6s claimed. Even our own director, Jack, has been on the radio talking about the whole quiz thing (among other things) on ‘The Museum of Curiosity‘. It’s a broad church.

For me, as a quiz writer, the essence is now boiled down to knowing what people know. I’m good at that now. Whichever people, in whatever setting, whether online, on a TV show, in a room, in a pub, that’s a skill I’ve got. It’s far from faultless, though. There’s as much joy in someone unexpectedly knowing something you thought would stump them, as there is despair in people using neither knowledge nor knowhow, and failing miserably when you least expect it.

Quizzes should always reward knowledge and knowhow – it’s a bit of a shame when people apply good reasoning to a question and still get it wrong. That applies to any quiz situation.

For some reason, this year, I’ve written a lot of questions, often in completely different contexts, about Ed Balls. Currently no man alive lends themselves better to slightly comical quiz questions. Thank you Ed Balls. And as my own little tribute to Ed Balls Day … Ed Balls.

I ran a quiz last week – a big old quiz for 200 people in a bar in London – an old routine I’d fallen out of but thankfully fell back into pretty quickly. My joy for the last year has been applying a fair bit of science and a little bit of art to question writing, initially on my own, then in close, limited collaboration. However, last week I remembered the joy of playing ‘Sound of da Police’ at high volume to a room full of tipsy but fiercely competitive business-folk, and, of course, I remembered the age-old rush of saying “And the year when they were all Number 1 is Nineteen …. ninety ……………. nine”

Missing a trick

/0 Comments/in , /by

There’s a fine line between a tricky question and a trick question sometimes.

Some quiz masters may be happy to write trick questions, but I try to avoid them if I can. It can be a little bit of a grey area sometimes, though.

There are various negative reactions a question/round/quiz might get. Here are a few –

– this is boring
– ghastly celebrity/soap tittle-tattle
– how could we be expected to know that?
– I wasn’t even born then
– oh, how clever-clever
– this is frustratingly hard
– this is unchallengingly easy
– this is for an unfairly specific and exclusive audience which doesn’t include me
– yuk, sport
– yuk, any other subject

As a quiz master, you certainly don’t mind some kinds of “negative” reactions – you don’t mind that fizzle of “damn, you fiend, you fooled me there” or “oh, how could i have missed that, I hate myself”, but what you don’t want is “hmph, that’s silly and unfair”.

Where’s the distinction between tricky (perhaps even tricksy) and trick questions? Tricky/tricksy questions can be very satisfactory for the quiz master, create enormous joy for the teams that got them right and a wry smile from those that get them wrong.  They’re a great part of a great quiz. It can’t be helped if sometimes some players don’t realise they haven’t, as such, been fooled, they’ve just been pleasantly outwitted. What’s a nice example?

Which 1972 Olympic gold medallist’s wedding was watched on TV the next year by a global audience of 100 million people?

90% of teams will be very pleased with themselves and put Princess Anne.

When they hear that the answer is Mark Phillips (an Olympic gold medallist, unlike his former wife) most let out a a phhh, a screech of disappointment even, but enjoy the question. Every now and then, someone will say “hmmph, that’s a trick”.

But it’s not a trick. All the facts are there in the question. Princess Anne is an altogether wrong answer. She is not an Olympic gold medallist.

A “trick” question … an unfair trick? Well, I suppose I’d define it as something where you can’t really fault the thinking of the players in their answer, but they are wrong on a technicality. A carefully worded question can eliminate those technicalities.

Here’s one people might have different opinions on:

What is the only English city whose name begins with the letter H?

Hereford is the answer. But Hull is what a lot of people put. But Hull is wrong, because the city is officially called Kingston upon Hull. Fine. It’s a trick of sorts. Plenty of people would be satisfied to miss that trick, and accept the answer. But some might say “look, everyone calls it Hull, it’s a city, that’s not really fair” …

I’m inclined to agree, or at least don’t want to court that disgruntlement. So, I might make the question ‘What is the only English city whose official name begins with the letter H?’. Once I’ve got the word “official” in, I wouldn’t have sympathy for any carping.

People have different views on this. Different quiz masters are sticklers for different things. I like tricky questions, and indeed a few tricksy questions, but I try to avoid trick questions if I can.

Testing Testing

/0 Comments/in /by

As a QuizQuizQuiz Quiz Master and main Question Writer, all aspects of my job require testing things to see if they’re at the right level.

Of course, I do a sound check before every quiz. This is just as important when using our own portable equipment as when plugging into a venue’s own AV system, though the challenges are slightly different.

With our own  system, the challenge is placing the speakers so that everyone can hear properly with no one being blasted with noise, making sure there’s no feedback, that my voice sounds clear and crisp etc. Some rooms we turn up at can provide more of a challenge than others.

With a venue’s own AV system, whether in a pub, a hotel or a conference room in a company’s offices, it ought to be simpler, but there are pitfalls to avoid. The system should have been perfectly set up to suit the room, and often it is. Often it really is a case of plug in, play a little music, say a word or two, yes, this’ll be perfect. But as a quiz master, we’re very aware of how much more volume is required in a room full of 100 people than an empty room; aware of it in a way that sometimes a venue’s own AV specialist isn’t. Often, one has to politely suggest “I think I’ll need a little more than that on the mic” and be told “No, no, this is fine” when I know full well that as the hum of 100 people chatting and cheering and drinking grows, I really do need a little more on the mic.

Likewise, every now and then, a conference room’s sound may sound fine and clear at mid-volume, but may begin to struggle at a slightly higher volume –  a bit of hiss, a bit of crackle. Experience has taught me the importance of a rigorous test – or as rigorous a test as possible.

Likewise, testing is important for a question writer. We update the database for our corporate events regularly, write 100s and 100s of new questions a year, and we want these questions not just to be ok, decent, forgettable questions, we want them to be great, memorable questions. I have a  pretty good idea when I’ve written a question if it’s a cracker or not, but, in many cases, questions I think will be surefire hits get a muted response if not quite used right, while seemingly innocuous ones bring the house down.

So, we ask all our quiz masters to provide us with feedback on new questions they’ve used when we send out new questions. We’re always swapping ideas and thoughts on how a question has gone down or can be slightly improved – we all want every question we run to be a bit better than the last one we ran.

And, finally, in my capacity as the writer of our multiple choice questions, getting the right level is of paramount importance. Often, because of budget and timing, that testing has to be internal. If I’ve written 5,000 questions for a game, all of which  require a difficulty level, I (and the client) must rely on my own experience, my own hard-earned sense of what people know and what they don’t. I’m pretty good at it. As well as me, there’ll always be at least one other experienced question writer, editing and checking my questions, and if they  feel I’ve mislevelled a question, they’ll let me know.

Sometimes, if the budget’s higher, if the questions are more specialised, we can build a thorough question test into our schedule. You may know we’ve done that recently, with a game we’ve written and which we’re rather excited about. Because of the nature of the game, it was important to test the difficulty ramping.

And, we’re lucky, we’ve got a ready-made focus group, our Friday Quiz mailing list, who we think we’ve got a pretty good relationship with and who, frankly, enjoy a bit of a quiz. So we put a message out for anyone who wanted to help us testing and got a big response. It was hugely helpful to test how quizzers responded to all the questions in the game and will hopefully improve it for the wider audience we hope it finds.

Of course, part of the  experience of surveys and testing is not to blindly accept what basic statistics tell you. Above all, at QuizQuizQuiz,  we trust our own experience. We think we know what makes a good quiz question, a well-balanced round, a fair subject matter. Whenever that confidence is put to the test, we’re happy to see it confirmed (or not – and we learn from it!)

Putting together a quiz night (Part 1)

/1 Comment/in /by

Here at QuizQuizQuiz, we don’t currently host any weekly pub quizzes, concentrating instead on corporate quizzes and question writing (as well as a few pretty exciting ideas in the pipeline). However, we put together hundreds of quiz nights every year and have written many, many pub quizzes in the past, so feel pretty well qualified to talk about how to put together a jolly good quiz for any kind of crowd.

In this post, I’ll limit myself to talking about quiz rounds as a whole, rather than specific questions (and their balanced distribution within a round and a quiz), which I’m sure we’ll come to at a later date. I’m talking about the overall construction of a quiz rather than the details.

How long should it be? How many rounds? How big should these rounds be? How hard should it be? What subjects to include? What should I avoid? What kind of rhythm should I establish within each round? What embellishments add to the magic?

Having listed all those questions, I realise that there’s rather too much there for one blog post. It goes without saying that these won’t be prescriptive answers, and that I, and no doubt you, will have been to plenty of excellent quizzes where the format was very different from what I lay out below. However, these suggestions reflect personal preference, a bit of common sense, and generally speaking, what we at QQQ have, over the years, discovered works best for us.

So, first of all, how long should a quiz be? Well, we get asked to run quizzes lasting anything from 20 minutes to 3 1/2 hours, and we like to think that, whatever the length, we’ll give our client just what they’re looking for (i.e. top notch quiz entertainment). However, quite often these shorter ones use the quiz as just one part of a bigger  showcase event or to be fitted in between courses of a formal meal, so I’ll concentrate on those where the quiz is the main focus of the event.

If you have an evening devoted to a quiz, whether a corporate event or a pub quiz, somewhere between 1 1/2 and 2 1/2 hours is ideal – I’d probably plump for 2 hours of quizzing with a break in the middle. This is enough time to fit in a wide variety of question types and subject matter, to build up a real momentum, to make people feel they’ve got their money’s worth, yet can be broken up into convenient chunks so participants who feel desperate for a cigarette or something radical like a conversation with their colleagues have an opportunity to do so.

If it’s less than an hour and a half, I often feel there are things we’ve missed, and more than two and a half hours, well, maybe for real enthusiasts, but it can be tiring for everyone (think about a film that is 2.5 hours long – tiring, and you are just sitting back and relaxing…), and if half the participants aren’t extremely drunk by the end, you’d be surprised.

And, on a similar topic, how many rounds should there be?

Somewhere between 5 and 8, I think, bearing in mind that one round should nearly always be a table round (pictures/puzzles, that kind of thing). 4 can sometimes feel like too few, like someone’s speciality will be missed out and they’ll feel unfavoured. This can be addressed by including plenty of different subjects within hybrid rounds, but nevertheless, I’m in favour of a good spread. You don’t want to have too many rounds though – people will forget what came where and just feel a little confused. It’s quite hard to answer this question, though, without moving to the next, which is

How big should these rounds be?

Here, I think the important answer is that it can, and should, vary. Although it might make practical sense sometimes, 6 rounds of 10 questions ad infinitum is rather a shame. Think in terms of time rather than number of questions – I don’t think a round should be much less than 15 minutes and I don’t think even the meatiest of rounds should be much longer than 25 minutes. In this range, this will give you 10-15 questions, but there might be quite a few multi-part questions with lots of different points available.

It’s very much part of our modus operandi to keep players on their toes – so you know roughly what’s coming, but not exactly – mixing up the the pace of the rounds and the number of questions and points per round is one technique that we use to achieve that.

Lots more to come, but for now, what’s the longest/shortest quiz you’ve ever been to? What is the ideal length and structure of a pub quiz?

 

Wikipedia for Quiz Question Writers

/0 Comments/in /by

In the olden days (well, 15 years ago) books were still the main way to check quiz questions. Even 5-10 years ago, the volume and quality of content available online wasn’t the same as it is now. Almost every famous person, film, band, TV show, organisation etc. now has an official site of some sort for a very reliable way of checking facts – but even then you find factual errors on official sites.

In the last few years, the biggest development for quiz writers has been the evolution of Wikipedia and its accuracy and reliable use of good citations. It is the fastest, most varied, most detailed and most well organised resource for quiz writers in the history of quiz question writing and to spurn Wikipedia is a very bold decision by any quiz writer (or at least a quiz writer who wants to work efficiently). Of course there are mistakes in Wikipedia, but there are also plenty of mistakes in other, supposedly/traditionally more reputable sources (both online and offline).

However, there is lots of nonsense on Wikipedia, and it does take a fair bit of experience/underlying knowledge/common sense to be able to spot the degrees of nonsense you might encounter. For example, with Wikipedia you always need to check the citations and external references, but no more so than with another source. The big difference is that Wikipedia is well enough organised, such that you almost always have citations and external references.

Oh, and of course Wikipedia is free. Anyone who writes pub and trivia quiz questions would, in my opinion, be a  fool to ignore Wikipedia. But you would also be a fool not to check everything in at least two sources, regardless of what your starting point is for the fact.

Do you use Wikipedia as a tool for quiz question writing? Is anything else as good?