Question: How difficult is it to write multiple-choice test questions?
The multiple choice quiz: it should be easy enough to write, right? The questions themselves seem so streamlined, so simple - just a question, some distractors, and the correct answer.
But effective multiple choice questions are notoriously difficult and time-consuming to construct. There are many ways to give away the correct answer. Students can read more into the question than you intended. It can be hard to find plausible distractors.
So, below are our top tips for designing effective, clear Multiple-Choice Questions, as well as a handy cheat sheet at the end to download, print and keep nearby for reference.
Hopefully these tips will help you to answer (b) - Challenging, but doable - next time we ask!
Let’s start with a basic definition and some terms.
According to Vanderbilt University’s Center for Teaching (Brame, 2013) a “multiple choice item consists of a problem, known as the stem, and a list of suggested solutions, known as alternatives. The alternatives consist of one correct or best alternative, which is the answer, and incorrect or inferior alternatives, known as distractors.”
Within a single multiple-choice question, educators need to knit many objectives and considerations, while at the same time maintaining clarity and keeping the learner at the forefront of their mind.
To quote The Learning Management Corporation:
Essentially, if your test-takers get a question wrong it should only be because they don't understand the concept being tested, not because the question itself is confusing.
The process of writing a multiple-choice question
1 Write the stem (the problem or question)
2 Write the correct answer
3 Write the distractors
4 Review and revise
5 Get a colleague to review and give feedback
1 Write the stem (the problem or question)
The stem should be meaningful by itself and should present a definite problem.
Too often, multiple choice items have a short, insubstantial stem that doesn’t give the learner enough information or context.
The stem needs to be a single, self-contained problem that uses the main idea in the question itself. A handy tip is to ask yourself, “will the learner know what the problem is without reading the response options?”
|Use complete sentences that end in a question mark:|
|What city is the capital of Sweden?||The capital city of Sweden is _______|
|Ensure the question poses a clear problem:
(avoid ambiguity and extra, irrelevant, reading)
|How long did the Italian Renaissance last for?||How long did the Italian Renaissance, which represented a rebirth in the arts and sciences, last for?|
|Try to avoid using negatives:
(if need to use them, put the negative at the end and bold it)
|Which statement regarding tidal volume is true?||Which statement is false regarding tidal volume?|
|Which statement regarding tidal volume is false?|
|Avoid questions that don't give any information:|
|Which statement regarding tidal volume is true?||Which of the following statements is true?|
- Ensure the question is relevant to your courses learning objectives.
- Try to use: “which statement” rather than “which of the following”.
- Higher level MCQs can start with a vignette (i.e. a set of information that sets up the question), from which one or more questions can refer to.
- Make sure all of the context is in the stem/question.
- Avoid true/false. Could use categories question instead.
- Put as much of the wording as possible into the stem rather than the distractors
|An increase in which of the following factors most increases metabolic rate?||Which of the following factors most increases metabolic rate?|
2. Physical activity
|1. An increase in stress
2. An increase in physical activity
3. An increase in temperature
2 Write the correct answer
The answer should be clear and concise.
According to Haladyna (2004), the correct choice is undeniably the one and only right answer.
That sounds easy, right? However, you might be surprised at how often a choice you thought was perfectly clear and concise can come across as confusing, even to someone who knows as much about the material as you do! A good way to test this is to ask a colleague to answer the questions and to give you feedback on their clarity and feasibility.
! Be aware that you aren’t giving away the correct answer accidentally!
|Often more care is taken to write the correct answer.||This can make the correct answer stand out against the distractors.|
|Wrong answers are sometimes bulked out to make them approx. the same length as the correct answer.||If you can take words out and still have the sentence make sense, this makes it standout as a wrong answer.|
|Distractors are written so they are plausible in relation to the correct answer.||If there is only one answer choice that can somehow be related to all the other choices makes it stand out as the correct answer.|
|Make sure other MCQs aren't giving away the answer to the MCQ.||Check in both the stems and answer choices for this.|
3 Write the distractors
Use plausible distractors (wrong-response options)
Okay. So, this is the tricky bit. A distractor is an unquestionably wrong answer. But deciding on suitable and plausible distractors is often the most challenging part of multiple choice question writing.
Ideally, each distractor should to be plausible to a test taker who has not yet learned the skill or knowledge that the test is trying to measure, but clearly wrong for those who do possess the knowledge. In other words, a good distractor should be selected by low achievers and not by high achievers.
|Have 3–4 choices (including the wright answer)
|Distractors should be plausible to those who are uninformed, yet should be clearly wrong to those who understand the concept being tested (i.e. don't write obviously wrong distractors).|
|Ensure your distractors are grammatically correct.|
|Make the distractors approx. the same length as the correct answer.|
|Do not use absolutes (e.g. "never", "always", "all" etc...).|
|Try to avoid "usually".|
|Do not use "all of the above".
|Do not use "none of the above".
|Order all answer choices alphabetically/numerically.|
|Do not introduce new terms that do not appear in the background, for example if something was only referred to by its description, use that description/alternatively, go back to the background page and introduce the term there.|
|Use either a single item or a full sentence.|
|Look back at what the question is asking - do the answer choices make sense in terms of the question?|
4 Review and revise
Encourage Higher-Order Thinking
Something that’s useful to consider when reviewing and revising your questions is the extent to which you want to promote higher-order thinking in your testing and learning outcomes.
There are many definitions of exactly what higher-order thinking means. Brookhart (2010) identifies definitions of higher-order thinking as falling into three categories: (1) those that define higher-order thinking in terms of transfer, (2) those that define it in terms of critical thinking, and (3) those that define it in terms of problem-solving.
In general, higher-order thinking is thinking on a level higher than just memorizing facts or repeating information back in exactly the same way it was told. Higher-order thinking requires students to do something with the facts: apply them, analyze them, organize them, align them to concepts and connect these to other concepts, question and critique them, and use them to creatively solve problems. Higher-order thinking encourages students to be independent, active and confident in the way they process and use the learned information.
You can promote and test for higher-order thinking within your multiple-choice questions.
To help with this, it’s useful to identify, organize, and clarify your educational goals, by using a framework like Bloom’s Taxonomy (1956). The 2001 revision of Bloom’s Taxonomy (Anderson and Krathwohl, 2001) provides a dynamic, action-orientated framework to classify and describe the cognitive processes and levels by which thinkers encounter and work with knowledge.
Very briefly, these levels are:
The idea is to think about how you can use these levels to test for understanding and ability that extends beyond the basics of remembering and understanding the information.
|Remembering||When did WWI begin?||Requires the student to simply recall when the war began.|
|Understanding||How did the killing of Archduke Ferdinand provide the spark for the war to begin?||Requires the student to understand why the assassination caused the war to begin.|
|Applying||Which of the following "hotspots" in the world today might be thrown into war with the assassination of a world leader?||Requires the student to understand the situation in a number of places in the world, and apply what they know about the causes of conflict to those situations.|
|Remembering||What are the symptoms of a cold?|
|Understanding||A patient has the following symptoms: fever, blocked nose, headache, cough. What ailment do they most likely have?|
|Applying||What tests would you conduct to determine if the patient has the flu?|
|Analyzing||How would you determine if the patient has a cold, the flu, or allergies?|
|Evaluating||[example of a process to test for the flu] what is the best justification for using this process to test for the flu?|
Here are some other ways to insert higher-order thinking into your multiple-choice questions:
Using novel material (e.g. questions based on examples/situations) can be a good way to test for understanding and higher level concepts.
|Lowest level example (remembering)||Which is the best definition of a metaphor?|
|Higher level example (understanding)||Which of the following is a metaphor?|
It can be useful to start with a short paragraph that provides context or a set of information that sets up the question or questions. Another possibility is to present a diagram or illustration and ask questions about it.
Tip: Real-world scenarios work well!
|Example (applying)||You arrive at the scene of an accident where people are panicked and yelling. Three people appear to be injured. What action will you take first?|
|Example (analyzing)||A 32-year-old man has a 4-day history of progressive weakness in his extremities. He has been healthy except for an upper respiratory tract infection 10 days ago. His temperature is 37.8 C (100 F), blood pressure is 130/80 mm Hg, pulse is 94/min, and respirations are 42/min and shallow. He has symmetric weakness of both sides of the face and the proximal and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes can be elicited; the plantar responses are flexor.
Which of the following is the most likely diagnosis?
The level of difficulty of an item can depend on what has been taught in the course:
The pegword method is a good approach for learning lists of items. What is the essential feature of the pegword method that makes it an effective mnemonic device?
- Analyzing - if the basic ideas behind mnemonic strategies and the information processing theory that lies behind it have not been taught
- Understanding - if the pegword approach has been taught, but not explained
- Remembering - if the features and why they work have been taught explicitly
Answer choices can also play a factor in determining the difficulty of the question:
The difficulty of the question can be determined by how far away the distractors are from the correct answer:
|What is the population of Dunedin?|
We hope these tips help you when you next set out to write multiple-choice questions!
The Lt Team
Download Cheat Sheet
- Vanderbilt University: Writing Good Multiple Choice Test Questions: https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/
- Brigham Young University: 14 Rules for Writing Multiple Choice Questions: https://testing.byu.edu/handbooks/14%20Rules%20for%20Writing%20Multiple-Choice%20Questions.pdf
- Johns Hopkins University: Tips for writing Effective Multiple Choice Questions: https://ii.library.jhu.edu/2016/12/15/tips-for-writing-effective-multiple-choice-questions/
- The University of Texas at Austin, Faculty Innovation Center: Writing Good Multiple-Choice Exams: https://facultyinnovate.utexas.edu/sites/default/files/writing-good-multiple-choice-exams-fic-120116.pdf
Andersen, L. W. & Krathwohl, D. R. (2001). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives, Abridged Edition. New York: Longman.
Bloom, B. S. (1956). Taxonomy of educational objectives. The classification of educational goals. Handbook 1: Cognitive domain. New York: David McKay.
Brame, C., (2013). Writing good multiple-choice test questions. Retrieved [02-25-18] from https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/
Brookhart, S., (2010), How to Assess Higher Order Thinking Skills in Your Classroom. Alexandria, VA: ASCD.
Haladyna, Thomas M., (2004). Developing and validating multiple-choice test items (3rd edition). Mahwah, N.J: Lawrence Erlbaum Associates.
Smith, J., (n/d). Writing Test Items. University of Otago, NP.
The Learning Management Corporation, (n/d). Writing Effective Questions. www.thelearningmanager.com.