Does the Question Format (Open-Ended vs Multiple-Choice) Affect Pass Marks Determined by Experts?

Let’s say we’ll carry out a case-based exam and therefore we need to determine the cut-off score for the exam. We ask standard setters, who are the subject-matter experts, what should be the cut-off score in this exam. Would the cut-off scores for pass/fail decisions be different if the question format is different? Would they determine a different pass mark for open-ended and multiple-choice questions even if the cases are the same?

Researchers from the UK conducted a study to shed light on this. They also investigated what happens to the cut-off scores if the right answers are provided to standard setters. It’s a really interesting study.

The flamingo is ready to present this study.

First of all, we need to provide some explanations for some terms to be on the same page altogether.

The Angoff method: It “involves a panel of expert judges making estimates on the proportion of borderline candidates that would answer each question correctly. These estimates are then averaged across all questions and judges to create a standard cut-off score.”

Single best answer question: This is a format that I believe many of our subscribers are familiar with. A case-based question is provided with five options that include only one true option.

Very short answer question: In this format, students write their own answers using one to five words in length instead of choosing from a list of options. It serves exams to refrain from the effect of guessing.

After these explanations, let’s jump into the study.

The researchers provided questions to medical teachers to determine cut-off scores for each question in an exam that included 50 questions. They have trained to set standards considering the Angoff method. There were four pairs of these 50 questions: Two were in single best answer format, and the other two were in very short answer format. Since the researchers also aimed to find out how providing the right answers affect standard setters’ cut-off score decisions, each of these two group of exams had also two groups: One with the right answers, and one without them. Remember, all exams had the same cases. The differences were formats and whether providing the right answers or not.

Now it’s time to see the results.

As you see in the video, there was no significant difference between providing and not providing the right answers in single best answer questions. Providing the right answers produced a significantly lower pass mark in very short answer format. When the answers were not provided, the cut-off scores were not significantly different between single best answer and very short answer formats.

I think the results are interesting and difficult to interpret. What do you think about the reasons behind these results? Why is it lower in the very short answer format when the right answers provided? Let me know your views in the comments.

If you want to read more, the link to the article at the description section below the video.

See you and adios para amigos. And also, don’t forget the flamingo.

Follow the Flamingo on Twitter: https://twitter.com/MedEdFlamingo