A. E. DUGDALE
where p = probability of each event and all the p values add up
to 1. log2(1/p) is the logarithm to the base 2 of 1/p.
Most assessments based on objective questions are designed
so that the average student will score about 70% correct an-
swers. This built-in bias lessens the amount of information we
gain. An extreme example of this effect is to start a question-
naire by asking a random sample of people “Are you male or
female”. If the subjects were random users of a shopping mall,
then the response would help in the analysis of later answers so
provides useful information. However if we took that same
questionnaire to a nunnery, it would be no surprise that 100%
answered “Female”. We knew this in advance, so gained no
new information from the question. The formula above gives us
a numerical measure of the information gained, allowing for
prior knowledge.
I shall use this basic formula to calculate the information in
the different types of objective question used in assessment of
medical education. To make the different types comparable, I
shall give each type a choice of four items, but this is not nec-
essary or even desirable.
Information Content of Different Types of
Objective Questions
Information in standard MCQ style question:
1) MCQ Type A: The standard MCQ style question
In a healthy person the common colour of urine is (mark
one box)
1 yellow [ ]
2 blue [ ]
3 red [ ]
4 brown [ ]
(the correct answer is yellow)
The candidate must pick one out of the four alternatives. If
the question is designed so that the candidate is equally likely
to choose any option, then the information content of this deci-
sion is 2 bits. In practice, the candidate knows that one option is
correct and the others wrong. From results of past examinations
the examiners know that about 70% of students will pick the
correct option and the others will be evenly divided among the
other three options. The student can make only one response
and the likelihoods have been knowingly weighted by the ex-
aminer. Using the formula above, we can show that the infor-
mation content of the student’s response is 1.36 bits. We have
effectively limited the candidate’s options, so we learn less
about his/her knowledge
2) MTFQ Type A: Multiple true/false question
The student could be asked this question in a different way
In a person with organic disease the urine may be (mark
one box in each line)
1 yellow true[ ] false[ ]
2 blue true[ ] false[ ]
3 red true[ ] false[ ]
4 brown true[ ] false[ ]
(the correct answers are yellow, red, brown)
In this form of objective question, the wording of the stem is
slightly different to allow multiple true responses. The answer
to each line must be independent of other lines. The work in-
volved for the examiner to set the question is almost the same
as MCQ Type A, but the student must answer four independent
true/false items. If the student were equally likely to answer
true or false to each line, than each response would contain 1
bit of information and the total information gained would be 4
bits. However, the student's knowledge of the subject makes the
correct answer more likely, say he has a 70% chance of giving
the correct answer. This limits his choices so the information
content of each answer drops from 1 to 0.88. The student an-
swers four independent questions, so the total information is
3.52 bits. By changing the format of the question so as to de-
mand an answer to each alternative, we have gained 3.52 bits of
information compared with 1.36 for the standard MCQ, more
than double the information (260%).
3) MTFQ with DK alternative
We can include a formal don’t know option in the MTFQ
In a person with organic disease the urine may be (mark
one box in each line)
1 yellow true[ ] false[ ] don’t know[ ]
2 blue true[ ] false[ ] don’t know[ ]
3 red true[ ] false[ ] don’t know[ ]
4 brown true[ ] false[ ] don’t know[ ]
(the correct answers are yellow, red, brown)
As in the previous format, each line tests a separate item of
knowledge, but there are now three alternatives. The don’t
know option gives the student a wider choice. Let us assume
that, on the average, students mark the correct box 70% of the
time, the wrong box 15% of the time and the don’t know box
15% of the time. The answer to each line will give 1.16 bits of
information, giving a total gain of 4.72 bits of information.
Quite apart from any philosophical value of giving the student a
“Don’t know” alternative, we have increased the information
gained about the student’s knowledge to 4.72 bits, compared
with 1.36 bits for the standard MCQ, a factor of 347% in the
return for the same time and effort by the examiner.
Sensitivity Analysis
In the descriptions so far, I have assumed that there is prior
knowledge that in standard MCQs the student will choose the
correct response rate of 70%, with other responses distributed
equally among the false options. Papers could be set with dif-
ferent expectations of student responses. The main changes in
information yielded come from the format of the question and
the expected number of correct responses. These are shown in
Table 1 where the number of expected correct responses is
given and the remainder of the answers evenly divided among
other options.
In all types, the greatest information yield is when the prior
expectation is for 50% correct responses. The information
gained decreases in all types of objective questions as the ex-
pected level of correct responses rises, but the relative advan-
tage of the MTFQ and MTFQ + DK increases. Under all levels
of expected correct answers, the MTFQ yields at least twice as
much information as the standard MCQ and the MTFQ+DK
has at least three times the yield.
Discussion and Conclusion
Objective questions have many advantages and are widely
used for feedback to teacherssessment of student and as
Copyright © 2013 SciRes.
40
