Towards More Efficient Assessments : Increasing Information from Objective Examinations *

Objective examination questions are widely used to assess students’ knowledge, but the standard MCQ with a stem followed by 4 6 possible answers one of which is chosen as correct is very inefficient. Simple changes to the format can treble the information gained. Information theory is the tool for assessing the information content of electronic and other communications. The “bit” is the unit of information and equals on true/false choice. I have applied basic information theory to objective questions. The standard MCQ with a stem, a choice of 4 possible answers (mark one answer true) and an expected correct answer rate of 70% yields 1.36 bits. A MTFQ with 4 choices where the student must answer true/false to every possible answer gives 3.52 bits of information. By adding a “don’t know” option the same MTFQ gives 4.72 bits of information, 350% of the standard MCQ. Thirty MTFQ with don’t know give the same information about students’ knowledge as 100 standard MCQs. The effort needed to set, sit and mark is the same for both. Small changes to the format of objective questions give large gains in efficiency. We should balance these gains against possible disadvantages.


Background
Education is the transfer of relevant information from teacher to student.In medical education, information is broadly classified into knowledge, skills and attitudes (KSA).Each needs different pathways; in this paper I shall consider only knowledge.In any transfer of information, there is a source (teacher, book etc.), the channel (speech, vision, electronic) and the recipient (student).Perfect transfer of information is not possible in theory or in practice, so all channels transmitting important information have feedback from recipient to source to check levels of completeness and accuracy of the data transmitted.In face-to-face teaching, we have immediate feedback; we can see signs of comprehension, confusion or boredom in our students.In other forms of learning, we lack such feedback, so we need formal methods to ensure that the student has received the information and has incorporated it into his/her knowledge base.In education we call this "assessment" and it can be done at the end of each session or at a later date.In a crowded curriculum where a few teachers are trying to transmit large amounts of information to many students, we should have channels that transmit information efficiently so that the times spent by teachers and students are minimised.This applies equally to the reverse process of feedback to check that the information has been transmitted, received and absorbed without major errors or omissions.Such feedback is vital for educational and regulatory reasons, but it does not add to the student's knowledge and is therefore often seen as a burden to be minimized.There is every incentive to make the feedback process as quick and painless as possible compatible with the needs and goals of such assessments.
The information we transmit to students is diverse, multilayered and complex, but we have largely simplified the feedback from the students by using "objective questions" with a narrow range of formats and responses.Unlike to complex information transferred from teacher to student, this simplified feedback can be tested for accuracy and completeness.Basic information theory shows great differences in the information content of various forms of objective question.Minor changes to the style of question can yield information up to 350% of the standard MCQ.

Information Theory-A Very Brief Overview
Advanced information theory is complex and highly mathematical, but the basic ideas are simple and need only high school algebra.The basis was published in a trade journal by Shannon (Shannon, 1948). .Many of the results derived from information theory are numerical results for ideas that are intuitively obvious.Information is measured in bits, one bit is the amount of information gained by choosing one of two equally likely alternatives such as Yes/No or True/False.If there are three equally likely choices, such as True/False/Don't Know then we can see that gain more information and this is confirmed by information theory.The basic equation is (Moulton, 2010) where p = probability of each event and all the p values add up to 1. log 2 (1/p) is the logarithm to the base 2 of 1/p.
Most assessments based on objective questions are designed so that the average student will score about 70% correct answers.This built-in bias lessens the amount of information we gain.An extreme example of this effect is to start a questionnaire by asking a random sample of people "Are you male or female".If the subjects were random users of a shopping mall, then the response would help in the analysis of later answers so provides useful information.However if we took that same questionnaire to a nunnery, it would be no surprise that 100% answered "Female".We knew this in advance, so gained no new information from the question.The formula above gives us a numerical measure of the information gained, allowing for prior knowledge.
I shall use this basic formula to calculate the information in the different types of objective question used in assessment of medical education.To make the different types comparable, I shall give each type a choice of four items, but this is not necessary or even desirable.

Information Content Different Types of Objective Questions
Information in standard MCQ style question: 1) MCQ Type A: The standard MCQ style question In a healthy person the common colour of urine is (mark one box) The candidate must pick one out of the four alternatives.If the question is designed so that the candidate is equally likely to choose any option, then the information content of this decision is 2 bits.In practice, the candidate knows that one option is correct and the others wrong.From results of past examinations the examiners know that about 70% of students will pick the correct option and the others will be evenly divided among the other three options.The student can make only one response and the likelihoods have been knowingly weighted by the examiner.Using the formula above, we can show that the information content of the student's response is 1.36 bits.We have effectively limited the candidate's options, so we learn less about his/her knowledge 2) MTFQ Type A: Multiple true/false question The student could be asked this question in a different way In this form of objective question, the wording of the stem is slightly different to allow multiple true responses.The answer to each line must be independent of other lines.The work involved for the examiner to set the question is almost the same as MCQ Type A, but the student must answer four independent true/false items.If the student were equally likely to answer true or false to each line, than each response would contain 1 bit of information and the total information gained would be 4 bits.However, the student's knowledge of the subject makes the correct answer more likely, say he has a 70% chance of giving the correct answer.This limits his choices so the information content of each answer drops from 1 to 0.88.The student answers four independent questions, so the total information is 3.52 bits.By changing the format of the question so as to demand an answer to each alternative, we have gained 3.52 bits of information compared with 1.36 for the standard MCQ, more than double the information (260%).
3 As in the previous format, each line tests a separate item of knowledge, but there are now three alternatives.The don't know option gives the student a wider choice.Let us assume that, on the average, students mark the correct box 70% of the time, the wrong box 15% of the time and the don't know box 15% of the time.The answer to each line will give 1.16 bits of information, giving a total gain of 4.72 bits of information.Quite apart from any philosophical value of giving the student a "Don't know" alternative, we have increased the information gained about the student's knowledge to 4.72 bits, compared with 1.36 bits for the standard MCQ, a factor of 347% in the return for the same time and effort by the examiner.

Sensitivity Analysis
In the descriptions so far, I have assumed that there is prior knowledge that in standard MCQs the student will choose the correct response rate of 70%, with other responses distributed equally among the false options.Papers could be set with different expectations of student responses.The main changes in information yielded come from the format of the question and the expected number of correct responses.These are shown in Table 1 where the number of expected correct responses is given and the remainder of the answers evenly divided among other options.
In all types, the greatest information yield is when the prior expectation is for 50% correct responses.The information gained decreases in all types of objective questions as the expected level of correct responses rises, but the relative advantage of the MTFQ and MTFQ + DK increases.Under all levels of expected correct answers, the MTFQ yields at least twice as much information as the standard MCQ and the MTFQ+DK has at least three times the yield.

Discussion and Conclusion
Objective questions have many advantages and are widely used for feedback to teachers sessment of student and as knowledge for regulatory and licensing bodies.Knowledge is reduced to fragments and selected so that a True/False or Yes/ No answer is possible and meaningful.This allows the mathematical measurement of the information content.This paper is about the information contained in each question, but not with the method of scoring the student's responses nor the subject matter tested, although I note that such testing is limited to material which is definitely True or False.This excludes newer material which may be important but is not yet fully established and a wide range of material where an answer will depend on surrounding circumstances.
Replacing the standard MCQ with the MTFQ involves no philosophical or educational change or innovation.It is a simple change of format, but one which doubles the information gained at no cost in time for examiner, the system or the students.The use of the MTFQ + DK involves an extension of the relevant definition of knowledge.Admitted ignorance is a recognised and valid state of knowledge, By omitting that formal option, we force the student to guess, which is not only logically unsound, it is undesirable and probably dangerous in later professional life.Guessing also complicates the scoring system of the examination (Harden et al., 1976;Ben-Simon et al., 1997).We all have areas of ignorance, it is a sign of wisdom to acknowledge ignorance and that should be encouraged by an appropriate marking scheme.A scoring that allots marks to Correct > Don't Know > Wrong is logically sound and will discourage guessing The standard MCQ format has lasted well.Many pre-tested and validated question banks are available.With no change in the philosophy of objective questions, but only a change from the MCQ format to the MTFQ format, we can increase the information content by a factor of two or more with no extra work for the examiner, thus gaining necessary data about the student more quickly, reducing the number of questions needed, and so easing the load for both examiners and students.If we acknowledge that admitted ignorance is a valid form of knowledge, then the MTFQ + DK will further increase the efficiency and information yielded by objective questions.The number of questions still needed will ensure that the assessment covers a wide range of the curriculum.Examiners must consider factors other than information content, but the efficiency, ease and reliability of collecting data about the students are important factors when preparing assessments.
answers are yellow, red, brown)

Table 1 .
The information yield for various types of objective questions showing the effect of varying the expected level of correct responses.