How Multiple-Choice Tests Work
“Close your books. Clear off your desk. Get your #2 pencil ready for the test.”
“Mark each question. One point for right. Zero for wrong and blanks.”
“There is no penalty unless you leave some blank.”
Students have heard this many times and believed it. They even count on a few lucky guesses for extra points as do at-risk teachers and administrators.
It is how those “extra points” appear that makes the scores for traditionally scored multiple-choice tests of decreasing value as the test score declines. Below 60%, chance can contribute as many right marks as the student. There is no way of knowing if a right mark represents what a student knows or is the result of chance.
Right-Mark only Scoring
A blank answer sheet is worth 20 points, on-average, at the start of a test with 100 questions with 5-option answers. This is on-average. This is not for certain every time. To get the 20 points, on-average, every question on the answer sheet must be marked with no need to look at the test questions.
These 20 points have nothing to do with what a student knows or can do. This is one-out-of-five right, on average, or 1/5 point per question.
A right mark is then worth 4/5 of a point. When added to the 1/5 from chance, the total is one point. If the mark is wrong, the 1/5 point is lost. (There is a built in penalty, on-average.)
The penalty can be more or less. At the pass/fail point of 60% right and 40% wrong, it can spread a number of students with the same ability over two letter grades. Half pass and half fail. This is not an honest basis for assigning grades.
Most classroom tests only have three functional answer options to each question even if the test is designed for 5-option answers. The test design of 5-option questions is static (1/5 point, on average, from chance).
Even though the test paper is static, the operation of testing is dynamic. By students eliminating two of the five options, the chance value increases from 1/5 to 1/3 point. The better prepared a class is, the higher the on-average chance value. However, at the mastery level above 90%, the test becomes a checklist and chance plays almost no part in the scoring. The test is a near perfect fit to student preparation.
Scoring Judgment
Chance can also be eliminated below the mastery level by making a simple change in the scoring and test instructions. Using good judgment to not mark a wrong answer (to omit, or leave blank) is given a value higher than zero.
“Select questions you know you can answer correctly. One point for right. Zero for wrong and 1/5 point for not marking a wrong answer (omit or blank).”
The blank test above is again worth 20 points but now for certain, not on-average. The test score starts at 20 points rather than zero. Now after marking all the questions a student can use to report what he/she knows or can do, the student stops marking.
A fixed value has been assigned to not marking a wrong answer (good judgment). Students do not need to gamble to obtain the remaining score from chance. Students receive scores for quantity (the number right) and for quality (the percent right of marked answers) that are combined for the test score.
A test score of 60% can have a quality score of 100% (no wrong answers). This student knows what he/she knows. The student and teacher can trust what this student knows. The student has a solid foundation on which to build future learning. And every student with similar ability gets the same score, an honest test.
Setting the Value for Judgment
The value for not making a wrong mark has a different meaning at different values. At zero, the test is a traditional “guess” test. At the design level for chance, it gives the student some control of chance. Either guess or accept a fixed test design value that is certain for each question. This value is not fair to the student as it is too low.
At the dynamic class fitness level for chance (about 33%), student judgment has complete control of chance. This is the lowest value for Knowledge and Judgment Scoring (KJS). Either mark a right answer or do not mark a wrong answer (omit) and take the fixed value for good judgment that is fair to the student. Chance no longer operates to produce a wide distribution of test scores for students of the same ability.
A value above the fitness level rewards students for the time and effort to use higher levels of thinking when taking the test. At 50%, knowledge and judgment have equal value. A right mark is scored 1/2 point for right and 1/2 point for not marking a wrong answer. Or alternately, start with 50 points and add one for right and subtract one for wrong. This is the highest value for good judgment that is compatible with traditional count-the-right-mark-only-scoring.
At the mastery level for judgment, 75% to 90%, an examinee does not mark a wrong answer (poor judgment) as more than a couple of wrong answers result in failure. This level is used in high risk occupations such as police, nursing, and nuclear power plant operation by Knowledge Factor, Inc.
Measure and Reward Student Development
Classroom tests and standardized tests can both use KJS. The test results show what each student knows and how much you can trust what each one knows. At the mastery level, you know what each student knows (understands and finds meaningful) regardless of the test score. KJS produces an honest accurate test score and item analysis. You can reward quantity and quality independently, as is done with essay tests.
The most important reason for using KJS is that it puts the student in charge of learning and reporting rather than just marking every question and then waiting for the teacher to report the right answers. Students learn to function at higher levels of thinking, as self-correcting, self-rewarding scholars, rather than as passive pupils. High quality students learn more, retain more, and get higher test scores.
TeachKJS is loaded with the final test administered to 106 students in a remedial general biology course. It presents a brief view of what Knowledge and Judgment Scoring can do. At the time this test was administered, 79% were functioning at the mastery level (90% +/-10% right on answers marked), 17% were struggling, and only 5% were still finding taking responsibility to report what they knew just too scary.
27 October 2008