Knowledge and Judgment Scoring
Welcome to a brief view of Knowledge and Judgment Scoring presented by a remedial biology class of 106 students. At the time this test was administered, 84 (79%) were functioning at the mastery level, that is, they were 90% +/-10% right on answers marked; 18 (17%) were struggling, and only 5 (5%) were still finding taking responsibility to report what they knew just too scary.
It takes about 100 students to generate data in one test that includes a full range of responses that also randomly occur in smaller classes. The smallest class size to generate meaningful results is 15 students.
Break Out reports all the information available in a multiple-choice test, from a student point of view, in a form that is easy for students and teachers to use. You can download the program to develop complete printouts for a full view and best perspective using the answer files in the zip file.
The snapshots below illustrate the benefits of using Knowledge and Judgment Scoring for three types of students found in every class.
|
Knowledge and Judgment Scoring Benefits |
||
|
Student Type |
Student Development |
Test Score |
|
High Performance High Quality |
They obtain a confirmation of the quality of their work and a specific list of what has yet to be mastered. |
Limited benefit. They are already using all levels of thinking in a cyclic self-correcting fashion. |
|
Medium Performance High Quality |
They now experience a reward for their ability to accurately report what they have mastered, what they can trust, as well as, for quantity. Their performance improves with practice. |
Maximum benefit. They have the knowledge and self-judgment needed to learn and test at all levels of thinking. |
|
Low Performance Low Quality |
They are made aware that successful students like taking responsibility for learning and testing and that doing so, with practice, improves their performance and satisfaction too. |
Variable benefit. Some never make the change from rote memory to question, answer, and confirm needed to be a self-correcting scholar. Others make the leap from D/F to A/B. |
The specific details that make sense of using Knowledge and Judgment Scoring rather than right-mark scoring are drawn from the Student Scores and the Student Counseling Mark Matrix sheets.
Student scores
The two top ranking (RK) students (answer sheets 21st and 82nd to be turned in) used poor judgment (PJ) to make two wrong marks and good judgment (GJ, omit to not make a wrong mark) on five questions. Students are rewarded for quantity (number of RT marks) and quality (%RT of marks) independently. These are combined for the test score (SCORE) using the given value for judgment (knowledge and judgment, have equal value on this test).
Student Counseling Mark Matrix
High Quality, High Performance
This abbreviated table only lists questions that one or more students omitted (@), to not mark in error, or they did mark in error (wrong answer listed). Questions are ranked easy to difficult from left to right at the bottom of the table. They are labeled from the Mark Analysis at the bottom of the table: expected (E), most students marked and were right; student discriminating (D), few students marked and were right; guessing (G), few students marked and were wrong; and misconception (M), most students marked and were wrong.
Each of these five top ranking students marked a unique combination of questions. These students know what they know and can trust. They did not miss any expected (E), easy, questions or the one misconception (M) question on the test.
High Quality, Medium Performance
These seven students, with the same test scores, rank at the center of the class. The students with the highest quality (%RT) finished the test first. The first five definitely know what they know and what they have yet to learn; traits of a self-correcting scholar. Their quality scores were near or higher than 80%.
Low Quality, Low Performance
The five lowest scoring students exhibited poor judgment by marking about the same number of wrong (PJ) and right (RT) answers. The first two students were trying Knowledge and Judgment Scoring. They failed the test. Their attempt to take responsibility for reporting what they knew did yield a few more points than the remaining three students who used right-mark scoring. These three avoided responsibility for what they knew. (The other two students using right-mark scoring earned scores of 79.6% and 57.1%, ranking 30th and 101st.)
Knowledge and Judgment Scoring adds a second dimension to multiple-choice tests: student responsibility to learn and report at all levels of thinking. If assessed, as the two students above explored, it is rewarded. The test is no longer “just multiple-guess”, only requiring that the student remember to bring a #2 pencil. Instead of getting credit for lucky guesses, students get credit for honest accurate reporting, the foundation for a self-correcting scholar, as they break out of old study and testing habits.
These test results show that students prefer Knowledge and Judgment Scoring after some experience doing something they never did with right-mark scoring. They are in charge of selecting which questions to use to report what they know and they become "masters" at it. Becoming a self-correcting scholar is an empowering experience. Student comments on how this change in learning and reporting improved their work in other courses prompted the creation of Break Out source code. Students also like “not having to waste time making a best guess” on questions they do not understand or can not even read.
Download Break Out source code or buy Break Out Plus and run to develop the complete printouts and view the results in their best perspective. When counseling a student about a missed question, the Student Mark Matrix shows which of the four classes of questions the students in the class have placed it, as well as, the individual student’s wrong answer. Use Power Up Plus to obtain the teacher/test-maker point of view based on a modern item analysis.
With only nine of the 49 questions functioning as Guessing or Misconception, the on-average number of functional answer options was reduced from six (five answers and omit) to just two (a fitness of 2.1 from the Mark Analysis). High quality students either marked the right answer or exercised good judgment to not mark a wrong answer.
The average dynamic minimum test score for this class was 0.50 (50%) rather than the static test design value of 0.20 (20%) for five-option questions. These students were well enough prepared to be able to get a test score of 50% by choosing from two options on each question on average.
25 October 2008