Most students initially choose right-mark scoring as it is the only scoring method they know for multiple-choice. They find taking responsibility for reporting what they know readily acceptable after one or two experiences with Knowledge and Judgment Scoring (KJS).
Break Out source code and Power Up Plus run in Microsoft and Macintosh Excel with the following six toolbar buttons:
Break Out imports a standard text (.txt) file from a scanner, Word, text editor or by manual entry of student marks onto Sheet 1:
Click Parse to create Sheet 2 where you can also enter student marks manually and edit the data before scoring and rescoring.
Click AnswerFileSave if you have any edits to save and then click Score.
Check for a very low score do to a scanner or other data entry error on Sheet 4:
Check for very difficult questions towards the lower right side of Sheet 3 that may contain errors. Resolve these by accepting them or giving all students the point or dropping the question on Sheet 2. Save the corrections and click Score for corrected results.
The nine digit Student ID can be coded in 3-digit parts for Test Number, Seat Number, and Student Number.
Sheet 5 presents the mark analysis and estimates of the average value for judgment when Knowledge and Judgment Scoring. Two estimates of student self-judgment are based on fitness (39.4% and 52.5%). Fitness is the average number of options students actually use on the test. This is rarely the design value of four or five options or 25% or 20%. Traditionally scored multiple-choice tests are almost always easier than the design value.
On this test the estimated average fitness is 45.6%. This is the average expected score if these students had used higher levels of thinking, discarded wrong options, and then guessed from the remaining items. This 4-option test actually functioned as a 2-option test with an average test score of 84.4%. The students were well prepared.
Click Omit_Value to set the reward for students electing Knowledge and Judgment Scoring rather than right-mark scoring. On this test of high scoring, mastery, students, the estimated value for judgment is between 39% and 52%, on-average. This justifies using 50% for judgment as the default rather than the original 33%
This value must be consistent with the test instructions, “Mark all items for right-mark scoring OR report only what you trust you know or can do for Knowledge and Judgment Scoring with a fixed value of 50% for good judgment (mastery)”, for example. This control has been left in the software in case anyone still wants to experiment with judgment values.
Power Up Plus (PUP) contains the above features and benefits as well as an item analysis terminating in a Test Performance Profile (master/easy, unfinished, and discriminating items).
These test results are real. Student names have been changed.
Validity Check: Copy and paste the data in Sheet 1 into your KJS software, sheet 1, cell A1, to check that it delivers the same results or load the answer (.ANS) file from the ZIP file.
27 February 2013