The Continual Assessment of Confidence or Knowledge with Hidden

The Continual Assessment of Confidence or Knowledge with Hidden MCQ? Short Paper W.I.P. Phil Davies School of Computing University of Glamorgan

Conference 1999 Learning Through Assessment Phil Davies School of Computing University of Glamorgan

Implementation of OLAL Tests Weeks 3,6,8,12 (60% Subject Mark) Percentage Marks 5%,15%,15%,25% “Beep Test” 2 Correct -1 Incorrect(Really -3)

Student Concentration Guessing – Why should the student care if incorrect? Allow TWO PASSES of the test By concentrating on first pass – will get reward on second pass – formative

Results of The Tests Improvement of Test Results Pass One to Two

Who benefited from the test? 10 1019 0 1 2029 5 3039 5 4049 20 5059 23 6069 27 7079 17 8089 10 90 0

Student Feedback Excellent way of gaining knowledge 2nd Test Allowed us to learn Good assistant to learning Encourages Revision Very good, promotes learning of a subject early on Better than sex!!!!

“There’s no confidence in multiple-choice testing ” CAA Conference 2002 Phil Davies

What decides the QUALITY of a question? In the 2001 World cup qualifier between England and Greece, who scored the all important equalizing goal? David Beckham Victoria Beckham Brooklyn Beckham Rebecca Loos

Rewards? Know it On the tip of my tongue If I see the answers I’ll know it Deduction Guess Know it oh no I’ve got it wrong . misinformation

Proportion of Questions Correct High Confidence 84% correct Fairly Confident 58% correct Not confident 47% correct 72% of all questions answered with high confidence

Student thoughts on Confidence “Eliminates guesswork” “I was waiting for a leggy blond to bring on my prize at the end” “separated the lucky students from me” “at least I didn’t lose marks for selecting no confidence” “are you testing my confidence or knowledge”

Quality of a multiple choice question? In the 2001 World cup qualifier between England and Greece, who scored the all important equalizing goal? Teddy Sheringham Michael Owen Robbie Fowler QUALITY OF QUESTION NOT DISTRACTERS

“Bite The Bullet” Use Confidence Testing for all four tests (10%, 15%, 20% & 25%) Weighting of Passes (50/50, 60/40, 70/30 & 80/20) How to judge if the students had ‘really’ learned? Identify improvement Select five questions that were answered ‘poorest’ in test one & pass through to test two Select five questions that were answered ‘poorest’ in test two & pass through to test three ( original five) . Etc.

How to identify ‘Poorest’ Using confidence, degree of correctness of answering Using confidence scores to provide a degree of ‘poorness’ i.e. 4, 2, 1, -6, -5, -4 High degree of ‘poorness’ could just be a ‘BAD’ question Swing between 1st and 2nd pass of degree of correctness should be ‘Reasonable’

6 25-29 7 30-34 7 35-39 12 40-44 15 45-49 13 50-54 11 55-59 6 60-64 5 65-69 2 70-74 75-79 80 1 3 Test # Average Pass One Average Pass Two Ratio First/Second Average Total Test One 10% 42.56% 72.74% 50 / 50 57.65% Test Two 15% 26.80% 66.62% 60 / 40 42.73% Test Three 20% 41.14% 76.40% 70 / 30 51.72% Test Four 25% 51.00% 81.50% 80 / 20 57.10%

Test 2 - 1 Test 3 - 2 Test 4 - 3 Test 4 - 1 80 -2 4 23 25.00 75-79 10.5 9 45.5 65.00 70-74 -31.8 43 9.6 20.80 65-69 -34.83 43.83 29.67 38.67 60-64 -45.27 53.27 16.55 24.55 55-59 -43.15 36.08 20.23 13.15 50-54 -36.2 48.8 21.53 34.13 45-49 -7.0 27.92 38.83 59.75 40-44 -45.43 30.00 28.43 13.00 35-39 -32.86 27.00 25.71 19.86 30-35 -47.33 44.5 31.33 28.50 25-29 -47.67 37.00 -13.00 -23.67 Average -30.25 33.70 23.12 26.56

Identifying Poor Questions Test One Weighting Difference Pass 2-1 Weighting Factor # Wrong Pass 2 # Correct Pass 2 Weighting Factor # Wrong Pass 1 # Correct Pass 1 Question Number Q5 27 61 -251 56 32 9 260 Q14 30 58 -218 60 28 67 285 Q15 34 54 -181 55 33 4 185 Q19 8 80 -356 76 12 223 579 Q20 33 55 -187 58 30 16 203 Q27 11 77 -373 58 30 21 394

Identifying the Poorest Questions Test two to three Weighting Difference Pass 2-1 Weighting Factor # Wrong Pass 2 # Correct Pass 2 Weighting Factor # Wrong Pass 1 # Correct Pass 1 Question Number Q2 14 74 -348 79 9 250 598 Q6 28 60 -222 45 43 -109 331 Q8 22 66 -255 70 18 145 400 Q10 28 60 -215 48 40 -66 281 Q11 20 68 -281 57 31 20 301 Q18 28 60 -200 79 9 227 427 Q19 31 57 -202 74 14 166 368 Q21 27 61 -235 57 31 32 267 Q27 32 56 -210 66 22 106 316 Q28 27 61 -256 77 11 238 494 Q32 23 65 -254 82 6 265 519

Identifying Poorest Questions from Three to Four Weighting Difference Pass 2-1 Weighting Factor # Wrong Pass 2 # Correct Pass 2 Weighting Factor # Wrong Pass 1 # Correct Pass 1 Question Number Q6 32 56 -212 63 25 73 285 Q11 27 61 -219 75 13 217 436 Q16 19 69 -313 60 28 58 371 Q26 30 58 -225 81 7 253 480 Q40 30 58 -218 72 16 183 401

Passed through to Test Four 13 of the poorest question 6 new questions based upon additional work between tests 3-4 21 questions (cross section of questions and topics from previous tests)

Improvement of Results Through Tests Questions Through Tests 400 300 Question 5 200 Question 6 Question 10 100 Question 11 Question 14 0 Question 15 1 -100 2 3 4 5 6 7 8 Question 16 Question 19 Question 20 -200 Question 21 Question 26 -300 Question 27 Question 40 -400 -500

Improvement Pass One of Questions Pass One Question Improvements 400 300 Question 5 200 Question 6 Question 10 100 Question 11 Question 14 0 Question 15 1 -100 2 3 4 Question 16 Question 19 Question 20 -200 Question 21 Question 26 -300 Question 27 Question 40 -400 -500

Improvement for Question 5 Stronger Students improved quickly through the tests Became more confident in answering the questions Weaker Students improved less quickly than stronger through the test progress

Is it just Confidence or Knowledge i.e. questions right or wrong? Category Average # of Questions Correct Pass 1 Average # of Questions Correct Pass 2 Average Total # of Questions Correct (1) 131.0 158.0 289.0 75-79 (2) 123.5 150.5 274.0 70-74 (3) 117.6 148.6 266.2 65-69 (4) 113.8 150.7 264.5 60-64 (5) 110.3 145.0 255.3 55-59 (6) 107.6 143.4 251.0 50-54 (7) 100.4 136.2 236.6 45-49 (8) 92.5 132.6 225.1 40-44 (9) 94.0 126.4 220.4 35-39 (10) 87.7 121.6 209.3 30-34 (11) 80.5 117.3 197.8 25-29 (12) 76.7 117.3 194.0 80

Some Concerns Category Average # of Questions Correct Pass 1 Average # of Questions Correct Pass 2 Average Total # of Questions Correct 12 81 127 208 (194) 3 102 136 238 (266) 7 78 117 195 (236) 8 112 141 253 (225) 10 97 137 234 (209)

Overall Comments Work in Progress Results look very positive Correctness Factor is dependant upon weighting of scores Guessing not so much of a problem Changed students mode of revising / study Fine granularity in the QUALITY OF A QUESTION Possibility of Computerized Adaptive Testing However note the quality of the question change throughout the course of the cumulative tests Does gender come into it?

Contact Information Email: [email protected] Phone: 01443 482247 School of Computing University of Glamorgan Pontypridd Mid – Glamorgan South Wales