CHAPTER 3 SCORING & MODERATION Strategies for scoring Reliability

CHAPTER 3 SCORING & MODERATION Strategies for scoring Reliability Scoring moderation

STRATEGIES FOR SCORING* The question format, options for how the student answers the question, and the way answers are scored should all measure the full range of complexity and depth of knowledge required in the learning target. Together, the question format, the options for how the student answers the question, and the way answers are scored provide a reasonable estimation of student mastery of the learning target. The connections between questions and their scoring criteria should be apparent to anyone familiar with the learning target. The rigor of scoring should increase in proportion to the stakes attached to the assessment and complexity of the item * As modified from Oregon Assessment Guideline (2014, p. 7)

SCORING FOR DIFFERENT ITEM TYPES Scoring is an activity to assign a value that can represent the degree of correct response to an assessment task Multiple-choice or Ordered Multiple-choice items Easy scoring; objective Short constructed response More time consuming to score, should have clear and objective scoring rubrics Extended constructed response Less objective, should have clear and objective scoring rubrics, more time consuming to score

RELIABLE SCORING The degree to which two or more scorers assign the same score to a single student response One type of reliability Component of Assessment Quality To be discussed in Chapter 6 Assessed by the use of two or more raters Use of scoring rubrics Can improve reliability since scores will be given consistently to a particular item Improve validity as a score is specifi cally given to one or a set of intended learning outcomes

ENSURING RELIABLE SCORING Selection of a set of responses Random samples of student work or Deliberately select from the high-, medium- and lowachieving groups of students Score by multiple people Check for exact matches, difference of one score, diff erence of two or more Discuss the results Evaluation of the eff ectiveness of scoring rubrics Training for additional scoring Evaluation of inter-rater reliability

OTHER ISSUES IN SCORING PROCESS Exemplar responses Responses that provide an excellent example of a particular scoring level Can be used to assist in the scoring process Borderline responses Even the best rubrics will result in some borderline responses Assign a score, perhaps with a /- What to do with missing responses Not-reached response Skipped response Can code as 0, or simply leave out (e.g. not use in percentage calculations)

RIGOR OF SCORING Four main options for scoring constructed response items, in order of ascending rigor: 1. Teachers directly score their own students’ work 2. Blind-scoring Teachers score randomized and anonymous selection of their own student work 3. Multi-rater A selection of student work is scored by two diff erent raters 4. Single or double scoring by third party Scoring is done by a separate, objective third party Most often related to high-stakes or standardized testing

SCORING MODERATION Activities: Ask two or more teachers to score the same portion of student work (for example, 25% of the total number of student work) A leader conducts a discussion of mismatched scores Discuss reasons for mismatched score to evaluate own scoring process Come to consensus Aims: To increase reliability Provide feedback for improving scoring rubrics For example, student responses in a single level may nevertheless look qualitatively diff erent – perhaps needing more than one level To promote professional development

BIBLIOGRAPHY E b e l , R. L . , & Fr i s b i e , D . A. ( 1 9 9 1 ) . E s s e n t i a l s o f e d u c a t i o n a l m e a s u re m e n t . E n g l e w o o d C l i ff s , N J : Pre n t i c e - H a l l . N i t ko , A. J . , & B ro o k h a r t , S . ( 2 0 0 7 ) . E d u c a t i o n a l a s s e s s m e n t o f s t u d e n t s . U p p e r S a d d l e R i v e r , N J : Pe a r s o n E d u c a t i o n , I n c . M c M i l l a n , J . H. ( 2 0 0 7 ) . C l a s s r o o m a s s e s s m e n t . P r i n c i p l e s a n d p r a c t i c e f o r e ff e c t i v e s t a n d a r d - b a s e d i n s t r u c t i o n ( 4 t h e d . ) . B o s t o n : Pe a r s o n - A l l y n & B a c o n . O re g o n D e p a r t m e n t o f E d u c a t i o n . ( 2 0 1 4 , J u n e ) . A s s e s s m e n t g u i d a n c e . Po p h a m , W. J . ( 2 0 1 4 ) . C r i t e r i o n - re f e re n c e d m e a s u re m e n t : A h a l f- c e n t u r y w a s t e d ? Pa p e r p re s e n t e d a t t h e A n n u a l M e e t i n g o f N a t i o n a l C o u n c i l o n M e a s u re m e n t i n E d u c a t i o n , P h i l a d e p h i a , PA. Po p h a m , W. J . ( 2 0 1 4 ) . C l a s s r o o m a s s e s s m e n t : W h a t t e a c h e r s n e e d s t o k n o w . S a n Fr a n c i s c o , C A: Pe a r s o n Ru s s e l l , M . K . , & A i r a s i a n , P. W. ( 2 0 1 2 ) . C l a s s r o o m a s s e s s m e n t : C o n c e p t s a n d a p p l i c a t i o n s . N e w Yo r k , N Y: M c G r a w - H i l l . S t e v e n s , D . & Le v i , A. ( 2 0 0 5 ) . I n t r o d u c t i o n t o r u b r i c s . A s a s s e s s m e n t t o o l t o s a v e g r a d i n g t i m e , c o n v e y e ff e c t i v e f e e d b a c k , a n d p r o m o t e s t u d e n t l e a r n i n g . S t e r l i n g : S t y l u s Pu b l i s h i n g , L L C Wi h a r d i n i , D . ( 2 0 1 0 ) . A s s e s s m e n t d e v e l o p m e n t I I . U n p u b l i s h e d m a n u s c r i p t . Re s e a rc h a n d D e v e l o p m e n t D e p a r t m e n t , B i n u s B u s i n e s s S c h o o l , J a ka r t a , I n d o n e s i a . Wi l s o n , M . ( 2 0 0 5 ) . C o n s t r u c t i n g m e a s u re s : A n i t e m re s p o n s e m o d e l i n g a p p ro a c h . N e w Yo r k : Ps y c h o l o g y Pre s s , Ta y l o r & Fr a n c i s G ro u p . Wi l s o n , M . , & S l o a n e , K . ( 2 0 0 0 ) . Fro m p r i n c i p l e s t o p r a c t i c e : A n e m b e d d e d assessment system. Applied Measurement in Education, 13 (2), pp. 181-208.

CREATIVE COMMONS LICENSE Scoring and Moderation PPT by the Oregon Department of Education and B e r ke l e y E v a l u a t i o n a n d A s s e s s m e n t Re s e a rc h C e n t e r i s l i c e n s e d u n d e r a C C BY 4 . 0. Yo u a r e f r e e t o : Sha re — cop y an d red ist rib u te th e ma te rial in any med iu m o r forma t Adapt — remix, transform, and build upon the material Under the following terms: A t t r i b u t i o n — Yo u m u s t g i v e a p p r o p r i a t e c r e d i t, p r o v i d e a l i n k t o t h e l i c e n s e , a n d i n d i c a t e i f c h a n g e s w e r e m a d e. Yo u m a y d o s o i n a n y r e a s o n a b l e m a n n e r , b u t n o t i n a n y w a y that suggests the licensor endorses you or your use. N o n C o m m e r c i a l — Yo u m a y n o t u s e t h e m a t e r i a l f o r c o m m e r c i a l p u r p o s e s. Sha reAlike — If you rem ix, tran sfo rm, or b uild u p on th e m at erial, yo u m ust distribu te you r contributions under the same license as the original. Oregon Department of Education welcomes editing of these resources and would g r e a t l y a p p r e c i a t e b e i n g a b l e t o l e a r n f r o m t h e c h a n g e s m a d e . To s h a r e a n e d i t e d version of this resource, please contact Cristen McLean, c r i s t e n . m c l e a n @ s t a t e . o r. u s.