Reliability and Validity in Research Bee Bornheimer, Robin

Reliability and Validity in Research Bee Bornheimer, Robin Fitzpatrick, Sarah Lehmann, Matt Pierce, and Maureen Whalen April 23, 2008 1

Believing what you read? “. . . there is a need for reliable and valid data on student learning outcomes.” “Validity concerns the degree to which inferences about students based on their test scores are warranted.” -Cameron, L, SL Wise, and SM Lottridge. 2007. The Development and Validation of the Information Literacy Test. College and research libraries 68 (3):229. 2

Reliable and Valid Data “The team’s combined goal was to produce a valid, reliable, authentic assessment of ICT literacy skills.” “The goal of the iSkills assessment is to measure the ICT literacy skills of students—higher scores on the assessment should reflect stronger skills.” -Katz, IR. 2007. Testing Information Literacy in Digital Environments: ETS's iSkills Assessment. Information technology and libraries 26 (3):3. 3

Reliability In statistics or measurement theory, a measurement or test is considered reliable if it produces consistent results over repeated testings. Refers to “how well we are measuring whatever it is that is being measured (regardless of whether or not it is the right quantity to measure).” -D. Rindskopf, Reliability: Measurement. In: Neil J. Smelser and Paul B. Baltes, Editor(s)-in-Chief, International Encyclopedia of the Social & Behavioral Sciences, Pergamon, Oxford, 2001, Pages 13023-13028. (http://www.sciencedirect.com/science/article/B7MRM-4MT09VJ2XN/1/083e3cc0b8b9d4e027b0ba214dcd9fa3) 4

Reliability Unlike the common understanding, in these contexts “reliability” does not imply a value judgment – Your car always starts/doesn’t start – Your friend is always/ never late 5

Classical Test Theory (CTT) A single trait or skill is being measured The trait or skill can be defined All items on the test measure the same trait or skill Formula for determining reliability Test is made more reliable by making it longer Limitation: reliability depends upon the sample group and is “not a characteristic of the test itself.” 6

Generalizability Theory (GT) Based on analysis of variance Unlike CTT, GT allows for multiple sources of error The test is designed to account for factors that researchers predict will influence scores Can compute multiple estimates of reliability 7

Item Response Theory (IRT) Like CCT, IRT measures a single trait or skill Relationship between the score on an individual test item and the skill/trait can be measured “Adaptive tests” – tests can be customized to the individual test-taker, e.g., the GRE Does not use the traditional concept of reliability 8

Observational Studies Some characteristics cannot be measured through a test Unobtrusiveness Multiple sources of error Reliability depends on the extent to which observers agree 9

Validity Evidence Content Validity: “that based on expert ratings of the items” in the test Construct Validity: “that based on the degree to which ILT scores statistically behave as we would expect a measure of information literacy to behave.” - Cameron, L, SL Wise, and SM Lottridge. 2007. The Development and Validation of the Information Literacy Test. College and research libraries 68 (3):229. 10

How can validity be established? Quantitative studies: – measurements, scores, instruments used, research design Qualitative studies: – ways that researchers have devised to establish credibility: member checking, triangulation, thick description, peer reviews, external audits 11

How can reliability be established? Quantitative studies? – Assumption of repeatability Qualitative studies? – Reframe as dependability and confirmability 12

"Reliability and validity are tools of an essentially positivist epistemology. While they may have undoubtedly proved useful in providing checks and balances for quantitative methods, they sit uncomfortably in research of this kind, which is better concerned by questions about power and influence, adequacy and efficiency, suitability and accountability. " Watling as cited in Simco & Warin, 1997, as cited in Winter, G. A comparative discussion of the notion of validity in qualitative and quantitative research. The Qualitative Report 4, nos. 3 and 4, (March 2000.). http://www.nova.edu/ssss/QR/QR4-3/winter.html. 13

Reliability and Validity Why do we bother? Terms used in conjunction with one another – Quantitative Research: R & V are treated as separate terms – Qualitative Research: R & V are often all under another, all encompassing term Semi-reciprocal relationship 14

Reliability Validity Valid Reliable Not Valid Not Reliable Not Valid 15

Winter states . . . “There is no single form, construct or concept that can universally be claimed to define or encompass the term. Neither, however, can validity be said to be a discreetly identifiable element of any research project, which is capable of being located at multiple and specific stages within research. The concept of ‘validity’ defies extrapolation from, or categorization within, any research project.” -Winter, G. A comparative discussion of the notion of validity in qualitative and quantitative research. The Qualitative Report 4, nos. 3 and 4, (March 2000.). http://www.nova.edu/ssss/QR/QR4-3/winter.html. 16

Questions 17