Assessing Critical Thinking Skills DAVID K. KNOX, PH.D., DIRECTOR OF

Assessing Critical Thinking Skills DAVID K. KNOX, PH.D., DIRECTOR OF CLEMSON THINKS 2 (CT 2 )

What is ”Critical” in Critical Thinking? κριτικός (Kritikos), relating to judging, fit for judging, skilled in judging, able to discern (Plato, Plutarch, Lucian, others). With the genitive of the object, for example: κριτικὸς ἐνθυμήσεων καί ἐννοιῶν καρδίας - able to judge the thoughts and intentions of the heart.

Attitude Knowledge Thinking Skills Critical Thinking “Proficiency in reading, writing, and arithmetic has traditionally been the entry-level threshold to the job market, but the new workplace requires more from its employees. Employees need to think critically, solve problems, innovate, collaborate, and communicate more effectively.” –– American Management Association, 2010 (quoted in Diane Halpern: Thought and Knowledge, page 5.)

Hart Research Associates Study: 2009* Intellectual and practical skills The ability to communicate effectively, orally and in writing (89%) Critical thinking and analytical reasoning skills (81%) The ability to analyze and solve complex problems (75%) Teamwork skills and the ability to collaborate with others in diverse group settings (71%) The ability to innovate and be creative (70%) The ability to locate, organize, and evaluate information from multiple sources (68%) The ability to work with numbers and understand statistics (63%) * http://www.aacu.org/sites/default/files/files/LEAP/2009 EmployerSurvey.pdf

Hart Research Associates Study: 2009* * http://www.aacu.org/sites/default/files/files/LEAP/2009 EmployerSurvey.pdf

Critical Thinking, Defined “Critical thinking is the process of purposeful, reflective judgment. Critical thinking manifests itself in giving reasoned and fair-minded consideration to evidence, conceptualizations, methods, contexts, and standards in order to decide what to believe or what to do.” (Facione, 2010). “The intellectually disciplined process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication, as a guide to belief and action”(Scriven and Paul, 1987).

Critical Thinking, Defined “Critical thinking is the use of those cognitive skills or strategies that increase the probability of a desired outcome. It is used to describe thinking that is purposeful, reasoned, and goal directed – the kind of thinking involved in solving problems, formulating inferences, calculating likelihoods, and making decisions, when the thinker is using skills that are thoughtful and effective for the particular context and type of thinking task.” (Halpern, 2014) “Being a critical thinker involves more than cognitive activities such as logical reasoning or scrutinizing arguments for assertions unsupported by empirical evidence. Thinking critically involves our recognizing the assumptions underlying our beliefs and behaviors. It means we can give justifications for our ideas and actions. Most important, perhaps, it means we try to judge the rationality of these justifications. We can do this by comparing them to a range of varying interpretations and perspectives. We can think through, project, and anticipate the consequences of those actions that are based on these justifications. And we can test the accuracy and rationality of these justifications against some kind of objective analysis of the "real" world as we understand it.” (Brookfield, 1987)

Critical Thinking, Defined “It is the mark of an educated mind to be able to entertain a thought without accepting it.” ― Aristotle, Metaphysics “All that we are is the result of what we have thought: it is founded on our thoughts and made up of our thoughts.” –– Buddha, Dhammapada “Learning without thought is labor lost; thought without learning is perilous” –– Confucius, Analects “Read not to contradict and confute; nor to believe and take for granted; nor to find talk and discourse; but to weigh and consider” –– Francis Bacon, “Of Studies”

Critical Thinking Mindset: Eagerness and Skills “A person with a strong disposition toward critical thinking has the consistent internal motivation to engage problems and make decisions by critical thinking. Operationally this means three things: The person consistently values critical thinking, believes that using critical thinking skills offers the greatest promise for reaching good judgments, and intends to approach problems and decisions by applying critical thinking skills as best as he or she can. This combination of values, beliefs, and intentions forms the habits of mind that dispose the person toward critical thinking.” –– Peter Facione, Think Critically, Chapter 2.1

Critical Thinking Mindset: Eagerness and Skills “Everyone thinks; it is our nature to do so. But much of our thinking, left to itself, is biased, distorted, partial, uninformed, or down-right prejudiced. Yet the quality of our life and that of what we produce, make, or build depends precisely on the quality of our thought. Shoddy thinking is costly, both in money and in quality of life. Excellence in thought, however, must be systematically cultivated” –– Richard Paul and Linda Elder, Critical Thinking Concepts and Tools, pg. 2

Critical Thinking Mindset: Eagerness and Skills Universal Intellectual Standards: Clarity: Could you elaborate further on that point? Could you express that point in another way? Could you give me an illustration?” Could you give me an example? Accuracy: Is that really true? How could we check that? How could we find out if that is true? Precision: Could you give me more details? Could you be more specific? Relevance: How is that connected to the question? How does that bear on the issue?

Critical Thinking Mindset: Eagerness and Skills Universal Intellectual Standards(cont’d) Depth: How does your answer address the complexities in the question? How are you taking into account the problems in the question? Are you dealing with the most significant factors? Breadth: Do we need to consider another point of view? Is there another way to look at this question? What would this look like from the point of view of X? Logic: Does this really make sense? Does this follow from what you said? How does that follow? Fairness: Are we considering all relevant viewpoints in good faith? Are we distorting some information to maintain our biased perspective? Are we more interested in our vested interests than the common good? –– Richard Paul and Linda Elder, Critical Thinking Concepts and Tools, pgs. 8,9

CT2 Points of Assessment: Classes Pre and Post California Critical Thinking Skills Tests CT Artifacts Additional questions for CT2 classes on Evaluation of Instructor forms ETS Proficiency Profile (Freshmen and Seniors) Clemson Educational Profile Transferrable Skills NSSE module Pecha Kucha

CT2 Points of Assessment:

Critical Thinking Tests Higher Education: General-Content, Multi-Aspect* ACT CAAP Test Module: Critical Thinking. . http://www.act.org/caap/index.html (retiring in January 2018) The California Critical Thinking Skills Test (CCTST): College Level. http://www.insightassessment.com The California Critical Thinking Dispositions Inventory (CCTDI). http://www.insightassessment.com Cambridge Thinking Skills Assessment. Cambridge Assessment http://www.admissionstests.cambridgeassessment.org.uk/adt/tsacambridge/about Collegiate Learning Assessment (CLA ). http://www.collegiatelearningassessment.org/ The Critical Thinking Assessment Test (CAT). http://www.tntech.edu/cat/using/ Cornell Critical Thinking Test, Level Z. https://www.criticalthinking.com/cornell-critical-thinking-test-level-z.html ETS Proficiency Profile. http://www.ets.org/proficiencyprofile/about *Adapted from: AN ANNOTATED LIST OF ENGLISH-LANGUAGE CRITICAL THINKING TESTS: Robert H. Ennis and Geoffrey Scott Chattin, University of Illinois UC

Critical Thinking Tests Higher Education: General-Content, Multi-Aspect* Halpern Critical Thinking Assessment (HCTA) https://sites.google.com/site/dianehalperncmc/home/research/halpern-critical-thinking-assessment Holistic Critical Thinking Scoring Rubric (HCTSR). http://www.insightassessment.com/ International Critical Thinking Basic Concepts & Understanding Online Test: Assessing Initial Understanding of Basic Critical Thinking Concepts and Principles. http://www.criticalthinking.org/pages/online-critical-thinking-basic-concepts-test/679 International Critical Thinking Essay Test. http://www.criticalthinking.org/store/products/international-critical-thinking-test/185 James Madison Test of Critical Thinking (part of the James Madison critical thinking course), https://www.criticalthinking.com/james-madison-critical-thinking-course-student-book.html . New Jersey Test of Reasoning Skills. http://cehs.montclair.edu/academic/iapc/ Oxford Thinking Skills Assessment. Cambridge Assessment. http://www.admissionstests.cambridgeassessment.org.uk/adt/tsaoxford/about Watson-Glaser II Critical Thinking Appraisal. https://us.talentlens.com/watson-glaser-critical-thinking-test *Adapted from: AN ANNOTATED LIST OF ENGLISH-LANGUAGE CRITICAL THINKING TESTS: Robert H. Ennis and Geoffrey Scott Chattin, University of Illinois UC

Critical Thinking Tests The California Critical Thinking Skills Test (CCTST) The California Critical Thinking Skills Test (CCTST) is an objective measure of the core reasoning skills needed for reflective decision making concerning what to believe or what to do. The CCTST is designed to engage the test-taker’s reasoning skills. Multiple choice items use everyday scenarios, appropriate to the intended test-taker group. Each item requires that the test-taker make an accurate and complete interpretation of the question. Any specialized information needed to respond correctly is provided in the question itself. The test items range in difficulty and complexity. Different questions progressively invite test-takers to analyze or to interpret information presented in text, charts, or images; to draw accurate and warranted inferences; to evaluate inferences and explain why they represent strong reasoning or weak reasoning; or to explain why a given evaluation of an inference is strong or weak. The instrument is typically administered in 45-50 minutes; the length of the instrument is set to permit maximum performance within the range of possible effort for the intended test-taker group.

Critical Thinking Tests The California Critical Thinking Skills Test (CCTST) Scale Descriptions Reasoning Skills - Overall: The Reasoning Skills Overall describes overall strength in using reasoning to form reflective judgments about what to believe or what to do. To score well overall, the test-taker must excel in the sustained, focused and integrated application of core reasoning skills including analysis, interpretation, inference, evaluation, explanation, induction and deduction. The Overall score predicts the capacity for success in educational or workplace settings which demand reasoned decision making and thoughtful problem solving. Analysis: Analytical reasoning skills enable people to identify assumptions, reasons and claims, and to examine how they interact in the formation of arguments. We use analysis to gather information from charts, graphs, diagrams, spoken language and documents. People with strong analytical skills attend to patterns and to details. They identify the elements of a situation and determine how those parts interact. Strong interpretation skills can support high quality analysis by providing insights into the significance of what a person is saying or what something means. Inference: Inference skills enable us to draw conclusions from reasons and evidence. We use inference when we offer thoughtful suggestions and hypotheses. Inference skills indicate the necessary or the very probable consequences of a given set of facts and conditions. Conclusions, hypotheses, recommendations or decisions that are based on faulty analyses, misinformation, bad data or biased evaluations can turn out to be mistaken, even if they have been reached using excellent inference skills.

Critical Thinking Tests The California Critical Thinking Skills Test (CCTST) Scale Descriptions, Continued Evaluation: Evaluative reasoning skills enable us to assess the credibility of sources of information and the claims they make. And, we use these skills to determine the strength or weakness of arguments. Applying evaluation skills we can judge the quality of analyses, interpretations, explanations, inferences, options, opinions, beliefs, ideas, proposals, and decisions. Strong explanation skills can support high quality evaluation by providing the evidence, reasons, methods, criteria, or assumptions behind the claims made and the conclusions reached. Deduction: Decision making in precisely defined contexts where rules, operating conditions, core beliefs, values, policies, principles, procedures and terminology completely determine the outcome depends on strong deductive reasoning skills. Deductive reasoning moves with exacting precision from the assumed truth of a set of beliefs to a conclusion which cannot be false if those beliefs are true. Deductive validity is rigorously logical and clear-cut. Deductive validity leaves no room for uncertainty, unless one alters the meanings of words or the grammar of the language. Induction: Decision making in contexts of uncertainty relies on inductive reasoning. We use inductive reasoning skills when we draw inferences about what we think must probably be true based on analogies, case studies, prior experience, statistical analyses, simulations, hypotheticals, and familiar circumstances and patterns of behavior. As long as there is the possibility, however remote, that a highly probable conclusion might be mistaken, the reasoning is inductive. Although it does not yield certainty, inductive reasoning can provide a solid basis for confidence in our conclusions.

Critical Thinking Tests The California Critical Thinking Skills Test (CCTST) Sample Disclosed Question (similar to actual question on test)

Pre-Test Post-Test

Core Critical Thinking Skills Measured by the CCTST

Critical Thinking Tests The Critical Thinking Assessment Test (CAT) The CAT instrument, developed at Tennessee Tech, is a unique tool designed to assess and promote the improvement of critical thinking and real-world problem solving skills. The instrument is the product of extensive development, testing, and refinement with a broad range of institutions, faculty, and students across the country. The National Science Foundation has provided support for many of these activities. The CAT instrument is designed to assess a broad range of skills that faculty across the country feel are important components of critical thinking and real world problem solving. The test was designed to be interesting and engaging for students. All of the questions are derived from real world situations. Most of the questions require short answer essay responses, and a detailed scoring guide helps ensure good scoring reliability.

Critical Thinking Tests The Critical Thinking Assessment Test (CAT) Skills Assessed by CAT Instrument Evaluating Information Separate factual information from inferences. Interpret numerical relationships in graphs. Understand the limitations of correlational data. Evaluate evidence and identify inappropriate conclusions. Creative Thinking Identify alternative interpretations for data or observations. Identify new information that might support or contradict a hypothesis. Explain how new information can change a problem. Learning and Problem Solving Separate relevant from irrelevant information. Integrate information to solve problems. Learn and apply new information. Use mathematical skills to solve real-world problems. Communication Communicate ideas effectively.

Critical Thinking Tests Sample Disclosed Question (similar to actual question on test)

Critical Thinking Tests Q. What did you like most about the CAT Test Scoring Session? Response Interacting with people from across campus Interaction with colleagues and fellow faculty members. The collegiality and opportunity to work with people from outside of my discipline. The conversations we had at our table, both formal and informal. Interacting with other faculty from this group The interaction with the other faculty members made the scoring sessions quite enjoyable. The camaraderie and the intellectual discussions The discussions at the table about the scoring, etc. Talking about how to apply the rubric in particular cases was, at times, eyeopening. Collegiality among all the participants The opportunity to exchange ideas with other colleagues. Meeting other faculty from other departments. Interaction with other like-minded CT2 faculty. Interaction with faculty while sharing a common goal The camaraderie and the questions. interaction with faculty from various disciplines Engaging in dialogue to reach consensus with colleagues

Critical Thinking Tests

Assessments: CCTST vs. CAT California Critical Thinking Skills Test (CCTST) 34 multiple choice questions 45 minutes Online or paper Graded online (online version) Multi-Lingual Versions 10 per test **Students are tested during the first and final weeks of classes Critical Thinking Assessment Test (CAT) Essay: 15 questions One hour Paper and pencil Faculty graded Faculty development potential 9.95 per test and 300 annual fee

Assessments: Profile ETS Proficiency As the only integrated test of general education skills, the ETS Proficiency test assesses four core skill areas - critical thinking, reading, writing and mathematics - in a single test that the Voluntary System of Accountability (VSA) has selected as a gauge of general education outcomes. Clemson University uses the ETS Proficiency Profile test to: Function as “bookends” in the CT2 assessment process measure and document program effectiveness to meet requirements for accreditation and program funding assess student proficiency in core academic skill areas to identify strengths, weaknesses and opportunities to improve curriculum and instruction compare performance against programs at more than 380 institutions nationwide conduct benchmark and trend analyses to evaluate program improvement efforts and overall learning outcomes Clemson University began in Fall 2011 to administer the ETS Proficiency Profile to all incoming freshmen. These same students are tested when they are seniors and we will be able to longitudinally track their progress. This provides us with a level of insight into the “value-added” aspect of a Clemson education never possible before.

Assessments: Profile ETS Proficiency ETS Proficiency Profile Standard Form The Standard form of the ETS Proficiency Profile test is intended to provide information about individual students as well as groups of students. It consists of 108 questions, divided into two sections of 54 questions each. The two sections may be administered either in a single two-hour testing session or in two separate, one-hour testing sessions. The Standard form includes: 27 questions testing critical thinking skills 27 questions testing reading skills 27 questions testing writing skills 27 questions testing mathematics skills Additionally, students taking the Standard form can earn a Certificate of Achievement based on how well they perform.

Assessments: Profile ETS Proficiency ETS Proficiency Profile Abbreviated Form The Abbreviated Form of the ETS Proficiency Profile test is intended to provide information about groups of at least 30 students. It is administered in a single 40-minute testing session. The Abbreviated Form does not provide information about individual students. The Abbreviated form is created by dividing the Standard Form into three smaller forms of 36 questions each. The three Abbreviated Forms are packaged in alternating sequence, so that each version is taken by one-third of the students. The 108 questions in the Standard Form are assigned to the three Abbreviated Forms to make each of them a miniature version of the Standard Form. Each of the three Abbreviated Forms includes: Nine questions measuring critical thinking skills Nine questions measuring reading skills Nine questions measuring writing skills Nine questions measuring mathematics skills

Artifacts Defined: ar·ti·fact noun \ˈär-ti-ˌfakt\ a: something created by humans usually for a practical purpose; especially : an object remaining from a particular period. Each course submits at least one type of artifact Can submit different artifacts Many classes have a specific assignment that will be designated the critical thinking artifact. Random sample from class is scored Scoring is performed by CT2 Faculty Scholars using the AAC&U Critical Thinking rubric Examples: Essay Project Critique Video Poster Presentation Portfolio

Artifacts: AAC&U Critical Thinking VALUE Rubric

Artifacts: “Visual Essay”

NSSE: Development of Transferrable Skills Topical Module

Pecha Kucha Pecha Kucha is a presentation format where you show 20 images, each for 20 seconds. The images advance automatically and you talk along to the images. A great tool for developing critical thinking skills. An effective tool for assessing critical thinking skills (using a rubric such as the AAC&U CT Rubric).

Thank You! Please visit our web site for further information: http://www.clemson.edu/thinks2 2016 Clemson Thinks2 Faculty Institute