Faculty Development Workshop East Tennessee State University College

Faculty Development Workshop East Tennessee State University College of Nursing Cognitive Code and Test Construction: Writing Questions that Promote Critical-Thinking Carolyn S. Merriman, RN, MS, APRN Associate Professor Fall 2017

At the end of this session the participants should be able to: 1. Understand what is a cognitive code and how it can guide writing criticalthinking questions; 2. The purpose of writing critical thinking questions in an academic setting; 3. Describe the components of writing a critical-thinking question; 4. Revise a test question to make it a higher level, critical thinking test item.

What is the purpose of tests at ETSU’s College of Nursing? Write down a few thoughts about this

What is the purpose of tests in the College of Nursing? Test basic nursing knowledge? Prepare graduates for standardized tests: NCLEX-RN? Internal curriculum evaluation? Promote critical thinking in practice?

What is a Cognitive Code? It deals with knowledge and higher level thinking. Includes: 1.Acquisition of facts and information 2.Use of knowledge in practice 3.Higher level cognitive skills (Oermann & Gaberson, 2017)

Bloom’s Taxonomy Old and New

Bloom’s Taxonomy Newest Version Remembering: Can the student recall or remember the information? define, duplicate, list, memorize, recall, repeat, reproduce, state Understanding: Can the student explain ideas or concepts? classify, describe, discuss, explain, identify, locate, recognize, report, select, translate, paraphrase Applying: Can the student use the information in a new way? choose, demonstrate, dramatize, employ, illustrate, interpret, operate, schedule, sketch, solve, use, write. Analyzing: Can the student distinguish between the different parts? appraise, compare, contrast, criticize, differentiate, discriminate, distinguish, examine, experiment, question, test. Evaluating: Can the student justify a stand or decision? appraise, argue, defend, judge, select, support, value, evaluate Creating: Can the student create new product or point of view? assemble, construct, create, design, develop, formulate, write.

Using Cognitive Code Important to decide on the level of cognitive learning needed Then develop outcomes and assessment methods for that level (Oermann & Gaberson, 2017)

sed Cognitive Code sification of Cognitive Levels from NCSBN College of Nursing’s Cognitive Code Guidelines taxonomy for the cognitive domain is used as a basis for writing and coding items for the examination (Bloom, et al., 1956; Anderson & Krathwohl, 2001). Sinc Semester Semester I Improvement % Application Analysis Semester II Improvement % Application Analysis Semester III Improvement % Application Analysis Semester IV Improvement % Application Analysis Knowledge Applicati on Analysis Synthesis 50% 40% 50% 10% 0% 25% 55% 70% 15% 5% 20% 55% 75% 20% 5% 10% 60% 85% 25% 5%

College of Nursing’s Cognitive Code Guidelines

Internal Curriculum Evaluation Process: Writing critical thinking test items. Measuring test items ability to discriminate between those who know the content and those who do not. Designing valid exams Measuring the reliability of exams.

Test construction is Multi-step: (Sax, 1997) 1. Identifying reason: Course Objectives 2. Determining best type of test items for evaluation of target audience 3. Creating a Test Blueprint

Test construction is Multi-step: 1. Identifying reason: Use course objectives and subobjectives to guide test construction

Test construction is Multi-step: 2. Determining best type of test items for evaluation of target audience What level are my students? – Beginners, Juniors, Seniors: Apply the Cognitive Code Guidelines

Test construction is Multi-step: 3. Test Blueprint: Content Validity Use course objectives Identify categories: Nursing Process, Client Needs Bloom’s Taxonomy, NCSBN, AACN, NLN, QSEN Determine mastery questions in advance (90% or above)

Faculty Made Test Blue Print 16 www.utexas.edu/provost/sacs/pdf/ Objectiv eA Objectiv eB Objectiv eC Objectiv eD Total 1 1 3 (7.5%) Remembe r 1 Understa nd 2 1 2 2 7 (17.5%) Apply 4 4 3 4 15 (37.5%) Analyze 3 2 3 2 10 (25%) 1 2 (5%) 2 1 3 (7/.5% 11 11 40 (100%) Evaluate 1 Create Total 10 8

What is so Important about a Test Blueprint? 1. 2. 3. Useful tool guiding the item writer so sufficient items developed at appropriate level. Provides accountability for educational outcomes produces. Can be used to inform students about the nature of the test and how they should prepare for it.

Skills needed to write critical thinking test items: 1. Sound clinical knowledge 2. Proper use of the English language, including grammar. 3. Ability to conduct a systematic exam analysis using item analysis data to evaluate worth of test items. 4. Editing ability to enhance test items’ proficiency in measuring students’ knowledge and problem solving ability.

Guidelines to Developing Critical-Thinking Exams 1. How do I get started? 2. What do I need to be a better critical- thinking test item writer? 3. What helpful hints will guide me?

How do I get started? A. Understand the effects of a sound curriculum design Instruction Objectives BALANCE Outcomes Evaluation

How do I get started? B. Gather resources for test ideas

How do I get started? B. Gather resources for test ideas Patient scenarios from clinical or practice Review books Test banks Case Studies

What do I need to be a better critical- thinking test item writer? Define CriticalThinking

What do I need to be a better critical- thinking test item writer? Define Critical-Thinking “The process of analyzing and understanding how and why a certain conclusion was reached.” Morrison, Nibert, Flick, 2006

How can the importance of critical thinking skills be conveyed to students?

Making Connections with Students Early and often talk to students about the role of critical thinking tests in nursing school and their ability to use critical thinking in practice.

What do I need to be a better critical- thinking test item writer? Critical-Thinking Tests 1. (Morrison, et.al. 2006) Rationale 2. Application or above cognitive level 3. Multilogical thinking 4. High level of discrimination/plausible alternatives

Critical-Thinking Tests: 1. Rationale Rationales should include why an option is correct and why the distracters are incorrect. Makes the exam process a teaching tool. Test review: Learning experience, faculty friendly, and legally defensible.

Critical-Thinking Tests: 2. Application or Above Bloom’s Taxonomy: Cognitive Levels Remembering Understanding Applying Analyzing Evaluating Creating

Critical-Thinking Tests: Application or Above Application: Verbs Apply, Calculate, Classify, Develop, Examine, Generalize, Illustrate, Show, Solve, Use

Critical-Thinking Tests: 3. Multi-logical thinking Thinking that requires knowledge of more than one fact to logically and systematically apply concepts to a clinical problem (Morrison, et. al., 2006) Viewing problems from more than one point of view or frame of reference.

Critical-Thinking Tests: Multi-logical thinking What intervention is best in preventing increased intracranial pressure? (Analyzing level: Cognitive Code VS What is the definition of intracranial pressure? (Understanding level: Cognitive Code)

Critical-Thinking Tests: 4. High level of Discrimination Use key words that require more than regurgitation of facts. All options should be plausible, with one being more important than others.

Key words that demonstrate high level of discrimination Best Most Important First Highest Priority

The nurse knows that which assessment finding is characteristic of a client with Parkinson’s disease? A. B. C. D. Night blindness. Pain in lower extremities. Shuffling gait. Incontinence.

The nurse knows that which assessment finding is characteristic of a client with Parkinson’s disease? A. B. C. D. Night blindness. Pain in lower extremities. Shuffling gait. Incontinence. What Cognitive Code level is this question?

The nurse knows that which assessment finding is characteristic of a client with Parkinson’s disease? A. B. C. D. Night blindness. Pain in lower extremities. Shuffling gait. Incontinence. Understanding: It is not Critical thinking. Not a high level of discrimination. Does not require multilogical thinking to answer.

The nurse is making a home visit to a 75-year-old male client who has had Parkinson’s disease for the past five years. Which finding has the greatest implication for this client’s care? A. The client’s wife tells the nurse that the grandchildren have not been to visit for over a month. B. The nurse notes that there are numerous throw rugs throughout the client’s home. C. The client has a towel wrapped around his neck that the wife uses to wipe her husband’s face. D. The client is sitting in an arm chair, and the nurse notes that he is gripping the arms of the chair.

Rationale Parkinson’s disease is characterized by a shuffling gait, and throw rugs throughout the home pose a safety hazard for this client (B). Visits from the grandchildren (A) may or may not be significant to this client, and the nurse should gather more information about the client’s feelings regarding such visits. Drooling is also characteristic of Parkinson’s disease, and a towel wrapped around the client’s neck (C) may be a good means of keeping the client dry, however, the nurse may need to counsel the family to be sure the client is dry and clean. Tremors are characteristic of Parkinson’s disease and grasping the arms of a chair (D) may help to control tremors in the hands and arms.

What Cognitive Code level is this question?

Cognitive Code Level: Applying Rationale Written at the application level Requires a high level of discrimination to choose the right answer from among plausible choices Requires multilogical thinking to answer--a shuffling gait (which puts the client at risk for falls) is a sign of Parkinson's’ disease, and must know what nursing care is most important for this client—promotion of safety. Throw rugs pose a safety hazard.

Testing Stems that Promote Critical Thinking Which intervention is most important? Which intervention, plan, assessment data is/are most critical to developing a plan of care? Which intervention should be done first? What action should the nurse take first? Which intervention, plan, nursing action has the highest priority? What response is best?

What do I need to be a better critical- thinking test item writer? Understand Terms and Use Specific Steps as a Guide Chapter 5 in Oermann and Gaberson Evaluating and Testing in Nursing Education

General Rules for Writing Test Items (Oermann & Gaberson, 2017) Make sure every item is measuring something important! Every item should have a correct answer. Use simple, clear, concise, precise, grammatically correct language. Avoid jargon, slang, or unnecessary abbreviations.

General Rules for Writing Test Items-con’t. (Oermann & Gaberson, 2017) Use positive wording. Don’t give irrelevant clues to the correct answer. Items should not depend on another. Eliminate extraneous information. Arrange for a critique and prepare more items than the blueprint specifies.

Helpful hints in test-writing? No proper names Eliminate ‘multiple’ multiples Non-sexist language Short and Sweet: – Cross out “of the following” – Delete scenarios Items cry out for Independence !

More helpful hints Ask a single, clearly defined question Viable options Stem-Option Relationships No Freebies Person-first language Clear Option Ranges Logical order Vary correct answer

And Even More Helpful Hints Eliminate “all of the above” and “none of the above” Rewrite any “all except” questions Ensure that alternatives do not overlap Style drug names uniformly Be aware of verbs and other writing issues

And the MOST IMPORTANT Helpful Hint Abide by the Written Testing Policy and Cognitive Code Guidelines for ETSU College of Nursing!

Use of Statistics Important to use statistics to evaluate individual items and overall test reliability and validity.

Writing Critical Thinking Test Questions: Hands-on Practice

Cognitive Code and Test Construction: Summary 1. Understand what cognitive code is and how to use it to write critical thinking tests. 2. Use guidelines to write critical thinking items. 3. Implement test-writing tips to revise and improve test questions 4. Keep practicing to get better!

Thank you for your attention !

References McDonald, M. (2002). Systematic assessment of learning outcomes: Developing multiple-choice exams. Sudbury, MA: Jones and Bartlett Publishers, Inc. and National League for Nursing. Morrison, S., Nibert, A., Flick, J. (2006). Critical thinking and test item writing (2nd ed.): Houston, TX: Health Education Systems, Inc. Oermann, M., and Gaberson, K. (2017). Evaluation and testing in nursing education (5th ed.): New York, NY: Springer Publishin Company. Sax, G. (1997). Principles of educational and psychological measurement and evaluation (4th ed.): Belmont, CA: Wadsworth Publishing Company.