Evaluation of Educational Innovations as Scholarship Presented

Evaluation of Educational Innovations as Scholarship Presented by: Andrea Berry and Magda Pasarica Developed by: Medical Education Scholarship Research & Evaluation (MESRE) Janet Riddle, Anna Cianciolo, Andrea Berry, Maria Blanco, Nicole Borges, Christy Boscardin, Kathryn Huggett, Steven Rougas, Sebastian Uijtdehaage, Anne Vo A section of the AAMC Group on Educational Affairs www.aamc.org/members/gea/gea sections/mesre/

OBJECTIVES Through analysis of the educational innovation case studies provided, workshop participants will be able to: – Utilizing a logic model, select program outcomes for their own innovations that will be valued by key stakeholders – Describe program processes that are likely to contribute to the selected outcomes – Describe a “theory-in-action” or rationale for how program processes lead to program outcomes – Identify features within the context in which the innovation is implemented that are likely to impact or generalize program outcomes

What is an innovation? New process or event Curricula Kanter SL. Toward better descriptions of innovations. Acad Med 2008 Aug;83(8):703-4. Solutions to problems New application or perspective Assessment tools

Evaluation: defined Systematic determination of the merit or worth of an object (educational intervention). (Scriven, 1967) Utilization-focused program evaluation: done for and with specific intended primary users for specific, intended uses. (Patton, 2008) What? So what? Now what? (Patton, 2008) Numerous additional definitions.

Evaluation of Educational Innovations Understanding how and why an innovation “works” Diagnose and document program strengths and weaknesses Inform the allocation of resources Determine that educational objectives met To determine next steps/make decisions To document success Dissemination of best practices

Going Beyond “Did it work?”* Planned Did it work? How did it work? Emergent What else happened? How is it happening? *Haji F, Morin M-P, Parker K. Rethinking programme evaluation in health professions education: beyond “did it work?’ Med Educ 2013;47:342-351

Evaluation Theory Logic Model Goal Oriented/Objective-Based (Tyler) Goals-free Evaluation (Scriven) Kirkpatrick’s 4-level model CIPP (Stufflebeam)

www.uwex.edu/ces/pande/evaluation/evallogicmodel.html

Inputs I-Pass study group education executive committee Support from Initiative for Innovation in Pediatric Education Support from institutional leadership Outputs Activities Team-building strategies Educational frameworks and guiding principles Development of curricular components 267 I-PASS faculty champions recruited 855 residents trained 888 faculty observations of resident handoffs 48 academic presentations Partnership with TeamSTEPPS leadership 1007 requests for curricular materials Grant funding PMID: 24871238 Participation Assumptions Outcomes – Impact Short Term Long Term Residents have positive perceptions of and accept the curriculum Impact on: I-PASS handoff process spreads within study sites Examining how: I-PASS handoff process adopted by other disciplines and provider types Modify quality of IPASS Handoff Bundle to be determined Medical errors Resident workflow Verbal and written miscommunications Hospital-level factors Patient-level factors External Factors Evaluation

SMALL GROUP EXERCISE

Now It’s Your Turn! Use the small group case vignette to fill out the blank logic model Consider: – How does the innovation “work”? problem – inputs – activities – outputs – outcomes – Why does (or might) this innovation “work”? context – assumptions – theory or rationale

The Associate Dean for Undergraduate Medical Education at your institution has charged all clerkship directors with developing a Wellness Curriculum for third-year medical students. You are the Clerkship Director for Family Medicine and have been appointed chair of the wellness curriculum development committee. To guide your efforts, the committee will use the Logic model for both conducting an initial needs assessment and planning for the eventual program evaluation. Medical students at your institution have favorable views of the clinical learning environment. Even so, most of students report increased fatigue and concerns about making a specialty choice during their clerkships. The members of the curriculum development committee are willing to work hard on this project and they expect that this curriculum development effort will result in multiple works of peer-reviewed scholarship.

Small Group Debrief What did you gain from working through the logic model? What did you think about the checklist? What was most challenging?

INNOVATIONS AS SCHOLARSHIP

https://www.mededportal.org/submit/instructions/

Why YOU, as an Educator, Are PERFECT for Scholarly Program Evaluation Careful, systematic documentation YOU have first-hand exposure to: Learners’ psychological experiences (e.g., via logic modeling) of your educational journey will help you Contextual factors generate very practical theory. That YOU are the expert on: is, to conduct SCHOLARSHIP! Subject matter of the intervention Opportunities for and constraints on teaching and learning Scholarly Program Evaluation is a form of “participatory action research”* Knowledge is built through a sequence of data collection, reflection, and action Educators participate as researchers Inquiry is context-bound *Baum F, MacDougall C, Smith D. Participatory action research. J Epidemiol Community Health 2006; 60(10): 854-7.

Common Limitations of Medical Education Program Evaluations* 1.Focus on Outcomes Only – No characterization of the psychological experience/educational process – Limited description of context – Limited description of other potential solutions 2.Insufficient Outcomes – No behavioral data (e.g., reactions, satisfaction, attitudes only) – Self-reported – Superficial measures of learning (e.g., knowledge quizzes) 3.Lack of a Comparison Group 4.Other Methodological Flaws – Small, non-representative sample – Non-validated outcome measures No generalizability/sustainability *1. Cook, DA.– If you teach them, they will learn: why medical education needs comparative effectiveness research. Adv Health Sci Educ 2012; 17: 305-10. 2. Cook DA, Beckman TJ. Reflections on experimental research in medical education. Adv Health Sci Educ 2010; 15: 455-64. 3. Kanter SL. Toward better descriptions of innovations. Acad Med 2008 Aug;83(8):703-4.

How Can We Make Program Evaluation ‘Count’ as Scholarship? Include Detailed Description of the Educational Program – Theory, Resources Used, Similar Programs Include Descriptions of the Process Outcomes – How was the Program Implemented & Resources Used – Use multiple outcomes data (e.g. subject scores, faculty evaluations, observations, social networks) Collaborate across multiples sites or multiple programs – Contextualize differential outcomes based on trainee characteristics or program implementation status – Deliberate case studies Review and include validity information on the tools – Conduct validation study whenever possible – Use literature review to support your validity information

PUBLISHING INNOVATIONS

Tips for Finding the Right Journal 1 Read the journal description & scope 2 Read the instructions for authors 3 Review a few examples from that journal category 4 If you are unsure, email the editor or editorial team

Reflection or Advice JGME Perspectives On teaching/on learning Medical Teacher Twelve tips Personal view Medical Education When I say Academic Medicine Last page Teaching and learning moments Journal of Interprofessional Care IPEP guides Teaching and Learning in Medicine Observations Curriculum MedEdPortal MedEdWorld Innovation JGME Educational innovation Brief report Medical Teacher Short communications Medical Education Really good stuff Academic Medicine Innovation reports Journal of Interprofessional Care Short reports Teaching and Learning in Medicine Educational case reports Medical Science Educator Innovation Short communications Adapted from: Blanchard RD, Nagler A, Artino AR Jr. Harvest the Low-Hanging Fruit: Strategies for Submitting Educational Innovations for Publication. J Grad Med Educ. 2015 Sep;7(3):318-22.

MESRE Annotated Bibliography https://www.aamc.org/download/456646/data/annotated-bibliography-of-journals-march2016.pdf

THANK YOU!