Best Practices in Assessment A Guide to the White Paper

Best Practices in Assessment A Guide to the White Paper on Meta-Competency Outcomes Assessment, and Assurance of Programmatic E ff e c t i v e n e s s May 2021

Best Practices in Assessment Dr. Jean-Nicolas Poirier Council Chair Program/Inst. Representative Dr. Gary Schultz Site Team Academy Committee (Chair) Ms. Jeannette Danner Director of Accreditation Services Dr. Craig Little President Council on Chiropractic Education (CCE) 2

Modus Operandi CCE Accredita tion Standard H Standard H: Curriculum, Competencies and Outcomes Assessment: “Best practices in assessment of student learning, regardless of instructional modality, measure student proficiency in the identified metacompetency outcomes and produce data that are utilized to guide programmatic improvements.” H.2: Assessment of Learning Outcomes and Curricular Effectiveness “The DCP employs best practices to assess and demonstrate each student’s achievement of metacompetency outcomes.” Council on Chiropractic Education (CCE) 3

Overview 1) Role of the Meta-competency Outcomes in Student Learning Assessment and Program Assessment 2) Interconnection between Standards A.3: Program Effectiveness and Standard H.2: Assessment of Learning Outcomes & Curricular Effectiveness 3) Best Practices in the Assessment of Clinical Competencies (Standard H.2) 4) Program Effectiveness Processes (Standard A.3) Metrics & Council on Chiropractic Education (CCE) 4

Assessment of Student Learning & Program Assessment Role of the Meta-Competency Outcomes (MCOs) Central role in both Standards A.3 & H.2 Achievement of the 8 CCE Meta-Competencies occurs through the Meta-Competency Outcomes (MCOs) Program learning outcomes (PLOs) align to the MCOs PLOs demonstrate achievement of the MCOs, plus additional program-specific competencies, when applicable Most frequent deficiencies-concerns: Standards Council on Chiropractic Education (CCE) 5 A.3 & H.2

Interconnection Between Standards A & H Student Achievement of the MetaCompetency Outcomes (MCOs) Curricular Effectiveness & Program Assessment (Standards A & H) (Standard H.2) Demonstrate achievement of 31 MCOs prior to graduation Define an end-of-program competency threshold Majority of MCOs are designed to be assessed in a clinical setting and near the end of the program Curricular effectiveness is subset of program effectiveness/assessment Program Effectiveness: Aggregate PLOs / MetaCompetency Outcomes Other Metrics (Standard A.3) Includes Program Learning Outcomes (PLOs) / MCOs Articulate roles of committees, data collection, reporting, and analysis Aggregated assessment data informs program effectiveness and improvement plans Evidence program improvements Requires additional program metrics Council on Chiropractic Education (CCE) 6

Best Practices in Assessment of Clinical Competencies (MCOs) Valid measures Multiple methods Multiple administrations Performance thresholds Frequent feedback Multiple assessors Reliable results Faculty/assessor training Rubrics Direct & indirect assessments Cycles for data collection & analysis Program-level thresholds An assessment plan that meets the CCE Standards must demonstrate it follows best practices in the assessment of clinical competencies. Council on Chiropractic Education (CCE) 7

Best Practices in Clinical Competency Assessment Multiple Methods: One method restricts the interpretations of student achievement. Multiple methods captures the dimensions of student learning. Multiple methods provides comparative data to evaluate methods. Valid Assessments: Measures the item intended to be measured. Appropriate format: direct observation vs. chart review vs. written assignment, etc. Achievement of clinical competencies should not be measured in the pre-clinical portion of the program. Evidence: Assessment methods must measure the MCOs Council on Chiropractic Education (CCE) 8

Best Practices in Clinical Competency Assessment Multiple Administrations: Established processes, timelines and the frequency of MCO assessments Repeated assessments provide regular feedback to students & monitors their progress. Multiple Assessors: Compensate for biases, halo effect, & leniency. Dedicated assessors, faculty-clinicians, CBI & preceptorship doctors. Evidence: Clinical assessment frequency schedule & associated performance thresholds - table. Assessment Plan, Clinic Manual, & course syllabi Council on Chiropractic Education (CCE) 9

Best Practices in Clinical Competency Assessment Reliable Results: Yields consistent results regardless of when it is used and who uses it. Train assessors on methods & rubrics. Rubrics: Descriptive rubrics provide clear & specific expectations for performance. Identifies the performance target & the specific MCOs being measured Council on Chiropractic Education (CCE) 10

Best Practices in Clinical Competency Assessment Direct vs. indirect assessments: Direct assessments: directly measure a student’s competencyperformance One or more direct assessment measures for each MCO Indirect assessments are perceptions of the student’s performance, but provide important feedback to the student Cycles for data collection & analysis: H.2 requires individual students’ MCO assessment data to be aggregated for cohort analysis & to inform curricular effectiveness Aggregated data is subject to program-level performance thresholds Data is used to inform program improvements, planning, and budgeting decisions Continuous process of quality improvement through established cycles Program-level vs. student-level Programs must establish both program-level and student-level thresholds: thresholds. Student-level thresholds must be met for the individual student to graduate. Program-level thresholds monitor the DCP’s performance & are used Council on Chiropractic Education (CCE) 11 in program and curricular effectiveness processes

Program Effectiveness (PE) Metrics Common PE Metrics: CCE Standard Requirements: A.3: PE metrics for academic and nonacademic operations A.4: Policy 56 metrics and thresholds retention/attrition rates GPA/academic standing course pass rates/course completion rates AATP academic & NBCE performance OSCE/gate-keeping exam performance alumni survey results employment rate H.2: MCO assessment data F.2, I.2, and J.2: measures and thresholds for Student Services, Research and Scholarship, and Service Some latitude in developing PE metrics based on the program, but must include PLOs/MCOs, NBCE performance, and DCP completion rate. clinic/patient care quality assurance metrics & performance patient satisfaction rates student satisfaction with the program, academic student services, etc. community service participation rates technology and facility utilization student and academic services/tutoring utilization rates NBCE performance – Policy 56 NBCE Parts I–IV & sub-test analysis DCP completion rate – budget vs. actual & composite financial index (CFI) Policy 56 PLO performance MCO achievement faculty research & scholarship output Title IV default rate Council on Chiropractic Education (CCE) 12

Program Effectiveness Processes Periodic Reviews Develop performance metrics Collect data on regular cycles Improved Performance Implement Program Improvem ents Use Data to Appropriate Changes Inform Program Improvements Analyze & Planning Data to Determine Program Systematic Reviews Effectivene ss Establish Threshol ds Distribute Data/ Reports to appropriat e people Program Effectiveness is an on-going monitoring process that utilizes data to inform program improvements and planning. Regularly Reviewed Timely Fashion Council on Chiropractic Education (CCE) 13

Break Council on Chiropractic Education (CCE) 14

Q & A Session Council on Chiropractic Education (CCE) 15

Optimal Practices in Reporting Standard H.2 MCO Assessment Plan: Verification of MCO Achievement: Provide a MCO assessment map Describe the DCP verification processes for student achievement of the MCOs, prior to completing the program Describe when & how competency achievement is measured Identify the MCOs measured on each assessment method Provide associated rubrics Provide copies of completed assessment methods/tools with examples of feedback to students Identify the MCO achievement threshold(s) Distinguish the MCO achievement assessments from the course-level assessments/pre-clinical assessments how, when, who Provide evidence of MCO achievement for each student, for the two most-recent graduating cohorts Utilization of Student Learning Assessment Data: Describe the different types of student learning assessment data analyzed by the DCP Give examples of how data led to curricular changes or program improvement activities Describe how these processes are tied to processes in16 Council on Chiropractic Education (CCE) Standard A re program effectiveness, planning, and

Optimal Practices in Reporting Standard A.3 Program Effectiveness Metrics & Data: Identify the DCP’s PE metrics Provide PE data/data reports and data analyses Provide examples of how PE data were utilized to inform program improvements, for example: curricular changes faculty additions in response to studentfaculty ratios changes/expansion of academic services & tutoring based on student surveys or utilization data faculty-clinician training in response to QAchart audit results Program Planning (A.2) action-steps in response to PE data Common Mistakes or Omissions: DCP does not differentiate planning and program effectiveness processes PLO and/or MCO achievement is not included in PE metrics PE data/data reports are provided without documented analysis of data Examples of program changes are not support by PE data Committee agendas/minutes lack details to demonstrate: dissemination of PE data, analysis of data, or decision-making/action-steps based on data Initiating new processes where necessary data/evidence is not available to the site team Council on Chiropractic Education (CCE) 17

A comprehensive self-study can be written in approximately 7090 pages of narrative, (not including attachments) Write to the Standards and sub-components (A.3, A.4, etc.) Self-study Report Optimal Practices: Standard F: Student Support Services: evidence of tracking, monitoring, & analyzing student complaints/grievances Standard G: Admissions – Policy 7: evidence of monitoring AATP students’ academic performance to determine if action/support is needed, e.g. academic services, tutoring, modified scheduling, etc. “Measures and Thresholds” F. Student Support Services I. Research & Scholarship J. Service cross reference section A, when applicable Council on Chiropractic Education (CCE) 18

H.3: Quality Patient Care: requires measurable outcomes and thresholds for performance provide the clinic QA plan provide the chart audit criteria/indicators hx, examination, diagnostics, diagnosis, management plans, outcomes assessment, patient safety, informed consent, etc., Self-study Report present vs not present insufficient Optimal Practices: describe how the chart audit results/data are used for improvements, e.g.: thresholds/performance targets provide the chart audit process: who, when/frequency faculty-clinician training feedback/guidance to students chart audit data compared with MCO assessment data to identify trends & patterns address compliance with state and federal laws/regulations, and accepted industry standards via incorporated into the clinic QA plan/chart audit indicators clinic operations/policies: patient info, HIPAA training, etc. Council on Chiropractic Education (CCE) 19

New - CCE website page: Guidance Materials CCE White Paper: Meta-Competency Outcomes Assessment and Assurance of Programmatic Effectiveness New videos: What is Student Assessment What is Program-level Assessment How are Student and Program Assessment Interconnected CCE Site Team Visit Instructional Video for Attachments and PDF Council on Chiropractic Education (CCE) 20