ASSESSMENT OF THE GEN ED PROGRAM – 2015-16 GEN ED 2.0 TASK FORCE 21

ASSESSMENT OF THE GEN ED PROGRAM – 2015-16 GEN ED 2.0 TASK FORCE 21 SEPTEMBER 2016

Overview Part 1: Why we assess. Part 2: How we currently assess the UMKC Gen Ed Core. Part 3: What we found in 2015-16. Part 4: What we are learning from the findings.

Three (or more) levels of assessment Course-level assessment Program-level assessment (degree, minor, certificate) Institutional-level assessment – the Gen Ed Program

Part 1: Why we assess.

The fine print Accountability – – State regulations HLC criteria

Why we assess Assessment helps us make sure that we are fulfilling the promises we make to our students and to society (Suskie, 2010; Finley, 2014) Our promises to students: – Institutional mission and values statements – Program’s mission statement – Program’s student learning outcomes

Purposes of Assessment Ensuring that we are delivering on what we care about: – Students get the best possible education – Students have access to the best possible learning environments, cocurricular programs, and support systems Ensuring that learning inside and outside the classroom is of appropriate scope, depth, and rigor Asking – and answering - questions we care about concerning student learning and development Identifying and preserving good practices in supporting student learning

The Important Questions Three questions to answer through assessment of student learning outcomes (SLOs): – What have our students learned? – Are we satisfied with what they’ve learned? – If not, what are we going to do about it? (Eder, 2010)

Part 2: How we currently assess the General Education Core1. 1 We will continue to assess the core we have.

Three methods – two direct, one indirect Authentic Assessment of Gen Ed SLOs – Review of student artifacts2 from gen ed courses ETS Proficiency Profile – standardized exam required of baccalaureate-seeking students prior to graduation – Reading, critical thinking, writing, and mathematics National Survey of Student Engagement (NSSE) and the Experiences with Writing Module 2 student work produced as a normal part of course requirements

How we got to where we are. GECC-Proposed Changes to Gen Education Assessment Spring 2016 – Passed by a vote of the faculty in April 2016

Process – Artifact Collection Reviewed number of courses offered in Spring 2016 – Discourse 200, Anchor 200, and Focus A grouping had more than others Drew random sample of 125 students for each course category Sent email to instructors with names of students drawn in sample with information on submitting student artifacts and assignment guidelines Docs sent to me, stripped of faculty and student identifiers

Process – Artifact Review Three rubrics developed based on AAC&U VALUE rubrics – reviewed by UAC and GECC (posted on Assessment Website) Review panel members recruited – Jerry Wyckoff and others Panels convened on May 24 and 25, 2016 – 18 faculty and staff panel members plus three facilitators – Panel members received honoraria of 200 meals Rubric norming – 3 groups (Anchor, Discourse, Focus) Each artifact reviewed by at least two panel members

Part 3: What we found in 2015-16.

Authentic Assessment: Results - Summary General Education Core Assessment – Spring 2016: Anchor 200, Discourse 200, Focus A Summary Data Tables Combined scores for two reviewers on the following scale: 8, 6-7, 4-5, 2-3, 0, with 8 indicating the highest level of achievement Table 1: Anchor 200: Culture and Diversity, Percentage of Students Achieving 4 or Higher on Each Criterion (N 75) % % % Achieving Achieving Achieving 4 or Higher 6 or Higher 8 Contextualize Information 69.3% 29.3% 0.0% Consider New Modes of Analysis 42.6% 5.3% 0.0% Think Beyond Discipline 21.4% 4.0% 0.0% Examine Human Cultures 46.6% 12.0% 0.0% Describe Global Culture 49.3% 9.3% 0.0% Describe Cultural Identity Factors 44.0% 12.0% 0.0% Table 2: Discourse 200, Percentage of Students Achieving 4 or Higher on Each Criterion (N 86) % % % Achieving Achieving Achieving 4 or Higher 6 or Higher 8 Context and Purpose 77.9% 37.2% 3.5% Content Development 75.6% 46.1% 3.5% Genre or Disciplinary Conventions 71.1% 31.5% 1.2% Sources and Evidence 59.3% 25.6% 2.3% Syntax and Mechanics 74.4% 30.2% 3.5% Cultural or Global Diversity 55.9% 18.7% 4.7% Table 3: Focus A: Arts and Humanities: Percentage of Students Achieving 4 or Higher on Each Criterion (N varies) % % % Achieving Achieving Achieving 4 or Higher 6 or Higher 8 Analyze the Human Condition – 68.0% 20.0% 4.0% Methods and Sources (n 25) Analyze the Human Condition – Quality and Clarity of Analysis 52.0% 24.0% 8.0% (n 25) Explain Context of Humanities 27.5% 10.0% 5.0% and/or Arts (n 40) Analyze Connections with Other 21.9% 9.7% 2.4% Areas of Inquiry (n 41)

General Education Core Assessment – Spring 2016: Anchor 200, Discourse 200, Focus A Summary Data Tables Outcomes for which students are meeting or exceeding expectations? Outcomes for which student are not meeting expectations?

Meeting expectations: – Discourse 200: Context and Purpose Content Development Genre or Disciplinary Conventions Syntax and Mechanics – Anchor 200: Contextualize Information 69% achieving 4 or higher OK for 200 level? – Focus A: (Small Ns) Analyze Human Condition – Methods and Sources 68% OK? Also developed in Discourse

Outcomes on which student not meeting expectations? – Interdisciplinary Thinking – Anchor 200: Think beyond the discipline - 21% achieved a 4 or higher Focus A: Analyze connection with other areas of inquiry - 22% – Culture and Diversity Anchor 200: Describe Global Culture – 49% Anchor 200: Describe Culture Identity Factors – 44% Discourse 200: Culture or Global Diversity – 56% Only courses in which specifically addressed in gen ed program

Some Lessons Learned Some Gen Ed SLOs - revise to ensure measurable – Version 2.0 must ensure outcomes are measurable Set targets for student achievement – What is good enough? Course design – – Course SLOs must be mapped to identified Gen Ed SLOs – Educational experiences must be present to support achievement of SLOs Assignments must be designed to elicit the identified SLOs Observation by Panel Members: The clearer and more detailed the assignment, the better the quality of student work

ETS Proficiency Profile ETS Proficiency Profile (EPP) Summary Results Administered to senior students – Eligible after complete 90 hours – Must sit for the exam prior to graduating Mean Scaled Total Scores and Skill Subscores for UMKC Seniors AY 2006-07 through AY 2015-16 Year 06-07 07-08 08-09 09-10 10-11 11-12 12-13 13-14 14-15 15-16 N 912 1,440 1,257 1,270 1,366 1,334 1,533 1,538 1,292 1,408 Total 451.09 451.09 452.11 452.02 449.65 450.55 453.87 452.97 452.25 448.72 Score Critical 114.27 113.93 114.37 114.23 113.72 113.73 114.21 113.69 113.36 112.27 Thinking Reading 120.34 120.13 120.82 120.42 119.98 120.32 120.65 120.53 120.38 119.27 Writing 115.05 115.20 115.27 115.25 114.8 115.06 116.11 116.03 115.82 115.20 Math 114.41 114.72 114.57 115.02 114.18 114.46 115.29 115.17 115.12 114.53 Note: Total Scaled Score possible range 400 to 500; Scaled Skill Subscores possible range 100 to 130.

About the ETS Proficiency Profile (EPP) a nationally-normed, multiple-choice test designed to assess students’ abilities in four areas: – mathematics, reading, writing, and critical thinking the learning outcomes are tested within the context of the humanities, social sciences, and natural sciences, although (disclaimer) the exam does not measure specific content in these areas, as all needed information is contained in the questions.

What are the data telling us? What are the institutional trend date telling us? – What are/should be our expectations for student achievement? What are the comparative data telling us? How can we use these data?

About the NSSE administered to first-year and senior students (every three year’s at UMKC) survey designed to evaluate – students’ engagement in programs that support their learning and personal development, and – the institution’s support of evidence-based good practices in undergraduate education, e.g., high-impact practices

Experience with Writing Module result of a collaboration between NSSE and the Council of Writing Program Administrators. Items touch on three aspects of good writing assignments — interactivity, meaning-making, and clarity. complements questions on the core survey about how much writing students do, the nature of their course assignments, and perceived gains in written expression.

What are the Experience with Writing data telling us? Are the results at a level you expected? Are the results at a level that is acceptable?

Part 4: What we are learning from the findings.

What are the implications of the data? Ensure Gen Ed SLOs are measurable Design of the Gen Ed Program: – ensure that there are sufficient and appropriate educational experiences to support achievement of the desired outcomes; – ensure that there are assignments designed to elicit the SLOs and that can be assessed across the program; and – set appropriate targets for student achievement in the aggregate of the SLOs (what is good enough). Ensure there is a feedback loop for assessment results and accountability for making programmatic/course alternations to enhance student learning