Academic Program Review Orientation 2023-24 Cycle

Academic Program Review Orientation 2023-24 Cycle

Program Review Basics

Rationale for Academic Program Review BOG Regulation 8.015 requires reviews of all academic degree programs by CIP code and degree program level at least once every seven years. Reviews must: Review mission and purpose of the program within the context of USF’s mission/strategic plan and the BOG’s strategic plan Assess how well the stated student learning outcomes and program objectives are being met Document how assessment results are being used for continuous program improvement Inform strategic planning, program development, and budgeting decisions at the university and state levels

Components of the Program Review Program self-study by CIP and degree program level (e.g., B.A./B.S. Geology, M.S. Geology, Ph.D. Geology, etc.) Department self-study Focuses on faculty & resources Dean’s report Includes questions for external reviewer External reviewer’s report Includes recommendations BOG summary report and improvement plan Prepared by department and submitted to ODS-IE for review and submission to BOG Progress report on actions taken to implement improvement plan one year after submission of the BOG summary report

Review of Program Hierarchy *Degree program: an organized curriculum leading to a degree in an area of study with at least one program major.

Types of Reviews On-campus Virtual (Teams) Document Review Three-day, in person review meeting faculty, staff, students, and touring facilities, etc. May be same format as the oncampus review or some half-day sessions based on external reviewer availability. Review of electronic documents and material available online.

Program Review Timeline for 2023-24 Spring 2023 Summer 2023 APR Q&A Session Provided by ODS-IE Spring 2023 ODS-IE Provides Data Sent to program by August 31, 2023 Fall 2024 Select External Reviewers Due to Jonna DeSantis August 1, 2023 Submit Program & Departmental Self-Studies Due to Dean/Associate Dean November 1, 2023 Dean Submits Report & Self Studies Due to Jonna DeSantis December 1, 2023 Spring 2024 Conduct Site Visit or Online Review with External Reviewer(s) Feb. 1 – April 14, 2024 Receive External Reviewer Report Due to Jonna 2-3 weeks after visit (e.g., May 5, 2024) Review External Report (Department, Dean, & Provost) May 19, 2024 Submit Departmental/Faculty Response (Optional) May 31, 2024 Submit BOG Summary Report/Improvement Plan to ODS-IE Upload Summary Report to BOG Portal (ODS-IE) Summer 2024 Fall 2024 Due to Jonna October 1, 2024 Due December 15, 2024 Submit Progress Report about Implementation of Action Items January 31, 2026 for inclusion in College Review

1-3 Program, Department Self-Studies, and Dean’s Report

Self-Study Templates Undergraduate Self-Study Graduate Self-Study

Self-Study Templates Departmental Self-Study Dean’s report

1. Program Self-Study Summary Data Sources from ODS Students Faculty Student Enrollments Student Term-to-Term Enrollments (5 semesters) New Admits by Fall Term Enrollments by Gender, Ethnicity, Nationality, and Student Level Degrees awarded by Academic Year, Gender, Ethnicity, and Student Type Student Credit Hours (SCH) Full-time (tenure / tenure track) Faculty Number of Adjunct Faculty Tenure track / non-tenure track Faculty by Rank, Ethnicity, and Gender Departmental SCH by Instructor / Instructor Type Graduate and Student Assistantships

Program Review Data

1. Program Self-Study Summary Data Sources from ODS Assessment Data (System for Assessment Management) Annual reports & executive summary data (three-years of data). Graduating Senior Survey Dashboard https://usfweb.usf.edu/ODS/secure/Surveys/GSS/GSS.aspx Academic Program Review Hub https://www.usf.edu/ods/accreditation/academic-program-review.aspx Post-graduation Outcomes Hub https://usfweb.usf.edu/ODS/Secure/Outcomes/OutcomesHub.aspx Note: Available behind single sign-on; if access is unavailable, contact Hennadii Balashov ([email protected]) for data export.

1. Undergraduate / Graduate Program SelfStudy Section 1 - Academic Program Overview Data Source Overview of degree program’s missions and purposes. List major changes since last review and the rationale for those changes. Describe modalities by which the program is offered, including the proportion of the program available through online learning. List all instructional sites for face-to-face instruction. List competing programs within the Florida SUS. Undergraduate Academic Catalog: https://catalog.usf.edu/content.php?catoid 13&navoid 1629. Describe the structure for student advising for academic program(s). Data provided by the Department / College. Information provided by the department. If new, refer to the degree proposal: https://academicplanning.usf.edu/curriculum/tracking.php. Data provided by the Department / College. If no online offerings, state that the program is not offered online. Data provided by the Department / College. For the Florida SUS, use the link: https://prod.flbog.net:4445/pls/apex/f?p 136:45:0::NO. For program outside of Florida, draw on experience from professional conferences to identify competing programs outside the state.

1. Undergraduate / Graduate Program SelfStudy Section 1 - Academic Program Overview Data Source Provide the program’s criteria for admission (if applicable). Info provided by the department

1. Undergraduate / Graduate Program SelfStudy Section 2 – Industry Demand for the Program Data Source Summarize current state of the discipline. Provide national, state and/or local data that support program need Burning Glass (provided by ODS) US Department of Labor statistics FETPIP results from Post-Graduation Outcomes Hub: https://usfweb.usf.edu/ODS/Secure/Outcomes/OutcomesHub.aspx. Indicate whether an industry or employer advisory council exists Info provided by the department Bureau of Labor Statistics Occupational Outlook Handbook O*Net Employ Florida Enterprise Florida Florida ExpertNet Florida Department of Economic Opportunity: Employment Statistics Florida Department of Economic Opportunity: Targeted Occupations List

1. Undergraduate / Graduate Program SelfStudy Section 2 – Student Demand for the Program Data Source Discuss the characteristics of the program’s students over time. Number of majors. Mix of full and part-time students. Mix of FTIC, transfer, and international students. Number of students participating in education abroad experiences. Number of students participating in undergraduate research experiences. ODS Data Table: Enrollment. Reflect on the growth or decline in student enrollment for your program and comment on the future potential demand and the program’s ability to meet this demand. Information provided by the department. Describe current recruitment/enrollment strategies for a diverse body of local, national, and international students Information provided by the department.

1. Undergraduate / Graduate Program SelfStudy Section 3 – Teaching and Student Learning Data Source List the student learning outcomes for the program. 3-year assessment summary (Information Provided by ODS) Summarize the results of the most recent three years of student learning outcome assessments. System for Assessment Management (Information provided by ODS) Describe how the results of the assessment of student learning outcomes have been used to implement program changes. Information provided by the department. Describe internal faculty led efforts to enhance: Innovative pedagogy. Student clubs / organizations. HIPs. (Information provided by the department). Describe the processes used to ensure quality is maintained in the curriculum and in the delivery of instruction. Information provided by the department.

1. Undergraduate / Graduate Program SelfStudy Section 3 – Teaching and Student Learning Data Source 3.7 Describe how student learning outcomes assessment ensures that students are professionally prepared for careers in the field and / or continued graduate education. Information provided by the department.

1. Undergraduate / Graduate Program SelfStudy Section 4 – Program Productivity Data Source Discuss the productivity of the program in terms of the following: ODS Data Table: Student Credit Hours & Faculty Data. Student credit hours generated. Student FTE. Degrees awarded (If program was designated as having “low productivity” in the annual Program Productivity Review, include the action plan for improvement). Time-to-degree. (Ph.D only) Percent of undergraduates without excess credit hours.

1. Undergraduate / Graduate Program SelfStudy Section 4 – Program Productivity Data Source Describe the national reputation or recognition of the program and list the sources of the reputation or ranking indices. Information provided by the department. Provide licensure / certification pass rates for the last three years (if applicable). Include the number of students attempting and passing the exam. Information provided by the department. Describe any significant student accomplishments. Information provided by the department. Provide information on program graduates including (1) employment rates, (2) earnings over time, and (3) continuing education. FETPIP or other related data (https://usfweb.usf.edu/ODS/UP Suite/Analytics/Outcomes.aspx) BOG - MyFloridaFuture) Reflect on student perceptions of program quality based on appropriate data sources. The Graduating Senior Survey at https://usfweb.usf.edu/ODS/secure/Surveys/GSS/GSS.aspx.

1. Undergraduate / Graduate Program SelfStudy Section 5 – Program Strengths and Weaknesses Data Source List program general / administrative goals and objectives. Reflect on whether these goals had been met during the last review period. Information provided by the department. Discuss potential opportunities in the following areas: Information provided by the department. Actions to improve program quality. New ventures to increase demand or improve competitiveness. Actions to achieve productivity gains. Actions to improve efficiency and reduce cost. Other opportunities.

1. Undergraduate / Graduate Program SelfStudy Section 5 – Program Strengths and Weaknesses Data Source Describe program barriers, threats, and unique vulnerabilities. Based on the SWOT analysis and current practice, list continuous quality improvement steps the program is taking to ensure the achievement of stated general / administrative goals and objectives. Information provided by the department. Information provided by the department.

2. Department Self-Studies Report Data Sources Section 1 – Department Overview Data Source Provide a brief history of the department and its academic program(s). Departmental website or department chair’s institutional memory Provide the department mission statement and a description of its alignment with the college mission, USF Strategic Plan, SUS strategic plan, and other state priorities. Describe the departmental vision. Describe departmental goals for teaching, research, and service. Describe the process used to prepare the selfstudy including the people involved.

2. Department Self-Study Report Section 2 - Benchmarking Data Source Compare the department to at least two peer departments with at least one department at an AAU peer institution. Select two departments that the department considers to be peers, and select two departments to which the departments aspires to be a peer. The USF institutional peer comparisons are located at https://usfweb.usf.edu/ODS/Secure/Performance/KeyPerformanceMetrics.aspx If using peers not included on the list of USF’s National Public Peers, please provide information to justify your choice(s). 2.2 Compare the department to at least two aspirational peer departments with at least one department at an AAU peer institution.

2. Department Self-Studies Report Data Sources Academic Analytics https://www.usf.edu/ods/data-tools/benchmarking portal.aspx Academic Analytics provides powerful data and analysis tools that help universities reach their research potential. USF uses Academic Analytics to allow faculty to find each other, form research networks and compare our programs to our peer list of institutions. The Academic Analytics database contains scholarly data on over 275,000 faculty members at over 400 US-based research universities. The data includes research grants, honorific awards, articles, books, conference proceedings, and more.

2. Department Self-Study Report Section 3 – Faculty Excellence Data Source Using the data supplied by ODS, provide an overview of faculty quality and number including: Data Provided by ODS. Number of full-time faculty Faculty mix (full-time vs. adjuncts) Racial / ethnic and gender makeup of the faculty SCH production by faculty type Ranks and tenure status Areas of expertise

2. Department Self-Study Report Section 3 – Faculty Excellence Data Source Describe any efforts to recruit, mentor, and retain talented diverse faculty. Information provided by Department Describe how faculty assignments are balanced among teaching, research, and services. Information provided by Department Describe the typical teaching load for full-time faculty in the program under review. Note any differences based on tenure status or other relevant factors. Information provided by Department Give an overview of faculty efforts related to research. Data extracted from Academic Analytics or other reputable, publicly available sources. Anticipated future directions Community-engaged research Research resulting in patents and licensing efforts

2. Department Self-Study Report Section 3 – Faculty Excellence Data Source Information provided by the Department Describe significant professional service activities of the faculty members who contribute to the program under review: Local, State, National, and International. Information provided by the Department Describe national or international awards or other evidence of noteworthy recognition and / or other accomplishment received during the review period by faculty who contribute to the program under review.

2. Department Self-Study Report Section 4 – Financial Sustainability Data Source Provide an overview of the sufficiency of institutional fiscal, physical, and human resources to support the program(s) under review. Include information on: Tuition revenue Graduate assistantships and waivers (if applicable) Administration support Laboratory space Laboratory equipment Department and classroom facilities Office space Technology support Library resources Online course / program support Academic Program Advisors Staff support Information provided by the department chair or program director based on known departmental practices.

2. Department Self-Study Report Section 4 – Financial Sustainability Data Source Describe any non-E&G sources of fund that support the program(s) under review (e.g. cost recovery, market rate, INTO, auxiliary funds, student fees). Information provided by the department chair or program director based on known departmental practices. Discuss the SCH produced by the department to the rest of the university, including service courses or programs in Enhanced General Education and interdisciplinary programs. Information provided by the department.

2. Department Self-Study Report Section 4 – Financial Sustainability Data Source List all documented community partnerships that are housed in the department. Information provided by Department Discuss any additional unmet resource needs of the program(s) under review. Information provided by Department

3. Dean’s Report The dean’s office will review the program selfstudy(ies) and department self-study and provide commentary and questions for external reviewers. Note: Specific questions from the dean to evaluators will help guide the evaluator’s program review.

Top Oversights in Self-Studies Data that should be included in a report are not included. Data are included in a report, but they are not summarized in a useful way for the external reviewer. Faculty CVs omitted Analysis results are meaningfully interpreted, but the report fails to document how findings were used to improve the program. Detailed descriptions of processes are provided, but with no apparent purpose, result, or recommendation. Reports combining multiple programs do not clearly differentiate between individual programs. Response fails to address the question being asked.

4. External Review and Reviewer Basics and Details

4. External Reviewer’s Report Selecting external reviewers The department identifies potential reviewers based on criteria provided in guidelines. The dean reviews potential reviewers. ODS-IE, in consultation with the Office of the Provost, confirms eligibility. The standard is one reviewer per CIP with three degree programs* and levels. (Requests for additional reviewers must be made to the Provost by the College Dean.) *Degree program: an organized curriculum leading to a degree in an area of study with at least one program major. Review Format On-campus Virtual (Teams) Document review

4. External Reviewer’s Report ODS-IE provides program and department self-studies and dean’s report (13) to the external reviewer(s) along with guidelines for the external review report (4). The reviewer produces a report that includes recommendations for improvements that do not focus solely on additional resources. (Program review is not a budget request process.) The report is submitted to Jonna DeSantis in ODS-IE within 2-3 weeks after the site visit. ODS-IE (Jonna) shares report with the department and the dean. The department notifies ODS-IE (Jonna) it has accepted the report. Upon acceptance of the report, the reviewer is paid for services.

4. External Review Basics ODS-IE arranges for payment to reviewers (needs six weeks notice). No reviewers may begin working until an approved Purchase Order is in the system. Note: USF Health programs cover all expenses, including travel and stipend.

4. External Review Basics Number of Degree Programs On-campus Virtual (Teams) Document Review 1,500 plus 1,000 travel 2,500 1,200 1,000 Additional degree program/BOG report (e.g., M.S. Geology) 500 400 375 Additional degree program/BOG report (e.g., Ph.D. Geology) 500 400 375 3,500 2,000 1,750 One degree program/BOG report (e.g., B.A./B.S. Geology) Totals

4. Typical External Review Timeline Day 1 Day 2 Day 3 Opening meeting/dinner General meetings and tours Follow-up meetings and exit meeting Coordinated and paid for by ODS-IE. Meetings with students, faculty, administrators, staff, and tour of relevant academic facilities. Lunch at Top of the Palms – Up to 5 people including reviewer. Reservation and payment coordinated and paid for by ODS-IE. Dinner – Coordinated and paid for by the academic department. Meetings coordinated by the department. Exit meeting coordinated by ODS-IE.

4. External Review – Preparing for the visit The reviewers are responsible for making their own air and hotel arrangements. The department determines schedule for Day 2 and any follow-up meetings on Day 3, other than the exit meeting. The department provides all ground transportation for the reviewer, including to and from the airport.

4. External Review – Itinerary Procedure Each program review will have an itinerary created in Box based on the itinerary template. It is imperative that any and all updates made to the itinerary are done in Box and not on a version saved on local computer. Access to the official itinerary will be granted once it is determined who will be responsible for adding meetings. Not required for Document Review

5-6 BOG Summary Report / Improvement Plan Progress Report

Program Review Timeline for 2023-24 Spring 2023 Summer 2023 APR Q&A Session Provided by ODS-IE Spring 2023 Select External Reviewers Due to Jonna DeSantis September 1, 2023 Submit Program & Departmental Self-Studies Due to Dean/Associate Dean November 1, 2023 Dean Submits Report & Self Studies Due to Jonna DeSantis December 1, 2023 Fall 2023 Spring 2024 Conduct Site Visit or Online Review with External Reviewer(s) Feb. 1 – April 14, 2024 Receive External Reviewer Report Due to Jonna 2-3 weeks after visit (e.g., May 5, 2024) Review External Report (Department, Dean, & Provost) May 19, 2024 Submit Departmental/Faculty Response (Optional) May 31, 2024 Submit BOG Summary Report/Improvement Plan to ODS-IE Summer 2024 Fall 2024 Due to Jonna October 1, 2024 Upload Summary Report to BOG Portal (ODS-IE) Submit Progress Report for Improvement Plan on January 31, 2026 for inclusion in College Review Due December 15, 2024

5. BOG Summary Report (DUE Oct. 1, 2023) 1. BACHELOR’S DEGREES ONLY: An Academic Learning Compact is available for each reviewed baccalaureate program. (Current ALC’s available at http://usfweb.usf.edu/DSS/SAM/Public/ALCs.aspx) 2. Was the program review conducted in conjunction with any specialized or programmatic accreditation reviews? (If yes, provide a description of the accreditor) 3. Name and affiliation of external reviewer. 4. Date of the last Academic Program Review for this program. 5. Briefly describe major changes made since the previous program review. 6. Summarize current strengths of the program. 7. Summarize current weaknesses of the program. 8. Summarize recommendations made by the external evaluator. 9. Describe improvement plans made as a result of the review. Note: https://www.usf.edu/ods/documents/academic-programs/master-bog-summary-report-2020-12-10.docx

5. BOG Summary Report (DUE Oct. 1, 2023) Change in BOG regulation requires A summary of the recommendations and/or proposed action plans made as a result of the review, including, but not limited to, areas in which the program goals/objectives are not being met, students are not achieving learning outcomes, and resources are not sufficient to achieve program goals/objectives.

6. Improvement Plans Progress Report (DUE Jan. 31, 2026) Status of improvement plan as described in summary report Improvement plan covers the implementation of program action items and recommendations Improvement Plan is due to 1-year month from submission of Summary Report (Due to BOG Dec 2023) Note: https://www.usf.edu/ods/documents/academic-programs/master-bog-summary-report-2020-12-10.docx

Academic Program Review Support Staff Joseph Boyd, Assistant Director, coordinates APR Process, serves as St. Petersburg campus liaison, [email protected] Jonna DeSantis, Executive Administrative Specialist, coordinates recruiting evaluators, document management, and scheduling; [email protected] Laura Brunkow, Fiscal and Business Specialist, coordinates contracting with and payment to evaluators; [email protected] Rebecca Gibbons, Assistant Director, serves as Sarasota-Manatee campus liaison, [email protected] Christopher Combie, Director, ODS-IE, facilitates academic program review process & troubleshoots problems; [email protected]

Questions? Electronic files of everything discussed are posted on the Academic Program Review Website: https://www.usf.edu/ods/accreditation/academicprogram-review.aspx . Thank you for your time and attention.