U N Pe a c e ke e p i n g T O T M o d u l e 3 Training Evaluation

U N Pe a c e ke e p i n g T O T M o d u l e 3 Training Evaluation & Reporting

Relevance Training evaluations help ensure a training programme has met its stated objectives Training facilitators are responsible to integrate various levels of evaluation throughout the training process UN Peacekeeping Training of Trainers 1

Module 3 Learning Objectives Learners will be able to: Identify the five levels of training evaluation, based on the ROI approach Design training evaluations at levels 1-3 Write an effective course report based on the recommended template UN Peacekeeping Training of Trainers 2

Learning Overview I. Five Levels of Training Evaluation II. Writing Course Reports UN Peacekeeping Training of Trainers 3

I. Five Levels of Training Evaluation Training evaluation is essential to ensure that a training programme has met its stated objectives. The purpose of training evaluation is to assess: Relevance Effectiveness Impact UN Peacekeeping Training of Trainers 4

I. Five Levels of Training Evaluation The UN approach to training evaluation is based on the ROI (return on investment) methodology. Level 5 Return on Investment (ROI) Level 4 Impact Level 3 Application Level 2 Learning Level 1 Reaction UN Peacekeeping Training of Trainers 5

Level 1 Reaction Level 1 measures participants’ reactions to the training and plans to use what they have learned. After a success training participants will: Perceive the training to be relevant to their job Perceive the training to be value added Rate the trainers as effective Recommend the training to others UN Peacekeeping Training of Trainers 6

Level 1 Reaction Level 1 data collection: Shorter trainings – may be collected at conclusion Longer trainings (more than 1-2 days) – should be collected periodically throughout Can be conducted in person or online UN Peacekeeping Training of Trainers 7

Level 1 Reaction Level 1 data collection: Likert scale Example: I intend to use concepts from the course in the near future. Strongly agree Agree Uncertain Disagree Strongly disagree 5 4 3 2 1 UN Peacekeeping Training of Trainers 8

Level 1 Reaction Level 1 data collection: true/false, multiple choice This training was relevant to my on-the-job tasks and responsibilities. True False The most useful aspects of the training were: A. Pre-course reading materials B. Group work and assignments C. Networking established with other participants D. Feedback received from the facilitators UN Peacekeeping Training of Trainers 9

Level 1 Reaction Level 1 data collection: open-ended/qualitative feedback What aspects of the course could use improvement? Overall, what is your reaction to the course? UN Peacekeeping Training of Trainers 10

Learning Activity 3.1 Designing Level 1 Evaluations Instructions: Design a Level 1 evaluation for the training you will deliver as part of your final evaluation Design questions to gauge the following: o Relevance o Usefulness o Intent to use the training o Overall satisfaction Time: 20 minutes UN Peacekeeping Training of Trainers 11

Level 2 Learning Level 2 evaluation measures the amount of learning that was gained as a result of the training. This should be based on the training’s stated learning objectives. Generating positive reactions is not enough: the goal of peacekeeping training is to increase competencies for mandate implementation. UN Peacekeeping Training of Trainers 12

Level 2 Learning Pre-tests establish a baseline to measure the amount of learning that has occurred as a result of the training. In order to measure growth, pre- and post-tests should utilize the same or similar questions. UN Peacekeeping Training of Trainers 13

Learning Activity 3.2 Creative Pre-Tests Instructions: Talk through/practice various options for conducting pre-tests: o Voting with your feet o Pollanywhere.com o Fishbowl o Role-play Time: 20 minutes UN Peacekeeping Training of Trainers 14

Level 2 Learning Continuous evaluation: a variety of formal and informal assessments should be integrated throughout the training programme. Formal assessments: written tests, selfassessments, demonstrations, other written work Informal assessments: observations, discussions, checks for understanding, monitoring participation UN Peacekeeping Training of Trainers 15

Level 2 Learning The design of Level 2 evaluations will depend on the complexity of the learning objectives. For example: Training at a low level of complexity may be assessed by testing recall or identification Training at higher levels of complexity may require use of simulation exercises or demonstrations UN Peacekeeping Training of Trainers 16

Learning Activity 3.3 Designing Level 2 Evaluations Instructions: Design a Level 2 evaluation for the lesson you will deliver as part of your final evaluation Create both pre- and post-test assessments Remember: assessments should be in line with stated learning objectives (refer to the objectives you wrote in the Module 1 activity) Time: 20 minutes UN Peacekeeping Training of Trainers 17

Level 3 Application Level 3 assessments focus on on-the-job application. This requires formulating application objectives. Application objectives: Identify behaviour, tasks and actions that are observable and measurable Are outcome-based and specific Specify what behaviour and actions the participant will change as a result of the training18 UN Peacekeeping Training of Trainers 18

Level 3 Application Sample application objectives POC application objectives – participants will: Demonstrate actions reflecting the mission POC strategy, guided by the DPO Operational Framework Respond to threats in a way that is appropriate with own roles and responsibilities without overlaps or gaps Utilize mission POC strategy effectively to achieve coherence in approach, minimize gaps, avoid duplication and maximize POC efforts UN Peacekeeping Training of Trainers 19

Level 3 Application Sample application objectives Rule of law application objectives – participants will: Provide legal advice and assistance guided by UN principles of peacekeeping Coordinate and convene work of international and national stakeholders engaged in rule of law in order to increase effectiveness Implement programmatic activities to improve effectiveness of justice system UN Peacekeeping Training of Trainers 20

Learning Activity 3.4a Designing Level 3 Application Objectives Instructions: Create application objectives based on the learning objectives you wrote in Module 1 Keep in mind the following guidelines. Application objectives should: o Identify behaviours, tasks, actions that are observable and measurable o Be outcome-based and specific o Specify what behaviours/actions the participant will change as a result of the training Time: 20 minutes UN Peacekeeping Training of Trainers 21

Level 3 Application Application level surveys are one of the main follow-on activities of training, as they seek to find answers to the question, “How can training support skill transfer?” Level 2 measures training outputs Level 3 measures outcomes, or on-the-job application UN Peacekeeping Training of Trainers 22

Converting application objectives into survey Level 3 Learning Application Level 3 Evaluation Questions of a POC training questions: Objectives of a POC training Participants are able to: Demonstrate actions reflecting the mission POC strategy, guided by the DPO Operational Framework Were you able to demonstrate actions reflecting the mission POC strategy, guided by the DPO Operational Framework? Yes No If answered “Yes”, what particular actions were you able to implement? (Select all that apply) Secured the voluntary, safe and dignified return of IDPs and refugees Facilitated access to humanitarian assistance Trained state security forces on human rights Other: please specify If answered “No”, what were the barriers? Barrier 1 Barrier 2 Barrier 3 Other, please specify UN Peacekeeping Training of Trainers 23

Level 3 Application When analyzing Level 3 data, consider enablers and barriers that may impact application, for example: Enablers Barriers Support from managers Increased motivation Usefulness of skills to support change processes Lack of support from managers Lack of time/resources Lack of opportunity to apply knowledge/skills UN Peacekeeping Training of Trainers 24

Level 3 Application A variety of qualitative and quantitative data can be used for Level 3 evaluations. 360 evaluations Focus groups Action plans Which data collection method is used depends on the nature of the training programme, the behaviour and action to be addressed and time/resources available. UN Peacekeeping Training of Trainers 25

Learning Activity 3.4b Designing Level 3 Application Surveys Instructions: Using the application objectives you wrote in activity 3.4a, draft corresponding survey questions Remember: Level 3 evaluation questions seek to ascertain how knowledge/skills have been transferred and through what actions. Start with survey questions. After, you may also consider other methods of data collection (focus groups, 360 evaluations, etc.) Time: 20 minutes UN Peacekeeping Training of Trainers 26

Level 3 Application Level 3 data should be compared to predetermined targets. For example: Indicators/objectives Target score Actual measured score “Ability to apply”: I have been able to apply what I learned in the course to my job. 4/5 4.2 “Improved job satisfaction”: my overall satisfaction with my job has improved since I attended the course. 4/5 4.7 “Transfer of knowledge & skills”: I have been able to help colleagues to improve their skills by what I have learned. 4/5 3.4 UN Peacekeeping Training of Trainers 27

Level 4 Impact Level 4 evaluations measure the impact of changes in on-the-job behaviours or actions. The success of a training programme should ultimately be measured in whether the impact objectives were reached or not. UN Peacekeeping Training of Trainers 28

Level 4 Impact Sample impact objectives: POC training: Respond to threats and violence efficiently Increase in number of cases detected early Increase in collaboration with local groups to collect information on rebel movements UN Peacekeeping Training of Trainers 29

Level 4 Impact Sample impact objectives (cont.) RoL training for Judicial Officers: provide legal advice and assistance guided by UN principles on peacekeeping Increased efficiency Enhanced communication Clearly established roles and responsibilities for mandate implementation UN Peacekeeping Training of Trainers 30

Level 4 Impact Sample impact objectives (cont.) Department of Operational Support course on Administration of Personnel Contracts: increase efficiency in administration of personnel contracts Improved time and quality (decrease in errors, increase in quality of work, decrease in processing time) UN Peacekeeping Training of Trainers 31

Level 4 Impact Improvements in work performance should not be solely linked to the training. Efforts to isolate the effects of the training may include: Control groups Trend lines Estimates on impact UN Peacekeeping Training of Trainers 32

Level 5 Return on Investment (ROI) Level 5 evaluations seek to determine the financial return on investment. This entails converting data on impact to monetary values to determine the return when compared with training costs. UN Peacekeeping Training of Trainers 33

Level 5 Return on Investment (ROI) Cost Items Financial returns Pro-rated cost of training Cost avoidance Savings Revenue materials Costs of training venues and equipment Travel-related costs Daily salaries/fees of consultants, trainers, trainees Costs for follow-up, evaluation and reporting UN Peacekeeping Training of Trainers 34

II. Course Reporting Reports should include the following as a minimum: Introduction/background Overall course objectives Facilitators and participants Course conduct and methodology Course evaluations at levels 1-3 Observations/recommendations Conclusions Annexes UN Peacekeeping Training of Trainers 35

Summary Conducting evaluations is essential to ensure objectives have been met UN approach to training evaluation is based on ROI methodology, which evaluates programmes on five levels: reaction, learning, application, impact & ROI At the conclusion of every training, facilitators will draft a course report, including training objectives, methodology, a list of facilitators and participants, as well as course evaluation results UN Peacekeeping Training of Trainers