Developing and Implementing a Monitoring & Evaluation Plan DRAFT 1

Developing and Implementing a Monitoring & Evaluation Plan DRAFT 1

Objectives of the Session By the end of this session, participants should be able to: Describe the resources and capacity necessary for developing and implementing an M&E plan Understand the key steps in the process of developing & implementing an M&E plan Describe how M&E findings can be effectively disseminated and utilized Assess how an M&E plan is working DRAFT 2

Overview of the Session Standards for an M&E plan Resource & capacity for an M&E plan Steps in developing &implementing an M&E plan Disseminating and using M&E findings Assessing how well the M&E plan is working DRAFT 3

Standards for an M&E plan DRAFT 4

Standards for an M&E Plan Utility - serve practical information needs of intended users Feasibility - be realistic, prudent, diplomatic and frugal Propriety - conducted legally, ethically, and with regard to those involved in and affected by the evaluation Accuracy - reveal and convey technically accurate information DRAFT 5

Capacity and Resources for M&E DRAFT 6

Capacity and Resources for M&E Technical capacity for M&E – M&E unit – Strategically located individuals who are motivated, committed, competent and interested in M&E Scale of program funding and resources for M&E – 10 percent of resources devoted to M&E – Costs related to data collection systems & information dissemination – Costs may be higher during the first year – Other resources include indicator guides, manuals, communication tools DRAFT 7

Key steps in developing an M&E plan DRAFT 8

Key Steps in Developing & Implementing an M&E Plan STAKEHOLDER CONSULTATION & PARTICIPATION - Define process for stakeholder involvement Translate problem statement, program goals and objectives into M&E frameworks – Establish scope of the M&E plan Develop M&E Framework – Determine elements to be monitored and evaluated Define indicators and identify data sources Determine M&E methods for data and information collection – Develop data collection plan – Determine M&E responsibilities Set targets Define reporting system, utilization and dissemination of results Plan for mid-course adjustments DRAFT 9

Stakeholder Consultation Advocate for the need for M&E Understand program goals and objectives Identify user needs and perspectives Learn about existing data collection systems & their quality Understand indicators that are being collected and used (or not used) Determine capacity for collecting and using data DRAFT 10

Stakeholder Participation Developing M&E framework Selecting indicators Setting targets Reviewing results Requires building consensus & commitment and maintaining effective relationships with intended users DRAFT 11

Exercise: Describe a Process to Involve Stakeholders Who are the stakeholders? When will they be involved? How will they be involved? What challenges might arise? How will you address those challenges? DRAFT 12

Translating Program Goals and Objectives M&E Frameworks Translate problem statement, program goal and objectives into frameworks that can be objectively measured – What do we want to know at the end of the program? – What do we expect to change by the end of the program? M&E framework outlines the scope of the M&E plan – Links program activities to expected outputs, outcomes, and population-level impacts – Informs types of information to be collected/used by different levels of the health system – Identifies what needs to be measured to guide indicator selection DRAFT 13

Elements to be Monitored & Evaluated Monitoring: – – – – – Resources (inputs) Quality of service Service statistics Service coverage Client/patient outcomes (behavior change/morbidity) Impact evaluation: – Attributing the change in outcomes to the program DRAFT 14

Developing M&E Framework Review program documents with stated goals and objectives Ensure that key factors that may influence program implementation and success are identified Achieve consensus among stakeholders DRAFT 15

Defining Indicators & Identifying Data Sources Indicators must be: SMART Linked to M&E framework Useful for program decision-making Consistent with international standards & other reporting requirements (as appropriate) Realistic to collect (feasible) Data sources: Existing vs. new DRAFT 16

Determining M&E Methodology Assess information systems capabilities to address strategic information needs Determine methods by which data will be collected, analyzed and reported – E.g., RHIS, surveys, sentinel surveillance systems, project information systems/records, new data collection Determine whether any special studies will be conducted and what design will be used – E.g., Qualitative, quantitative or combination of both DRAFT 17

Assigning M&E Responsibilities: Implementing the M&E Plan Monitoring data – Who will collect data? – Who will analyze data? – Who will report data? Special studies – Who will oversee data collection? – Who will conduct analysis and reporting? – Is anyone else planning a similar evaluation? DRAFT 18

Role of the M&E Unit Consensus building among all stakeholders Coordination – between various program components – between other stakeholders (avoid duplication of data collection) Data manipulation Reporting Information dissemination and review DRAFT 19

Data Collection Plan/Matrix DRAFT Indicators Who responsible Timing Data quality notes 20

Class Activity: Data Collection Plan DEVELOP A DATA COLLECTION MATRIX FOR YOUR PROGRAM. CONSIDER THE FOLLOWING ISSUES: Who will be responsible for data collection and its supervision? Who will be responsible for ensuring data quality at each stage? How will data quality be checked at every stage? How often will the data be collected, compiled, sent, and analyzed? What indicators will be derived from each data source? How will the data be sent (raw, summary)? What tools/forms will be used, if any? What resources (staff, office supplies, computers, transportation) will be needed at each stage? Who will analyze the data? How often will analysis occur? How often will the results be compiled into reports? To whom and how often will the results be disseminated? DRAFT 21

Setting Targets Focus on what the program should achieve Orient stakeholders to the task to be accomplished Motivate individuals Monitor whether anticipated progress is being made DRAFT 22

Factors to Consider When Setting Targets What can realistically be achieved given the resources and the program context? – – – – DRAFT Baseline levels Past trends Needs and gaps in services capacity and logistics 23

Approaches to Setting Targets Establish final target then plan progress in between Establish annual/intermediate targets Assess progress in attaining targets and readjust, if necessary DRAFT 24

Useful Information for Setting Targets Past trends Expert opinions Research findings What has been accomplished elsewhere Client expectations DRAFT 25

Developing a Data Dissemination & Utilization Plan Define users of M&E findings Define feedback mechanism to meet user needs – Strategically timed user meetings/workshops – Annual report and review meeting – Database to manage data and facilitate access and use (e.g., PIMS, DSS) DRAFT 26

Planning for Mid-Course Adjustments Program changes can affect the M&E plan performance monitoring and impact evaluation Internal M&E capacity facilitates adjustments Flexibility and regular review of program results necessary DRAFT 27

How Well is the M&E Plan Working? Are M&E activities progressing as planned? Are the evaluation questions being answered sufficiently? Have other evaluation questions been raised and should they be incorporated into the plan? Are there any methodological or evaluation design issues that need to be addressed? Are there any outside factors (political, environment) that are affecting the plan? Are appropriate staff and funding still available to implement the M&E plan? Are M&E finding being disseminated and used by stakeholders for decision making and program improvement? DRAFT 28

Remember that an M&E plan is a living document and needs to be adjusted when a program is modified DRAFT 29

Developing & Implementing an M&E Plan: Logic Model Inputs Human resources Understanding of the program Authority and mandate Stakeholders DRAFT Processes Advocate Assess strategic information needs Assess information systems capabilities Achieve consensus and commitment Develop mechanism for M&E plan review Prepare document for final approval Output M&E Plan Document Outcomes Short-term M&E System for obtaining Strategic Information decision making Long-term Evidence-based decisions for improving programs Impacts Improved health status 30

Summary: The Basics Start early Involve stakeholders at all stages in the process Assess strategic information needs for intended users Assess current capacity and use what is already available Avoid duplication of data collection and reporting Do not collect information that will not be used Review progress/results regularly and make adjustments to M&E plan, if necessary DRAFT 31

References Adamchak S et al. (2000). A Guide to Monitoring and Evaluating Adolescent Reproductive Health Programs. Focus on Young Adults, Tool Series 5. Washington D.C.: Focus on Young Adults. Bertrand J et al. (1996). Evaluating Family Planning Programs. The Evaluation Project. Curtis, S (YEAR?). Building M&E Plans. FHI (YEAR?). Core Module 3: Developing a Monitoring and Evaluation Work Plan. Rommelmann V, P Setel, Y Hemed, H Mponezya, G Angeles, T Boerma (2003). Costs and Results of Information Systems for Poverty Monitoring, Health Sector Reform, and Local Government Reform In Tanzania. DRAFT 32