Logic Models How to Integrate Data Collection into your Everyday Work

Logic Models How to Integrate Data Collection into your Everyday Work

Agenda Introduction to Logic Models Developing a Basic Logic Model Exercise One Using a Logic Model to Plan for Evaluation Exercise Two

Introduction to Logic Models

Logic Models: An Overview A systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan, and the changes or results you hope to achieve. Resources Needed to accomplish activities (“Inputs”) Activities Process, tools, events, technology, and actions Outputs Results of activities (e.g., counts, types, levels of services delivered) Outcomes Short-term and long-term specific and observable changes (S.M.A.R.T) Impact Organizational, community, or system level changes Measurement tool (indicator) required for all activities, outputs, outcomes, impacts

Resources Definition: inputs Factors that enable resources: Funding Existing organizations Potential collaborating partners Existing organizational or interpersonal networks Staff & volunteers Time Facilities Equipment Supplies Factors that limit resources: Negative attitudes Lack of resources Policies Laws Regulations Geography

Activities Definition: Processes or actions to accomplish objectives Products Infrastructure Promoting materials Enhancing Providing educational infrastructure Building relationships, curricula collaborating Building capacity (for Services Educating and training desired results) Counseling Health screening

Outputs Definition: direct products of activities Number of classes taught, meetings held, or materials produced and distributed Program participation rates and demography Hours of each type of service provided

Outcomes Definition: observable and measurable changes in a specific population, community, or category (S.M.A.R.T) Changes in: Attitudes Behaviors Knowledge Skills Status Level of functioning

Impacts Definition: organizational, community, and/or system level changes expected to result from program activities. May include improved conditions, increased capacity, and/or changes in policy. Outcomes and impacts should be S.M.A.R.T. Specific Measurable Attainable Realistic Timely

Logic Model Example Planning a family vacation Resources Holiday flight schedule s Family schedules Frequent flyer holiday options Holiday weather Activities Create family schedule Get holiday flight info Get tickets Arrange ground transport Outputs Tickets for all family members Frequent flyer miles used Money saved Outcomes Family members enjoy vacation Impact Continued good family relations

Developing a Basic Logic Model for Your Program: The Why

Why use a Logic Model? It is a useful planning tool It helps you identify and collect the data needed to monitor and improve programming It presents program information and progress towards goals in ways that inform, advocate for a particular program approach, and teach program stakeholders.

Benefits of a Logic Model Program Elements Planning and Design Criteria for Program Success Program goals and objectives, and important side effects are well defined ahead of time. Benefits of Program Logic Models Finds “gaps” in the theory or logic of a program and work to resolve them. Program goals and objectives are Builds a shared understanding of both plausible and possible. what the program is all about and how the parts work together. Program Implementation and Management Relevant, credible, and useful performance data can be obtained. Focuses attention of management on the most important connections between action and results. Evaluation, Communication, and Marketing The intended users of the evaluation results have agreed on how they will use the information. Provides a way to involve and engage stakeholders in the design, processes, and use of evaluation.

Exercise One In groups, discuss your organization’s existing logic models and how they are utilized in evaluation.

Planning Evaluations with Logic Models: The How

Types of Evaluations Formative Examines program service delivery or technology, the quality of its implementation, and assessment of organizational context, inputs Focuses on program activities, outputs, and short-term outcomes in order to make mid-course corrections Provides suggestions to strengthen program implementation Summative Examines effects (results) or outcomes of program Focuses on program’s short and long-term outcomes and impact Describes quality and effectiveness of program on participants and community

First Determine What Will Be Evaluated Program: (1) Context (2) Program Implementation (Process) (3) Outcomes (Results) Program Results Program Implementation Resources Inputs Activities Processes or actions Outputs Direct products of activities Outcomes Measurable changes for specific population, community Measurement tool (indicator) required Impact Organizational, community, system-level impacts

Context Evaluation Questions Intent of Evaluation: assess how a program functions within the economic, social, and political landscape of its community to determine factors that might influence ability to perform the work as planned Examples Can we secure a donated facility? How many medical volunteers can we recruit? What is it about the free clinic that supports its ability to reduce the numbers of patients seeking care in Memorial Hospital’s ER?

Implementation (Process) Evaluation Questions Intent of Evaluation: assess the extent to which activities were executed as planned Examples How many patients were seen in the clinic each night, month, year? Which organizations most frequently referred patients to the clinic? What supplies were donated to the clinic?

Outcome Evaluation Questions Intent of Evaluation: determine the extent to which progress is being made toward the desired changes (results) in individuals, organizations, communities or systems Examples What was the cost/visit in the free clinic? How does this compare to the cost/visit in Memorial’s ER? How satisfied were clinic patients with the care they received? Did the reading program improve the reading scores of the youth who participated in the program?

Evaluation Questions and Planning 1. 2. 3. 4. 5. 6. Focus Area: From your program logic model, list the components of the most important aspects of your program. Audience: Identify the key audiences for each focus area. Who has an interest in your program? Questions: For each focus area and audience, list the questions they may have about your program. Information Use: For each audience and question you have identified, identify the ways you will use the evaluation information. Indicators: Describe what information could be collected that would indicate the status of your program and its participants for each question. Technical Assistance: Indicate the extent to which your organization has the evaluation and data management expertise to collect and analyze the data that relates to this indicator.

Focusing Your Evaluation Questions Evaluation Focus Area Determine what will be evaluated Audience Question Information Use For each focus area identified, list the audiences that are most likely to be the most interested in that area For each focus area and audience identified, list the questions they might have about your program For each audience and questions identified, list the ways and extent to which you plan to use the evaluation information

Audience Questions and Evaluation Use Audience Typical Questions Evaluation Use Program Management and Staff Are we reaching our target population? Are our participants satisfied with our program? Is the program being run efficiently? How can we improve our program? Programming decisions, day-to-day operations Participants Did the program help me and people like me? What would improve the program next time? Decisions about continuing participation Community Members Is the program suited to our community needs? What is the program really accomplishing? Decisions about participation and support Public Officials Who is the program serving? What difference has the program made? Is the program reaching its target population? What do participants think about the program? Is the program worth the cost? Decisions about commitment and support. Knowledge about the utility and feasibility of the program approach. Funders Is what was promised being achieved? Is the program working? Is the program worth the cost? Accountability and improvement of future grantmaking efforts

Establishing Indicators Evaluation Focus Area Indicators (examples) How to Evaluate Resources Logs or reports of financial/staffing status. Compare actual resources acquired against anticipated resources. Activities Description of planned activities. Logs or reports of actual activities. Descriptions of participants. Compare actual activities provided and/or types of participants reached against what was proposed. Outputs Logs or reports of actual activities. Actual products delivered. Compare the quantity of what was actually delivered against what was expected. Outcomes & Impacts Participant attitudes, knowledge, skills, intentions, and/or behaviors intended to result from your activities Compare the measures before and after the program.

Exercise Two Discuss the sample logic model and its similarity to the logic model developed for your organization. Discuss any challenges you faced – or may possibly face – in developing a logic model like the sample.