Using the Logic Model to Develop Evaluation Research Design

Using the Logic Model to Develop Evaluation Research Design and Measures

Program Planning and Evaluation Solid program planning is vital to a well designed evaluation. Community partners (stakeholders) need to have a meaningful way to provide input to program planning. Extension staff must be able to communicate outcomes to funders and stakeholders.

Logic Models Well developed, easy to use tool for planning educational programs, using stake holder input. Provide a common language Simple, graphic way to communicate program theory Provides planners with a road map – asking them to determine where they want to end up (goals) and then chart their course of action

Outcomes based Outcomes must be clearly written and the relationship between learning, action and long-term outcomes must be logical. Forces planners to think about and articulate their goals and objectives. Is an easy model for stakeholders to understand and provide feedback on.

Generic logic model Strategy Inputs » Impact Outputs Activities » Outcomes » Outcomes » Outcomes Participation » CONTEXT A diagram of the theory of how a program is supposed to work A graphic depiction of relationships between activities and results Source: University of Wisconsin – Extension http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

University of New Hampshire Cooperative Extension Began staff development efforts in 2001 Provided in-services, mouse pads, web support and even one-on-one support to staff Began presenting the model to advisory committees Some resistance from staff, but many liked the model More staff are using it now to obtain grant funding Suggested model for communities proposing CYFAR – New Communities projects

Benefits of logic modeling to partnerships Provides framework to discuss and articulate joint work Keeps end outcome upfront and center Helps facilitate conversation with focus on agreed upon goal that might not happened otherwise Utilizes local needs assessments and works with community partners to inform Logic Model development Separates indicators of achievement - # papers published, # and type of experiments completed – from theory of change. Indicators of achievement are part of evaluation plan for the partnership.

Case Examples Seacoast Youth Leadership Program Hillsboro After School Program

Seacoast Youth Leadership Program This project will evaluate the efficacy of the youth diversion program’s ability to achieve improvements in the following objectives: Leadership Future Aspirations/Goal Setting Social Responsibility Decision Making/Social Skills Social Connectedness Media Literacy

SYLP – Logic Model Inputs Outputs Participants Activities Learning Outcomes Action Outcomes Condition Outcomes H.S. graduation rates and living wage crime social capital among youth and adults safe, positive environments for youth

SYLP – Logic Model, cont Inputs Outputs Participants Activities Learning Outcomes Action Outcomes Improved school attendance & performance Healthy, drug free lifestyle Demonstrate care and concern for others Positive decision making Condition Outcomes H.S. graduation rates and living wage crime social capital among youth and adults safe, positive environments for youth

SYLP – Logic Model, cont Inputs Outputs Participants Activities Learning Outcomes Action Outcomes life skill knowledge & competencies Improved school attendance & performance technology intelligence & media literacy Healthy, drug free lifestyle Improved prosocial norms, citizenship & commitment to peers, family & community Goal setting, ‘Can do’ attitude Demonstrate care and concern for others Positive decision making Condition Outcomes H.S. graduation rates and living wage crime social capital among youth and adults safe, positive environments for youth

SYLP – Logic Model, cont Inputs Outputs Participants Activities Learning Outcomes Action Outcomes Research-based, best practice curriculum life skill knowledge & competencies Improved school attendance & performance Life skills through experiential learning – leadership and service learning technology intelligence & media literacy Healthy, drug free lifestyle Support & learning for parents Mobilize community partnership Improved prosocial norms, citizenship & commitment to peers, family & community Goal setting, ‘Can do’ attitude Demonstrate care and concern for others Positive decision making Condition Outcomes H.S. graduation rates and living wage crime social capital among youth and adults safe, positive environments for youth

SYLP – Logic Model, cont Inputs Outputs Participants Activities Youth/ parents High-risk, low-income families & communities Lower Seacoast Youth Collaborative Learning Outcomes Action Outcomes Research-based, best practice curriculum life skill knowledge & competencies Improved school attendance & performance Life skills through experiential learning – leadership and service learning technology intelligence & media literacy Healthy, drug free lifestyle Support & learning for parents Mobilize community partnership Improved prosocial norms, citizenship & commitment to peers, family & community Goal setting, ‘Can do’ attitude Demonstrate care and concern for others Positive decision making Condition Outcomes H.S. graduation rates and living wage crime social capital among youth and adults safe, positive environments for youth

SYLP – Logic Model, cont Inputs Outputs Participants Activities Human Resources Youth/ parents Material Resources High-risk, low-income families & communities Community Partnership Lower Seacoast Youth Collaborative Learning Outcomes Action Outcomes Research-based, best practice curriculum life skill knowledge & competencies Improved school attendance & performance Life skills through experiential learning – leadership and service learning technology intelligence & media literacy Healthy, drug free lifestyle Support & learning for parents Mobilize community partnership Improved prosocial norms, citizenship & commitment to peers, family & community Goal setting, ‘Can do’ attitude Demonstrate care and concern for others Positive decision making Condition Outcomes H.S. graduation rates and living wage crime social capital among youth and adults safe, positive environments for youth

Hillsboro After school Program This project will evaluate the efficacy of the after school program’s ability to achieve improvements in the following objectives: Academic Performance Future Aspirations Social Connectedness Social Responsibility Sense of Self Parenting Efficacy Social Skills/Decision Making Community Involvement with Youth/School

Hillsboro After School – Future Aspirations Inputs Outputs Participants Activities Learning Outcomes Action Outcomes Impact Youth pursue postsecondary education and apprenticeship -type experiences to attain what they need to enter and succeed in a chosen field.

Future Aspirations, cont Inputs Outputs Participants Activities Learning Outcomes Action Outcomes Impact Youth articulate a positive view of their future, including planned career/work goals/life goals. Youth pursue postsecondary education and apprenticeship -type experiences to attain what they need to enter and succeed in a chosen field. Youth make connections/ network within the community.

Future Aspirations, cont Inputs Outputs Participants Activities Learning Outcomes Youth learn about various career options related to their interests, talents, skills. Youth develop a new skill and explore related careers. Action Outcomes Impact Youth articulate a positive view of their future, including planned career/work goals/life goals. Youth pursue post-secondary education and apprenticeshiptype experiences to attain what they need to enter and succeed in a chosen field. Youth make connections/ network within the community.

Future Aspirations, cont Inputs Outputs Participants Activities 7-8 week series of special interest 4H projects using 4-H CCS, JASON, Robotics, NASA and computer literacy curriculum Field trips within community to learn about local careers and jobs. Learning Outcomes Youth learn about various career options related to their interests, talents, skills. Youth develop a new skill and explore related careers. Action Outcomes Impact Youth articulate a positive view of their future, including planned career/work goals/life goals. Youth pursue post-secondary education and apprenticeshiptype experiences to attain what they need to enter and succeed in a chosen field. Youth make connections/ network within the community.

Future Aspirations, cont Inputs Outputs Participants Activities Youth Program Staff 4-H Volunteers Business & industry members from the community & region 7-8 week series of special interest 4H projects using 4-H CCS, JASON, Robotics, NASA and computer literacy curriculum Field trips within community to learn about local careers and jobs. Learning Outcomes Youth learn about various career options related to their interests, talents, skills. Youth develop a new skill and explore related careers. Action Outcomes Impact Youth articulate a positive view of their future, including planned career/work goals/life goals. Youth pursue post-secondary education and apprenticeshiptype experiences to attain what they need to enter and succeed in a chosen field. Youth make connections/ network within the community.

Future Aspirations, cont Inputs UNHCE staff & volunteers Access to school library/ Computers Program space 4-H CCS, JASON, NASA & computer literacy curriculum Material, supplies Snacks Transportation College student interns Outputs Participants Activities Youth Program Staff 4-H Volunteers Business & industry members from the community & region 7-8 week series of special interest 4-H projects using 4-H CCS, JASON, Robotics, NASA and computer literacy curriculum Field trips within the community to learn about local careers and jobs. Learning Outcomes Youth learn about various career options related to their interests, talents, skills. Youth develop a new skill and explore related careers. Action Outcomes Impact Youth articulate a positive view of their future, including planned career/work goals/life goals. Youth pursue post-secondary education and apprenticeshiptype experiences to attain what they need to enter and succeed in a chosen field. Youth make connections/ network within the community.

Hillsboro After School – Community Involvement Inputs Outputs Participants Activities Learning Outcomes Action Outcomes Impact Citizens recognize and act to ensure community wide support and involvement in school, after school and youth/family programs After school program is expanded and sustained

Hillsboro After School – Community Involvement, cont Inputs Outputs Participants Activities Learning Outcomes Action Outcomes Youth and adults work together to set and reach goals. youth serving as equal partners on community boards & committees Farmers’ Market participation with growers accepting Food Stamp program Impact Citizens recognize and act to ensure community wide support and involvement in school, after school and youth/family programs After school program is expanded and sustained

Hillsboro After School – Community Involvement, cont Inputs Outputs Participants Activities Learning Outcomes Action Outcomes knowledge of jobs, careers & resources in community. Youth and adults work together to set and reach goals. Youth & adults learn to work in partnership. Adults learn the importance & value of youth as a resource to community. Youth learn components of an effective & healthy community. youth serving as equal partners on community boards & committees Farmers’ Market participation with growers accepting Food Stamp program Impact Citizens recognize and act to ensure community wide support and involvement in school, after school and youth/family programs After school program is expanded and sustained

Hillsboro After School – Community Involvement, cont Inputs Outputs Participants Activities Field trips / visits with community leaders. Training in community youth development Steering committee reviews/ sets program goals and shares vision Adults & youth partner on after school program & Teen Center development Learning Outcomes Action Outcomes knowledge of jobs, careers & resources in community. Youth and adults work together to set and reach goals. Youth & adults learn to work in partnership. Adults learn the importance & value of youth as a resource to community. Youth learn components of an effective & healthy community. youth serving as equal partners on community boards & committees Farmers’ Market participation with growers accepting Food Stamp program Impact Citizens recognize and act to ensure community wide support and involvement in school, after school and youth/family programs After school program is expanded and sustained

Hillsboro After School – Community Involvement, cont Inputs Outputs Participants Activities Community adults Youth Program Staff & 4-H volunteers Social services & health care workers T own government Business leaders, regional planning commission Main Street project leaders Field trips / visits with community leaders. Training in community youth development Steering committee reviews/ sets program goals and shares vision Adults & youth partner on after school program & Teen Center development Learning Outcomes Action Outcomes knowledge of jobs, careers & resources in community. Youth and adults work together to set and reach goals. Youth & adults learn to work in partnership. Adults learn the importance & value of youth as a resource to community. Youth learn components of an effective & healthy community. youth serving as equal partners on community boards & committees Farmers’ Market participation with growers accepting Food Stamp program Impact Citizens recognize and act to ensure community wide support and involvement in school, after school and youth/family programs After school program is expanded and sustained

Hillsboro After School – Community Involvement, cont Inputs Program space Snacks Tech – AV, computers Community development plan CYD resources Community gardens Staff from various disciplines Outputs Participants Activities Community adults Youth Program Staff & 4-H volunteers Social services & health care workers Town government Business leaders, regional planning commission Main Street project leaders Field trips / visits with community leaders Training in community youth development Steering committee reviews/ sets program goals and shares vision Adults & youth partner on after school program & Teen Center development Learning Outcomes Action Outcomes knowledge of jobs, careers & resources in community Youth and adults work together to set and reach goals Youth & adults learn to work in partnership Adults learn the importance & value of youth as a resource to community Youth learn components of an effective & healthy community youth serving as equal partners on community boards & committees Farmers’ Market participation with growers accepting Food Stamp program Impact Citizens recognize and act to ensure community wide support and involvement in school, after school and youth/family programs After school program is expanded and sustained

Seacoast Youth Leadership Program This project will evaluate the efficacy of the youth diversion program’s ability to achieve improvements in the following objectives: Leadership Future Aspirations/Goal Setting Social Responsibility Decision Making/Social Skills Social Connectedness Media Literacy

Hillsboro After school Program This project will evaluate the efficacy of the after school program’s ability to achieve improvements in the following objectives: Academic Performance Future Aspirations Social Connectedness Social Responsibility Sense of Self Parenting Efficacy Social Skills/Decision Making Community Involvement with Youth/School

Collapsing extensive outcome goals and objectives into manageable and meaningful evaluation research conceptual categories for design and measurement development

Identifying existing measures for conceptual categories LEADERSHIP INDEPENDENCE (WRIGHTSMAN, 1964,1974) MASTERY (PEARLIN, 1981) SELF-ESTEEM (ROSENBERG, 1965) FUTURE ASPIRATIONS (PEER LEADER SURVEY & DYS) SOCIAL CONNECTEDNESS SOCIAL SUPPORT (VAUX) ATTACHMENT TO PARENTS & SCHOOL (RESNICK, 1997; ADD HEALTH SURVEY)

Collaboration with Extension staff to determine appropriate evaluation design and measurement EVALUATION AS A PROCESS COMMUNICATION IS KEY

Utilization of triangulation of methods to provide depth and breadth to the evaluation research process

Contact Information: Scott R. Meyer, MSW, Ph.D - (603) 535-2377 [email protected] Stephanie Halter,M.A., Research Associate, (603) 536-5725 [email protected] Rick Alleva, Extension Educator, (603) 679-5616 [email protected] Dan Riedy, Extension Educator, (603) 641-6060 [email protected] Penny Turner, Extension Educator, (603) 641-6060, [email protected] Lisa Townson, Extension Specialist, (603) 862-1031 [email protected]