W.K. Kellogg Foundation Logic Model Development Guide<\/p>\n
Using Logic Models to Bring Together Planning, Evaluation, and Action<\/p>\n
Logic Model Development Guide<\/p>\n
W.K. KELLOGG<\/p>\n
foundation<\/p>\n
To help people help themselves through the practical application of knowledge and resources to improve their quality of life and that of future generations.<\/p>\n
Updated January 2004<\/p>\n
W.K. Kellogg Foundation One East Michigan Avenue East Battle Creek, Michigan 49017-4012 www.wkkf.org<\/p>\n
Copyright \u00a9 1998 by the W.K. Kellogg Foundation<\/p>\n
All rights reserved. This publication may be freely utilized by the general public for educational, charitable, nonprofit, and\/or other non-commercial purposes. Such use does not require prior or written permission. Such use should include acknowledgment that it is \u201cwith the permission of the W.K. Kellogg Foundation\u201d. Any other use is prohibited without the prior written permission of the W.K. Kellogg Foundation. For permission requests, please contact the Communications Department at communications@wkkf.org or 269-969-2079.<\/p>\n
Page I<\/p>\n
Contents<\/p>\n
Introduction……………………………………………………………………………………………III<\/p>\n
Chapter 1<\/p>\n
Introduction to Logic Models …………………………………………………………………………….1<\/p>\n
The What and Why of the Logic Model………………………………………………………………1<\/p>\n
Logic Model Definition ………………………………………………………………………………..1<\/p>\n
Logic Model Purpose ……………………………………………………………………………………3<\/p>\n
Trip Planning Logic Model Example………………………………………………………………..3<\/p>\n
Why Use a Logic Model?……………………………………………………………………………………5<\/p>\n
Program Success…………………………………………………………………………………………….5<\/p>\n
Program Investments ……………………………………………………………………………………..6<\/p>\n
Simple Logic Model Basics ………………………………………………………………………………..7<\/p>\n
Logic Model Development………………………………………………………………………………7<\/p>\n
Reading a Logic Model ………………………………………………………………………………….7<\/p>\n
Other Logic Model Examples …………………………………………………………………………….8<\/p>\n
Theory Model …………………………………………………………………………………………….10<\/p>\n
Outcomes Model…………………………………………………………………………………………11<\/p>\n
Activities Model…………………………………………………………………………………………..12<\/p>\n
Chapter 2<\/p>\n
Developing a Basic Logic Model for Your Program………………………………………………15<\/p>\n
Demonstrating Progress Toward Change…………………………………………………………….16<\/p>\n
Exercise 1 \u2014 Describing Results…………………………………………………………………………16<\/p>\n
Exercise 1 Checklist…………………………………………………………………………………………20<\/p>\n
Exercise 2 \u2014 Describing Actions…………………………………………………………………………21<\/p>\n
Exercise 2 Checklist…………………………………………………………………………………………23<\/p>\n
Program Implementation Template \u2014 Exercise 1 & 2 …………………………………………..25<\/p>\n
Chapter 3<\/p>\n
Developing a Theory-of-Change Logic Model for Your Program…………………………….27<\/p>\n
Exercise 3 – Constructing a Program Theory……………………………………………………….28<\/p>\n
Program Planning…………………………………………………………………………………………28<\/p>\n
Exercise 3 Checklist……………………………………………………………………………………….33<\/p>\n
Program Planning Template \u2014 Exercise 3…………………………………………………………..34<\/p>\n
Page II<\/p>\n
Chapter 4<\/p>\n
Using Your Logic Model to Plan for Evaluation ………………………………………………….35<\/p>\n
Exercise 4 \u2014 Posing Evaluation Questions …………………………………………………………..35<\/p>\n
Formative\/Summative Evaluation Questions ……………………………………………………35<\/p>\n
Evaluation Vantage Points Context, Implementation, Outcomes…………………………36<\/p>\n
Focus Areas, Audiences, Questions, Information Use…………………………………………38<\/p>\n
Audiences and Evaluation………………………………………………………………………………42<\/p>\n
Exercise 4 Checklist……………………………………………………………………………………..43<\/p>\n
Evaluation Planning Template \u2014 Exercise 4…………………………………………………………44<\/p>\n
Exercise 5 \u2014 Establishing Indicators……………………………………………………………………45<\/p>\n
Indicators of Success……………………………………………………………………………………..45<\/p>\n
Exercise 5 Checklist……………………………………………………………………………………..47<\/p>\n
Indicators Development Template \u2014 Exercise 5 ………………………………………………..48<\/p>\n
Resource Appendix …………………………………………………………………………….49<\/p>\n
Forms Appendix ……………………………………………………………………………………53<\/p>\n
Page III<\/p>\n
Introduction<\/p>\n
If you don\u2019t know where you\u2019re going, how are you gonna\u2019 know when you get there?<\/p>\n
-Yogi Berra<\/p>\n
In line with its core mission – To help people help themselves through the practical application of knowledge and resources to improve their quality of life and that of future generations – the WK. Kellogg Foundation has made program evaluation a priority. As our staff and grantees work on a spectrum of social improvement programs, the need for shaping and contributing to the body of knowledge regarding evaluation becomes increasingly clear. Our first guide, the W.K. Kellogg Foundation Evaluation Handbook, was published in 1998, and has been made available to nearly 7,500 people. The Evaluation Handbook is a practical, step-by-step manual for conducting evaluations. With the Handbook, we introduced the concept of the program logic model and the ways in which applying this concept has added value to our own work.<\/p>\n
The program logic model is defined as a picture of how your organization does its work – the theory and assumptions underlying the program. A program logic model links outcomes (both short- and long-term) with program activities\/processes and the theoretical assumptions\/principles of the program.<\/p>\n
The W.K. Kellogg Foundation Logic Model Development Guide, a companion publication to the Evaluation Handbook, focuses on the development and use of the program logic model. We have found the logic model and its processes facilitate thinking, planning, and communications about program objectives and actual accomplishments. Through this guide, we hope to provide an orientation to the underlying principles and language of the program logic model so it can be effectively used in program planning, implementation, and dissemination of results.<\/p>\n
The premise behind this guide – and our view of the role of evaluation in programming – is simple: Good evaluation reflects clear thinking and responsible program management. Over the years, our experience in using logic models in initiatives such as the Kellogg Youth Initiative Partnerships, Devolution, ENLACE (Engaging Latino Communities for Education), and the Native American Higher Education Initiative, to name just a few, has provided ample evidence of the effectiveness of these methods.<\/p>\n
Learning and using tools like logic models can serve to increase the practitioner\u2019s voice in the domains of planning, design, implementation, analysis, and knowledge generation. The process of developing the model is an opportunity to chart the course. It is a conscious process that creates an explicit understanding of the challenges ahead, the resources available, and the timetable in which to hit the target. In addition, it helps keep a balanced focus on the big picture as well as the component parts.<\/p>\n
In general, logic modeling can greatly enhance the participatory role and usefulness of evaluation as a management and learning tool. Developing and using logic models is an important step in building community capacity and strengthening community voice. The ability to identify outcomes and anticipate ways to measure them provides all program participants with a clear map of the road ahead. Map in hand, participants are more confident of their place in the scheme of things, and hence, more likely to actively engage and less likely to stray from the course – and when they do, to do so consciously and intentionally. Because it is particularly amenable to visual depictions, program logic modeling can be a strong tool in communicating with diverse audiences – those who have varying world views and different levels of experience with program development and evaluation.<\/p>\n
Page IV<\/p>\n
Introduction<\/p>\n
The Logic Model Development Guide contains four chapters and two comprehensive appendices.<\/p>\n
Chapter 1 presents a basic introduction to the logic model as an action-oriented tool for program planning and evaluation. It also offers an array of sample logic models.<\/p>\n
Chapter 2 consists of exercises and examples focused on the development of a simple program logic model. Exercises include practical examples, checklists for reviewing content quality, and a template for developing a logic model.<\/p>\n
Chapter 3 gives instructions on how to expand a basic logic model to explore and explain the theory-of-change that describes the rationale for your program. A template and checklist are provided.<\/p>\n
Chapter 4 offers two exercises that afford the reader with an introduction to how the basic logic modeling techniques introduced in the previous chapters can be applied to inform thinking about what should be included in an evaluation plan. Templates and checklists are also provided.<\/p>\n
The Resources Appendix provides logic model development resources – references and Web sites worth visiting. The Forms Appendix includes blank templates to copy when developing your own logic models.<\/p>\n
Acknowledgements<\/p>\n
This work builds on the experience of many at the WK. Kellogg Foundation who pioneered the application of logic modeling to their initiatives. For example, logic models were first used with the Kellogg Youth Initiative Partnerships (KYIP). In this application, the models were instrumental in helping staff establish program direction, implementation, an evaluation framework, and outcomes across three sites. In KYIP, logic modeling was used to facilitate and guide the development of the specific assumptions and processes that ultimately led to the transition of the initiative from a WKKF-operated program to a community-owned program. WKKF program staff, including Tyrone Baines, Phyllis Meadows, Gerald Smith, Judy Watson Olson, Steve Peffers, Joyce Brown, and John Seita were instrumental in these efforts.<\/p>\n
Our work in developing the Logic Model Development Guide began at the request of Kellogg Foundation Program Director Blas Santos who expressed a need for user-friendly tools and processes to support the work of grantees in Latin America and the Caribbean.<\/p>\n
The Logic Model Development Guide represents a collaborative effort. We particularly want to acknowledge the efforts of the Kellogg Foundation\u2019s former director of evaluation, Ricardo Millett, and his team of evaluation managers, including Astrid Hendricks-Smith and Mark Lelle, who have since left the organization. Their tireless work among staff and grantees continues to promote the use of logic models to plan, design, and manage initiatives. Dale Hopkins and Karin Ladley were instrumental in bringing the material to print. We also wish to acknowledge the work of the Kellogg Foundation Vice Presidents of Programs Rick Foster, Gail McClure, Dan Moore, and Gloria Smith, along with Senior Vice President of Programs Anne Petersen, who have underscored the importance of evaluation, embraced the logic model approach, and adopted it as a valued program support tool.<\/p>\n
Special thanks are extended to Cynthia Phillips, a primary writer and consultant throughout the development of this guide, and Work Volk Consultants, LLP, for formatting and editorial assistance. Thanks, also, to Beverly Parsons of In Sites; Andrew Hahn and the students at the Florence Heller Graduate School for Advanced Studies in Social Welfare, Brandeis University; Marc Osten, Summit Consulting Collaborative; Sally Bond, The Program Evaluation Group; Joel Meister and Eva Moya, University of Arizona; Amy Coates-Madsen and staff at Maryland Association of Nonprofit Organizations; and Gail Randall, Greater Worchester Community Foundation.<\/p>\n
-The Program Staff of the WK. Kellogg Foundation<\/p>\n
Page 1<\/p>\n
A program logic model is a picture of how your program works \u2014 the theory and assumptions underlying the program. …This model provides a road map of your program, highlighting how it is expected to work, what activities need to come before others, and how desired outcomes are achieved (p. 35).<\/p>\n
W.K. Kellogg Foundation Evaluation Handbook (1998)<\/p>\n
Chapter 1 Introduction to Logic Models<\/p>\n
Chapter One defines logic models and explains their usefulness to program stakeholders. You will learn the relevance of this state-of-the-art tool to program planning, evaluation, and improvement.<\/p>\n
Effective program evaluation does more than collect, analyze, and provide data. It makes it possible for you \u2014 program stakeholders \u2014 to gather and use information, to learn continually about and improve programs that you operate in or fund. The W.K. Kellogg Foundation believes evaluation \u2014 especially program logic model approaches \u2014 is a learning and management tool that can be used throughout a program\u2019s life \u2014 no matter what your stake in the program. Using evaluation and the logic model results in effective programming and offers greater learning opportunities, better documentation of outcomes, and shared knowledge about what works and why. The logic model is a beneficial evaluation tool that facilitates effective program planning, implementation, and evaluation.<\/p>\n
The What and Why of the Logic Model<\/p>\n
The WHAT: Logic Model Definition<\/p>\n
Basically, a logic model is a systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan, and the changes or results you hope to achieve.<\/p>\n
Figure 1. The Basic Logic Model.<\/p>\n
The most basic logic model is a picture of how you believe your program will work. It uses words and\/or pictures to describe the sequence of activities thought to bring about change and how these activities are linked to the results the program is expected to achieve.<\/p>\n
Page 2<\/p>\n
Most of the value in a logic model is in the process of creating, validating, and modifying the model… The clarity of thinking that occurs from building the model is critical to the overall success of the program (p. 43).<\/p>\n
W.K. Kellogg Foundation Handbook (1998)<\/p>\n
Chapter 1<\/p>\n
The Basic Logic Model components shown in Figure 1 above are defined below. These components illustrate the connection between your planned work and your intended results. They are depicted numerically by steps 1 through 5.<\/p>\n
YOUR PLANNED WORK describes what resources you think you need to implement your program and what you intend to do.<\/p>\n
1. Resources include the human, financial, organizational, and community resources a program has available to direct toward doing the work. Sometimes this component<\/p>\n
is referred to as Inputs.<\/p>\n
2. Program Activities are what the program does with the resources. Activities are the<\/p>\n
processes, tools, events, technology, and actions that are an intentional part of the program implementation. These interventions are used to bring about the intended program changes or results.<\/p>\n
YOUR INTENDED RESULTS include all of the program\u2019s desired results (outputs, outcomes, and impact).<\/p>\n
3. Outputs are the direct products of program activities and may include types, levels and targets of services to be delivered by the program.<\/p>\n
4. Outcomes are the specific changes in program participants\u2019 behavior, knowledge, skills, status and level of functioning. Short-term outcomes should be attainable within 1 to 3 years, while longer-term outcomes should be achievable within a 4 to 6 year timeframe. The logical progression from short-term to long-term outcomes should be reflected in impact occurring within about 7 to 10 years.<\/p>\n
5. Impact is the fundamental intended or unintended change occurring in organizations, communities or systems as a result of program activities within 7 to 10 years. In the current model of WKKF grantmaking and evaluation, impact often occurs after the conclusion of project funding.<\/p>\n
The term logic model is frequently used interchangeably with the term program theory in the evaluation field. Logic models can alternatively be referred to as theory because they describe how a program works and to what end (definitions for each employed by leading evaluation experts are included in the Resources Appendix).<\/p>\n
The What. How to \u201cRead\u201d a Logic Model<\/p>\n
When \u201cread\u201d from left to right, logic models describe program basics over time from planning through results. Reading a logic model means following the chain of reasoning or \u201cIf…then…\u201d statements which connect the program\u2019s parts. The figure below shows how the basic logic model is read.<\/p>\n
Page 3<\/p>\n
Sample Factors influencing the trip:<\/p>\n
\u2022 Family members\u2019 school and work schedules<\/p>\n
\u2022 The holidays<\/p>\n
\u2022 Winter weather<\/p>\n
\u2022 Frequent Flier availability<\/p>\n
Sample Activities:<\/p>\n
\u2022 Creating\/checking family schedules<\/p>\n
\u2022 Gathering holiday flight and FF information<\/p>\n
\u2022 Getting airport transportation<\/p>\n
\u2022 Notifying Iowa relatives<\/p>\n
Figure 2. How to Read a Logic Model.<\/p>\n
The WHY: Logic Model Purpose and Practical Application<\/p>\n
The purpose of a logic model is to provide stakeholders with a road map describing the sequence of related events connecting the need for the planned program with the program\u2019s desired results. Mapping a proposed program helps you visualize and understand how human and financial investments can contribute to achieving your intended program goals and can lead to program improvements.<\/p>\n
A logic model brings program concepts and dreams to life. It lets stakeholders try an idea on for size and apply theories to a model or picture of how the program would function. The following example shows how the logic model approach works. (If you are familiar with logic models, you may wish to skip ahead to the section entitled \u201cWhy Use A Logic Model?\u201d)<\/p>\n
An Example:<\/p>\n
We are proposing an inexpensive family trip from Charleston, South Carolina, to Des Moines, Iowa, to visit relatives during December school holidays. The seasonal trip we dream of taking from Charleston to Des Moines is the \u201cprogram.\u201d Basic assumptions about our trip \u201cprogram\u201d are:<\/p>\n
\u2022 We want to visit relatives between 12\/10\/00 and 1\/5\/01 while the children are out of school.<\/p>\n
\u2022 We will fly from South Carolina to Iowa because it takes less time than driving and because frequent flier (FF) miles are available.<\/p>\n
\u2022 Using frequent flier miles will reduce travel costs.<\/p>\n
We have to determine the factors influencing our trip, including necessary resources, such as, the number of family members, scheduled vacation time, the number of frequent flier miles we have, round trip air reservations for each family member, and transportation to and from our home to the airport. The activities necessary to make this happen are the creation of our own family holiday schedule, securing our Iowa relative\u2019s schedule, garnering air line information and reservations and planning for transportation to and from the airport.<\/p>\n
Page 4<\/p>\n
You can\u2019t do \u201cgood\u201d evaluation if you have a poorly planned program.<\/p>\n
Beverly Anderson Parsons (1999)<\/p>\n
Chapter 1<\/p>\n
In this example, the results of our activities \u2014 or outputs \u2014 are mostly information, such as family schedules, flight schedules, and cost information based on the time frame of the trip. This information helps identify outcomes or immediate goals. For instance, if we make reservations as soon as possible, we are able to find flights with available frequent flier slots and probably have more options for flights that fit within the time frame. Knowing this, our outcomes improve \u2014 reservations made well in advance result in flight schedules and airline costs that suit our timeline and travel budget. Longer-term impact of our trip is not an issue here, but might be projected as continued good family relationships in 2010.<\/p>\n
Using a simple logic model as a trip-planning tool produced tangible benefits. It helped us gather information to influence our decisions about resources and allowed us to meet our stated goals. Applying this process consistently throughout our trip planning positions us for success by laying out the best course of action and giving us benchmarks for measuring progress \u2014 when we touch down in Charlotte and change planes for Cincinnati, we know we\u2019re on course for Des Moines.<\/p>\n
Typical logic models use table and flow chart formats like those presented here to catalogue program factors, activities, and results and to illustrate a program\u2019s dimensions. Most use text and arrows or a graphic representation of program ideas. This is what our trip planning \u201cprogram\u201d could look like in logic model format.<\/p>\n
It was easy to organize travel plans in a flow chart, but we could also choose to organize and display our thinking in other ways. A logic model does not have to be linear. It may appear as a simple image or concept map to describe more complex program concepts. Settling on a single image of a program is sometimes the most difficult step for program stakeholders.<\/p>\n
Page 5<\/p>\n
If program planners don\u2019t have any hypotheses guiding them, their potential for learning from the initiative is low, and the program is probably in trouble (p. 1).<\/p>\n
Everything You Wanted to Know About Logic Models but Were Afraid to Ask,<\/p>\n
Connie Schmitz and Beverly Anderson Parsons (1999)<\/p>\n
Why Use a Logic Model?<\/p>\n
As you can see from the travel plan example, logic models are useful tools in many ways. Because they are pictorial in nature, they require systematic thinking and planning to better describe programs. The visual representation of the master plan in a logic model is flexible, points out areas of strength and\/or weakness, and allows stakeholders to run through many possible scenarios to find the best. In a logic model, you can adjust approaches and change courses as program plans are developed. Ongoing assessment, review, and corrections can produce better program design and a system to strategically monitor, manage, and report program outcomes throughout development and implementation.<\/p>\n
Effective evaluation and program success rely on the fundamentals of clear stakeholder assumptions and expectations about how and why a program will solve a particular problem, generate new possibilities, and make the most of valuable assets. The logic model approach helps create shared understanding of and focus on program goals and methodology, relating activities to projected outcomes.<\/p>\n
The bane of evaluation is a poorly designed program.<\/p>\n
Ricardo Millett, Director, WKKF Evaluation Unit<\/p>\n
Logic Models Better Position Programs For Success<\/p>\n
Many evaluation experts agree that use of the logic model is an effective way to ensure program success. Using a logic model throughout your program helps organize and systematize program planning, management, and evaluation functions.<\/p>\n
1. In Program Design and Planning, a logic model serves as a planning tool to develop program strategy and enhance your ability to clearly explain and illustrate program concepts and approach for key stakeholders, including funders.<\/p>\n
Logic models can help craft structure and organization for program design and build in self-evaluation based on shared understanding of what is to take place. During the planning phase, developing a logic model requires stakeholders to examine best practice research and practitioner experience in light of the strategies and activities selected to achieve results.<\/p>\n
2. In Program Implementation, a logic model forms the core for a focused management plan that helps you identify and collect the data needed to monitor and improve programming.<\/p>\n
Using the logic model during program implementation and management requires you to focus energies on achieving and documenting results. Logic models help you to consider and prioritize the program aspects most critical for tracking and reporting and make adjustments as necessary.<\/p>\n
3. For Program Evaluation and Strategic Reporting, a logic model presents program information and progress toward goals in ways that inform, advocate for a particular program approach, and teach program stakeholders.<\/p>\n
Page 6<\/p>\n
There are many ways to conduct evaluations, and professional evaluators tend to agree that there is no \u201cone best way\u201d to do any evaluation. Instead, good evaluation requires carefully thinking through the questions that need to be answered, the type of program being evaluated, and the ways in which the information generated will be used. Good evaluation, in our view, should provide useful information about program functioning that can contribute to program improvement.<\/p>\n
W.K. Kellogg Foundation Evaluation Unit<\/p>\n
Chapter 1<\/p>\n
We all know the importance of reporting results to funders and to community stakeholders alike. Communication is a key component of a program\u2019s success and sustainability. Logic models can help strategic marketing efforts in three primary ways:<\/p>\n
\u2022 Describing programs in language clear and specific enough to be understood and evaluated.<\/p>\n
\u2022 Focusing attention and resources on priority program operations and key results for the purposes of learning and program improvement.<\/p>\n
\u2022 Developing targeted communication and marketing strategies.<\/p>\n
The Table below describes the relationship between a successful program and the benefits derived from the use of logic models.<\/p>\n
How Logic Models Better Position Programs Toward Success.<\/p>\n
Program Elements<\/p>\n
Criteria for Program Success1<\/p>\n
Benefits of Program Logic Models2<\/p>\n
Planning and Design<\/p>\n
Program goals and objectives, and important side effects are well defined ahead of time.<\/p>\n
Finds \u201cgaps\u201d in the theory or logic of a program and work to resolve them.<\/p>\n
Program goals and objectives are both plausible and possible.<\/p>\n
Builds a shared understanding of what the program is all about and how the parts work together.<\/p>\n
Program<\/p>\n
Implementation and Management<\/p>\n
Relevant, credible, and useful performance data can be obtained.<\/p>\n
Focuses attention of management on the most important connections between action and results.<\/p>\n
Evaluation, Communication, and Marketing<\/p>\n
The intended users of the evaluation results have agreed on how they will use the information.<\/p>\n
Provides a way to involve and engage stakeholders in the design, processes, and use of evaluation.<\/p>\n
1 Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (Eds.). (1994). Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass Publishers.<\/p>\n
2 Barley, Z., Phillips, C., & Jenness, M. (1998). Decoding Program Logic Models. Workshop presented at the Annual Meeting of the American Evaluation Association, Chicago, IL, November, 1998.<\/p>\n
Logic Models Strengthen the Case for Program Investment<\/p>\n
Clear ideas about what you plan to do and why \u2014 as well as an organized approach to capturing, documenting, and disseminating program results \u2014 enhance the case for investment in your program.<\/p>\n
Page 7<\/p>\n
LOGIC MODEL IF… THEN Assumptions:<\/p>\n
\u2022 Certain resources are needed to operate your program.<\/p>\n
\u2022 If you have access to them, then you can use them to accomplish your planned activities.<\/p>\n
\u2022 If you accomplish your planned activities, then, you will, it is hoped, deliver the amount of product and\/or service that you intended.<\/p>\n
\u2022 If you accomplish your planned activities to the extent intended, then your participants will benefit in specific ways.<\/p>\n
\u2022 If these benefits to participants are achieved, then certain changes in organizations, communities, or systems might occur under specified conditions.<\/p>\n
Developing a Program Logic Model Requires a Simple Image and a Straightforward Approach<\/p>\n
A picture IS worth a thousand words. The point of developing a logic model is to come up with a relatively simple image that reflects how and why your program will work. Doing this as a group brings the power of consensus and group examination of values and beliefs about change processes and program results.<\/p>\n
Logic Models Reflect Group Process and Shared Understanding<\/p>\n
Frequently, a professional evaluator is charged with developing a logic model for program practitioners. But a logic model developed by all stakeholders \u2014 program staff, participants, and evaluators \u2014 produces a more useful tool and refines program concepts and plans in the process. We recommend that a logic model be developed collaboratively in an inclusive, collegial process that engages as many key stakeholders as possible. This guide provides a step-by-step process to assist program planners.<\/p>\n
Like Programs, Logic Models Can Change Over Time<\/p>\n
As a program grows and develops, so does its logic model. A program logic model is merely a snapshot of a program at one point in time; it is not the program with its actual flow of events and outcomes. A logic model is a work in progress, a working draft that can be refined as the program develops.<\/p>\n
Simple Logic Model Basics <\/p>\n
Creating a logic model: What they look like and what needs to be included<\/p>\n
Logic models come in as many sizes and shapes as the programs they represent. A simple model focuses on project-level results and explains five basic program components. The elements outlined below are typical of the model promoted by United Way of America to support an outcomes-based approach to program planning and evaluation.<\/p>\n
Developing and Reading a Basic Logic Model<\/p>\n
Read from left to right, logic models describe program basics over time, beginning with best practice information or knowledge about \u201cwhat works\u201d from successful program practitioners and other trusted authorities. Reading a logic model means following the chain of reasoning or \u201cIf…then…\u201d statements which connect the program\u2019s parts. The gray box in the left column defines the assumptions stated in \u201cIf…then…\u201d terms.<\/p>\n
Page 8<\/p>\n
Chapter 1<\/p>\n
Building a Logic Model by Basic Program Components<\/p>\n
As you conceptualize your program, begin by describing your basic assumptions and then add the following program components in the order that they should occur.<\/p>\n
1. Factors are resources and\/or barriers, which potentially enable or limit program effectiveness. Enabling protective factors or resources may include funding, existing organizations, potential collaborating partners, existing organizational or interpersonal networks, staff and volunteers, time, facilities, equipment, and supplies. Limiting risk factors or barriers might include such things as attitudes, lack of resources, policies, laws, regulations, and<\/p>\n
geography.<\/p>\n
2. Activities are the processes, techniques, tools, events, technology, and actions of the planned program. These may include products \u2014 promotional materials and educational curricula; services \u2014 education and training, counseling, or health screening; and infrastructure \u2014 structure, relationships, and capacity used to bring about the desired results.<\/p>\n
3. Outputs are the direct results of program activities. They are usually described in terms of the size and\/or scope of the services and products delivered or produced by the program. They indicate if a program was delivered to the intended audiences at the intended \u201cdose.\u201d A program output, for example, might be the number of classes taught, meetings held, or materials produced and distributed; program participation rates and demography; or hours of each type of service provided.<\/p>\n
4. Outcomes are specific changes in attitudes, behaviors, knowledge, skills, status, or level of functioning expected to result from program activities and which are most often expressed at an individual level.<\/p>\n
5. Impacts are organizational, community, and\/or system level changes expected to result from program activities, which might include improved conditions, increased capacity, and\/or changes in the policy arena.<\/p>\n
Thinking about a program in logic model terms prompts the clarity and specificity required for success, and often demanded by funders and your community. Using a simple logic model produces (1) an inventory of what you have and what you need to operate your program; (2) a strong case for how and why your program will produce your desired results; and (3) a method for program management and assessment.<\/p>\n
Other Logic Model Examples<\/p>\n
In practice, most logic models are more complex and fall into one of three categories: the theory approach model (conceptual), outcome approach model, or activities approach model (applied) \u2014 or a blend of several types. It is not unusual for a program to use all three types of logic models for different purposes. No one model fits all needs, so you will<\/p>\n
Page 9<\/p>\n
need to decide exactly what you want to achieve with your logic model \u2014 and where you are in the life of your program \u2014 before deciding on which model to use.<\/p>\n
Types of Logic Models: Emphasis and Strengths<\/p>\n
Types of Logic Models: Emphasis and Strengths<\/p>\n
A program is a theory and an evaluation is its test. In order to organize the evaluation to provide a responsible test, the evaluator needs to understand the theoretical premises on which the program is based (p. 55).<\/p>\n
Carol Weiss (1998)<\/p>\n
Descriptions of Three Approaches to Logic Models: Which Fits Your Program?<\/p>\n
1. Theory Approach Models emphasize the theory of change that has influenced the design and plan for the program. These logic models provide rich explanation of the reasons for beginning to explore an idea for a given program. Sometimes they have additional parts that specify the problem or issue addressed by the program, describe the reasons for selecting certain types of solution strategies, connect proven strategies to potential activities, and other assumptions the planners hold that influence effectiveness. These models illustrate how and why you think your program will work. They are built from the \u201cbig picture\u201d kinds of thoughts and ideas that went into conceptualizing your program. They are coming to be most often used to make the case in grant proposals. Models describing the beginnings of a program in detail are most useful during program planning and design.<\/p>\n
Page 10<\/p>\n
The purpose of using program logic models in WKKF grantmaking is to help internal and external stakeholders understand how the Foundation\u2019s investment will contribute to achieving the intended goals. This understanding should help these various stakeholders make informed decisions about program priorities, funding priorities, assistance to grantees, evaluation of programming impact, and marketing, communication, and marketing strategies.<\/p>\n
W.K. Kellogg<\/p>\n
Foundation Evaluation Handbook (1998)<\/p>\n
Chapter 1<\/p>\n
2. Outcomes Approach Models focus on the early aspects of program planning and attempt to connect the resources and\/or activities with the desired results in a workable program. These models often subdivide outcomes and impact over time to describe short-term (1 to 3 years), long-term (4 to 6 years), and impact (7 to 10 years) that may result from a given set of activities. Although these models are developed with a theory of change in mind, this aspect is not usually emphasized explicitly. Models that outline the approach and expectations behind a program\u2019s intended results are most useful in designing effective evaluation and reporting strategies.<\/p>\n
3. Activities Approach Models pay the most attention to the specifics of the implementation process. A logic model of this type links the various planned activities together in a manner that maps the process of program implementation. These models describe what a program intends to do and as such are most useful for the purposes of program monitoring and management. This type provides the detailed steps you think you will need to follow to implement your program. It shows what you will actually do in your community if your proposal is funded. Models that emphasize a program\u2019s planned work are most often used to inform management planning activities.<\/p>\n
Working Through Theory Approach Logic Models Emphasizes Assumptions<\/p>\n
A theory approach logic model links theoretical ideas together to explain underlying program assumptions. The focus here is on the problem or issue and the reasons for proposing the solution suggested in your program\u2019s approach. Remember, the theory logic model is broad and about \u201cbig ideas,\u201d not about specific program \u201cnuts and bolts.\u201d<\/p>\n
Noted evaluator and program theorist Carol Weiss (1998) explains that for program planning, monitoring, and evaluation, it is important to know not only what the program expects to achieve but also how. We must understand the principles on which a program is based, a notion not included in evaluation until recently. Discussions about the whethers, hows, and whys of program success require credible evidence and attention to the paths by which outcomes and impacts are produced.<\/p>\n
The theory logic model is suitable for use by funders and grantees. A case example of its use is provided below.<\/p>\n
In this case, the model describes a WKKF cluster initiative\u2019s (Comprehensive Community Health Models of Michigan) programming strategy or its theory of change. Notice that this model places emphasis on \u201cYour Beginnings\u201d by including the assumptions identified by program planners as the principles behind the design of the initiative.<\/p>\n
Page 11<\/p>\n
These models help build a common understanding between managers and evaluators…. Such agreement is a prerequisite for evaluation work that is likely to be useful to management. [These models] display the key events (inputs, activities, outcomes) that could be monitored and the assumed causal linkages that could be tested in evaluations of the program.<\/p>\n
Joseph S. Wholey, Harry P. Hatry, and K.E. Newcomer (1994)<\/p>\n
Example of a Theory Logic model (Adapted from WKKF\u2019s Comprehensive Community Health Models of Michigan).<\/p>\n
Working with Outcome Approach Models Highlights Activities and Program Implementation<\/p>\n
Outcome approach logic models display the interrelationships between specific program activities and their outcomes. On the next page is an example drawn from the Calhoun County Health Improvement Program, funded under the Comprehensive Community Health Models of Michigan initiative.<\/p>\n
This linear, columnar model emphasizes the causal linkages thought to exist among program components. The arrows show which sets of activities program developers believed would contribute to what outcomes. These statements serve as logical assertions about the perceived relationship among program operations and desired results and are the hallmark of the logic model process.<\/p>\n
Notice that this model emphasizes \u201cYour Intended Results\u201d in the greatest relative detail and anticipates achievement outside the time allotted for the initiative.<\/p>\n
Page 12<\/p>\n
Not only will a logic model clarif each element of your program, it will enable you to respond to the question: \u201cTo what do I want to be held accountable?\u201d.<\/p>\n
The Evaluation Forum<\/p>\n
(1999)<\/p>\n
Example of an Outcome Approach model (example drawn from the Calhoun County Health Improvement Program, funded under the Comprehensive Community Health Models of Michigan initiative).<\/p>\n
Chapter 1<\/p>\n
Using the Activities Approach Models to Track Outcomes<\/p>\n
The activities approach logic model also connects program resources and activities to desired results but does so in very great detail. Each outcome is usually dealt with separately by the activities and events that must take place to keep the program on track. The model emphasizing \u201cYour Planned Work\u201d can be used as a work plan or management tool for program components and in conjunction with other models.<\/p>\n
Notice how it points out what program activities need to be monitored and what kind of measurements might indicate progress toward results. Below is one model describing the connections between project tasks and outcome achievement for the community coverage strand from the outcome approach example provided earlier.<\/p>\n
Page 13<\/p>\n
Adapted from the Calhoun County Health Improvement Program, one site of WKKF\u2019s Comprehensive Community Health Models of Michigan initiative<\/p>\n
There Is No Best Logic Model<\/p>\n
Try several on for size. Choose the model that fits your program best and provides the information you need in the format that is most helpful. Like anything else, it takes practice to use logic models as effective program tools. We learn through trial and error to find what works best for what program. Don\u2019t hesitate to experiment with program logic model design to determine what works best for your program. And don\u2019t be concerned if your model doesn\u2019t look like one of the case examples.<\/p>\n
The following show how the logic model forms gather information that can be used throughout your program\u2019s life \u2014 from defining the theory on which your program rests to evaluating program impact.<\/p>\n
Page 14<\/p>\n
Chapter 1<\/p>\n
How to use a Logic Model Through the Life of Your Program:<\/p>\n
1. Program Planning<\/p>\n
For more detail, see the Program Planning Template on p. 57.<\/p>\n
CLARIFYING PROGRAM THEORY:<\/p>\n
1. PROBLEM OR ISSUE STATEMENT: Describe the problem(s) your program is attempting to solve or the issue(s) your program will address.<\/p>\n
2. COMMUNITY NEEDS\/ASSETS: Specify the needs and\/or assets of your community that led your organization to design a program that addresses the problem.<\/p>\n
3. DESIRED RESULTS (OUTPUTS, OUTCOMES AND IMPACTS): Identify desired results, or vision of the future, by describing what you expect to achieve near- and long-term.<\/p>\n
4. INFLUENTIAL FACTORS: List the factors you believe will influence change in your community.<\/p>\n
5. STRATEGIES: List general successful strategies or \u201cbest practices\u201d that have helped communities like yours achieve the kinds of results your program promises.<\/p>\n
6. ASSUMPTIONS: State the assumptions behind how and why the change strategies will work in your community.<\/p>\n
2. Program Implementation<\/p>\n
RESOURCES<\/p>\n
ACTIVITIES<\/p>\n
OUTPUTS SHORT<\/p>\n
SHORT & LONG-TERM OUTCOMES<\/p>\n
IMPACT<\/p>\n
In order to accomplish our set of activities we will need the following:<\/p>\n
In order to address our problem or asset we will accomplish the following activities:<\/p>\n
We expect that once accomplished these activities will produce the following evidence or service delivery:<\/p>\n
We expect that if accomplished these activities will lead to the following changes in 1-3 then 4-6 years:<\/p>\n
We expect that if accomplished these activities will lead to the following changes in 7-10 years:<\/p>\n
For more detail, see the Program Implementation Template on p. 54.<\/p>\n
DEMONSTRATING YOUR PROGRAM\u2019S PROGRESS:<\/p>\n
1. OUTPUTS: For each program activity, identify what outputs (service delivery\/implementation targets) you aim to produce.<\/p>\n
2. OUTCOMES: Identify the short-term and long-term outcomes you expect to achieve for each activity.<\/p>\n
3. IMPACT: Describe the impact you anticipate in your community in 7 to 10 years with each activity as a result of your program.<\/p>\n
4. ACTIVITIES: Describe each of the activities you plan to conduct in your program.<\/p>\n
5. RESOURCES: Describe the resources or influential factors available to support your program activities.<\/p>\n
3. Program Evaluation<\/p>\n
For more detail, see the Evaluation Planning Template on p. 59.<\/p>\n
For more detail, see the Indicators Development Template on p. 61.<\/p>\n
PROGRAM EVALUATION QUESTIONS AND INDICATORS:<\/p>\n
1. FOCUS AREA: From your program theory logic model, list the components of the most important aspects of your program.<\/p>\n
2. AUDIENCE: Identify the key audiences for each focus area. Who has an interest in your program?<\/p>\n
3. QUESTIONS: For each focus area and audience, list the questions they may have about your program.<\/p>\n
4. INFORMATION USE: For each audience and question you have identified, identify the ways you will use the evaluation information.<\/p>\n
5. INDICATORS: Describe what information could be collected that would indicate the status of your program and its participants for each question.<\/p>\n
6. TECHNICAL ASSISTANCE: Indicate the extent to which your organization has the evaluation and data management expertise to collect and analyze the data that relates to this indicator.<\/p>\n
Page 15<\/p>\n
Over the past few years, I have markedly changed my approach to logic modeling. I have become convinced that it makes a considerable difference if you do the outcomes before planning the activities.<\/p>\n
I definitely advocate doing the outcomes first! I find that people come up with much more effective activities when they do. Use the motto, \u201cplan backward, implement forward.\u201d<\/p>\n
Beverly Anderson Parsons, WKKF Cluster Evaluator<\/p>\n
Chapter 2 Developing a Basic Logic Model For Your Program<\/p>\n
Drawing a picture of how your program will achieve results<\/p>\n
Whether you are a grant seeker developing a proposal for start-up funds or a grantee with a program already in operation, developing a logic model can strengthen your program. Logic models help identify the factors that will affect your program and enable you to anticipate the data and resources you will need to achieve success. As you engage in the process of creating your program logic model, your organization will systematically address these important program planning and evaluation issues:<\/p>\n
\u2022 Cataloguing of the resources and actions you believe you will need to reach intended results.<\/p>\n
\u2022 Documentation of connections among your available resources, planned activities and the results you expect to achieve.<\/p>\n
\u2022 Description of the results you are aiming for in terms of specific, measurable, action-oriented, realistic and timed outcomes.<\/p>\n
The exercises in this chapter gather the raw material you need to draw a basic logic model that illustrates how and why your program will work and what it will accomplish. You can benefit from creating a logic model at any point in the life of any program. The logic model development process helps people inside and outside your organization understand and improve the purpose and process of your work.<\/p>\n
Chapter 2 is organized into two sections \u2014 Program Implementation, and Program Results. The best recipe for program success is to complete both exercises. (Full-size masters of each exercise and the checklists are provided in the Forms Appendix at the back of the guide for you to photocopy and use with stakeholder groups as you design your program.)<\/p>\n
Exercise 1: Program Results. In a series of three steps, you describe the results you plan to achieve with your program.<\/p>\n
Exercise 2: Program Resources and Activities by taking you through three steps that connect the program\u2019s resources to the actual activities you plan to do.<\/p>\n
The Mytown Example<\/p>\n
Throughout Exercises 1 and 2 we\u2019ll follow an example program to see how the logic model steps can be applied. In our example, the folks in Mytown, USA, are striving to meet the needs of growing numbers of uninsured residents who are turning to Memorial Hospital\u2019s Emergency Room for care. Because that care is expensive and not the best way to offer care, the community is working to create a free clinic. Throughout the chapters, Mytown\u2019s program information will be dropped into logic model templates for Program Planning, Implementation, and Evaluation.<\/p>\n
Page 16<\/p>\n
Chapter 2<\/p>\n
Novice logic modelers may want to have copies of the Basic Logic Model Template in front of them and follow along. Those readers with more experience and familiarity may want to explore the text and then skip ahead to the completed Basic Logic Model for the Mytown Example on page 34.<\/p>\n
Demonstrating Progress Toward Change<\/p>\n
The Importance of Documenting Progress<\/p>\n
According to many funders, grant applications frequently lack solid descriptions of how programs will demonstrate their effectiveness. Some grantees think activities are ends unto themselves. They report the numbers of participants they reach or the numbers of training sessions held as though they were results.<\/p>\n
Conducting an activity is not the same as achieving results from the accomplishment of that activity. For example, being seen by a doctor is different from reducing the number of uninsured emergency room visits. Tracking data like meetings held or patients enrolled does monitor your program\u2019s implementation and performance, but those data are outputs (activity data), not outcomes (which refer to the results you expect to achieve in future years).<\/p>\n
\u201cDo the outcomes first\u201d is sage advice. Most logic models lack specific short- and long-term outcomes that predict what will be achieved several years down the road. Specifying program milestones as you design the program builds in ways to gather the data required and allows you to periodically assess the program\u2019s progress toward the goals you identify. For that reason, Exercise 1 isn\u2019t filled out from left to right. This exercise asks you to \u201cdo the outcomes first.\u201d We will focus our attention first on what we have called \u201cyour intended results.\u201d<\/p>\n
As you implement your program, outcome measures enhance program success by assessing your progress from the beginning and all along the way. That makes it possible to notice problems early on. The elements (Outputs, Outcomes, and Impact) that comprise your intended results give you an outline of what is most important to monitor and gauge to determine the effectiveness of your program. You can correct and revise based on your interpretation of the collected data.<\/p>\n
Exercise 1 \u2014 Describing Results<\/p>\n
Describe the results you desire \u2014 Outputs, Outcomes and Impact<\/p>\n
If you were running the Mytown Free Clinic, how would you show that your desired outcome (a reduction in uninsured emergency care) didn\u2019t result from a mass exodus of uninsured residents from Mytown, USA, or a sudden increase in number of employees offered health insurance coverage by local businesses?<\/p>\n
Page 17<\/p>\n
How will you demonstrate that your program contributed to the change you intend? A well-crafted logic model can assert it is reasonable to claim that your program made a substantive contribution to your intended change. When programs operate in real communities where influences and forces are beyond your control, evaluation is generally more about documenting a program\u2019s contribution than about proving something. Community-based initiatives operate in complex environments where the scientific certainty of \u201cproof\u201d is seldom attainable. This is where logic models can be especially helpful.<\/p>\n
INSTRUCTIONS: Exercise 1 will use the Basic Logic Model Development Template. In particular, you will use the information presented in the gray text boxes that follow about the Mytown example program to determine what results are intended for this program. Example information about outcomes, impacts, and outputs are provided. You will fill in the blank Basic Logic Model Development Template to illustrate first the outcomes and impacts sought and then the outputs. You can then look at the completed template on page 25 to see compare your interpretation with that produced by the Mytown folks.<\/p>\n
Exercise 1 uses the Basic Logic Model Development Template<\/p>\n
Resources<\/p>\n
Activities<\/p>\n
Outputs<\/p>\n
Short- & Long-Term Outcomes<\/p>\n
Impact<\/p>\n
In order to accomplish our set of activities we will need the following:<\/p>\n
In order to address our problem or asset we will conduct the following activities:<\/p>\n
We expect that once completed or under way these activities will produce the following evidence of service delivery:<\/p>\n
We expect that if completed or ongoing these activities will lead to the following changes in 1-3 then 4-6 years:<\/p>\n
We expect that if completed these activities will lead to the following changes in 7-10 years:<\/p>\n
Outcomes and Impacts should be SMART:<\/p>\n
\u2022 Specific<\/p>\n
\u2022 Measurable<\/p>\n
\u2022 Action-oriented<\/p>\n
\u2022 Realistic<\/p>\n
\u2022 Timed<\/p>\n
Page 18<\/p>\n
Chapter 2<\/p>\n
Some logic models number the lists within a column to aid discussion. Some tabular logic models use rows to order and show the relationships among components. Some logic models, like the outcome and activity examples provided in Chapter One, use a box and arrow format to illustrate the \u201ccausal linkages\u201d demonstrating how your resources, activities, outputs, outcomes, and impact connect to form chains. These depictions add to the clarity of your logic model\/evaluation plan. However, for the most basic of logic models, the inventory approach we illustrate is sufficient to capture your thinking about how a program will work. The other techniques will improve its utility, but the most important task is to first get the component parts categorized and described. Once you have completed the inventory table for this and Exercise 2 feel free to experiment with identifying the relationships among the items across columns.<\/p>\n
Short-term outcomes are results you expect to achieve one to three years after a program activity is under way.<\/p>\n
Short-term outcomes are specific changes in things like attitudes, behaviors, knowledge, skills, status, or level of functioning expected to result from program activities. These usually are expressed at an individual level among program participants.<\/p>\n
EXAMPLES: Signed Memorandum of Agreement from the local technical college donating clinic space, change in participants\u2019 attitudes about the need for a medical home, increase in numbers of scheduled annual physicals, increased patient follow-up visits, change in staff\u2019s awareness of patient scheduling challenges, increased appropriate referrals from ER\u2019s.<\/p>\n
Insert Mytown\u2019s short-term outcomes in the Short- and Long-Term Outcomes Column of the Basic Logic Model Development Template.<\/p>\n
Long-term outcomes are results you expect to achieve in four to six years.<\/p>\n
Long-term outcomes are also specific changes in things like attitudes, behaviors, knowledge, skills, status, or level of functioning expected to result from program activities. These usually build on the progress expected by the short-term outcomes.<\/p>\n
EXAMPLES: The clinic serves as a medical home for 500 uninsured patients. The clinic has sustained funding sources: patient co-payments ($10\/visit) provide 20% of the Clinic\u2019s operating costs, United Way provides 20%, Memorial Hospital donates 20%, the Medical Society contributes 20% and an endowment established at the Community Foundation provides the final 20%. An annual golf tournament organized by the Kiwanis Club funds special clinic projects. There has been a 25% reduction in uninsured emergency care since Mytown Free Clinic opened five years ago. In the Clinic\u2019s fifth year there is a 15% reduction in uninsured ER visits. Seventy-five volunteer administrators and 300 volunteer medical professionals regularly serve at the clinic each year. Five companies donate all necessary medical supplies. Grant funds purchase the computers and software needed to create electronic patient records. For five years patient satisfaction ratings have been 90%.<\/p>\n
Insert Mytown\u2019s long-term outcomes in the Short- and Long-Term Outcomes column of the Basic Logic Model Development Template.<\/p>\n
Page 19<\/p>\n
Impact refers to the results expected seven to ten years after an activity is under way \u2014 the future social change your program is working to create.<\/p>\n
Impacts are the kinds of organizational, community, or system level changes expected to result from program activities and which might include improved conditions, increased capacity, and\/or changes in the policy arena.<\/p>\n
EXAMPLES: Specific reduction in inappropriate emergency room use, increased donations of clinic supplies to meet identified needs, a stable supply of medical volunteers, an endowment supporting 35% of the clinic\u2019s operating funds, 900 patients served\/year.<\/p>\n
Insert Mytown\u2019s impacts in the Impact Column of the Basic Logic Model Development Template.<\/p>\n
Outputs are data about activities.<\/p>\n
They are the direct results of program activities. They are usually described in terms of size and scope of the services or products delivered or produced by the program. They indicate whether or not a program was delivered to the intended audiences at the intended \u201cdose.\u201d A program output, for example, might include the number of classes taught, meetings held, materials distributed, program participation rates, or total service delivery hours.<\/p>\n
EXAMPLES: Number of patients referred to the Free Clinic from Memorial ER\/year, the number of patients screened\/year, the number of qualified patients enrolled in the Free Clinic\/year, the average number of patient visits\/day, the total number of patient visits\/year, the number and specialties of medical volunteers, the number of volunteer administrators trained, the number and locations of clinic posters distributed, the number of potential patients calling for information\/ month.<\/p>\n
Insert Mytown\u2019s outputs in the Outputs Column of the Basic Logic Model Development Template.<\/p>\n
Page 20<\/p>\n
Chapter 2<\/p>\n
Exercise 1 Checklist:<\/p>\n
Review what you have created using the checklist below to assess the quality of your draft.<\/p>\n
Progress Toward Results Quality Criteria<\/p>\n
Yes<\/p>\n
Not<\/p>\n
Yet<\/p>\n
Comments Revisions<\/p>\n
1. A variety of audiences are taken into consideration when specifying credible outputs, outcomes, and impacts.<\/p>\n
2. Target participants and\/or partners are described and quantified as outputs (e.g. 100 teachers from 5 rural high schools).<\/p>\n
3. Events, products, or services listed are described as outputs in terms of a treatment or dose (e.g. 30 farmers will participate in at least 3 sessions of program, or curriculum will be distributed to at least 12 agencies).<\/p>\n
4. The intensity of the intervention or treatment is appropriate for the type of participant targeted (e.g. higher-risk participants warrant higher intensities).<\/p>\n
5. The duration of the intervention or treatment is appropriate for the type of participant targeted (e.g. higher-risk participants warrant longer duration).<\/p>\n
6. Outcomes reflect reasonable, progressive steps that participants can make toward longer-term results.<\/p>\n
7. Outcomes address awareness, attitudes, perceptions, knowledge, skills, and\/ or behavior of participants.<\/p>\n
8. Outcomes are within the scope of the program\u2019s control or sphere of reasonable influence.<\/p>\n
9. It seems fair or reasonable to hold the program accountable for the outcomes specified.<\/p>\n
10. The outcomes are specific, measurable, action-oriented, realistic, and timed.<\/p>\n
11. The outcomes are written as change statements (e.g. things increase, decrease, or stay the same).<\/p>\n
12. The outcomes are achievable within the funding and reporting periods specified.<\/p>\n
13. The impact, as specified, is not beyond the scope of the program to achieve.<\/p>\n
Page 21<\/p>\n
I would emphasize that people may well change their minds about the activities that are the most useful after having done the results work.<\/p>\n
Beverly Anderson Parsons, WKKF Cluster Evaluator<\/p>\n
Exercise 2 \u2014 Describing Actions Linking It All Together<\/p>\n
Exercise 2 illustrates exactly how you plan to put your program theory to work. It leads you to identify the resources and activities your program will need to achieve your intended results. This exercise documents your knowledge of the community resources you have available and specific activities your program will implement.<\/p>\n
Program rationales in grant proposals are usually strong. Grantees tend to have a very good sense of what they want to do. However, they frequently fail to make specific connections between their program and related best practice literature and practitioner wisdom that could and should support their approach and their work.<\/p>\n
To connect actions to program results, this exercise links your knowledge of what works with specific descriptions of what your program will do. It requires you to anticipate what will be needed to support program activities. The elements that comprise your program implementation act as a game plan for the program you propose.<\/p>\n
Most logic models list activity items and resources (like planning meetings, curriculum purchase or design, training workshops, and service delivery). Depending on the nature of your effort, other types of products and processes may be included. Management-oriented logic models also include program and evaluation development, staff and volunteer training, recruitment of partners and participants, and the publicity needed to support your work along the way.<\/p>\n
As mentioned earlier, if your program addresses multiple issues you may find it helpful to go through the exercises for each issue in turn and then aggregate them into a larger model that highlights the relationships among issues.<\/p>\n
We recommend referring to a literature review on the problem your program is designed to address when you specify program activities. From this explicit knowledge of what works, you can more clearly connect the abstract strategies supporting the program to its concrete activities.<\/p>\n
When Exercise 2 is complete and you are satisfied that you have an accurate inventory of the Mytown program\u2019s component parts, transfer the information to the Basic Logic Model Development Template. Remember you have already filled in the three columns on the right with what you have learned about the intended results for the Mytown program example.<\/p>\n
What activities are planned? Based on what you know about effective ways to solve problems or build assets, what specific activities have you planned?<\/p>\n
Page 22<\/p>\n
Chapter 2<\/p>\n
EXAMPLES: Personnel Committee launches and completes search for full-time director. Director is hired and oriented to the board and the community. Board and staff visit the Anywhere Free Clinic to learn from its experience and to select documents to replicate (i.e., policies and procedures, job descriptions, equipment needs, budgets, funding strategies, volunteer and patient records). Board and staff conduct program-planning retreat. Based upon Anywhere\u2019s funding plan, board secures Free Clinic\u2019s first-year funding. Marketing Committee creates public relations campaign in collaboration with Volunteer Committee to secure volunteers and patients. Facility Committee creates and completes MOA with technical college to secure a clinic facility. Quality Assurance Committee creates evaluation plan in cooperation with Memorial Hospital\u2019s Emergency Room staff and the local Chamber of Commerce.<\/p>\n
Summarize Mytown\u2019s activities in the Activities column of the Basic Logic Model Development Template<\/p>\n
What resources are needed? Once you have specified what you plan to do, determine the resources you will need to support the solutions your program proposes. For some types of programs, it may also be helpful to describe the influential factors you are counting on to support your efforts in the community.<\/p>\n
EXAMPLES: Medical Society\/Memorial Hospital Task Force for the Uninsured will become a Free Clinic Board of Directors and secure a 501(c)(3) status from the IRS. The Board will recruit 7-10 additional representatives from drug companies, the local technical school, Mytown\u2019s United Way, the Chamber of Commerce, the Community Foundation, the Volunteer Center, the Nurses Association, etc. During a 6-month planning period, board committees will be launched; staff will be recruited\/hired\/oriented; a site visit will be conducted; and the Clinic\u2019s first-year\u2019s funding ($150,000\/year) will be secured. Committees will create an MOA with Memorial Hospital and the Medical Society to secure equipment required: 5 exam tables, 7 desks, 5 blood pressure cuffs, 5 otoscopes, 5 stethoscopes, 5 PDR\u2019s, 1 set of scales, 10 thermometers, three computers, one first aid emergency kit.<\/p>\n
Summarize Mytown\u2019s resources in the Resources column of the Basic Logic Model Development Template.<\/p>\n
Page 23<\/p>\n
Exercise 2 Checklist:<\/p>\n
Review what you have created using the checklist below to assess the quality of your draft.<\/p>\n
Theory into Action Quality Criteria<\/p>\n
Yes<\/p>\n
Not<\/p>\n
Yet<\/p>\n
Comments\/Revisions<\/p>\n
1. Major activities needed to implement the program are listed.<\/p>\n
2. Activities are clearly connected to the specified program theory.<\/p>\n
3. Major resources needed to implement the program are listed.<\/p>\n
4. Resources match the type of program.<\/p>\n
5. All activities have sufficient and appropriate resources.<\/p>\n
Page 24<\/p>\n
Chapter 2<\/p>\n
Here we include a flowchart that summarizes the steps to complete your basic logic model. Keep in mind that you could use this inventory style template to then further describe the relationships among the components using numbered items, rows, or boxes and arrows as we mentioned earlier.<\/p>\n
Flowchart for Exercises 1 & 2 – Describing Results, Resources, and Activities<\/p>\n
Exercise 1 Describing Results<\/p>\n
Step 1.1<\/p>\n
For each of the specific activities you have planned to do, what short-term and then long-term outcomes do you expect to achieve as indicators of the progress made by your program toward its desired results?<\/p>\n
Step 1.2<\/p>\n
For each of the specific activities that you have planned to do, what outputs (service delivery or implementation targets) do you hope to reach through the operation of your program?<\/p>\n
Step 1.3<\/p>\n
For each of the specific activities you have planned to do, what impact do you expect to achieve in your community?<\/p>\n
Exercise 2 Describing Resources and Activities<\/p>\n
Step 2.1<\/p>\n
Knowing what you know about what works to solve problems or Duild assets as specified in the theory of change for your pro-ram, what specific activities have you planned to do?<\/p>\n
Step 2.2<\/p>\n
What resources are available to your program to support the specific activities you have planned to do (for some programs, it may also be important to state those influential factors you are counting on to support your work)?<\/p>\n
Page 25<\/p>\n
Logic Model Development Program Implementation Template \u2014 Exercise 1 & 2<\/p>\n
RESOURCES<\/p>\n
ACTIVITIES<\/p>\n
OUTPUTS<\/p>\n
SHORT-AND LONG-TERM OUTCOMES<\/p>\n
IMPACT<\/p>\n
In order to accomplish our set of activities we will need the following:<\/p>\n
In order to address our problem or asset we will accomplish the following activities:<\/p>\n
We expect that once accomplished these activities will produce the following evidence or service delivery:<\/p>\n
We expect that if accomplished these activities will lead to the following changes in 1-3 then 4-6 years:<\/p>\n
We expect that if accomplished these activities will lead to the following changes in 7-10 years:<\/p>\n
\u2022 IRS 501(c)(3) status<\/p>\n
\u2022 Diverse, dedicated board of directors representing potential partners<\/p>\n
\u2022 Endorsement from Memorial Hospital, Mytown Medical Society, and United Way<\/p>\n
\u2022 Donated clinic facility<\/p>\n
\u2022 Job descriptions for board and staff<\/p>\n
\u2022 First year\u2019s funding ($150,000)<\/p>\n
\u2022 Clinic equipment<\/p>\n
\u2022 Board & staff orientation process<\/p>\n
\u2022 Clinic budget<\/p>\n
\u2022 Launch\/complete search for executive director<\/p>\n
\u2022 Board & staff conduct Anywhere Free Clinic site visit<\/p>\n
\u2022 Board & staff conduct planning retreat<\/p>\n
\u2022 Design and implement funding strategy<\/p>\n
\u2022 Design and implement volunteer recruitment and training<\/p>\n
\u2022 Secure facility for clinic<\/p>\n
\u2022 Create an evaluation plan<\/p>\n
\u2022 Design and implement PR campaign<\/p>\n
\u2022 # of patients referred from ER to the clinic\/year<\/p>\n
\u2022# of qualified patients enrolled in the clinic\/year<\/p>\n
\u2022 # of patient visits\/year<\/p>\n
\u2022 # of medical volunteers serving\/year<\/p>\n
\u2022# of patient flyers distributed<\/p>\n
\u2022 # of calls\/month seeking info about clinic<\/p>\n
\u2022 Memorandum of Agreement for free clinic space<\/p>\n
\u2022 Change in patient attitude about need for medical home<\/p>\n
\u2022 Change in # of scheduled annual physicals\/follow-ups<\/p>\n
\u2022 Increased # of ER\/physician referrals<\/p>\n
\u2022 Decreased volume of un-reimbursed emergencies treated in Memorial ER<\/p>\n
\u2022 Patient co-payments supply 20% of clinic operating costs<\/p>\n
\u2022 25% reduction in # of uninsured ER visits\/year<\/p>\n
\u2022 300 medical volunteers serving regularly each year<\/p>\n
\u2022 Clinic is a United Way Agency<\/p>\n
\u2022 Clinic endowment established<\/p>\n
\u2022 90% patient satisfaction for 5 years.<\/p>\n
\u2022 900 patients served\/year<\/p>\n
This page was intentionally left blank. <\/p>\n
Page 27<\/p>\n
Chapter 3 Developing a Theory-of-Change Logic Model For Your Program<\/p>\n
Drawing a picture of why your program should succeed<\/p>\n
Whether you are a grant seeker developing a proposal for start-up funds or a grantee with a program already in operation, developing a logic model can strengthen your program. Logic models help identify the factors that will affect your program and enable you to anticipate the data and resources you will need to achieve success. As you engage in the process of creating your program logic model, your organization will systematically address these important program planning and evaluation issues:<\/p>\n
\u2022 Description of the change strategy that your program supports.<\/p>\n
\u2022 Definition of the problem you are attempting to address.<\/p>\n
\u2022 Quantification of the scope of the needs or assets that make the case for your selection of the problem you address.<\/p>\n
\u2022 Acknowledgement of the factors that may influence your ability to create change in your community.<\/p>\n
\u2022 Application of best practice research that supports plausible solution strategies for identified problem area.<\/p>\n
\u2022 Statement of your assumptions about why your selected strategies will work in your community in the ways you described.<\/p>\n
Exercise 3: Program Planning constructs a program theory. Successful programs create change and are built on a solid knowledge of what works \u2014 your program\u2019s theory. Exercise 3 guides you through a series of six steps that diagram the fundamental theory that supports your program. This supports and builds upon the basic logic model. In most cases, if you are developing a new program, this step should come first to inform your preliminary thinking. We have placed it after basic logic models because it is a slightly more complex exercise.<\/p>\n
Page 28<\/p>\n
Chapter 3<\/p>\n
Exercise 3 \u2014 Constructing a Program Theory Program Planning<\/p>\n
The Importance of Framing Your Problems or Issues with Sound Program Theory<\/p>\n
Imagine you work for a funding organization. Each quarter you review a mountain of grant proposals from worthy organizations doing good work. All of them need money. What information would you require to determine which programs to fund? Funders tell us they look for organizations that have done their homework by clearly defining the problem they plan to address, describing the reasons behind their approach, and outlining how they anticipate measuring their achievements. Funding decisions are more favorable if you can demonstrate clearly how and why they will succeed. Logic models help you do just that.<\/p>\n
Most grantees know what they want to do in their communities; Exercise 1 makes a sound case for how and why funders should invest in your program.<\/p>\n
It is crucial to begin program design with the basics. Funders encourage grantees to start by clearly and succinctly explaining the problems they plan to address. Completing Exercise 1 describes the issues your program will address, identifies the needs and assets of your community that are related to your issues, and specifies why certain results are desired. Funders and donors generally limit their investments to certain areas of interest, so if your program addresses several issues, you may want to construct a logic model for each one.<\/p>\n
Exercise 3 Uses The Theory-of-Change Template<\/p>\n
Logic Model Development<\/p>\n
Program Planning Template – Exercise 1<\/p>\n
Page 29<\/p>\n
INSTRUCTIONS: Exercise 3 will use the Theory-of-Change Template. In particular, you will use the information presented in the gray text boxes that follow about the Mytown example program to determine what theory-of-change was used to design and develop this program. Example information about influential factors, the problem, community needs\/assets, strategies, and assumptions are provided. You will fill in the blank Theory-of-Change Template provided in the Forms Index (p. 57) to illustrate the program theory for the Mytown example. You can then look at the completed template on page 34 to compare your interpretation with that produced by the Mytown folks.<\/p>\n
What problems are you attempting to solve or what issues are you striving to address? A well-constructed program theory points toward your program\u2019s eventual effectiveness. Begin your problem statement explaining concisely the issue you will address, stating the issue either as a community problem or asset. Your theory-of-change logic model will be built upon this statement, which illustrates how the program will function and what it expects to achieve in your community. It is smart to refer to research about your program\u2019s problem or issue in your statement; Internet searches can provide other successful program or \u201cbest practice\u201d information.<\/p>\n
PROBLEM STATEMENT EXAMPLE: There are increasing numbers of uninsured male workers, aged 40-55, in Mytown, USA, due to local plant closings. As the bottom line of hospitals shrink, the costs of uninsured care in local emergency rooms are negatively affecting local health systems. To meet the human and financial needs of Mytown, USA, an accessible, free medical home must be created to offer medical care and health education for Mytown\u2019s uninsured residents.<\/p>\n
Insert Mytown\u2019s Problem or Issue in the Problem or Issue box of the Theory-of-Change Template<\/p>\n
What needs or assets led you to address this issue? If a community needs assessment has been conducted or if you have prioritized community needs and capacity, data exist that make your case stronger and more specific by identifying and targeting your program\u2019s participants and activities. Documentation of community needs and assets also helps your evaluation plan later on. It can become a baseline providing indicators that measure progress made by your program over time. (Discussed in more detail in Chapter 4.)<\/p>\n
DOCUMENTED NEEDS\/ASSETS EXAMPLE: Memorial Hospital\u2019s Annual Report states that 28% of uninsured male patients, aged 40-55, received emergency room care in the previous year. Last year\u2019s United Way Community Needs Assessment identified health care for the uninsured as the #1 community health care issue. The Medical Society and Memorial Hospital\u2019s Task Force on the Uninsured is researching ways to address the needs of the uninsured AND reduce costly, inappropriate ER use.<\/p>\n
Insert Mytown\u2019s community needs\/assets in the Community Needs\/Assets box of the Theory-of-Change Template.<\/p>\n
Page 30<\/p>\n
Chapter 3<\/p>\n
What are your desired results? Identify what you expect your program to achieve in the near and longer term. These become your outputs, outcomes and impact.<\/p>\n
DESIRED RESULTS EXAMPLE: Increase accessible, affordable health care for the uninsured and reduce the incidence of un-reimbursed care provided in emergency rooms. Create a free clinic that combines an appropriate, accessible, free medical home and patient education to reduce the numbers of uninsured males, aged 40-55, seeking care in emergency rooms. Anticipate a 15% increase in males, aged 40-55, with a free medical home and a 25% decrease in the incidence of uninsured men seeking care in the ER within 5 years.<\/p>\n
Insert Mytown\u2019s desired results (notice these are targeting men, which is more specific than in the basic logic model example) in Desired Results box of the Theory-of-Change Template.<\/p>\n
What influential factors (protective and\/or risk) could influence change in your community? What are the potential barriers and\/or supports that might impact the change you hope for? Are there policies or other factors that could affect your program?<\/p>\n
INFLUENTIAL FACTOR EXAMPLE: There is documented need for a free clinic. In its Report for the New Millennium, the Mytown Chamber of Commerce projects a 35% increase in the number of small businesses unable to afford employee health care benefits over the next five years. There is strong community support for a free clinic. At the request of Mytown United Way, Memorial Hospital and The Medical Society have created a joint task force to explore the creation of a free clinic.<\/p>\n
Insert Mytown\u2019s influential factors in the Influential Factors box of the Theory-of-Change Template.<\/p>\n
Page 31<\/p>\n
Program Planning \u2014 Clarifying Program Theory Flowchart for Exercise 3<\/p>\n
(1) Problem or Issue Statement<\/p>\n
Describe the problem(s) your program is attempting to solve or the issue(s) your program will address.<\/p>\n
(2) Community Needs\/Assets<\/p>\n
Specify the needs and\/or assets of your community that led your program to address the problem(s) or issue(s).<\/p>\n
(3) Desired Results (Outputs, Outcomes, and Impacts)<\/p>\n
Identify your desired results, or vision of the future, by describing what you expect to achieve, near- or long-term, if your program is funded.<\/p>\n
(4) Influential Factors<\/p>\n
List the factors (e.g., protective or risk factors, existing policy environment, or other factors) you believe will influence change in your community.<\/p>\n
(5) Strategies<\/p>\n
List general, successful strategies or \u201cbest practices\u201d your research identified that have helped communities like yours achieve the kinds of results your program promises.<\/p>\n
(6) Assumptions<\/p>\n
State the assumptions behind how and why the identified change strategies will work in your community (e.g., principles, beliefs, ideas).<\/p>\n
For more detail, see the Program Planning Template \u2014 Exercise 3 on p. 34.<\/p>\n
Page 32<\/p>\n
Chapter 3<\/p>\n
Why do you believe your program will work. Look for strong rationale based on \u201cbest practice\u201d research that connects what you plan to do with why your approach will succeed. Funders are eager for evidence that supports why you propose the solutions you do. It\u2019s a good idea to relate your approach to similar change strategies that have proven effective in communities like yours. Reviewing literature and past evaluation reports from other programs (or your own work) will provide you with ample information to construct your program rationale. The Internet makes it easier to research effective program strategies.<\/p>\n
PROGRAM STRATEGY EXAMPLE: A clinic using volunteer medical professionals reduced emergency room care visits in Another town, USA, in 1997 by 25%. A free clinic in Mytown, USA, using volunteer medical professionals could provide crucial, affordable medical homes for growing numbers of uninsured residents preventing costly, inappropriate emergency room use by males 40-55 experiencing coronary emergencies.<\/p>\n
Insert Mytown\u2019s strategies in the Strategies box of the Theory-of-Change Template.<\/p>\n
Why will your approach be effective. After you make the case for selecting a specific strategy from among the alternatives you researched, state out loud why this strategy is needed and why it will work in your community. It is important early on to document instances that describe the general condition of public reaction to your problem\/issue and possible solutions.<\/p>\n
You should draw direct conclusions about the statement of need and capacities in your community in your assumption. In addition, it should be quite apparent how your program intends to function as an intervention \u2014 to solve identified problems or build existing assets.<\/p>\n
We list assumptions last in this exercise because in this abstracted learning format, the logic modeler has the benefit of all the information that supports assumptions. They are easier to spot and articulate with all the facts in front of you. In real-world conditions, assumption are best stated up-front \u2014 much earlier in the logic model development process \u2014 many basic logic models we have seen include a supporting page with the diagram that lists the assumptions that belie the model drawn.<\/p>\n
ASSUMPTION EXAMPLE: As proven in Anothertown, access to affordable medical care reduces the incidence of emergency visits by providing appropriate, preventive primary care. A free medical clinic should prove successful in Mytown, because of its history of extraordinary volunteerism. Mytown\u2019s Medical Society officially encourages its 400 medical professional members to volunteer 20 hours each year to help medically underserved community residents. Mytown\u2019s Nursing Association is also interested in collaborating with a free clinic. Memorial Hospital has agreed to assist in planning and funding a free clinic. There is precedence for lending free facilities to medical projects serving those in need. Mytown\u2019s technical college donates space for Mytown\u2019s volunteer dental clinic. Mytown\u2019s Free Clinic will be strongly supported by the people, businesses and institutions of Mytown, USA.<\/p>\n
Insert Mytown\u2019s assumptions in the Assumptions box of the Theory-of-Change Template.<\/p>\n
Page 33<\/p>\n
Exercise 3 Checklist:<\/p>\n
After completing Exercise 3 and constructing your program theory, you can use the following checklist to assess the quality of your draft. It\u2019s helpful if someone other than the model\u2019s creators reviews the first program draft and completes the checklist, too.<\/p>\n
Exercise Three Checklist<\/p>\n
Yes<\/p>\n
Not<\/p>\n
Yet<\/p>\n
Comments Revisions<\/p>\n
1. The problems to be solved\/or issues to be addressed by the planned program are clearly stated<\/p>\n
2. There is a specific, clear connection between the identified community needs\/assets and the problems to be solved (or issues to be addressed).<\/p>\n
3. The breadth of community needs\/assets has been identified by expert\/practitioner wisdom, a needs assessment and\/or asset mapping process.<\/p>\n
4. The desired results\/changes in the community and\/or vision for the future ultimately sought by program developers are specific.<\/p>\n
5. Influential factors have been identified and cited from expert\/practitioner wisdom and\/or a literature review.<\/p>\n
6. Change strategies are identified and cited from expert\/practitioner wisdom and\/or literature review.<\/p>\n
7. The connection among known influential factors and broad change strategies has been identified.<\/p>\n
8. The assumptions held for how and why identified change strategies should work in the community are clear.<\/p>\n
9. There is consensus among stakeholders that the model accurately describes the proposed program and its intended results.<\/p>\n
Page 34<\/p>\n
Logic Model Development Program Planning Template – Exercise 3<\/p>\n
Page 35<\/p>\n
Chapter 4 Using Your Logic Model to Plan for Evaluation<\/p>\n
Thinking through program evaluation questions in terms of the logic model components you have developed can provide the framework for your evaluation plan. Having a framework increases your evaluation\u2019s effectiveness by focusing on questions that have real value for your stakeholders.<\/p>\n
\u2022 Prioritization of where investment in evaluation activities will contribute the most useful information for program stakeholders.<\/p>\n
\u2022 Description of your approach to evaluation.<\/p>\n
There are two exercises in this chapter; Exercise 4 deals with posing evaluation questions and Exercise 5 examines the selection of indicators of progress that link back to the basic logic model or the theory-of-change model depending on the focus of the evaluation and its intended primary audiences.<\/p>\n
Exercise 4 – Posing Evaluation Questions<\/p>\n
The Importance of \u201cProve\u201d and \u201cImprove\u201d Questions<\/p>\n
There are two different types of evaluation questions \u2014 formative help you to improve your program and summative help you prove whether your program worked the way you planned. Both kinds of evaluation questions generate information that determines the extent to which your program has had the success you expected and provide a groundwork for sharing with others the successes and lessons learned from your program.<\/p>\n
Benefits of Formative and Summative Evaluation Questions 3<\/p>\n
3 Adapted from Bond, S.L., Boyd, S. E., & Montgomery, D.L.(1997 Taking Stock: A Practical Guide to Evaluating Your Own Programs, Chapel Hill, NC: Horizon Research, Inc. Available online at http:\/\/www.horizon-research.com.<\/p>\n
Formative Evaluation – Improve<\/p>\n
Summative Evaluation – Prove<\/p>\n
Provides information that helps you improve your program. Generates periodic reports. Information can be shared quickly.<\/p>\n
Generates information that can be used to demonstrate the results of your program to funders and your community.<\/p>\n
Focuses most on program activities, outputs, and short-term outcomes for the purpose of monitoring progress and making mid-course corrections when needed.<\/p>\n
Focuses most on program\u2019s intermediate-term outcomes and impact. Although data may be collected throughout the program, the purpose is to determine the value and worth of a program based on results.<\/p>\n
Helpful in bringing suggestions for improvement to the attention of staff.<\/p>\n
Helpful in describing the quality and effectiveness of your program by documenting its impact on participants and the community.<\/p>\n
Page 36<\/p>\n
Chapter 4<\/p>\n
Looking at Evaluation from Various Vantage Points<\/p>\n
How will you measure your success? What will those investing in your program or your target audience want to know?<\/p>\n
A clear logic model illustrates the purpose and content of your program and makes it easier to develop meaningful evaluation questions from a variety of program vantage points: context, implementation and results (which includes outputs, outcomes, and impact).<\/p>\n
What Parts of Your Program Will Be Evaluated?<\/p>\n
Using a logic model to frame your evaluation questions.<\/p>\n
Remember you can draw upon the basic logic model in Exercises 1 and 2 and the theory-of-change model in Exercise 3. Feasibility studies and needs assessments serve as valuable resources for baseline information on influences and resources collected during program planning.<\/p>\n
Context is how the program functions within the economic, social, and political environment of its community and addresses questions that explore issues of program relationships and capacity. What factors might influence your ability to do the work you have planned? These kinds of evaluation questions can help you explain some of the strengths and weakness of your program as well as the effect of unanticipated and external influences on it.<\/p>\n
Page 37<\/p>\n
Sample CONTEXT QUESTIONS: Can we secure a donated facility? With the low morale created by high unemployment, can we secure the financial and volunteer support we need? How many medical volunteers can we recruit? How many will be needed each evening? How will potential patients find out about the clinic? What kind of medical care will patients need? How can we let possible referral sources know about the clinic and its services? What supplies will we need and how will we solicit suppliers for them? What is it about the free clinic that supports its ability to reduce the numbers of patients seeking care in Memorial Hospital\u2019s ER?<\/p>\n
Implementation assesses the extent to which activities were executed as planned, since a program\u2019s ability to deliver its desired results depends on whether activities result in the quality and quantity of outputs specified. They tell the story of your program in terms of what happened and why.<\/p>\n
SAMPLE IMPLEMENTATION QUESTIONS: What facility was secured? How many patients were seen each night\/month\/year? What organizations most frequently referred patients to the clinic? How did patients find out about the clinic? How many medical volunteers serve each night\/month\/year? What was the value of their services? What was the most common diagnosis? What supplies were donated? How many patients per year did the Clinic see in its first\/second\/third year?<\/p>\n
Outcomes determine the extent to which progress is being made toward the desired changes in individuals, organizations, communities, or systems. Outcome questions seek to document the changes that occur in your community as a result of your program. Usually these questions generate answers about effectiveness of activities in producing changes in magnitude or satisfaction with changes related to the issues central to your program.<\/p>\n
SAMPLE OUTCOME QUESTIONS: How many inappropriate, uninsured patients sought medical care in Memorial\u2019s ER in the Clinic\u2019s first\/second\/third year? Was there a reduction in unfunded ER visits? How did the number of uninsured patients compare to previous years when the clinic was not operating? What was the cost\/visit in the Free Clinic? What is the cost\/visit in Memorial\u2019s ER? How do they compare? What were the cost savings to Memorial Hospital? How satisfied were Clinic patients with the care they received? How satisfied were volunteers with their service to the Clinic?<\/p>\n
Page 38<\/p>\n
The use of program theory as a map for evaluation doesn\u2019t necessarily imply that every step of every possible theory has to be studied. …Choices have to be made in designing an evaluation about which lines of inquiry to pursue. …The theory provides a picture of the whole intellectual landscape so that people can make choices with a full awareness of what they are ignoring as well as what they are choosing to study… Weiss (1998)<\/p>\n
Evaluation<\/p>\n
Chapter 4<\/p>\n
Creating Focus<\/p>\n
Though it is rare, you may find that examining certain components of your program is sufficient to satisfy your information needs. Most often, however, you will systematically develop a series of evaluation questions, as shown in the Flowchart for Evaluation Question Development.<\/p>\n
Flowchart for Evaluation Question Development<\/p>\n
(1) Evaluation Focus Area<\/p>\n
What is going to be evaluated? List those components from your theory and\/or logic model that you think are the most important aspects of your program. These areas will become the focus of your evaluation.<\/p>\n
(2) Audience<\/p>\n
What key audience will have questions about your focus areas? For each focus area you have identified, list the audiences that are likely to be the most interested in that area.<\/p>\n
(3) Question<\/p>\n
What questions will your key audience have about your program? For each focus area and audience that you have identified, list the questions they might have about your program.<\/p>\n
(4) Information Use<\/p>\n
If you answer a given question, what will that information be used for? For each audience and question you have identified, list the ways and extent to which you plan to make use of the evaluation information.<\/p>\n
For more detail, see the Evaluation Planning Template \u2014 Exercise 4 on p. 44.<\/p>\n
Page 39<\/p>\n
What is going to be evaluated?<\/p>\n
For each area on which your program focuses, list the most important aspects of your program theory and logic model. Focus your evaluation on them.<\/p>\n
Focus Area Examples:<\/p>\n
Context Examples \u2014 Evaluating relationships and capacity. How will the Free Clinic recruit and train effective board and staff members? What is the best way to recruit, manage, retain and recognize medical and administrative volunteers and other Clinic partners? What is the most effective way to recruit and retain uninsured patients? How will the operation of a Free Clinic affect Memorial Hospital\u2019s expenses for providing uninsured medical care in its ER? How many patients can Clinic volunteers effectively serve on a regular basis? What is the ideal patient\/volunteer ratio?<\/p>\n
Insert focus areas into Focus Area Column of Evaluation Questions Development Template for Evaluation Planning, Exercise 4.<\/p>\n
Implementation Examples – Assessing quality and quantity. How many major funding partners does the clinic have? How are volunteers and patients scheduled? How many medical volunteers serve Clinic patients on a regular basis? What is the value of their services? What is the most common diagnosis at the Clinic? What is the most common diagnosis of uninsured patients seen in Memorial\u2019s ER? How long do patients wait to be seen at the Clinic? Is there a patient or volunteer waiting list?<\/p>\n
Insert focus areas into Focus Area Column of Evaluation Questions Development Template for Evaluation Planning, Exercise 4.<\/p>\n
Outcomes – Measuring effectiveness, magnitude and satisfaction. Has the clinic increased access to care for a significant number of Mytown\u2019s uninsured citizens? How many residents of Mytown, USA, do not have health insurance? How many patients does the Clinic serve on a regular basis? What is that ratio? What is the cost per visit in the Clinic and Memorial\u2019s ER? How do the costs compare? What is the satisfaction level of Clinic patients and volunteers with Clinic services and facilities? How many donors does the Clinic have? What is their satisfaction with Clinic services and facilities? How effectively is the Clinic educating, engaging and involving its partners? What organizations have officially endorsed the Clinic? What is the board and staff\u2019s satisfaction with clinic operations, facilities and services?<\/p>\n
Insert focus areas into Focus Area Column of Evaluation Questions Development Template for Evaluation Planning, Exercise 4.<\/p>\n
The benefits of asking and answering evaluation questions depend on how clear you are about the purpose of your evaluation, who needs to know what when, and the resources you have available to support the evaluation process.<\/p>\n
Page 40<\/p>\n
Chapter 4<\/p>\n
What Information Will Your Program\u2019s Audiences Want?<\/p>\n
As shown below, program audiences will be interested in a variety of different kinds of information. Donors may want to know if their money did what you promised it would. Patients might want to know how many patients the clinic serves and how many volunteers it has. Physicians donating their time and talent could be interested in the financial value of their contributions. If you ask your audiences what they want to know, you\u2019ll be sure to build in ways to gather the evaluation data required.<\/p>\n
Audience<\/p>\n
Typical Questions<\/p>\n
Evaluation Use<\/p>\n
Program Management and Staff<\/p>\n
Are we reaching our target population? Are our participants satisfied with our program? Is the program being run efficiently? How can we improve our program?<\/p>\n
Programming decisions, day-to-day operations<\/p>\n
Participants<\/p>\n
Programming decisions, day-to-day operations Did the program help me and people like me? What would improve the program next time?<\/p>\n
Decisions about continuing participation.<\/p>\n
Community Members<\/p>\n
Is the program suited to our community needs?<\/p>\n
What is the program really accomplishing?<\/p>\n
Decisions about participation and support.<\/p>\n
Public Officials<\/p>\n
Who is the program serving? What difference has the program made? Is the program reaching its target population? What do participants think about the program? Is the program worth the cost?<\/p>\n
Decisions about commitment and support. Knowledge about the utility and feasibility of the program approach.<\/p>\n
Funders<\/p>\n
Is what was promised being achieved?<\/p>\n
Is the program working?<\/p>\n
Is the program worth the cost?<\/p>\n
Accountability and improvement of future grantmaking efforts.<\/p>\n
How often do you have to gather data? Whether a question is more formative or summa-tive in nature offers a clue on when information should be collected.<\/p>\n
\u2022 Formative information should be periodic and reported\/shared quickly to improve your efforts.<\/p>\n
\u2022 Summative tends to be \u201cbefore and after\u201d snapshots reported after the conclusion of the program to document the effectiveness and lessons learned from your experience.<\/p>\n
Page 41<\/p>\n
Involve Your Audience in Setting Priorities<\/p>\n
Program developers often interview program funders, participants, staff, board and partners to brainstorm a list of all possible questions for a key area identified from their program theory or from their logic models. That list helps determine the focus of the evaluation. Involving your audience from the beginning makes sure you gather meaningful information in which your supporters have a real interest.<\/p>\n
Prioritization is a critical step. No evaluation can answer all of the questions your program\u2019s audiences may ask. The following questions can help you narrow your number of indicators: How many audiences are interested in this information? Could knowing the answer to this question improve your program? Will this information assess your program\u2019s effectiveness?<\/p>\n
The final focus for your evaluation is often negotiated among stakeholders. It is important to keep your evaluation manageable. It is preferable to answer a few important questions thoroughly than to answer several questions poorly. How well you can answer your questions will depend on the time, money, and expertise you have at your disposal to perform the functions required by the evaluation.<\/p>\n
What key audiences will have questions about your evaluation focus areas?<\/p>\n
For each focus area that you identified in the previous step, list the audiences that are likely to be most interested in that area. Summarize your audiences and transfer to the Audience Column of the Evaluation Questions Development Template for Evaluation Planning, Exercise 4.<\/p>\n
Context \u2014 Relationships and Capacity<\/p>\n
Example audiences: Medical professionals, Memorial Hospital Board and Staff (especially ER staff), Medical associations, Foundations, The Chamber of Commerce, United Way, The Technical College, uninsured residents, medical supply companies, local media, public officials.<\/p>\n
Implementation \u2014 Quality and Quantity<\/p>\n
Example audiences: Funders, In-kind donors, Medical and administrative volunteers, Board, Staff, Patients, Public Officials, The media, Medical associations, Local businesses, Health care organizations.<\/p>\n
Outcomes \u2014 Effectiveness, Magnitude, and Satisfaction<\/p>\n
Example audiences: Funders, In-kind donors, Volunteers, Board, Staff, Patients, Public Officials, The media, Medical associations, Local businesses, Health care organizations.<\/p>\n
Page 42<\/p>\n
Chapter 4<\/p>\n
What questions will key audiences ask about your program?<\/p>\n
For each focus area and key audience you identified in the previous step, list the questions your stakeholders ask about your program. Insert summaries in the Question Column of the Evaluation Questions Development Template for Evaluation Planning, Exercise 4 (on page 44).<\/p>\n
Sample of Key Audience Questions:<\/p>\n
\u2022 Who are the collaborative partners for this program? What do they provide?<\/p>\n
\u2022 What is the budget for this program?<\/p>\n
\u2022 How many staff members does the program have?<\/p>\n
\u2022 How many patients does the clinic serve?<\/p>\n
\u2022 How many visits per year does the average patient have?<\/p>\n
\u2022 What is the most common diagnosis?<\/p>\n
\u2022 Does the clinic save the hospital money?<\/p>\n
\u2022 How does the organization undertake and support program evaluation?<\/p>\n
\u2022 How are medical volunteers protected from lawsuits?<\/p>\n
\u2022 How satisfied are patients, volunteers, board and staff with the clinic\u2019s services?<\/p>\n
\u2022 What do experts say about the clinic?<\/p>\n
\u2022 How many uninsured patients still seek inappropriate care in the ER? Why?<\/p>\n
How will the evaluation\u2019s information be used?<\/p>\n
For each question and audience you identified in the previous step, list the ways and extent to which you plan to make use of the evaluation information. Summarize audience use of information. Insert in the Use Column of the Evaluation Questions Development Template for Evaluation Planning, Exercise 4.<\/p>\n
Context – Relationships and Capacity Examples<\/p>\n
\u2022 Measure the level of community support.<\/p>\n
\u2022 Assess effectiveness of community outreach.<\/p>\n
\u2022 Assess sustainability of Clinic funding sources.<\/p>\n
\u2022 Improve volunteer and patient recruitment methods.<\/p>\n
\u2022 Secure additional Clinic partners.<\/p>\n
Implementation – Quality and Quantity Examples<\/p>\n
\u2022 Assess optimal number of volunteers and patients to schedule per session to improve operating effectiveness while maintaining patient and volunteer satisfaction.<\/p>\n
\u2022 Measure patient, volunteer, staff, board, donors and community satisfaction with clinic.<\/p>\n
\u2022 Determine cost savings per visit. Share information with local medical and business groups to encourage their support.<\/p>\n
Page 43<\/p>\n
Outcomes and Impact – Examples of Effectiveness, Magnitude, and Satisfaction<\/p>\n
\u2022 Cost savings of Clinic \u2014 use to obtain additional volunteer and financial support from Memorial Hospital.<\/p>\n
\u2022 Patient satisfaction survey results \u2014 use to improve patient services and satisfaction.<\/p>\n
\u2022 Analysis of most frequent referral sources \u2014 use to present information seminars to ER staff, social service workers and unemployment insurance clerks to increase patient referrals and intakes.<\/p>\n
\u2022 Analysis of most prevalent patient diagnoses \u2014 use to create relevant patient health education newsletter. Patient tracking system will measure impact of education program.<\/p>\n
Exercise 4 Checklist: After completing Exercise 4 you can use the following checklist to assess the quality of your draft.<\/p>\n
Posing Questions Quality Criteria<\/p>\n
Yes<\/p>\n
Not Yet<\/p>\n
Comments Revisions<\/p>\n
1. A variety of audiences are taken into consideration when specifying questions.<\/p>\n
2. Questions selected are those with the highest priority.<\/p>\n
3. Each question chosen gathers useful information.<\/p>\n
4. Each question asks only one question (i.e. \u201cextent of X, Y, and Z\u201d is not appropriate).<\/p>\n
5. It is clear how the question relates to the program\u2019s logic model.<\/p>\n
6. The questions are specific about what information is needed.<\/p>\n
7. Questions capture lessons learned about your work along the way.<\/p>\n
8. Questions capture lessons learned about your program theory along the way.<\/p>\n
Page 44<\/p>\n
Logic Model Development Evaluation Planning Template \u2014 Exercise 4<\/p>\n
Evaluation Focus Area<\/p>\n
Audience<\/p>\n
Question<\/p>\n
Use<\/p>\n
Relationships<\/p>\n
Funders<\/p>\n
Is the program cost effective?<\/p>\n
Cost benefits\/fundraising<\/p>\n
Are volunteers & patients satisfied with Clinic services?<\/p>\n
Program promotion\/fundraising<\/p>\n
Medical Volunteers<\/p>\n
What is the most common diagnosis?<\/p>\n
Quality assurance\/Planning<\/p>\n
How will medical volunteers be protected from lawsuits?<\/p>\n
Volunteer recruitment<\/p>\n
Patients<\/p>\n
Am I receiving quality care?<\/p>\n
Program improvement & planning<\/p>\n
How long can I receive care here?<\/p>\n
Staff<\/p>\n
Are we reaching our target population?<\/p>\n
Evaluation\/program promotion<\/p>\n
How do patients find us? What\u2019s our best promotional approach?<\/p>\n
Evaluation and\/or improvement<\/p>\n
Funders\/Donors<\/p>\n
Program Budget?<\/p>\n
Cost benefit analysis<\/p>\n
Outcomes<\/p>\n
Cost\/visit?<\/p>\n
Volunteers<\/p>\n
Visits\/month\/year?<\/p>\n
Annual Report\/program promotion\/public relations<\/p>\n
Cost savings for Memorial Hospital?<\/p>\n
Annual Report\/program promotion\/fundraising<\/p>\n
Patients<\/p>\n
Volunteers\/year?<\/p>\n
Annual Report\/volunteer recruitment<\/p>\n
Patient satisfaction<\/p>\n
Program improvements\/staff training<\/p>\n
Staff<\/p>\n
Patient & volunteer satisfaction<\/p>\n
Common DRG(?)<\/p>\n
Page 45<\/p>\n
The biggest problem is usually that people are trying to accomplish too many results. Once they engage in a discussion of indicators, they start to realize how much more clarity they need in their activities.<\/p>\n
I also find that it is important that the program, not the evaluator, is identifying the indicators. Otherwise, the program can easily discredit the evaluation by saying they don\u2019t think the indicators are important, valid, etc.<\/p>\n
Beverly Anderson Parsons, WKKF Cluster Evaluator<\/p>\n
Exercise 5 \u2014 Establishing Indicators<\/p>\n
One of the biggest challenges in developing an evaluation plan is choosing what kind of information best answers the questions you have posed. It is important to have general agreement across your audiences on what success will look like. Indicators are the measures you select as markers of your success.<\/p>\n
In this last exercise you create a set of indicators. They are often used as the starting point for designing the data collection and reporting strategies (e.g., the number of uninsured adults nationally, statewide, in Mytown, USA, or the number of licensed physicians in Mytown). Often organizations hire consultants or seek guidance from local experts to conduct their evaluations. Whether or not you want help will depend on your organization\u2019s level of comfort with evaluation and the evaluation expertise among your staff.<\/p>\n
Examples and Use of Indicators.<\/p>\n
Focus Area<\/p>\n
Indicators<\/p>\n
How to Evaluate1<\/p>\n
Influential Factors<\/p>\n
Measures of influential factors – may require general population surveys and\/or comparison with national data sets2.<\/p>\n
Compare the nature and extent of influences before (baseline) and after the program.<\/p>\n
Resources<\/p>\n
Logs or reports of financial\/staffing status.<\/p>\n
Compare actual resources acquired against anticipated.<\/p>\n
Activities<\/p>\n
Descriptions of planned activities. Logs or reports of actual activities. Descriptions of participants.<\/p>\n
Compare actual activities provided, types of participants reached against what was proposed.<\/p>\n
Outputs<\/p>\n
Logs or reports of actual activities. Actual products delivered.<\/p>\n
Compare the quality and quantity of actual delivery against expected.<\/p>\n
Outcomes & Impacts<\/p>\n
Participant attitudes, knowledge, skills, intentions, and\/or behaviors thought to result from your activities3.<\/p>\n
Compare the measures before and after the program4.<\/p>\n
1 This table was adapted from A Hands-on Guide to Planning and Evaluation (1993) available from the National AIDS Clearinghouse, Canada.<\/p>\n
2 You may want to allocate resources to allow for the assistance of an external evaluation consultant to access national databases or perform statistical analyses.<\/p>\n
3 Many types of outcomes and impact instruments (i.e. reliable and valid surveys and questionnaires) are readily available. The Mental Measurement Yearbook published by the Buros Institute (http:\/\/www.unl.edu\/buros\/) and the ERIC Clearinghouse on Assessment and Evaluation (http:\/\/ericae.net\/) are great places to start.<\/p>\n
4 You may need to allocate resources to allow for the assistance of an external evaluation consultant.<\/p>\n
Page 46<\/p>\n
Chapter 4<\/p>\n
Our advice is to keep your evaluation simple and straightforward. The logic model techniques you have been practicing will take you a long way toward developing an evaluation plan that is meaningful and manageable.<\/p>\n
Determine the kinds of data you will need and design methods to gather the data (i.e., patient registration forms, volunteer registration forms, daily sign-in sheets, national, state and local statistics). Sometimes, once an indicator (type of data) is selected, program planners set a specific target to be reached as an agreed upon measure of success (for example 25% decrease in the numbers of inappropriate ER visits).<\/p>\n
As in the previous exercises use the space below to loosely organize your thoughts. Then, once the exercise is completed and assessed, use the Indicator Development Template on page 61 to record your indicators and technical assistance needs.<\/p>\n
Filling in the Flowchart for Indicator Development<\/p>\n
What information will be gathered to \u201cindicate\u201d the status of your program and\/or its participants?<\/p>\n
Column 1: Focus Areas \u2014 From the information gathered in Exercise 4, transfer the areas on which your evaluation will focus into column one (for example, patient health, volunteer participation, sustaining supporting partnerships).<\/p>\n
Column 2: Questions \u2014 transfer from Exercise 4 the major questions related to each focus area \u2014 big questions your key audiences want answered. Remember to keep your evaluation as simple as possible.<\/p>\n
Column 3: Indicators \u2014 Specify the indicators (types of data) against which you will measure the success\/progress of your program. It\u2019s often helpful to record the sources of data you plan to use as indicators (where you are likely to find or get access to these data).<\/p>\n
Column 4: Technical Assistance \u2014 To what extent does your organization have the evaluation and data management expertise needed to collect and analyze the data that related to each indicator? List any assistance that would be helpful \u2014 universities, consultants, national and state data experts, foundation evaluation departments, etc.<\/p>\n
Page 47<\/p>\n
Exercise 5 Checklist: <\/p>\n
Review what you have created using the checklist below to assess the quality of your evaluation plan.<\/p>\n
Establishing Indicators Quality Criteria<\/p>\n
Yes<\/p>\n
Not<\/p>\n
Yet<\/p>\n
Comments Revisions<\/p>\n
1. The focus areas reflect the questions asked by a variety of audiences. Indicators respond to the identified focus areas and questions.<\/p>\n
2. Indicators are SMART-Specific, Measurable, Action-oriented, Realistic, and Timed.<\/p>\n
3. The cost of collecting data on the indicators is within the evaluation budget.<\/p>\n
4. Source of data is known.<\/p>\n
5. It is clear what data collection, management, and analysis strategies will be most appropriate for each indicator.<\/p>\n
6. Strategies and required technical assistance have been identified and are within the evaluation budget for the program.<\/p>\n
7. The technical assistance needed is available.<\/p>\n
Page 48<\/p>\n
Logic Model Development<\/p>\n
Indicators Development Template – Exercise 5<\/p>\n
Focus Area<\/p>\n
Question<\/p>\n
Indicators<\/p>\n
Technical Assistance Needed<\/p>\n
Relationships<\/p>\n
Are volunteers & patients satisfied w\/ clinic care?<\/p>\n
\u2022 Patient satisfaction surveys<\/p>\n
\u2022 Volunteer satisfaction tests<\/p>\n
Anywhere\u2019s patient satisfaction surveys Anywhere\u2019s volunteer survey<\/p>\n
Outcomes<\/p>\n
Are we reaching our target population?<\/p>\n
\u2022 % of clinic patients vs. % of uninsured citizens in Mytown, USA<\/p>\n
\u2022# of qualified clinic patients\/year<\/p>\n
Reports from Chamber of Commerce Patient database creation<\/p>\n
How do patients find the clinic?<\/p>\n
\u2022 Annual analysis of telephone referral log<\/p>\n
\u2022 Referral question on patient intake form<\/p>\n
Telephone log database Anywhere\u2019s patient intake form<\/p>\n
Does the clinic save the community $?<\/p>\n
\u2022 Cost\/visit<\/p>\n
\u2022 # of uninsured patients seen in hospital ER -beginning the year before clinic opened<\/p>\n
Budget figures; patient service records<\/p>\n
Tracking database software Strategic direction for analysis<\/p>\n
What does the clinic provide?<\/p>\n
\u2022 Most common diagnosis<\/p>\n
\u2022 Hospital cost\/visit for common diagnosis<\/p>\n
DRG workbook\/tables (hospital staff) Input from hospital billing staff<\/p>\n
How has volunteering affected doctors, nurses, administrators and patients?<\/p>\n
\u2022 Annual volunteer survey<\/p>\n
\u2022 Patient satisfaction survey<\/p>\n
\u2022 # of volunteers\/year<\/p>\n
\u2022 # of volunteers donating to clinic operations<\/p>\n
Anywhere surveys and analysis instruments<\/p>\n
Volunteer management database Donor database (Raiser\u2019s Edge?)<\/p>\n
Page 49<\/p>\n
Resources Appendix<\/p>\n
This Appendix provides information on print and electronic resources available to support you in your logic model development process.<\/p>\n
1. Logic Model Information and Examples<\/p>\n
University of Nevada, Reno Western CAPT web site http:\/\/www.unr.edu\/colleges\/educ\/captta\/prev\/evaluate.htm<\/p>\n
BJA Evaluation web site http:\/\/www.bja.evaluationwebsite.org<\/p>\n
Schmitz, C. C. & Parsons, B. A. (1999). Everything you wanted to know about logic models but were afraid to ask. Battle Creek, MI: W.K. Kellogg Foundation.<\/p>\n
2. United Way of America\u2019s Outcome Models<\/p>\n
United Way of America web site http:\/\/www.unitedway.org\/outcomes\/contents.htm<\/p>\n
Measuring program outcomes: A practical approach. United Way of America 701 North Fairfax Street Alexandria, VA 22314 (703) 836-7100<\/p>\n
3. Definitions and Information on Program Theory and Evaluation<\/p>\n
Program Theory Definitions<\/p>\n
\u2022 A plausible and sensible model of how a program is supposed to work (Bickman, 1987, p. 5).<\/p>\n
\u2022 The set of assumptions about the relationships between the strategy and tactics the program has adopted and the social benefits it is expected to produce (Rossi, Freeman, & Lipsey, 1999, p.98).<\/p>\n
\u2022 The full chain of objectives that links inputs to activities, activities to … outputs, … outputs to … outcomes, and… outcomes to ultimate goals constitutes a program\u2019s theory<\/p>\n
(Patton, 1997, p. 218).<\/p>\n
\u2022 A set of interrelated assumptions, principles, and\/or propositions to explain or guide social actions (Chen, 1990, p. 40).<\/p>\n
\u2022 An explanation of the causal links that tie program inputs to expected program outputs (Weiss, 1998, p. 55).<\/p>\n
\u2022 A chain of causal assumptions linking program resources, activities, intermediate outcomes, and ultimate goals (Wholey, 1987, p. 78).<\/p>\n
Page 50<\/p>\n
Resources Appendix<\/p>\n
Bickman, L. (Ed.). (1987). Using program theory in evaluation. New Directions for Program Evaluation Series (no. 33). San Francisco: Jossey-Bass.<\/p>\n
Chen, H. T. (1990). Theory driven evaluations. Newbury Park, CA: Sage.<\/p>\n
Wholey, J. S. (Ed). (1987). Organizational excellence: Stimulating quality and communicating value. Lexington, MA: Lexington Books.<\/p>\n
Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (Eds.). (1994). Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass Publishers.<\/p>\n
Weiss, C. H. (1998). Evaluation: Methods for studying programs and policies. (2nd Ed). Upper Saddle River, NJ: Prentice Hall.<\/p>\n
4. W.K. Kellogg Foundation Logic Model Examples<\/p>\n
W.K. Kellogg Foundation Web site http:\/\/www.wkkf.org<\/p>\n
W.K. Kellogg Foundation Evaluation Handbook.<\/p>\n
Parsons, B. A. (1999). Making logic models more systemic. A paper presented at the Annual Meeting of the American Evaluation Association, Orlando, FL, November 1999.<\/p>\n
Parsons, B. A., Schmitz, Co (1999) Everything You Wanted to Know About Logic Models But Were Afraid to Ask. A paper presented at the Annual Meeting of the American Evaluation Association, Orlando, FL, November 1999.<\/p>\n
5. Information about Logic Model Development and Use<\/p>\n
The Evaluation Forum (1999). Outcomes for success. The Evaluation Forum 1932 First Avenue, Suite 403<\/p>\n
Seattle, WA 98101<\/p>\n
(206) 269-0171<\/p>\n
Page 51<\/p>\n
Freddolino, P. P. (1999). The program logic model: What it is and how to teach it. A preconference workshop presented at the 1999 Conference of the Michigan Association for Evaluation, East Lansing, MI, May.<\/p>\n
Targeting Outcomes of Programs. http:\/\/deal.unl.edu\/TOP\/synopsis.htm<\/p>\n
Innovation Network, Inc. electronic logic model development tool http:\/\/www.inetwork.org<\/p>\n
6. Evaluation Planning Information<\/p>\n
W.K. Kellogg Foundation Web site http:\/\/www.wkkf.org<\/p>\n
W.K. Kellogg Foundation Evaluation Handbook. Taking Stock.<\/p>\n
http:\/\/www.horizon-research.org<\/p>\n
The Evaluation Forum (1994). A field guide to outcome-based program evaluation.<\/p>\n
The Evaluation Forum 1932 First Avenue, Suite 403 Seattle, WA 98101 (206) 269-0171<\/p>\n
Rossi, P. H., Freeman, H. E., & Lipsey, M. W. (1999). Evaluation: A systematic approach. Thousand Oaks, CA: Sage.<\/p>\n
Patton, M. Q. (1997). Utilization-focused evaluation: The new century text. Thousand Oaks, CA: Sage.<\/p>\n
Page 52<\/p>\n
This page was intentionally left blank.<\/p>\n
Page 53<\/p>\n
Forms Appendix<\/p>\n
This Appendix provides the worksheet templates and checklists for exercises 1-5:<\/p>\n
Logic Model Development Program Planning and Implementation<\/p>\n
Exercises 1 and 2 Template Exercise 1 Checklist Exercise 2 Checklist<\/p>\n
Theory-of-Change Logic Model Development Planning<\/p>\n
Exercise 3 Template Exercise 3 Checklist<\/p>\n
Logic Model Development Evaluation and Indicators Development<\/p>\n
Exercise 4 Template Exercise 4 Checklist Exercise 5<\/p>\n
Exercise 5 Checklist<\/p>\n
Copyright \u00a9 1998 by the W.K. Kellogg Foundation<\/p>\n
All rights reserved. This publication may be freely utilized by the general public for educational, charitable, nonprofit, and\/or other non-commercial purposes. Such use does not require prior or written permission. Such use should include acknowledgment that it is \u201cwith the permission of the W.K. Kellogg Foundation\u201d. Any other use is prohibited without the prior written permission of the W.K. Kellogg Foundation. For permission requests, please contact the Communications Department at communications@wkkf.org or 269-969-2079.<\/p>\n
Page 54<\/p>\n
Forms Appendix<\/p>\n
Logic Model Development<\/p>\n
Program Implementation Template \u2014 Exercise 1 & 2<\/p>\n
RESOURCES<\/p>\n
ACTIVITIES<\/p>\n
OUTPUTS<\/p>\n
SHORT- & LONG-TERM OUTCOMES<\/p>\n
IMPACT<\/p>\n
In order to accomplish our set of activities we will need the following:<\/p>\n
In order to address our problem or asset we will accomplish the following activities:<\/p>\n
We expect that once accomplished these activities will produce the following evidence or service delivery:<\/p>\n
We expect that if accomplished these activities will lead to the following changes in 1-3 then 4-6 years:<\/p>\n
We expect that if accomplished these activities will lead to the following changes in 7-10 years:<\/p>\n
Page 55<\/p>\n
Exercise 1 Checklist<\/p>\n
Progress Toward Results Quality Criteria – 1<\/p>\n
Yes<\/p>\n
Not Yet<\/p>\n
Comments\/ Revisions<\/p>\n
1. A variety of audiences are taken into consideration when specifying credible outputs, outcomes, and impacts.<\/p>\n
2. Target participants and\/or partners are described and quantified as outputs (e.g. 100 teachers from 5 rural high schools).<\/p>\n
3. Events, products, or services listed are described as outputs in terms of a treatment or dose (e.g. 30 farmers will participate in at least 3 sessions of program, or curriculum will be distributed to at least 12 agencies).<\/p>\n
4. The intensity of the intervention or treatment is appropriate for the type of participant targeted (e.g. higher-risk participants warrant higher intensities).<\/p>\n
5. The duration of the intervention or treatment is appropriate for the type of participant targeted (e.g. higher-risk participants warrant longer duration).<\/p>\n
6. Outcomes reflect reasonable, progressive steps that participants can make toward longer-term results.<\/p>\n
7. Outcomes address awareness, attitudes, perceptions, knowledge, skills, and\/ or behavior of participants.<\/p>\n
8. Outcomes are within the scope of the program\u2019s control or sphere of reasonable influence.<\/p>\n
9. It seems fair or reasonable to hold the program accountable for the outcomes specified.<\/p>\n
10. The outcomes are specific, measurable, action-oriented, realistic, and timed.<\/p>\n
11. The outcomes are written as change statements (e.g. things increase, decrease, or stay the same).<\/p>\n
12. The outcomes are achievable within the funding and reporting periods specified.<\/p>\n
13. The impact, as specified, is not beyond the scope of the program to achieve.<\/p>\n
Page 56<\/p>\n
Forms Appendix<\/p>\n
Exercise 2 Checklist<\/p>\n
Theory into Action Quality Criteria<\/p>\n
Yes<\/p>\n
Not Yet<\/p>\n
Comments\/Revisions<\/p>\n
1. Major activities needed to implement the program are listed.<\/p>\n
2. Activities are clearly connected to the specified program theory.<\/p>\n
3. Major resources needed to implement the program are listed.<\/p>\n
4. Resources match the type of program.<\/p>\n
5. All activities have sufficient and appropriate resources.<\/p>\n
Page 57<\/p>\n
Logic Model Development<\/p>\n
Program Planning Template \u2014 Exercise 3<\/p>\n
Page 58<\/p>\n
Forms Appendix<\/p>\n
Exercise 3 Checklist<\/p>\n
Exercise Three Checklist<\/p>\n
Yes<\/p>\n
Not Yet<\/p>\n
Comments\/ Revisions<\/p>\n
1. The problems to be solved\/or issues to be addressed by the planned program are clearly stated<\/p>\n
2. There is a specific, clear connection between the identified community needs\/assets and the problems to be solved (or issues to be addressed).<\/p>\n
3. The breadth of community needs\/assets has been identified by expert\/practitioner wisdom, a needs assessment and\/or asset mapping process.<\/p>\n
4. The desired results\/changes in the community and\/or vision for the future ultimately sought by program developers are specific.<\/p>\n
5. Influential factors have been identified and cited from expert\/practitioner wisdom and\/or a literature review.<\/p>\n
6. Change strategies are identified and cited from expert\/practitioner wisdom and\/or literature review.<\/p>\n
7. The connection among known influential factors and broad change strategies has been identified.<\/p>\n
8. The assumptions held for how and why identified change strategies should work in the community are clear.<\/p>\n
9. There is consensus among stakeholders that the model accurately describes the proposed program and its intended results.<\/p>\n
Page 59<\/p>\n
Logic Model Development<\/p>\n
Evaluation Planning Template \u2014 Exercise 4<\/p>\n
Evaluation Focus Area<\/p>\n
Audience<\/p>\n
Question<\/p>\n
Use<\/p>\n
Page 60<\/p>\n
Forms Appendix<\/p>\n
Exercise 4 Checklist<\/p>\n
Posing Questions Quality Criteria<\/p>\n
Yes<\/p>\n
Not<\/p>\n
Yet<\/p>\n
Comments\/<\/p>\n
Revisions<\/p>\n
1. A variety of audiences are taken into consideration when specifying questions.<\/p>\n
2. Questions selected are those with the highest priority.<\/p>\n
3. Each question chosen gathers useful information.<\/p>\n
4. Each question asks only one question (i.e. \u201cextent of X, Y, and Z\u201d) is not appropriate).<\/p>\n
5. It is clear how the question relates to the program\u2019s logic model.<\/p>\n
6. The questions are specific about what information is needed.<\/p>\n
7. Questions capture lessons learned about your work along the way.<\/p>\n
8. Questions capture lessons learned about your program theory along the way.<\/p>\n
Page 61<\/p>\n
Logic Model Development<\/p>\n
Indicators Development Template – Exercise 5<\/p>\n
Focus Area<\/p>\n
Question<\/p>\n
Indicators<\/p>\n
Technical Assistance Needed<\/p>\n
Page 62<\/p>\n
Forms Appendix<\/p>\n
Exercise 5 Checklist<\/p>\n
Establishing Indicators Quality Criteria<\/p>\n
Yes<\/p>\n
Not Yet<\/p>\n
Comments\/ Revisions<\/p>\n
1. The focus areas reflect the questions asked by a variety of audiences. Indicators respond to the identified focus areas and questions.<\/p>\n
2. Indicators are SMART-Specific, Measurable, Action-oriented, Realistic, and Timed.<\/p>\n
3. The cost of collecting data on the indicators is within the evaluation budget.<\/p>\n
4. Source of data is known.<\/p>\n
5. It is clear what data collection, management, and analysis strategies will be most appropriate for each indicator.<\/p>\n
6. Strategies and required technical assistance have been identified and are within the evaluation budget for the program.<\/p>\n
7. The technical assistance needed is available.<\/p>\n
This page was intentionally left blank. <\/p>\n
W.K.KELLOGG<\/p>\n
FOUNDATION<\/p>\n
One Michigan Avenue East Battle Creek, MI 49017-4012 USA<\/p>\n
269-968-1611 TDD on site<\/p>\n
Facsimile: 269-968-0413 www.wkkf.org<\/p>\n
EV3815 Item #1209 0305 5M SCG Printed on Recycled Paper<\/p>\n","protected":false},"excerpt":{"rendered":"
W.K. Kellogg Foundation Logic Model Development Guide Using Logic Models to Bring Together Planning, Evaluation, and Action Logic Model Development Guide W.K. KELLOGG foundation To help people help themselves through the practical application of knowledge and resources to improve their quality of life and that of future generations. Updated January 2004 W.K. Kellogg Foundation One […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[10],"class_list":["post-106686","post","type-post","status-publish","format-standard","hentry","category-research-paper-writing","tag-writing"],"_links":{"self":[{"href":"https:\/\/papersspot.com\/blog\/wp-json\/wp\/v2\/posts\/106686","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/papersspot.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/papersspot.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/papersspot.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/papersspot.com\/blog\/wp-json\/wp\/v2\/comments?post=106686"}],"version-history":[{"count":0,"href":"https:\/\/papersspot.com\/blog\/wp-json\/wp\/v2\/posts\/106686\/revisions"}],"wp:attachment":[{"href":"https:\/\/papersspot.com\/blog\/wp-json\/wp\/v2\/media?parent=106686"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/papersspot.com\/blog\/wp-json\/wp\/v2\/categories?post=106686"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/papersspot.com\/blog\/wp-json\/wp\/v2\/tags?post=106686"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}