Qualitative methods quantitative impact evaluation often leaves us with reduced form results sometimes difficult to understand what aspects of the intervention worked and why qualitative methods allow us to delve more deeply into select areas of impact through. This is followed by a survey of impact evaluation studies carried out in key development sectors including education, health, infrastructure, finance, and agriculture, among others. Approaches, tor and methods for impact evaluation interact. Choosing evaluation methods world health organization. Evaluation methods evaluation methods can be either qualitative e. The program stage and scope will determine the level of effort and the methods to be used.
There are a variety of evaluation designs, and the type of evaluation should match the development level of the program or program activity appropriately. Measurement of impact of any developmental programme is a difficult task. To further understand why a randomised evaluation is the most credible, consider two other commonly used impact evaluation methods that suffer from poor counterfactual measures. Evaluation concepts and methods are of growing importance to public health organizations, as well as to education and social services programs. Expert judgment quantitative physical and mathematical models cumulative impact assessment matrices and interaction diagrams rapid impact assessment matrix riam battelle environmental evaluation system. Choosing an impact evaluation method 187 determining which method to use for a given program 187 how a programs rules of operation can help choose an impact evaluation method 188 a comparison of impact evaluation methods 193 finding the smallest feasible unit of intervention 197 chapter 12. This approach to evaluation focuses on the practicalities of defining successful outcomes and success cases brinkerhoff, 2003 and uses some of the processes from theorydriven evaluation to determine the linkages, which may take the form of a logic model, an impact.
This work is increasingly important as the global development community works to reduce poverty and achieve the. Approaches and methods of theory based impact evaluation. How should we choose methods for impact evaluation. This work is increasingly important as the global development community works to reduce poverty and achieve the 2030 sustainable development goals. Pdf on mar 25, 2019, valerie pattyn published towards appropriate impact evaluation methods find, read and cite all the research you need on.
Overview of impact evaluation page 2 an impact evaluation should only be undertaken when its intended use can be clearly identified and when it is likely to be able to produce useful findings, taking into account the availability of resources and the timing of decisions about the programme or policy under investigation. The survey focuses on what types of development interventions have been evaluated. An impact evaluation relies on rigorous methods to determine the changes in outcomes which. Although other kinds of evaluation may cover one or more of the features in the list below, an impact evaluation would normally be expected to cover all of them stern 2015, perrin 2012, rogers 2012, actionaid 2016. Data collection and analysis methods in impact evaluation, methodological briefs. Methods and limitations of evaluation and impact research reinhard hujer, marco caliendo, dubravko radi.
A discussion paper for ausaid practitioners september 2012. The causal effect or impact of a programme p on an outcome. Chapter ii describes the fundamental problem underlying impact evaluation, which poses a substantial challenge to development researchers and practitioners. Third, it presents six wellknown impact evaluations as case studies. It then discusses the merits and limitations of various impact evaluation methods. Quantitative and qualitative methods in impact evaluation gsdrc. The impact evaluation model and the self evaluation form the ofsted self evaluation form sef is an opportunity for schools and their partners to demonstrate the positive impact that workforce reform and extended services are making on the lives on children and young people. Impacts may be positive or negative, intended or unintended, direct or indirect. Impact evaluation methods for youth employment interventions ilo.
Impact evaluation is the main means for empirically testing what actually happens when interventions are implemented. Lini wollenberg, cgiar research program on climate change, agriculture and food security and gund institute, university of vermont lorta program inception workshop, 2426 july 2018 impact evaluation methods. Usaid definition of impact evaluation part 1 impact evaluations measure the change in a development outcome focused on higher level results outcomes rather than inputs and outputs e. While many formative evaluations focus on processes, impact evaluations can be used formatively if an intervention is ongoing. Participants can be difficult to track long term and individual or group factors which draw people to participate may also cause drop out or irregular participation from a project and even more so from a structured study. Sample impact evaluation questions once the purpose and focus of the evaluation are determined, you should identify specific evaluation questions. More information about each of the methods can be found on the website. Impact evaluation a model guidance and practical examples. Quantitative and qualitative methods in impact evaluation. The evaluation questions you choose will guide your selection of an appropriate evaluation design. A primer on geospatial impact evaluation methods, tools, and. Mar 30, 2017 impact evaluation provides the causal effect of a programme on the outcome.
Notes 3 and 4 set out the elements that need to be in place and the decisions to be taken before initiating an evaluation. The four types of analysis that we will discuss in this series are. Impact evaluation is an assessment of how the intervention being evaluated affects. Choosing appropriate designs and methods for impact evaluation. Outline of principles of impact evalation discussion dra. Impact evaluation methods for youth employment interventions 3 in other words, impact evaluations try to asthe attribution challenge sess whether, to what extent and why observed changes in outcomes of interest can be attributed to an intervention or project. This note is a guide to implementing an impact evaluation for youth employment. This approach to evaluation focuses on the practicalities of defining successful outcomes and success cases brinkerhoff, 2003 and uses some of the processes from theorydriven evaluation to determine the linkages, which may take the form of a logic model, an impact model, or a results map. This means that an impact evaluation must establish what has been the cause of observed changes in this case impacts referred to as causal attribution also referred to as causal inference. The following are some sample policy impact evaluation questions. Section 3 outlines the methods of dealing with endogeneity. Impact evaluation methods of arrafunded energy code initiatives michael rovito, ers, new york city, ny tim pettit, dnvgl, washington, d. Impact evaluation in practice is an essential resource for evaluators, social programs, ministries, and others committed to making decisions using good evidence.
Impact assessment methods this document outlines several common impact assessment methods, including. Impact evaluation seeks to determine the longer term results that are generated by policy decisions, often through interventions, projects or programmes. The focus of this note is the socalled counterfactual framework on which quantita. Impact evaluation methods experimental design presented by. The formal literature on impact evaluation methods and practices is large, with a few useful overviews. Information about evaluation methods in guidance note 1 has been drawn.
Formative impact evaluations are undertaken to inform decisions in regard to making changes to a programme or policy. Evaluation is a complex task and involves several steps. Clarifying the terminology 7 4 utilising quantitative and qualitative approaches in impact evaluation 11 4. Impact evaluation 10, unicef office of research, florence. Note 5 is essential prereading to introduce readers to the different options for impact evaluation. The impact evaluation model can support this process by helping. An impact evaluation provides information about the impacts produced by an intervention positive and negative, intended and unintended, direct and indirect.
The limitations of an exclusive reliance on quant or qual evaluation approaches 3 1. Quantitative and qualitative methods in impact evaluation and measuring results i. The meaning of impact evaluation differs depending on how impact. Impact evaluation methods in public economics harvard dash. The designations employed in ilo publications, which are in. It sets ie into the wider setting of evidencebased policy. The benefits of a mixed methods approach 4 part ii. Mar 01, 2019 information about evaluation methods in guidance note 1 has been drawn from betterevaluation, an international collaboration for sharing information to improve evaluation. However, care must be taken about generalizing from a specific context.
This brief benefited from the guidance of many individuals. Note 5 impact evaluation methods for youth employment. Statistical parameters lever of confidence, power, 6. Jul 17, 2015 tive impact evaluation methods, we will start by placing them in the broader results context, which also includes monitoring and other types of evaluation. Spanish version available at facultysupplemental%20files metodosdeevaluacionde. This chapter aims to help readers identify what is distinctive about impact evaluation. Purpose of impact evaluation impact evaluation serves both objectives of evaluation. Choosing appropriate designs and methods for impact. The choice of methods and designs for impact evaluation of policies and.
A primer on geospatial impact evaluation methods, tools. Yet there is a need to put the theory into practice in a handson fashion for practitioners. In parallel to these trends, behavioral economics has increasingly attracted. To establish causality between a programme and an outcome, impact evaluation methods are used. Impact evaluation designs are identified by the type of methods used to generate the counterfactual and can be broadly classified into three categories experimental, quasiexperimental and nonexperimental designs that vary in feasibility, cost, involvement during design or after implementation phase of the intervention, and degree of. Methods and limitations of evaluation and impact research. Module 9 introduction to impact evaluation usaid learning lab. Training evaluation usually includes a combination of these methods and reframes our thinking about evaluation in that measurements are aimed at different levels of a system. This is the second edition of the impact evaluation. Gies are not appropriate for all types of development interventions. Evaluation types when to use what it shows why it is useful. In a new guide on impact evaluation, stern makes this.
It begins with a brief statement of the evaluation problem section 2. In a new guide on impact evaluation, stern makes this point about when a fully. An impact evaluation relies on rigorous methods to determine the changes in outcomes which can be attributed to a specific intervention based on causeandeffect analysis. This is the power of randomisation and is the reason why these evaluations provide the most accurate impact estimates. Oct 02, 20 multiple evaluation questions treatment groups 2. This article introduces geospatial impact evaluation gie methods, which have opened up new opportunities to understand what works, what doesnt, and why at a substantially lower time and financial cost.
Multivariable regression matching instrumental variables regression discontinuity these methods rely on being able to mimic the counterfactual under certain assumptions problem. What is geospatial impact evaluation gie and when can it be used. Quantitative and qualitative methods in impact evaluation and. Impact evaluations need to account for the counterfactual what would have occurred without the intervention through the use of an experimental or quasiexperimental design using comparison and treatment groups. Geospatial impact evaluations gies rely on subnationally georeferenced intervention, outcome, and covariate data and quasiexperimental methods of causal inference to measure the intended or unintended impacts of development programs. A brief introduction to randomized evaluations and comparison with other methods.
Pdf on apr 2, 2017, abhisek mishra and others published methods of impact evaluation. One categorization of impact evaluation methods is the following. Impact evaluation methods in public finance a brief introduction to randomized evaluations and comparison with other methods may 2015 dina pomeranz harvard university and nber1 abstract recent years have seen a large expansion in the use of rigorous impact evaluation techniques. Abstract under the american recovery and reinvestment act arra. Information about evaluation methods in guidance note 1 has been drawn from betterevaluation, an international collaboration for sharing information to improve evaluation.
1546 1248 1754 77 1183 1763 181 721 1698 404 338 1310 833 1452 695 1179 981 1099 587 1101 117 335 438 122 441 1115 1561 463 956 794