This report aims to support impact evaluation in the energy sector by assessing what has been produced against what might be possible. It reviews 85 impact evaluations of energy interventions in developing countries, summarizes findings on outcomes, identifies evidence gaps, and proposes ways forward. A majority of studies has so far focused on rural electrification and improved cookstoves, while most investments are in power generation and transmission. After considering the body of evidence available, directions and approaches are proposed for future impact evaluations to best contribute new evidence regarding the sector.
This report aims to support impact evaluation in the energy sector by assessing what has been produced against what might be possible. It reviews 85 impact evaluations of energy interventions in developing countries, summarizes findings on outcomes, identifies evidence gaps, and proposes ways forward. It notes that impact evaluation has so far focused on rural electrification and improved cookstoves, while most investments are in power generation and transmission. It highlights where future impact evaluation investments can best contribute new evidence and how such studies could be approached.
The second edition of the Impact Evaluation in Practice handbook is a comprehensive and accessible introduction to impact evaluation for policy makers and development practitioners. First published in 2011, it has been used widely across the development and academic communities. The book incorporates real-world examples to present practical guidelines for designing and implementing impact evaluations. Readers will gain an understanding of impact evaluations and the best ways to use them to design evidence-based policies and programs. The updated version covers the newest techniques for evaluating programs and includes state-of-the-art implementation advice, as well as an expanded set of examples and case studies that draw on recent development challenges. It also includes new material on research ethics and partnerships to conduct impact evaluation. The handbook is divided into four sections: Part One discusses what to evaluate and why; Part Two presents the main impact evaluation methods; Part Three addresses how to manage impact evaluations; Part Four reviews impact evaluation sampling and data collection. Case studies illustrate different applications of impact evaluations. The book links to complementary instructional material available online, including an applied case as well as questions and answers. The updated second edition will be a valuable resource for the international development community, universities, and policy makers looking to build better evidence around what works in development.
Impact evaluation is an empirical approach to estimating the causal effects of interventions, in terms of both magnitude and statistical significance. Expanded use of impact evaluation techniques is critical to rigorously derive knowledge from development operations and for development investments and policies to become more evidence-based and effective. To help backstop more use of impact evaluation approaches, this book introduces core concepts, methods, and considerations for planning, designing, managing, and implementing impact evaluation, supplemented by examples. The topics covered range from impact evaluation purposes to basic principles, specific methodologies, and guidance on field implementation. It has materials for a range of audiences, from those who are interested in understanding evidence on "what works" in development, to those who will contribute to expanding the evidence base as applied researchers.
This Energy Efficiency Program Impact Evaluation Guide describes the common terminology, structures, and approaches used for determining (evaluating) energy and demand savings as well as avoided emissions and other non-energy benefits resulting from facility (non-transportation) energy efficiency programs that are implemented by local governments, states, utilities, private companies, and nonprofits. While this guide does not recommend specific approaches, it provides context, planning guidance, and discussion of issues that determine the most appropriate evaluation objectives and best practices approaches for different efficiency portfolios. By using standard evaluation terminology and structures and best practices approaches, evaluations can support the adoption, continuation, and expansion of effective efficiency actions. The primary audiences for this guide are energy regulators; public and private energy efficiency portfolio administrators such as utilities, nonprofit organizations, and government agencies; program implementers; and evaluators looking for guidance on the following: - The evaluation process and approaches for determining program impacts - Planning evaluation efforts - Key issues associated with establishing evaluation frameworks for improving the efficacy of energy efficiency portfolios, documenting the impacts of such portfolios, and comparing demand- and supply-side resources. Introductory portions and appendices are also intended for policymakers seeking general information about efficiency program impact evaluation as well as the basic principles of process and market evaluations and cost-effectiveness analyses. Although the guide is not directly intended for expert evaluation practitioners who can rely on more detailed and specific resources that are referred to in this guide, it offers introductions to and summaries of evaluation topics that can be useful for explaining concepts and standard practices to clients, new staff, stakeholders, and others who could benefit from a refresher on principles
Developed as a product of the State and Local Energy Efficiency Action Network (SEE Action), facilitated by the U.S. Department of Energy/U.S. Environmental Protection Agency.
This publication aims to support impact evaluation in the transport sector by assessing what has been produced against what might be possible. It reviews 91 impact evaluations of transport interventions in developing countries, summarizes findings on outcomes, identifies evidence gaps, and proposes ways forward. Most of the studies reviewed find significant effects on at least one of the outcomes investigated. However, impact evaluation has given relatively little coverage to major areas of investment, such as urban and sustainable transportation, transport corridors, and efficiency enhancing measures. New methods and increasing openness of geospatial data provide scope to generate more innovative impact studies.
Universal household electrification is a key component of the United Nations Sustainable Development Goals, but the evidence base for social and economic impacts of electricity access remains unclear. Here we report results from a systematic review of impact evaluations of household electrification. We only find 31 studies that conduct statistical hypothesis tests to assess impacts. Among these, seven draw on a randomized experiment designed for causal inference. The randomized experimental studies generate fewer positive results than observational or quasi-experimental studies, such as correlational, instrumental variable, and difference-in-differences designs. These results call for a reassessment of what we know about the impacts of household electrification. They also call for major investment in impact evaluation of electricity access using randomized controlled trials, with a particular focus on when and how energy access interventions can furnish large benefits to their intended beneficiaries. Large-scale impact evaluations using experimental methods will require close collaboration between policymakers and researchers.
Impact evaluation aims to answer whether and to what extent a development intervention has delivered its intended effects, thus enabling evidence-based policy making. The desire for more hard evidence of the effectiveness of development interventions has fueled a growing interest in rigorous impact evaluation in the international development community. This report discusses the fundamental challenge of impact evaluation, which is to credibly attribute the impact, if any, to the intervention concerned. It then discusses the merits and limitations of various impact evaluation methods. It also presents a survey of recent applications of impact evaluation, focusing on the typical evaluation problems looked at, methods used, and key findings. The report includes six case studies and outlines practical steps in implementing an impact evaluation.
This document aims to provide a Toolbox for the use of Counterfactual Impact Evaluation (CIE) methods in the case of energy State aid interventions. In particular, it seeks to assist Member State (MS) policy officers when devising an evaluation plan to be submitted to the European Commission for adoption and also for writing the Terms of Reference for the tendering of evaluation reports. State aid interventions take place alongside various economic and social activities. When evaluating the effect of a State aid intervention, one must isolate the policy effect from other effects that may happen at the same time. Otherwise, one may over/underestimate the effect of the intervention. Various econometric methods may be employed to carry this out, which this toolbox outlines.
