1. Purpose and use of the guidance: Better criteria, better evaluation
This chapter explains the purpose of the document and how it can support readers in understanding the adapted criteria definitions and use them in their work. It also explains how the guidance was developed, including the role of global evaluation stakeholders in informing its design and content. The chapter then considers how the criteria should be applied thoughtfully to improve both the delivery and design of evaluations.
The OECD’s Development Assistance Committee (DAC) first laid out five evaluation criteria in 1991 (OECD, 1991[1]) and provided definitions for them in 2002 (OECD, 2002[2]). These five criteria – relevance, effectiveness, efficiency, impact and sustainability – have become a common reference point for evaluators in development co-operation and beyond.
Prompted by a desire to respond to the 2030 Agenda and the opportunity to draw on extensive experience and learning since the criteria were first defined, a wide-ranging consultation and adaptation process was undertaken in 2017-19 and revised criteria definitions were endorsed by the DAC in December 2019 (OECD, 2019[3])1. During the consultation process, users indicated that they found many aspects of the evaluation criteria to be valuable and would like to continue to use them in their assessment future interventions. The feedback showed that the evaluation criteria are widely used and understood, allowing for consistency and a certain level of comparability.2 They have created a common, simple and neutral language as well as a normative framework in the sector that is adaptable, results-focused and comprehensively covers the key areas required for accountability and learning. As a result, the revised criteria reflect support for continuity and build on what worked well with the original definitions.
The guidance should be read in conjunction with the OECD’s Quality Standards for Development Evaluation (OECD, 2010[4]) and the Principles for Evaluation of Development Assistance (OECD, 1991[1]). The Principles focus on the management and institutional setup of evaluation systems whilst the Standards inform evaluation processes and products. This guidance helps users to operationalise the definitions of the evaluation criteria by unpacking them in more detail, providing examples and discussing the practical challenges commonly encountered when they are implemented during evaluations. Box 1.1 notes future publications that will further support application and understanding of the criteria.
Over the last 25 years, much has been learned about how to use the criteria set out by the DAC. The main challenge identified during the 2017-19 consultation was their practical application. The discussions underlined the importance of how the criteria should be used and not just how they are defined. For example, the criteria are sometimes used in a mechanistic way and not tailored to context. The idea of the guidance is therefore to support different user groups in the thoughtful application of the criteria in practice, contributing to higher quality, useful evaluation.
Accordingly, the document includes language and guidance to help readers understand and apply the criteria definitions, including the two principles elaborated to guide the use of the criteria (OECD, 2019[3]). This guidance expands on these two principles and explains each criterion’s definition, enabling users to interpret and apply the criteria in their work.
This document provides users with a succinct guide on how to interpret the criteria and apply them thoughtfully in their work. While the criteria are primarily applied in evaluation, they have wide applicability in monitoring and results-based management, the design of an intervention, or in strategic planning. It is intended to help users think through some of the challenges that can arise and where to look for solutions. It helps readers understand how to unpack the definitions, how the criteria relate to each other, the challenges when applying them and how these challenges can be addressed. Practical examples have been selected to illustrate how the criteria can be thoughtfully applied in real-world settings.
The guidance is not intended to be used as an evaluation manual, template or checklist. This guide is one source to draw on when evaluation plans are being developed and may also help to complement institution-specific policies, standards and guidance on which evaluators rely.
Equally, the guidance does not replace existing ethical guidelines and standards, as set out by other institutions or proposed by the OECD DAC (OECD, 1991[1]; OECD, 2010[4]). The guidance complements international standards used by the United Nations and other multilateral agencies, such as, the United Nations Evaluation Group (UNEG) norms and standards, Evaluation Co-operation Group (ECG) good practice standards and Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP) guidance. Other useful resources include the websites of bilateral, UN agency and other multilateral evaluation units, the Better Evaluation website and the websites of EvalPartners and its related agencies, which focus on specific areas such as the International Initiative for Impact Evaluation (3ie).
In addition to revisiting this guidance based on experience with the new definitions, EvalNet and partners are also developing specific guidance that will go into more depth on particular topics, including guidance for use of the criteria with a gender equality lens as well as the application of the criteria in humanitarian settings. In collaboration with EvalNet, ALNAP is also in the process of updating its widely used guidance for humanitarian evaluation and use of the criteria (Beck, 2016[5]).
Translations of the criteria definitions and principles
Working with key partners, and in line with the mandate to support learning and evaluation capacity development, the OECD is currently in the process of translating the criteria definitions into various languages in addition to the official languages of the OECD, English and French. Translations into Spanish and Chinese have been completed with new versions in Arabic, Dari, Pashto, Portuguese, Russian, Thai, and Urdu expected in 2021.
2nd Edition of the Glossary of Key Terms in Evaluation and Results Based Management
Definitions of the five original criteria were first published in the OECD’s Glossary of Key Terms in Evaluation and Results Based Management (OECD, 2002[2]). A second edition of the Glossary is currently being developed, reflecting updates to the criteria and other terms. The (forthcoming) 2nd Edition will be a useful companion to this guidance, as it provides succinct definitions of words used throughout the text, including: results, outcomes, impacts, theory of change, and beneficiary.
By the same token, the guidance, like the criteria themselves, is intended to prompt self-reflection and critical thinking and is not to be used in a mechanistic way. Such an approach is out of sync with best practice and the principles of 2030 Agenda (described further in Chapter 3). The guidance does not prescribe specific methodological approaches or tools, as the criteria are not a methodology. Some signposts to other sources that do elaborate on methods are included in the document as a starting point.
The guidance can be seen as a bridge between the overarching, formal documents such as the definitions of the criteria (OECD, 2019[3]) and OECD Quality Standards (OECD, 2010[4]), which provide consistent and clear definitions and terminology, and the day-to-day application of the criteria in particular institutions in a thoughtful and appropriate way, with the many context-specific decisions that this requires.
In addition to providing examples of how the criteria have been applied in real evaluations, the guidance includes elements for analysis that explain the concepts contained within each definition. These elements are not intended to be definitive or used as discrete or sub-criteria, but rather they illustrate and help to unpack each definition. They will assist users in finding the best contextual interpretation of each criterion for an individual evaluation. During the design phase, dialogue between evaluators, commissioners, and intended users of the evaluation’s findings can also explore how the different elements of each criterion should be assessed and which are most important in drawing overall conclusions, as this will vary depending on the intervention, context and intended use of an evaluation.
Chapter 2 introduces the role of evaluation criteria and how these fit within broader norms and standards. Chapter 3 provides guidance on how to interpret the criteria and apply them thoughtfully, and will also help evaluators in using the criteria within their policy context. In Chapter 4, each definition is described in more detail, common challenges are highlighted and practical examples given. Reflecting the 2030 Agenda policy priority to “leave no one behind”, Chapter 4 includes an overview of key issues related to inclusion to consider for each criterion.
As most of the examples are drawn from evaluations based on the original definitions, and the new definitions and principles are only now being rolled out, this guidance is considered as a working draft. The guidance will be used during a testing phase of initial implementation and then be revised after several years to capture additional learning and up-to-date examples. The Secretariat is particularly interested in gathering more examples from beyond the EvalNet membership and lessons on the use of the criteria for local and national evaluations.
The guidance is applicable for use across a range of contexts – local, national and international. The criteria are deliberately broad and have been designed for applicability to a range of interventions and sectors. They are relevant beyond the traditional sovereign official development assistance interventions. If applied thoughtfully, the criteria can be equally useful to interventions related to the private sector, blended finance or similar policy areas such as trade, amongst others.
This guidance is intended primarily for evaluators, evaluation managers and commissioners. It is suitable for students or young professionals using the criteria for the first time, as well as seasoned practitioners carrying out complex evaluations. It will also be of interest to guide intervention design, strategic planning and results management. Better evaluation depends partly on an effective integration of monitoring, evaluation and learning systems within the programme or policy cycle. The guidance will be useful in helping evaluators and operational partners develop a shared vision of and language about what success looks like at the outset (i.e. during the design phase of an intervention), making interventions more assessable, while improving overall results.
References
[5] Beck, T. (2016), Evaluating Humanitarian Action using the OECD-DAC Criteria, ALNAP, https://www.alnap.org/help-library/evaluating-humanitarian-action-using-the-oecd-dac-criteria (accessed on 11 January 2021).
[3] OECD (2019), Better Criteria for Better Evaluation: Revised Evaluation Criteria Definitions and Principles for Use, DAC Network on Development Evaluation, OECD Publishing, Paris, https://www.oecd.org/dac/evaluation/revised-evaluation-criteria-dec-2019.pdf (accessed on 11 January 2021).
[4] OECD (2010), Quality Standards for Development Evaluation, DAC Guidelines and Reference Series, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264083905-en.
[2] OECD (2002), Evaluation and Aid Effectiveness No. 6 - Glossary of Key Terms in Evaluation and Results Based Management (in English, French and Spanish), OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264034921-en-fr.
[1] OECD (1991), Principles for Evaluation of Development Assistance, Development Assistance Committee, OECD Publishing, Paris, https://www.oecd.org/development/evaluation/2755284.pdf (accessed on 11 January 2021).