Chapter 1. Towards a systemic approach to policy evaluation
Governments need to understand how and why a policy has the potential to succeed, and to ensure the efficient allocation of their financial resources. However, the understanding of the different practices used to assess whether government actions have met their expected goals and how they may complement each other, remains limited. This chapter provides an of policy evaluation across OECD countries and underlines the importance of developing a systemic approach in this area. The chapter discusses the relevance of policy evaluation and its distinctive role in the public sector and analyses countries’ definitions of policy evaluation. The chapter also introduces the three components of policy evaluation systems: institutionalisation, quality and use of evaluation.
Key findings
Countries generally express strong commitment towards policy evaluation: There is a shared concern to understand and improve government's performance and outputs, as well as to promote evidence-informed policy-making, and improve the quality of public services.
Policy evaluation is part of a range of practices geared to ensuring government’s effectiveness and efficiency: these include monitoring, spending reviews, and performance management. Not only do these practices complement each other, but policy evaluation also has a distinctive role to play in providing credible evidence for various public management efforts, such as monitoring or performance budgeting.
More than half of the countries (27 out of 41) have a formal definition of policy evaluation: 14 of them have one definition applicable across the government, while in 13 several definitions coexist.
While countries’ definitions on policy evaluation reflect their own institutional set up, common elements are present, including what should be measured (policies, programs, plans, reforms), why an evaluation should be conducted (aims), when (ex-ante or ex-post), and the actors involved.
The most common criteria for evaluation are outputs and outcomes, followed by policies processes and impacts.
Countries face several challenges for promoting policy evaluation across government such as the limited use of evaluation results in policy-making, the absence of a strategy for policy evaluation that promotes a whole of government approach, the limited availability of human resources (capacities and capabilities) and the lack of an integrated approach to evidence management, including data.
A systemic approach, relying on mutually supportive elements in terms of institutionalisation, quality and use is most likely to ensure a methodologically rigorous and systematic adoption of evaluations throughout the policy cycle, and use of findings by decision-makers.
Introduction
This chapter provides a first overview of the nature of policy evaluation across survey respondents and introduces the importance of developing sound policy evaluation systems. The first section discusses the relevance of policy evaluation for countries, outlines why policy evaluation matters, and addresses governments’ main objectives for conducting evaluations. The second section analyses countries’ definitions of policy evaluation, adopting an empirical approach. The last section aims to introduce this paper’s approach to policy evaluation systems and its three components: institutionalisation, quality of evaluations and use of results.
Why does policy evaluation matter?
Governments are facing increasingly complex economic, social and environmental challenges, known as the VUCA, Volatile, Uncertain, Complex and Ambiguous, which require systemic approaches and system thinking. These challenges are compounded by increased citizen demand and higher expectations, fragmentation in knowledge, higher perception of corruption, skill gaps in the civil service budgetary constraints and an erosion of trust in public institutions (OECD, 2018[1]). Lessons learned from OECD experience also highlight that it is more difficult for governments to identify outcomes, trade-offs, as well as winners and losers of an implemented policy due to the increase complexity, divergent values and interdependent processes, structures and actors that are related to major policy challenges (OECD, 2018[2]).
In this context, governments should demonstrate that their decisions and policies are informed by evidence, that they set realistic expectations about various policy choices, and spend public resources adequately. Thus, policy evaluation has a critical role to ensure these goals as well as to avoid policy failure (Howlett, 2019[3]). By evaluating performance and results, policymakers have a deeper understanding of the underlying policy problems and can make informed decisions about the feasibility of continuing the policy or initiating a new one.
Policy evaluation facilitates learning as it helps to understand why and how a policy was or has the potential to be successful or not, by providing an assessment about the reasons and causal mechanisms leading to policy success or failure. It contributes moreover to the quality of decision-making by providing insights on how to improve links between policy formulation, implementation and outcomes (OECD, 2017[4]). Simultaneously, policy evaluation has the potential to improve policy accountability and transparency, and provide legitimacy for the use of public funds and resources as it provides citizens and other stakeholders with information whether the efforts carried out by the government, including allocated financial resources, are producing the expected results (OECD, 2018[2]).
Government core objectives for policy evaluation
The OECD survey on policy evaluation provides an overview of governments’ stated objectives when conducting evaluations (See Figure 1.1). According to these results, countries ranked most of the objectives between 9 and 10, showing their strong commitment toward policy evaluation. The results show no clear-cut priorities among these objectives aside from their shared concern to measure government's performances/outputs and the resources required to achieve them, as well as to promote evidence-informed policy-making (OECD, 2018[2]) and improve the quality of public services in all respondents and OECD countries. Countries are also concerned about conducting evaluations to improve policies value-for-money, to enhance trust in public institutions, and to encourage transparency in the allocation of public resources –albeit apparently to a slightly lesser extent.
Learning is also often an important objective of policy evaluation, even if it does not appear as such in the results below. It is often crucial and has been identified as such by lead experts (Lazaro, 2015[5]). While the results in terms of learning are often less likely to achieve media impact or strong public attention, they are also potentially the most useful in that they can help improve policies and understand why policies work or don’t, and what kind of adjustment may be needed.
Note: n=42 (35 OECD member countries). Answers reflect responses to the question, “What are the government’s main objectives for conducting evaluations?”, where 0 indicates "least important objective", 5 is "Neutral", and 10 is a "principal objective".
Source: OECD Survey on Policy Evaluation (2018).
Defining policy evaluation
What is the distinctive role of policy evaluation in the public sector?
For the purpose of this report, and as a reference for the survey respondents, policy evaluation is defined as a “structured and objective assessment of an ongoing or completed policy or reform initiative, its design, implementation and results. Its aim is to determine the relevance and fulfilment of objectives, efficiency, effectiveness, impact and sustainability as well as the worth or significance of a policy” 1. While this definition may not be universally accepted by the evaluator community, it has offered a starting point to start the analysis and the questionnaire design. From an empirical perspective, a number of countries define evaluation, and others specifically do regarding policy evaluation. Box 1.1 presents some of these examples.
The Netherlands: “Policy evaluation is an examination of the efficiency (the extent to which the optimum effect is achieved with as few costs as possible and undesirable side effects) and effectiveness (the extent to which the policy objective is realized through the use of the policy instruments examined) of policy.” (Ministry of Finance of The Netherlands, 2018[6]).
United States: “Evaluation means an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency.” (115th Congress, 2019[7]).
Canada: “Evaluation is the systematic and neutral collection and analysis of evidence to judge merit, worth or value. Evaluation informs decision-making, improvements, innovation and accountability. Evaluations typically focus on programs, policies and priorities and examine questions related to relevance, effectiveness and efficiency. Depending on user needs, however, evaluations can also examine other units, themes and issues including alternatives to existing interventions. Evaluations generally employ social science research methods” (Canada Treasury Board, 2016[8]).
Source: 115th Congress of the United States (115th Congress, 2019[7]), Ministry of Finance of The Netherlands (2018[6]), Canada Treasury Board (Canada Treasury Board, 2016[8]).
Promoting the comprehension of why and how a policy was or has the potential to succeed (i.e. learning) and improving the efficient allocation of financial resources (i.e. accountability) is becoming a priority across public administrations. However, there is a lack of awareness from practitioners and stakeholders about the different practices developed to assess whether government actions have met their expected goals (monitoring, spending reviews, or performance management), and how they differ from one another or support/complement each other. For example, while in some cases, policy evaluations can come close to performance audit, these two practices still differ in fundamental ways as professional disciplines.
This is why it is important to distinguish between monitoring and evaluation. Monitoring seeks to check progress against planned targets and can be defined as the formal reporting of evidence to show that resources are adequately spent, outputs are successfully delivered and milestones met (HM Treasury, 2011[9]). (Table 1.1).
Additionally, over the past years OECD governments have developed a range of tools that focus on strengthening the alignment of budget decision making with the government policy cycle, with a view towards improving performance and overall public sector effectiveness (OECD, 2019[12]), such as:
Spending reviews, which aim to increase the fiscal space available to government to finance its policy priorities. Initially, after the global financial crisis, spending reviews were identified as a tool for extracting savings from agencies in a way that would seek to improve the quality of public expenditure. (OECD, 2011[13]). Since then, reviews have evolved to identify whether a line agency’s activities align to a government’s priorities and the implementation challenges that an agency faces (OECD, 2019[12]). By contrast, policy evaluations focus not only on measuring whether costs are justified in terms of efficiency and value for money (Smismans, 2015[14]), but also on assessing the extent to which public intervention causes an observed effect, and its relation to intended objectives. Therefore, although these two practices assess public policy programs or activities based on criteria such as efficiency, spending reviews have a specific focus on improving the quality of public expenditure and on proposing reallocations (The World Bank, 2018[15]). Evaluations can have a broader learning function, helping to assess performance with regard to the policy objectives that were initially fixed. Evaluation can also be a crucial tool to inform the results of spending reviews, without which these may become a purely mechanical exercise.
Performance management, which is defined as a process by which an agency involves its employees, as individuals and members of a group, in improving organisational effectiveness (Walker and Moore, 2011[16]). This practice, like policy evaluation, aims to increase accountability and provide quality data on a reliable basis to inform decision-making (OECD, 2019[12]). Nonetheless, although performance management seeks to ensure that a programme is operating as intended in a timely manner and with efficient use of resources, it cannot explain performance variations (Kroll and Moynihan, 2018[17]). On the contrary, policy evaluation can help make sense of performance outcomes, and create a sense of “policy memory” by taking into account challenges from experiences and good practices that could be incorporated into current performance efforts (Acquah, Lisek and Jacobzone, 2019[18]; OECD, 2008[19]).
Audit, aims to determine whether the information collected or actual conditions correspond to established criteria, including compliance with financial or legal rules. Auditing helps to ensure that public-sector entities and public servants will perform their functions effectively, efficiently, ethically and in accordance with the applicable laws and regulations (International Organisation of Supreme Audit Institutions, 2019[20]). For instance, independent external bodies such as the Supreme Audit Institutions (SAIs) play a role in overseeing and holding government to account for its use of resources, together with the legislature and other oversight bodies. Thus, SAIs support policy evaluation by providing valuable evidence on key government functions, as well as by ensuring accountability (OECD, 2016[21]), and they can play a double role both as providers of audits, compliance audits and performance audits, as well as of evaluations (see Chapter 3).
How can policy evaluation be defined? Countries’ approaches
Governments can benefit from adopting a clear definition of policy evaluation to distinguish it from other practices. Such a definition would also help create a shared understanding within the public sector of the aims, tools and features of policy evaluation. Countries’ definitions could therefore include what is policy evaluation, the type of knowledge it should produce, how and why it should be conducted and the actors that are involved.
Figure 1.2 shows that more than half of the survey respondents (27 countries) have adopted a formal definition of policy evaluation: 14 of them have one definition applicable across government, while 13 have several. In terms of OECD countries, a higher share of respondents define policy evaluation (23 of 35), either by having one definition (11) or several (12).
Note: n=41 (35 OECD member countries). Kazakhstan answered that they do not know if there is a formal definition for policy evaluation. Answers reflect responses to the question, “Does your government have a formal definition of policy evaluation?” and "Please provide the definition/s and the reference to the relevant documents".
Source: OECD Survey on Policy Evaluation (2018).
In some cases, the definition is embedded in a legal document. For instance, Japan presents the definition in a law, the Government policy evaluations Act (Act No. 86 of 2001). Argentina defines evaluation in the decree 292/2018, which designates the body responsible for preparing and executing the annual monitoring and evaluation plan for social policies and programmes (2018[22]). In Latvia, the definition is framed in the development planning system law. Finally, some countries define evaluation in guidelines or manuals as is the case of Mexico (general guidelines for the evaluation of the general public administration programmes (2007[23])), Costa Rica (manual of evaluation for public interventions (2018[24])), and Colombia (guide for the evaluation of public policies (2016[25])).
Key concepts found in country definitions
The analysis for this report has aimed to identify elements of consistency across definitions of evaluation in countries. In fact, while the definitions of policy evaluation diverge across survey respondents, they share several characteristics. The definitions were clustered and mapped across various conceptual dimensions, to highlight elements of a shared understanding.
For the purpose of this report, the key conceptual terms found in the definitions supplied by countries were clustered across three main categories: (1) criteria for evaluation, (2) type of public interventions evaluated, and (3) characteristics (Figure 1.3). This exercise seeks to provide a broad picture of the different approaches for policy evaluation across responding countries, and compare those with practitioners and academic definitions.
Note: n=27 (23 OECD member countries). 14 countries (12 OECD member countries) answered they do not have a formal definition for policy evaluation. Moreover, one country (no OECD member country) answered he does not know if there is a formal definition for policy evaluation. Answers reflect affirmative responses to the question, “Does your government have a formal definition of policy evaluation?” and "Please provide the definition/s and the reference to the relevant documents". Aims of policy evaluations.
Source: OECD Survey on Policy Evaluation (2018).
Criteria for policy evaluation
Policy evaluation evaluate different criteria, such as the relevance; effectiveness; efficiency; sustainability and/or impact of a specific intervention (See Box 1.2.).
Relevance — to what extent do the (original) objectives (still) correspond to needs and issues?
Effectiveness — to what extent did a policy/public intervention generate observed effects and changes? To what extent do the observed effects correspond to the objectives?
Efficiency — were the costs involved justified, given the changes and effects achieved?
Sustainability — does the policy/public intervention present net benefits at the long term?
Impact — what are the effects produced by an intervention (i.e. positive or negative, primary and secondary long-term effects produced, directly or indirectly, intended or unintended)?
Source: OECD-DAC (2002[26]), European Environment Agency (2017[27]) , Smismans, (2015[14]), and Gasper (2018[28]).
Most of countries’ definitions express that the aim of policy evaluation is to assess the effectiveness and efficiency of a policy or program. Findings from the last OECD performance budgeting survey also highlight the interest of countries in evaluating the effectiveness and efficiency of an intervention. Sixteen countries answered that they “usually” evaluate programme effectiveness and efficiency, while seven countries “always” measure effectiveness and five do so for efficiency. This can be explained by governments’ needs to identify not which policy options generate the highest impact, but also options that are the most cost-effective (Heider, 2017[29]).
Seven countries (Austria, Great Britain, Japan, Lithuania, Latvia, Mexico, and Slovakia) refer to impact as criteria. A number of countries directly refer to impact assessment or impact evaluation rather than to evaluation. This could show the misunderstanding of some countries when trying to implement evaluations throughout the policy cycle, but only carrying them out after the implementation of the policy. Seven countries (Australia, Canada, Costa Rica, Great Britain, Japan, México, and Slovakia) also define the aims of policy evaluation in terms of relevance.
Lastly, only Mexico, Slovakia, Great Britain and Costa Rica incorporate the criteria of sustainability in their definitions. One of the reasons for this can be the lack of a stringent and clear definition of sustainability and its association with environmental measurements. According to OECD-DAC evaluation criteria (2002[26]), sustainability includes the “examination of the financial, economic, social, environmental, and institutional capacities of the systems needed to sustain net benefits over time”. Another explanation may be that these results indicate that sustainability may be less of a pressing challenge for sampled countries (mostly OECD countries).
Types of public interventions set out in the definitions
Regarding which kind of public interventions are present in the different definitions, countries generally focus on programmes, interventions and policies. Some of them additionally consider activities such as regulations and processes, adopting a wider definition of policy evaluation. These findings may also demonstrate some conflation between monitoring and evaluation terms, such as referring explicitly to ongoing operational decision-making (processes). In general, survey answers reveal a countries’ relative difficulty in defining the concept ‘policy’’, as respondents refer to interventions, programmes and initiatives as falling under that category. A good example of definition, where they are clearly differentiate between different types of public interventions, is the definition of Costa Rica (see Box 1.3).
In its manual for the evaluation of public interventions (2018[24]), the Ministry of National Planning and Economic Policy from Costa Rica differentiates four different types of public interventions (policy, plan, program and project), depending on the characteristics of the problem to be addressed (magnitude of the problem, resources available to respond to it, scope, target population, etc.). This classification can be summarised as follows:
Policies: Defined course of action to guide or achieve an objective, expressed in guidelines, strategic aims and actions on a specific topic.
Plans: Integrated set of programmes that respond to the fulfilment of objectives and goals, which are executed in the short, medium and long term, and include dedicated resources.
Programmes: Set of interrelated projects that look to achieve specific and common objectives.
Projects: A set of activities that aim to achieve specific objectives, with a given budget and by a specific date, mainly oriented to the production of goods and services.
Source: Ministry of National Planning and Economic Policy of Costa Rica (2018[24]).
Key characteristics of policy evaluation should be the evaluator
Definitions can also include quality attributes (systematic, objective, and rigorous); time setting, “when” (Ex-post vs. Ex-ante) and who (internal evaluation vs. external evaluation).
As will be explained in the chapter on Quality and use of policy evaluations, the quality of evaluations depends on both their methodological rigor and their trustworthiness. Reflecting both these aspects, the most common characteristics found in countries’ definitions relating to quality are the following:
Systematic: An evaluation should be carried out using a planned and organised procedure or an agreed set of methods. For instance, decisions need to be based on systematic approaches (i.e. theory-based approach: logic, reasoning, and by an accurate guide or principle) instead of on unfounded assumptions (Gasper, 2018[30]).
Rigour: evaluations should be developed using well-designed and well-implemented methods tailored to the target question (Commission on Evidence-Based Policymaking, 2017[31]).
Objective: An evaluation should be conducted from an impartial position, without any personal or political factor influencing (from a researcher or policymaker) the research design and its implementation (Parkhurst, 2017[32]).
According to the survey (Figure 1.3), 15 of the respondents (12 OECD countries) mentioned at least one of the terms above in their definition. A majority of them include the fact that policy evaluation must be systematic (14 of all the survey respondents and 11 OECD countries). Only seven countries of the total respondents (from which six are OECD countries) specify a policy evaluation must be objective. Canada and Spain take into account characteristics similar to objectiveness such as neutral (in Canada definition), and reasoned (in Spain definition). Only Great Britain, Japan, Argentina and Mexico mention rigorous (see further examples in Box 1.4).
Argentina: “The evaluation of policies, programmes, plans and projects with social impact, comes from a form of applied, systematic, planned and rigorous social research; aimed at identifying, obtaining and providing data and valid and reliable information about them; which will allow improving both its design and its implementation, and ensure access to the human rights it seeks to promote (Decree 292/2018)”.
Mexico: “Evaluation is a systematic and objective analysis of federal programs whose purpose is to determine the relevance and achievement of its objectives and goals, as well as its efficiency, quality, results, impact and sustainability”.
Lithuania: “Evaluation is a systematic and objective determination of the suitability, effectiveness, efficiency, usefulness and long-term impact of the planned, executed or completed programs” (Resolution on Strategic Planning methodology No 827 approved on 2002 June 6).
Source: Poder Ejecutivo Nacional de Argentina (2018[33]) and Consejo Nacional de Evaluación de la Política de Desarrollo Social de México (2007[23])).
The time setting (ex-ante vs. ex-post) is a key criterion for differentiation, both from an analytical and methodological perspective. The term ex-post evaluation refers to a retrospective evaluation that can be interim (i.e. at the mid-term of an initiative), final (at its conclusion), or ex post in the strict sense (placed several years after the intervention has finished) (Smismans, 2015[34]). Ex post evaluation can be a tool for accountability, also close to performance audit and control, for example when it is performed through supreme audit institutions, or internal inspection bodies. Yet, ex post evaluation is also important to facilitate learning, to understand if and when the objectives of policies where attained and spell out a theory of change. Ex-ante evaluation on the other hand refers to a set of rules, instructions and procedures that enable public institutions to have a portfolio of socially profitable investment initiatives before their implementation (Ministerio de Desarrollo Social y Familia de Chile, 2019[35]). It provides an assessment whether the strategy and objectives proposed are relevant to target population needs; and whether the assumptions concerning expected results and impacts are realistic and in line with the resources available (The European Network for Rural Development, 2014[36]).
New approaches in policy evaluation are trying to disentangle the challenges of linking ex-ante and ex-post appraisal, and apply a policy evaluation system focused on the entire policy cycle (thus covering these two types of evaluations), in all the policy areas (Mergaert and Minto, 2015[37]; Smismans, 2015[34]). For survey respondents, the policy evaluation definition is more commonly linked to the second term (ex-post), related to an already implemented policy. Seven OECD countries of the survey respondents refer to ex-post, and only five OECD countries as well to ex-ante. Specific countries defy the norm such as Norway which specifies that evaluations can be undertaken prior to (ex-ante), during, or after implementation (ex-post) (2006[38]). This also depends on the context and use of policy evaluation. Thus, while evaluation of public expenditure, policies and programmes tends to be overwhelmingly ex-post, by contrast, in the regulatory area, the focus is most often on ex-ante evaluation of regulations.
The third common characteristic is related to “who” carries out an evaluation: external evaluations (also known as “informal”, “outside” or “society-driven” evaluation) or internal evaluations (also known as “formal”, “inside” or “government-driven” evaluation) (Schoenefeld and Jordan, 2017[39]; Weiss, 1993[40]; Hildén, 2014[41]). Survey findings (See Figure 1.3) report that only four OECD countries of the total respondents mention external evaluations in their definitions, and five OECD countries internal evaluation.
External evaluation refers to an evaluation of an intervention conducted by entities and/or individuals outside the government. This type of evaluation could be considered to be more independent, it can take a more critical look at the policy being studied and their results can be potentially more trusted (Schoenefeld and Jordan, 2017[39]). However, as much as this type of evaluation could be independent from government actions, it can also be limited by the knowledge of the evaluator about the context and political process, as well as access to relevant data.
The notion of internal evaluation refers to an evaluation of a development intervention conducted by a governmental institution. Internal evaluators may have more knowledge about a public policies, provide a more accurate assessment according to local contexts and have easier access to inside data than what an external evaluator could supply (Weiss, 1993[40]; Schoenefeld and Jordan, 2017[39]). Nonetheless, in the process of conducting an evaluation, the internal evaluators can be under political pressure and time constraints to show good results, which can affect the validity of the findings of the evaluation and its public deliberation.
Overall, there is still no evidence to determine which type of evaluation is better or preferred. In practice, the differences between internal and external evaluations can sometimes be blurred as hybrid approaches, mixing internal evaluations combined with some external evaluation for specific or more technical aspects of a policy or programme. Moreover, a government can commission the evaluation to an external organisation (e.g. NGO, universities), while still ensuring that civil servants control the research questions addressed by the evaluation (i.e. principle agent relationships (Schoenefeld and Jordan, 2017[39]).
The selection of “internal vs. external” evaluations will depend on how each approach fits with the goals of the policy evaluation and the overall social-political circumstances (Schoenefeld and Jordan, 2017[39]). Some countries have adopted clear criteria for determined which approach fits best in what circumstances. For example, the Cabinet Implementation Unit from Australia (2014[42]) specifies that the availability of resources and capacity will determine whether the evaluation is conducted internally or externally.
The concept of external evaluation also covers a variety of actors. Figure 1.4 presents an overview of what type of actors typically carry out evaluations in surveyed countries. For instance, although the evaluation of government-wide policy priorities (GWPP) are still mainly carried out by the government (26 countries of all the respondents, and 21 OECD countries), eight countries choose to commission evaluations to civil society organisations or universities and seven countries choose the private sector.
Note: n=29 (24 OECD member countries). Four countries (all OECD member countries) answered that they do not have government-wide policy priorities. Moreover, nine countries (7 OECD member countries) answered that they do not evaluate their government-wide policy priorities. Answers reflect affirmative responses to the question, “Evaluations of government–wide policy priorities are carried out by ". The option "Other" is not included.
Source: OECD Survey on Policy Evaluation (2018).
At the sector level, ministries stand out as carrying out in average about 53% of their evaluations internally: either by the own institution (26 countries in Health and 17 in the Public Sector Reform (PSR)) or by a central unit (15 countries in Health and 5 in PSR). Ministries only do external evaluations around 36% of cases: either with civil society organisations or universities (12 countries in Health and 8 in PSR) or with the private sector (13 countries in Health and 10 in PSR).
What polices are being evaluated?
According to Figure 1.5, most of the countries evaluate policies that have formal requirements, such as policies with evaluation clauses into laws, policies identified by government institutions and policies defined as government priorities in a national plan or program. The influence of international commitments can be seen in some countries, such as Austria, Germany, Finland, Spain, and Greece. Lastly, few countries evaluate all policies; which can be due to an evaluation can be time consuming and entails a costly process.
Note: n=42 (35 OECD member countries). Answers reflect responses to the question, “Which policies are evaluated?". The option "Other" is not included.
Source: OECD Survey on Policy Evaluation (2018).
Methodologies and tools used in policy evaluation
All phases of the policy chain can be evaluated through different types of policy evaluation (see Figure 1.6). This report defines the different stages of the policy chain as follows:
1. The input level refers to the resources employed to implement a policy (OECD, 2016[43]) such as staff, money, time, equipment, etc.
2. The process level refers to the activities that were undertaken in a policy (OECD, 2016[43]).
3. The output level refers to a first level of results, directly associated with the products delivered by the policy implemented.
4. The outcome level refers to the medium-term (directly) consequences of the policy implemented (OECD, 2016[43]).
5. The impact level looks at the long-term consequence of a policy initiative (OECD, 2016[43]).
Source: OECD
The chain of “input-activities-output-outcome-impact” and its causality is particularly complex. Evaluating a single initiative, versus a comprehensive action plan, requires different tools and can probably reach different levels of understanding (OECD, 2016[43]). As shown in Figure 1.6, each of these elements can be compared to one another to evaluate different aspects of a policy/public intervention. For instance, a cost effectiveness analysis will require the comparison between the cost of an intervention (from the inputs or resources employed) and the outcomes obtained. Table 1.2 presents in more detail some of the evaluations and methodologies used to respond these target questions.
Survey data (see Figure 1.7) shows that on average the most evaluated elements in government-wide policy priorities (GWPP) and in policies on charge of ministries of Health and PSR are outputs (87%) and outcomes (88%), followed by process (78%), impact (76%), and inputs (75%). On the other hand, there is a major variance in the impact element. Impacts are more commonly evaluated for GWPP (86% in all the survey respondents and 88% in OECD countries) and for the policies of PSR ministries (80% in all the survey respondents and 85% in OECD countries), compared to the policies of Health ministries (61% in all and OECD countries respondents).
Note: The chart is expressed as a percentage of responding countries as number of respondents differ for the main institution, health and PSR. n=29 (24 OECD member countries). 4 countries (all OECD member countries) answered that they do not have government-wide policy priorities. Moreover, 9 countries (7 OECD member countries) answered that they do not evaluate their government-wide policy priorities. For the Health ministries n=31 (28 OECD member countries). 9 countries (7 OECD member countries) did not participate on this survey. Moreover, 2 countries (1 OECD member country) are not included as they answered that none of the policies that fall in their institution's responsibility are evaluated. For the PSR ministries n=25 (20 OECD member countries). 11 countries (10 OECD member countries) did not participate on this survey. Moreover, 6 countries (5 OECD member countries) are not included as they answered that none of the policies that fall in their institution's responsibility are evaluated. Answers reflect responses to the question, “Which elements are evaluated by your institution? (Check all that apply)".
Source: OECD Survey on Policy Evaluation (2018).
More generally, given that evaluations are a costly activity, it is important to justify the need for evaluation and the resources that it will require, which would call governments to establish some sets of criteria for determining when and what type of evaluation is needed. Setting threshold and proportionality criteria is something already well embedded in some countries concerning the Regulatory Impact Assessment process.
This may also explain why, despite the potential of randomised control trials (RCT) or quasi-experimental designs (QED) to provide rigorous findings, the evidence shows that they are comparatively less used (see further details in). In addition, these results could be related to the not always practical use of RCTs due to legal, political and ethical considerations such as ensuring participant selection procedures are fair, there is an acceptable balance of benefits and harms, and participants provide an informed consent (Acquah, Lisek and Jacobzone, 2019[18]; Goldstein et al., 2018[51]).
Towards sound policy evaluation systems: promoting institutionalisation, quality and use
Despite the growing interest for and acknowledgment of its contribution to improve the design and implementation of public policies, policy evaluation often constitutes the weakest link in the policy cycle. The reasons for this are manifold. Firstly, some countries face technical barriers for carrying out evaluations such as the challenges of governments to create and share verifiable, accurate, useable and unbiased data within and outside public administration (Rutter, 2012[52]).
Findings from the OECD survey suggest (see Figure 1.8) that the four main challenges for promoting policy evaluation across government are: the limited use of evaluation results in policy-making, the absence of a strategy for policy evaluation that promotes a whole of government approach, the limited availability of human resources (capacities and capabilities) for policy evaluation. Ministries perceive issues related to the quality of the evidence, and related to the political interest in, demand for, policy evaluation.
Note: For the main institution n=42 (35 OECD member countries). For the Health ministries n=31 (28 OECD member countries). 9 countries (7 OECD member countries) did not participate on this survey. Moreover, 2 countries (1 OECD member country) are not included as they answered that none of the policies that fall in their institution's responsibility are evaluated. For the PSR ministries n=25 (20 OECD member countries). 11 countries (10 OECD member countries) did not participate on this survey. Moreover, 6 countries (5 OECD member countries) are not included as they answered that none of the policies that fall in their institution's responsibility are evaluated. Answers reflect responses to the questions, “What are the government's current challenges for promoting policy evaluations?” for the main institution and “What are current challenges for promoting policy evaluation in your institution?" for Health and PSR, where 0 indicates that is a "rare challenge", 5 is "Neutral", and 10 is a "principal challenge".
Source: OECD Survey on Policy Evaluation (2018).
These four main challenges perceived by countries can be considered – to a certain extent and depending of the institutional context – as inter-dependents. The lack of human resources in terms of capabilities and capacities (for instance to commission or undertake evaluations) can probably affect the quality of the evaluations in a negative way. As will be analysed in the subsequent chapters, the quality of evaluations might also influence the use of its results. Moreover, considering these elements, the development an integrated and whole-of-government strategy to promote policy evaluation is not an easy task, due to the aforementioned limited capacities and capabilities, the lack of political support, and probably also to the absence of analytical frameworks to develop such strategies. Because of these interdependencies, this report adopts a systematic approach of the promotion of policy evaluation within governments.
Beyond these challenges, another key element is the issue of the timeliness of the evaluation results, that is, whether they arrive at the time of decision-making. Usually, the time span of Ministers and political life implies a very short lead-time to make decisions. Evaluations, on the other hand, require time. This challenge may call for two kinds of reactions. The first is a rapid adaptive response, doing a quick evaluation within the available timeframe, and working out proxies and similar studies. Another approach is to invest upfront, and to have a certain reserve of “evaluation capacity” and evaluative studies that can be ready for when the demand arises. These allow to draw on the existing stock of knowledge and to provide answers to the short-term demands when it arrives. However, this requires an investment ex ante, the capacity to manage the stock of knowledge in a strategic manner in light of anticipated demand. This issue is crucial for establishing a well-functioning evaluation system.
The value of a systemic whole-of-government approach
This report promotes a systems’ approach to policy evaluation. A system can be defined as “elements linked together by dynamics that produce an effect, create a whole new system or influence its elements” (OECD, 2017[53]). A policy evaluation system can be defined, following Lazaro, as: one in which evaluation is a regular part of the life cycle of public policies and programmes, it is conducted in a methodologically rigorous and systematic manner, in which its results are used by political decision-makers and managers, and those results are also made available to the public” (2015[54]). Thus, a system calls for constant adjustment throughout the policy cycle, with implications for the ways in which institutions, processes, skills and actors are organised (OECD, 2017[53]).
In order to develop and/or implement a strategy for promoting a whole-of-government approach on policy evaluation, this report adopts a tiered approach toward through the triple lens of institutionalisation, quality and use (see Box 1.5)
This report approaches the idea of a sound policy evaluation system through the following three dimensions:
Institutionalisation: the systematic process of embedding evaluation practices into more formal and systematic approaches. It can include establishing an evaluation system in governmental settings through specific policies or strategies (Lázaro, 2015[54]; Gaarder and Briceño, 2010[55]).
Quality: defined as policy evaluations that are technically rigorous as well as well governed; that is be independent and appropriate for the decision-making process (Picciotto, 2013[56])
Use: which is defined under three conditions (Ledermann, 2012[57]):
Symbolic use (also known as persuasive), occurs when the results of evaluations are taken up to justify or legitimise a pre-existing position, without changing it;
Conceptual use happens when evaluation results lead to an improved understanding or a change in the conception of the subject of evaluation;
Instrumental use is when evaluation recommendations inform decision-making and lead to an alteration in the object of evaluation.
Source: Lázaro (2015[54]), Gaarder and Briceño (2010[55]), Ledermann (2012[57]), and Picciotto (2013[56]).
References
[7] 115th Congress (2019), Public Law No: 115-435 (01/14/2019) - Foundations for Evidence-Based Policymaking Act of 2018, https://www.congress.gov/bill/115th-congress/house-bill/4174.
[18] Acquah, D., K. Lisek and S. Jacobzone (2019), “The Role of Evidence Informed Policy Making in Delivering on Performance: Social Investment in New Zealand”, OECD Journal on Budgeting, Vol. 19/1, https://dx.doi.org/10.1787/74fa8447-en.
[8] Canada Treasury Board (2016), Policy on results, https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=31300 (accessed on 20 September 2019).
[31] Commission on Evidence-Based Policymaking (2017), The Promise of Evidence-Based Policymaking: Report of the Commission on Evidence-Based Policymaking, https://www.cep.gov/report/cep-final-report.pdf (accessed on 6 August 2019).
[23] CONEVAL (2007), Lineamientos generales para la evaluación de los Programas Federales de la Administración Pública Federal, https://www.coneval.org.mx/rw/resource/coneval/eval_mon/361.pdf (accessed on 18 June 2019).
[45] Crowley, D. et al. (2018), “Standards of Evidence for Conducting and Reporting Economic Evaluations in Prevention Science”, Prevention Science, Vol. 19/3, pp. 366-390, https://doi.org/10.1007/s11121-017-0858-1.
[25] Departamento Nacional de Planeación (2016), ¿Qué es una Evaluación?, https://sinergia.dnp.gov.co/Paginas/Internas/Evaluaciones/%C2%BFQu%C3%A9-es-Evaluaciones.aspx.
[42] Department of the Prime Minister and Cabinet (2014), Guide to Implementation Planning, Australian government, https://www.pmc.gov.au/sites/default/files/files/guide-to-implementation-planning.pdf (accessed on 12 July 2019).
[27] European Environment Agency (2017), “EEA guidance document-policy evaluation”, https://www.researchgate.net/publication/317594615.
[50] Flay, B. et al. (2005), “Standards of Evidence: Criteria for Efficacy, Effectiveness and Dissemination”, Prevention Science, Vol. 6/3, pp. 151-175, https://doi.org/10.1007/s11121-005-5553-y.
[55] Gaarder, M. and B. Briceño (2010), “Institutionalisation of government evaluation: balancing trade-offs”, Journal of Development Effectiveness, Vol. 2/3, pp. 289-309, https://doi.org/10.1080/19439342.2010.505027.
[28] Gasper, D. (2018), “Policy Evaluation: From Managerialism and Econocracy to a Governance Perspective”, in International Development Governance, Routledge, https://doi.org/10.4324/9781315092577-37.
[30] Gasper, D. (2018), “Policy Evaluation: From Managerialism and Econocracy to a Governance Perspective”, in International Development Governance, Routledge, https://doi.org/10.4324/9781315092577-37.
[51] Goldstein, C. et al. (2018), Ethical issues in pragmatic randomized controlled trials: A review of the recent literature identifies gaps in ethical argumentation, BioMed Central Ltd., https://doi.org/10.1186/s12910-018-0253-x.
[29] Heider, C. (2017), Rethinking Evaluation - Efficiency, Efficiency, Efficiency, https://ieg.worldbankgroup.org/blog/rethinking-evaluation-efficiency.
[41] Hildén, M. (2014), “Evaluation, assessment, and policy innovation: exploring the links in relation to emissions trading”, Environmental Politics, Vol. 23/5, pp. 839-859, https://doi.org/10.1080/09644016.2014.924199.
[9] HM Treasury (2011), The Magenta Book: Guidance for evaluation, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/220542/magenta_book_combined.pdf (accessed on 18 June 2019).
[3] Howlett, M. (2019), “Policy analytical capacity and evidence‐based policy‐making: Lessons from Canada”, https://doi.org/10.1111/j.1754-7121.2009.00070_1.x.
[20] International Organisation of Supreme Audit Institutions (2019), ISSAI 100: Fundamental Principles of Public-Sector Auditing, http://www.issai.orghttp://www.intosai.org (accessed on 8 January 2020).
[17] Kroll, A. and D. Moynihan (2018), “The Design and Practice of Integrating Evidence: Connecting Performance Management with Program Evaluation”, Public Administration Review, Vol. 78/2, pp. 183-194, https://doi.org/10.1111/puar.12865.
[5] Lazaro, B. (2015), Comparative Study on the Institutionalization of Evaluation in Europe and Latin America, Eurosocial Programme.
[54] Lázaro, B. (2015), Comparative study on the institutionalisation of evaluation in Europe and Latin America, Eurosocial Programme, Madrid, http://sia.eurosocial-ii.eu/files/docs/1456851768-E_15_ENfin.pdf (accessed on 9 July 2019).
[57] Ledermann, S. (2012), “Exploring the Necessary Conditions for Evaluation Use in Program Change”, American Journal of Evaluation, Vol. 33/2, pp. 159-178, https://doi.org/10.1177/1098214011411573.
[10] McDavid, J., I. Huse and L. Hawthorn (2006), Program Evaluation and Performance Measurement: An Introduction to Practice, https://study.sagepub.com/mcdavid3e (accessed on 28 January 2020).
[37] Mergaert, L. and R. Minto (2015), “Ex Ante and Ex Post Evaluations: Two Sides of the Same Coin? The Case of Gender Mainstreaming in EU Research Policy”, Symposium on Policy Evaluation in the EU, https://doi.org/10.1017/S1867299X0000427X.
[35] Ministerio de Desarrollo Social y Familia de Chile (2019), Evaluación Social Ex Ante, http://sni.ministeriodesarrollosocial.gob.cl/evaluacion-iniciativas-de-inversion/evaluacion-ex-ante/ (accessed on 5 July 2019).
[24] Ministerio de Planificación Nacional y Política Económica (2018), Manual de Evaluación para Intervenciones Públicas, https://documentos.mideplan.go.cr/share/s/6eepeLCESrKkft6Mf5SToA (accessed on 5 August 2019).
[38] Ministry of Finance (2006), The Norwegian Government Agency for Financial Management (DFØ), https://www.regjeringen.no/en/dep/fin/about-the-ministry/etater-og-virksomheter-under-finansdepartementet/subordinateagencies/the-norwegian-government-agency-for-fina/id270409/ (accessed on 12 July 2019).
[6] Ministry of Finance of The Netherlands (2018), Arrangements for periodic evaluation research, https://wetten.overheid.nl/BWBR0040754/2018-03-27 (accessed on 12 July 2019).
[49] Morton, M. (2009), Applicability of Impact Evaluation to Cohesion Policy 1 Report Working Paper of, https://ec.europa.eu/regional_policy/archive/policy/future/pdf/4_morton_final-formatted.pdf (accessed on 8 August 2019).
[12] OECD (2019), Budgeting and Public Expenditures in OECD Countries 2019, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264307957-en.
[11] OECD (2019), Open Government in Biscay, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/e4e1a40c-en.
[1] OECD (2018), Building Capacity for Evidence Informed Policy Making: Towards a Baseline Skill Set, http://www.oecd.org/gov/building-capacity-for-evidence-informed-policymaking.pdf (accessed on 3 September 2019).
[47] OECD (2018), Cost-Benefit Analysis and the Environment: Further Developments and Policy Use, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264085169-en.
[2] OECD (2018), Draft Policy Framework on Sound Public Governance, http://www.oecd.org/gov/draft-policy-framework-on-sound-public-governance.pdf (accessed on 8 July 2019).
[59] OECD (2018), OECD Performance Budgeting Survey.
[4] OECD (2017), Government at a Glance - OECD, OECD, Paris, http://www.oecd.org/gov/govataglance.htm (accessed on 9 July 2019).
[53] OECD (2017), Systems Approaches to Public Sector Challenges: Working with Change, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264279865-en.
[43] OECD (2016), Open Government: The Global Context and the Way Forward, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264268104-en.
[21] OECD (2016), Supreme Audit Institutions and Good Governance: Oversight, Insight and Foresight, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/9789264263871-en.
[13] OECD (2011), “Typology and implementation of spending reviews”, OECD SBO Meeting on Performance and Results, http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=GOV/PGC/SBO(2011)9&doclanguage=en (accessed on 1 August 2019).
[48] OECD (2010), “DAC Guidelines and Reference Series: Quality Standards for Development Evaluation”, https://www.oecd.org/development/evaluation/qualitystandards.pdf (accessed on 9 July 2019).
[19] OECD (2008), Performance Budgeting: A Users’ Guide, https://www.oecd.org/gov/budgeting/Performance-Budgeting-Guide.pdf (accessed on 2 August 2019).
[58] OECD-DAC (2009), “Guidelines for Project and Programme Evaluations”, https://www.entwicklung.at/fileadmin/user_upload/Dokumente/Projektabwicklung/Englisch/Guidelines_for_Project_and_Progamme_Evaluations.PDF (accessed on 20 September 2019).
[26] OECD-DAC (2002), Glossary of Key Terms in Evaluation and Results Based Management, https://www.oecd.org/dac/evaluation/18074294.pdf (accessed on 18 June 2019).
[32] Parkhurst, J. (2017), The politics of evidence : from evidence-based policy to the good governance of evidence, Routledge, London, http://researchonline.lshtm.ac.uk/3298900/ (accessed on 23 November 2018).
[56] Picciotto, R. (2013), “Evaluation Independence in Organizations”, Journal of MultiDisciplinary Evaluation, Vol. 9/20, p. 15.
[33] Poder Ejecutivo Nacional (2018), Decreto 292/2018: Evaluación de Políticas y Programas Sociales, 11-04-2018, http://servicios.infoleg.gob.ar/infolegInternet/verNorma.do?id=308653 (accessed on 8 July 2019).
[22] Poder Ejecutivo Nacional de Argentina (2018), Decreto 292/2018: Evaluación de Políticas y Programas Sociales.
[52] Rutter, J. (2012), Evidence and Evaluation in Policy making, Institute for Government, https://www.instituteforgovernment.org.uk/sites/default/files/publications/evidence%20and%20evaluation%20in%20template_final_0.pdf.
[39] Schoenefeld, J. and A. Jordan (2017), “Governing policy evaluation? Towards a new typology”, Evaluation, Vol. 23/3, pp. 274-293, https://doi.org/10.1177/1356389017715366.
[14] Smismans, S. (2015), “Policy Evaluation in the EU: The Challenges of Linking Ex Ante and Ex Post Appraisal”, Symposium on Policy Evaluation in the EU, https://doi.org/10.1017/S1867299X00004244.
[34] Smismans, S. (2015), “Policy Evaluation in the EU: The Challenges of Linking Ex Ante and Ex Post Appraisal”, Symposium on Policy Evaluation in the EU, https://doi.org/10.1017/S1867299X00004244.
[46] Steuerle, E. and L. Jackson (2016), Advancing the power of economic evidence to inform investments in children, youth, and families, National Academies Press, https://doi.org/10.17226/23481.
[36] The European Network for Rural Development (2014), The Ex Ante Evaluation 2014-2020 RDPs, https://enrd.ec.europa.eu/evaluation/publications/guidelines-ex-ante-evaluation-2014-2020-rdps_en (accessed on 29 January 2020).
[15] The World Bank (2018), Spending Review Manual: Bulgaria.
[44] Vammalle, C. and A. Ruiz Rivadeneira (2017), “Budgeting in Chile”, OECD Journal on Budgeting, Vol. 16/3, https://dx.doi.org/10.1787/budget-16-5jfw22b3c0r3.
[16] Walker, K. and K. Moore (2011), Performance Management and Evaluation: What’s the difference?, https://www.childtrends.org/wp-content/uploads/2013/06/2011-02PerformMgmt.pdf (accessed on 18 July 2019).
[40] Weiss, C. (1993), “Where politics and evaluation research meet”, Evaluation Practice, Vol. 14/1, pp. 93-106, https://doi.org/10.1016/0886-1633(93)90046-R.
Note
← 1. This definition is adapted from the Open Government: The Global Context and the Way Forward (2016[43]), which is based on “OECD-DAC Glossary” in Guidelines for Project and Programme Evaluation (2009[84])
This document, as well as any data and map included herein, are without prejudice to the status of or sovereignty over any territory, to the delimitation of international frontiers and boundaries and to the name of any territory, city or area. Extracts from publications may be subject to additional disclaimers, which are set out in the complete version of the publication, available at the link provided.
https://doi.org/10.1787/89b1577d-en
© OECD 2020
The use of this work, whether digital or print, is governed by the Terms and Conditions to be found at http://www.oecd.org/termsandconditions.
