Executive Summary
Policy evaluation contributes fundamentally to sound public governance. It can help governments improve the design and implementation of public policies that can, in turn, lead to prosperity for their country and well-being for its citizens. Policy evaluation contributes to promoting public accountability, learning and increased public sector effectiveness through improved decision-making.
The report provides a broad analysis across 42 countries of the institutionalisation of policy evaluation, its quality and use. This systemic perspective allows for a full discussion of how evaluation can contribute to the policy cycle as well as the tools that rely upon evaluation, such as budgeting and regulation. A sound institutional set-up can help align isolated and unplanned evaluation efforts into more formal and systematic approaches. Promoting both better quality and use can have a greater impact in ensuring that evaluation fully achieves its purpose.
Generally, countries show strong commitment to policy evaluation. Some countries have embedded policy evaluations in their constitutions, and around two-thirds of responding countries have developed some kind of legal framework for policy evaluation. Similarly, most countries have adopted guidelines on policy evaluation applicable across government.
The term “evaluation” can cover a range of practices. Around two-thirds of the countries surveyed have at least one definition of evaluation. While these definitions of policy evaluation often reflect the specificities of a country’s institutional setting, common concepts can be found across definitions, in particular regarding what should be measured (policies, programmes, plans, reforms), why an evaluation should be conducted (aims), and when (ex-ante or ex-post).
Most countries face significant challenges in promoting policy evaluation across government. These challenges are mainly related to the limited use of evaluation results in policy making, the absence of a coherent whole-of-government strategy for policy evaluation, and the lack of human resources – whether in terms of skills, capacity or capability.
Sound institutional set-ups can provide incentives to ensure that evaluations are effectively conducted. They can promote transparency and accountability in the management of evaluations, and contribute to improving both the comparability and consistency of results as well as the use of results in policy processes. For this reason, countries are using a range of institutional approaches to anchor policy evaluations in their governance apparatus. They may also co-ordinate their approaches and frameworks for policy evaluation with those related to evidence and data governance.
In addition to legal frameworks, about half of the countries have also developed policy frameworks that provide principles and strategic direction for evaluations. Such policy frameworks provide high-level guidance and clarity for institutions by outlining overarching best practices and goals.
The centre of government provides strategic direction for policy evaluation in two-thirds of the countries surveyed. As such, the centre plays a crucial role in embedding a whole-of-government approach to policy evaluation and it often has the broadest mandate to do so. Ministries of finance also have significant responsibility in 60% of countries. These results show the strong economic impetus for policy evaluation and the close connection between incentives to enhance the quality of public expenditure and incentives to deliver results. Ministries of public sector reform and planning also play a role in some countries, along with autonomous agencies. Agencies are often a source of good evaluation practices, particularly in the Nordic countries, even if their role remains decentralised. They are generally well placed to conduct independent, transparent and accountable policy evaluations.
Co-ordination mechanisms such as commissions and integrated cross-departmental services, when they exist, can help strengthen evaluation systems, for example in terms of staffing and capabilities. Locating the office in charge of policy evaluation close to political decision-making power may allow it to be more effective in commissioning policy evaluations and following up on commitments by ministries. At the same time, ensuring that evaluation systems are independent, transparent and accountable can help bolster citizens’ trust in the results.
Policy evaluation can only be truly effective if it is of high quality and its results are used. Quality control and quality assurance are key to ensuring the robustness of evaluations. Standards play an important role in quality assurance, and guidelines exist in three-quarters of the countries surveyed. Other quality control mechanisms, such as peer review, systematic reviews, and competency requirements for evaluators are relatively common. Up to half of the countries organise training for evaluators, and a majority recognise the importance of developing evaluator competences.
While using the results of evaluations is a challenge, it is paramount to achieving impact. Countries are relying on a range of organisational and institutional mechanisms to promote their use and to create a marketplace for evaluations. Some of these measures consist of a co-ordination platform (in about one-third of the sample) or a management response (in one-quarter of the sample). Rating and grading systems are also used to a limited extent. Finally, most mechanisms to promote skills and competences are aimed at evaluators and managers, rather than to improve the capacity of policy makers and decision makers to use evidence.
The heterogeneity of country approaches suggests that the creation of an evaluation marketplace depends on the local political and cultural context. Evaluation can also be embedded into policy planning and policy-making processes. Half of the countries incorporate evaluation results in the budget cycle. In particular, many OECD countries use spending reviews. The area of regulatory policy is also one where the use of evaluation is well developed, with significant requirements for evaluation embedded in the regulatory impact assessment process.
The role of institutions outside of the executive branch remains limited, both in the promotion of quality and use of evaluations, beyond their involvement in the budgetary cycle – although Supreme Audit Institutions are a key actor in terms of the supply of evaluations.
