Chapter 4. Monitoring, evaluation and audit institutions
Monitoring and evaluation systems are essential for assessing to what extent policies and resource allocations for implementing the SDGs result in meaningful outcomes. This chapter illustrates ways in which solid monitoring and evaluation systems, and the strategic use of the information they generate throughout the policy and budget cycle, can foster a range of objectives such as policies’ value for money, accountability, and overall transparency of a policy-making process. Sound monitoring and evaluation systems become even more important given the complex and interconnected nature of the SDGs.
Monitoring and evaluation for effective delivery of the SDGs
Given the aspirational and global nature of the SDGs, governments are called to set targets, taking into account different circumstances, capacities and priorities. A whole-of-government strategic approach and strong stakeholder engagement can help to set these targets in a meaningful way and foster ownership across society. In addition, incorporating the SDGs into the budget process guarantees that resource allocation reflects policy priorities and supports policy coherence across policy goals and electoral cycles. When policy-makers want to assess whether their policy targets and the subsequent resource allocation result in meaningful changes and achievements, monitoring and evaluation (M&E) come into play.
Solid M&E and the strategic use of the information it generates throughout the policy and budget cycle can foster a range of objectives such as policies’ value for money, accountability and overall transparency of a policy-making process. Taking into account the complex and interconnected nature of the SDGs, a sound M&E system is of particular importance. In addition, considerations such as inclusiveness (the ‘leaving no-one behind’ principle) and sustainability – as opposed to more traditional considerations such as efficiency or effectiveness – ask for innovative approaches when assessing the merit and achievements of policy initiatives in support of the SDGs.
Collaborating with a host of other international organisations and governments from 38 countries within the framework of the Praia Group on Governance statistics, the OECD has been at the forefront of promoting evidence-based policy recommendations for improving monitoring and evaluation in SDG governance (Praia Group, 2015[1]).
Notwithstanding their complementarity, monitoring and evaluation are two different things. Policy monitoring refers to a continuous function that uses systematic data collection on specific indicators to provide policy makers and stakeholders with information regarding progress and achievements of an ongoing public policy initiative and/or the use of allocated funds (OECD, 2019[2]; OECD, 2016[3]). It contributes to planning and operational decision-making, as it provides evidence to measure performance and can help to raise specific questions in order to identify implementation delays or bottlenecks. It can also strengthen accountability regarding the use of resources, the efficiency of internal management processes, or the outputs of a given policy initiative.
Policy evaluation refers to a structured, in-depth assessment of an intended, ongoing or completed policy initiative. The aim is to determine the relevance and fulfilment of policy objectives, as well as to assess dimensions such as public policies’ efficiency, effectiveness, impact or sustainability. As such, policy evaluation focuses on determining the worth or significance of a policy initiative (OECD, 2019[2]; OECD, 2016[3]). It serves three main purposes that are highly relevant for reaching the SDGs. Firstly, it fosters learning by helping policy makers to understand why and how a policy was (ex-post) - or is expected to be (ex-ante) - successful or not. Secondly, it contributes to strategic decision-making, by providing insights on how to improve the links between policy decisions and outcomes. And finally, it promotes accountability as it provides not only government, but also citizens and a broad range of stakeholders, with information whether policy initiatives, including the financial resources mobilised, are producing the expected results.
While governments are responsible for setting up their M&E system, it is important to acknowledge (a) a division of labour within government (e.g. Centre of Government versus sector actors engaging in M&E activities), which can result in different institutional set-ups; and (b) M&E responsibilities of actors outside the executive (e.g. Parliament and Supreme Audit Institutions).
Monitoring mechanisms to keep track of SDG implementation
Performance monitoring can be defined as “the continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing policy or reform initiative with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds” (OECD, 2016[3]). It is a critical tool to inform governments on how they are progressing along the path to achieving their stated policy goals.
Countries have developed different monitoring mechanisms to ensure efficient policy-making. The monitoring of financial performance and budget execution can help governments to assess the effectiveness of public spending against their strategic objectives and adjust the allocation of financial resources in case of unforeseen implementation challenges or misspending. Linkages between the government’s strategic objectives (as identified for example in its plurennial development planning), spending results areas in the national budget and clear performance goals help to ensure meaningful monitoring.
These links take on added importance as governments pursue the translation of the SDGs into their national contexts through a triangulation exercise that aligns the SDGs with national strategic planning objectives and spending results areas in the national budget. Translating the SDGs into national development goals complemented with key performance targets and indicators can enable the government to reflect the SDGs in their own strategic planning. Governments can then align their national strategic objectives (now reflecting the SDGs) with the spending results areas in the national budget. This has the potential to enable governments to assess how their financial allocations and spending decisions are advancing the country down the path of achieving the SDGs in a way that reflects national development priorities and objectives.
In many countries, the Centre of Government (CoG) plays a direct role in overseeing the monitoring of the SDGs’ implementation across government and the communication of progress and achievements, though different institutional set-ups can be considered. Some of the potential challenges for countries to consider include:
-
The blending of pre-existing national policy priorities (e.g. national development goals) and their established indicators with SDG targets and indicators.
-
The coordination between CoG institutions and sector ministries (i.e. issues of ownership of the monitoring and reporting cycle, knowledge asymmetry regarding the performance and deliverables to be monitored etc.).
-
The capacity to collect reliable data in a cost-effective way and ensure the large flow of generated information can be easily absorbed.
-
The need to make relevant information publicly available and engage in a debate regarding the observed performance with a wide range of actors going well beyond government (e.g. engagement with citizens and other stakeholders, NGOs, Parliament, etc.).
-
The need to ‘close’ the policy cycle, or in other words act upon the information collected through the SDG implementation monitoring process, by adjusting planning and budgeting decisions accordingly.
In 2016, the United States government introduced the online platform “U.S. National Statistics for the UN Sustainable Development Goals”. The platform presents relevant national statistical data associated to 244 indicators across the 17 SDGs. This is the result of collaborative work between the U.S. Office of Management and Budget, the Office of Information and Regulatory Affairs, the U.S. Departments of State, the Office of International Organizations, the U.S. General Services Administration and the U.S. Office of Science and Technology Policy.
The objective of the initiative is to make data available and accessible to all citizens. In addition, the platform is an open source and allows multiple users to use it concurrently. It can be used and adapted by other nations and localities according to their own reporting needs. The United Kingdom was the first to clone and customize it, launching their NRP in late 2017. The US and UK have since combined best features into “Open SDG” and continues to collaborate to incorporate other features such as SDMX (available soon). This will enable other countries that have also adapted the platform (e.g. Poland, Ghana, Armenia) to simply transfer newly-developed features.
The platform includes instructions for developers, data providers and policy makers on how to customize the site. Currently the site presents information for 41% of all the indicators. It is work in progress and other data sources are being explored. It is available in English and Spanish and will become available in French later this year, eventually also being translated to the other official UN languages.
By making data from the public sector available, this initiative from the United States government fosters transparency and accountability and has the potential to increase stakeholder participation. Moreover, the initiative provides other nations a valuable tool to keep track on their own progress to achieve the SDGs.
Sources: U.S. Government (n.d.[4]), U.S. National Statistics for the U.N. Sustainable Development Goals, https://sdg.data.gov/ (accessed on 17 May 2°19); IISD (2017[5]), US Presents National SDG Monitoring Platform, https://sdg.iisd.org/news/us-presents-national-sdg-monitoring-platform/.
Monitoring the impact of public procurement on broader policy objectives
Demonstrating the impacts of public procurement in a way that is consistent and reliable across countries is challenging. The OECD study “Productivity in Public Procurement” introduces a framework that systematically assesses the effects of public procurement activity and the success of government policy. The framework has been tested in two OECD member countries, Finland and Chile, to validate its applicability to different types of systems (OECD, 2019[6]). The framework has also been used in Malta to design a set of performance indicators to measure efficiency and effectiveness in the context of the economic impact of public procurement (OECD, 2018[7]) (Box 4.2).
The public procurement system in Malta is in a state of active reform and change. The institutional framework for public procurement has been built over time as part of the change programme. Reforms have been brought about principally in the regulatory realm and as a result there have been some wide-reaching structural changes to organisations. The OECD review of the public procurement processes in Malta identified a number of areas for potential improvements:
-
Increased competition, reduced administrative burdens, shortened duration and increased compliance levels through e-procurement.
-
Increased value from use of framework agreements and/or consolidated contracts.
-
Decreased cost of procurement staff over time as levels of efficiency increase in areas of automation such as e-procurement, the use of framework agreements and consolidated contracts.
-
Decrease in unsuccessful procurement processes.
-
Improved public procurement planning as reflected in decreased time spent on certain parts of the procurement lifecycle (specification development, clarifications, evaluation).
To demonstrate the improved efficiency and effectiveness of the procurement function, it was necessary to create a new set of metrics and define the data that was needed for measurement.
Source: OECD (2018[7]), Public Procurement in Malta: Re-engineering the Department of Contracts, http://www.oecd.org/gov/public-procurement/.
One tool that seeks to inform evidence-based policy-making with a view towards strategic procurement in support of the SDGs is the Methodology for Assessing Procurement Systems (MAPS) (Box 4.3). MAPS is intended to provide a harmonised tool for use in the assessment of public procurement systems. The methodology is designed to enable a country, with or without the support of external partners, to conduct an assessment of its procurement system in order to determine its strengths and weaknesses. The resulting information can serve as the basis for harmonised system development and reform initiatives that can improve capacity and address any weaknesses. The assessment also provides the country with information it can use to monitor the performance of its system and evaluate the success of the reform initiatives in improving performance. By identifying weaknesses in a country’s current system, it also offers external partners information that can help them determine risks to the funds they provide to partner countries.
The MAPS analytical framework consists of a core assessment methodology and six supplementary modules. The MAPS core methodology provides a comprehensive approach for assessing procurement systems. It defines the structure to conduct a country context analysis, presents a refined indicator system for assessing the quality and performance of the system in terms of outcomes and results and describes the key elements of the assessment process. An innovation in the MAPS are quantitative assessment criteria that reflect Key Performance Indicators (KPIs) used by countries to track the performance of their public procurement systems.
Source: ©MAPS (n.d.), The MAPS Analytical Framework, http://www.mapsinitiative.org/ (accessed in 2019).
MAPS was recently revised to match today’s challenges. The new version of MAPS is timely in the wake of the launch of the SDGs. Like the SDGs, MAPS will be relevant for all countries, irrespective of income level or development status. MAPS is related to Goal 12, which calls for the promotion of sustainable procurement practices in line with national priorities and policies, and Goal 16, which calls for effective and accountable institutions. In addition, MAPS is anchored in the 2015 OECD Recommendation of the Council on Public Procurement and is reflective of leading international procurement frameworks such as the United Nations Commission on International Trade Law (UNCITRAL) Model Law on Public Procurement (2011), the European Union (EU) Directives on Public Procurement (2014), and the procurement frameworks used by multilateral development banks, countries and implementing institutions. It provides a holistic assessment framework, establishing the criteria of an effective and efficient procurement system that all countries should strive to achieve.
Policy evaluation to foster strategic decision-making, learning and accountability
While the specific objectives, scope and methodology of individual evaluations may vary widely, policy evaluations in essence analyse the link between policy decisions and outcomes. By aiming to support strategic decision-making, learning and accountability, policy evaluation should be part and parcel of the policy cycle, including for those policies impacting the achievement of the SDGs. However, given the resource-intensive nature of evaluations (both in terms of human and financial means), a structural approach to policy evaluation would still imply a deliberate, strategic choice to carry out specific evaluations and target specific information needs.
The stock of regulations in OECD countries is far larger than the flow, yet scant attention is often paid to regulatory proposals once they have become laws. Ex-post evaluation is thus a crucial tool to ensure that regulations remain fit for purpose, that businesses are not unnecessarily burdened, and that citizens’ lives are protected. Yet despite this, there has only been a minor increase in the number of countries that have formal requirements and a comprehensive methodology in place for ex-post evaluations. Improving the stock of regulation would help ensure that regulations are still relevant, do not impose unnecessary costs on society and do not lead to unintended consequences.
Source: OECD (2018[8]), OECD Regulatory Policy Outlook 2018, https://doi.org/10.1787/9789264303072-en.
Preliminary findings of ongoing OECD research regarding the institutionalisation and governance of policy evaluation across government (OECD, forthcoming[9]) suggest that:
-
A variety of institutional set-ups exist to organise policy evaluation across government (e.g. the coordinating institution could be the Prime Minister’s Office or equivalent, Ministry of Finance, Ministry of Planning, a dedicated agency; sector ministries could have different levels of autonomy to take evaluation-related initiatives).
-
The legal framework for carrying out policy evaluation can vary substantially (e.g. constitutional provisions, legal provisions, dedicated policy on policy evaluation).
-
The use of policy evaluation results in decision-making remains a particular challenge for many countries.
Promoting policy evaluation across the policy cycle entails more than ticking the box that evaluations are produced. Ensuring the systematic production of policy evaluations is a necessary but not sufficient condition to enhancing the quality of public governance and service delivery. Poor quality evaluations will hardly contribute to better learning, higher accountability, or better decision-making and policy design. Likewise, high-quality evaluations may still be completely ignored for actual policy decisions. While fostering an evaluation culture is a long-term endeavour, concrete actions can be taken to promote the relevance and uptake of policy evaluations. These can for instance include the promotion of political commitment and stakeholder engagement.
Given that the quality of policy evaluations is an essential factor to guarantee the robustness and validity of any policy evaluation effort, both quality control (deliverable oriented) and quality assurance (process oriented, i.e. doing the right things in the right way) are important. A preliminary analysis of the OECD Survey regarding the institutionalisation and governance of policy evaluation across government indicates that governments are focusing on the following areas to foster the quality of their policy evaluations:
-
The skills and capacities within the public service to conduct or commission policy evaluations.
-
The role of stakeholders, to ensure that evaluations are targeted properly and that recommendations for improvement are practical and user-centred.
-
The realisation of meta-evaluations and the role of Supreme Audit Institutions to audit the policy evaluation system.
Whereas most countries have established a monitoring framework for SDG implementation (or are in the process of doing so), few countries have addressed the issue of how to evaluate SDG achievements. A review of 43 Voluntary National Reviews refers in this respect to “a general lack of reflection and understanding about how evaluation should be included into the SDGs’ national follow-up and review processes” (Partners for Review, 2018, p. 4[10]). This indicates that mainstreaming evaluation of the SDGs is still largely unchartered territory, with a great potential to be developed strategically in the future. Building upon existing building blocks of a country’s evaluation eco-system might be a good starting point in this respect (as an illustration, see Annex D for the case of Costa Rica, where a National Evaluation Policy with an explicit cross-reference to the SDG agenda was recently developed).
Supreme Audit Institutions and the oversight of the national SDG agenda
Through their traditional role in external oversight of government accounts, Supreme Audit Institutions (SAIs) fulfil a crucial role as key democratic institutions by holding government to account for its use of public resources. In addition to these traditional activities, SAIs may contribute to the formulation, implementation and evaluation of policies. These activities take the form of assessments carried out by the SAIs on key functions of the policy cycle, as well as by the provision of insight and foresight.
Yet, this is not the only channel through which SAIs can support the implementation of the overall 2030 Agenda. The SDGs, by their nature, have a medium to long-term horizon for the achievement of outcomes. Within their traditional role of providing oversight, SAIs are well-equipped to monitor and report on policies for implementing SDGs beyond electoral cycles and assess whether public finances are managed in such a way as to achieve the goals. As to their role in the provision of insight and foresight, SAIs can inform the Centre of Governments and policy-makers at different stages of the policy cycle (formulation, implementation, evaluation and oversight). SAIs can, therefore, through their audits – and consistent with their mandates and priorities – make valuable contributions to national efforts to track progress, monitor implementation and identify improvement opportunities across the full set of the SDGs.
Due to their position as independent institutions that monitor performance across a wide range of policy fields, SAI’s have an overall picture of financial and policy measures taken by various institutions on SDG-relevant issues. They are in a unique position to ensure an integrated and coherent approach to SDG implementation across the economic, social and environmental dimensions of sustainable development and to flag important cross-sectoral issues (UN, 2018[11]).
The International Organization of Supreme Audit Institutions (INTOSAI) has identified four approaches through which INTOSAI and SAIs can contribute to the implementation of SDGs, namely by:
-
Assessing national governments’ preparedness for SDG implementation.
-
Conducting performance audits of key government programmes that contribute to specific SDGs.
-
Contributing to the implementation of SDG 16.
-
Acting as models of transparency and accountability in their own operations.
SAIs can assess the preparedness of national governments to implement, monitor and report on the progress of the SDGs. For example, the Netherlands Court of Audit conducted a ‘preparedness review’ of the government’s commitment to achieve the SDGs. Similarly, the Federal Court of Accounts of Brazil (TCU) conducted several pilot audits to assess the Brazilian government’s readiness to implement specific SDGs, addressing in particular the process of institutionalisation of the SDGs and governance mechanisms within the centre of government, and the monitoring of SDG targets and indicators. SAIs have also shown commitment to taking steps in strengthening their own capabilities to carry out SDG-related audits. For example, in 2016, a coordinated audit involving SAIs from 11 Latin American countries, conducted by the TCU, evaluated governments’ preparedness to implement the SDGs (TCU, 2017[12]), with a focus on target 2.4 (sustainable food production systems).
SAIs can also help ensure that the SDGs are being properly implemented, benefitting the whole of society and restoring public trust in institutions. As an example, the Swedish National Audit Office (NAO) audited the government’s gender equality initiative, recommending the development of an institutional structure to strengthen the government’s gender mainstreaming strategy.
Furthermore, SAIs can support the implementation of SDG 16, which relates to transparent, efficient and accountable institutions. Indeed, SAIs play a critical role in evaluation and oversight through their audits, evaluations and advice, thus holding the government to account for the use of public resources. They produce evidence to inform what works and what does not work, provide insight into potential duplications and fragmentations across government, and can expose where a policy does not serve the public interest or clearly advantages a narrow interest group.
Finally yet importantly, SAIs should act as models of transparency and accountability in their own auditing and reporting operations. This may be particularly challenging in countries where the achievement of the SDGs is most crucial: SAIs may have limited resources, lack of technical capacity, or may be subject to political interference, all of which may limit the effectiveness of the auditing body. Capacity building programmes and knowledge sharing networks among SAIs, such as the INTOSAI Development Initiative, are particularly relevant in this respect.
Drawing on the successful experience of SAI’s, some countries intend to build similar capacities in their Internal Audit institutions, as evidenced by the Argentina case study in Annex D.
Lessons learned from country experiences
Monitoring and Evaluation (M&E) can guide operational and strategic decision-making and furthermore support learning and accountability objectives. As such, countries recognise the importance of integrating M&E in the governance framework for implementing the SDGs. While the executive government has a clear responsibility in designing such an M&E framework, the case studies presented in Annex D show the importance of engaging a broader range of stakeholders and provide examples of the important role other actors (in particular SAIs) can play. The case material furthermore indicates that embedding SDG-focused M&E efforts in broader, ongoing efforts to promote government-wide policy priorities (e.g. in the context of a national development plan) can be helpful to create traction. As a general observation, monitoring efforts appear to have been much more firmly established across governments for the time being, than evaluation.
Effective independent external audit institutions are critical to making sure that governments are operating in an optimal way, without waste or fraud, to deliver better policies and programmes that benefit citizens. The OECD’s international peer reviews have shown that the trend is clearly towards a role for SAIs that focuses on broader governance aims than their traditional oversight role. SAIs increasingly assess the preparedness of government to address long-term economic, social and environmental policy challenges. SAIs can provide a unique, horizontal view on government policies and programmes by providing critical evidence for more informed policy formulation, implementation and evaluation.
The case studies in Annex D show various results from monitoring and evaluation exercises. In Austria, the evaluation by the national Supreme Audit Institution has revealed the lack of coordination regarding SDG implementation across government, and the absence of an SDG perspective in the budgetary process. By contrast, Colombia’s performance is monitored by the National Planning Department (NPD) while the national monitoring (and evaluation) system, SINERGIA, tracks process, output, and outcome indicators set in the National Development Plan (NDP). Similarly, Costa Rica’s National Evaluation Policy will focus on policy evaluation in relation to the SDGs. Other countries, such as Egypt and Luxembourg, have so far only implemented monitoring mechanisms for SDG targets and indicators. Of the countries covered in the case studies, only in Brazil, Finland and the Netherlands has the national Supreme Audit Institutions taken up the role to act as external, independent evaluators of the SDG implementation process. The Ireland case highlights the role of regulatory impact assessments, and, finally, India’s SDG Index shows national efforts to track progress at both national and state level.
The OECD is collaborating with SAIs and INTOSAI to extend its review of external, independent evaluation mechanisms to other countries.
The OECD Recommendation of the Council on Regulatory Policy and Governance is the first international instrument to address regulatory policy, management and governance as a whole-of-government activity. It sets out the measures by which governments can implement or advance regulatory reform, arguing that Regulatory Impact Analysis (RIA) is both a tool and a decision process for informing political decision makers on whether and how to regulate to achieve public policy goals. As a tool supporting decision-making, RIA focuses on ensuring that a systematic and rigorous process of identification and assessment of the potential impacts of government actions is undertaken and on quantifying the expected costs and benefits of a regulatory measure; on assessing the effectiveness of the measure in achieving its policy goals; and on determining whether there are superior alternative approaches available to governments. As a decision process, RIA complements other key elements of regulatory policy, such as public consultation, by developing a better understanding of the likely impact of regulatory options and communicating this information to policy makers, at a time and in a form that can be used to guide regulatory decision-making in relation to both proposed and existing regulations (OECD, 2009).
In addition, drawing from the OECD Network of Economic Regulators, and building on the OECD Framework for Regulatory Policy Evaluation, the OECD Survey on the institutionalisation and governance of policy evaluation, and the OECD Regulatory Policy Outlook, the OECD is well placed to support countries in the design of effective M&E systems, which will also be an integral part of the proposed Global Hub on the Governance for the SDGs. In particular, the Framework for Regulatory Policy Evaluation assists countries in systematically evaluating the design and implementation of regulatory policy against the achievement of strategic regulatory objectives, such as those derived from the SDGs.
References
[5] IISD (2017), US Presents National SDG Monitoring Platform, International Institute for Sustainable Development, https://sdg.iisd.org/news/us-presents-national-sdg-monitoring-platform/.
[2] OECD (2019), Open Government in Biscay, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/e4e1a40c-en.
[6] OECD (2019), Productivity in Public Procurement - A Case Study of Finland: Measuring the Efficiency and Effectiveness of Public Procurement, OECD, Paris, http://www.oecd.org/gov/public-procurement/publications/productivity-public-procurement.pdf.
[8] OECD (2018), OECD Regulatory Policy Outlook 2018, OECD Publishing, Paris, https://doi.org/10.1787/9789264303072-en.
[7] OECD (2018), Public Procurement in Malta: Re-engineering the Department of Contracts, OECD, Paris, http://www.oecd.org/gov/public-procurement/.
[3] OECD (2016), Open Government: The Global Context and the Way Forward, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264268104-en.
[9] OECD (forthcoming), OECD Survey on the Institutionalisation and Governance of Policy Evaluation, OECD, Paris.
[10] Partners for Review (2018), “Comparative analysis of 43 VNRs submitted to the HLPF 2017”, http://www.partners-for-review.de/wp-content/uploads/2018/05/P4R-Comparative-analysis-of-2017-VNRs.pdf.
[1] Praia Group (2015), Praia Group on Governance Statistics, http://ine.cv/praiagroup/index.html (accessed on 17 May 2019).
[12] TCU (2017), SDG Coordinated Audit, https://portal.tcu.gov.br/biblioteca-digital/auditoria-coordenada-ods.htm.
[4] U.S. Government (n.d.), U.S. National Statistics for the U.N. Sustainable Development Goals, https://sdg.data.gov/ (accessed on 17 May 2019).
[11] UN (2018), Working Together: Integration, Institutions and the Sustainable Development Goals, World Public Sector Report 2018, Division for Public Administration and Development Management, Department of Economic and Social Affairs (DPADM), United Nations.