6. Provision 5: Monitoring and evaluation
Develop and implement monitoring, evaluation and learning mechanisms for open government strategies and initiatives by:
(i) Identifying institutional actors to be in charge of collecting and disseminating up-to date and reliable information and data in an open format;
(ii) Developing comparable indicators to measure processes, outputs, outcomes and impact in collaboration with stakeholders;
and (iii) Fostering a culture of monitoring, evaluation and learning among public officials by increasing their capacity to regularly conduct exercises for these purposes in collaboration with relevant stakeholders.
Given their multidimensional and cross-cutting nature, open government policies are inherently difficult to monitor and evaluate (OECD, 2019[1]). Notwithstanding this complexity, the necessity of being able to prove the positive impacts of open government reforms, including a more concrete understanding of their dynamics and effects, has made monitoring and evaluation (M&E) particularly relevant (OECD, 2019[1]). Solid M&E mechanisms can help ensure that policies are achieving their intended goals, contribute to the identification of policy design and implementation barriers and orient policy choices by building on past experiences. M&E is also instrumental to initiating changes and communicating policy results in a timely and accessible manner (OECD, 2019[1]). Last but not least, by feeding into further policy design, M&E results can improve policy effectiveness and value for money (OECD, 2016[2]).
As outlined under the assessment of provision 3, most Open Government Offices (Offices that co-ordinate the OGP Action Plan) were initially set up to co-ordinate the OGP Action Plan cycle but have over time taken over a mandate for other substantive responsibilities relating to openness. Moreover, these offices continue to have a broad set of responsibilities for the OGP Action Plan. This often includes monitoring the implementation of the OGP Action Plan (87.9%) and evaluating its impact (84.8%) (OECD, 2020[3]). Open Government Offices are often not the only actors responsible for collecting and disseminating information for the purpose of monitoring the open government agenda (also see the assessment of provision 4). In fact, across Respondents, the most important responsibility of Multi-Stakeholder Fora is to monitor the implementation of the action plan (88.9%).
Brazil, the coordinator of each commitment included in the OGP Action Plan needs to prepare an Execution Status Report (CSR) every two months. In addition, monitoring meetings take place for every commitment every three months and a general meeting with all commitments coordinators is organised every six months. These monitoring meetings may also involve civil society stakeholders. The Open Government Portal of the Controller General of the Union (CGU) serves as the main mechanism to display the information collected through the monitoring meetings and the CSRs. For each commitment, the Portal provides information on associated milestones, the percentage of execution of each milestone, commitment-related information and documents as well as the minutes of the monitoring meetings of each commitment.
Source: OECD (2022[4]).
Only a small number of Adherents have designed indicators in the field of open government.
The design of indicators to measure the implementation of open government policies and practices can be a key input for analytical work (OECD, 2011[5]). For the time being, most existing indicators in Adherents focus only on inputs and/or processes and they are often linked to Adherents’ OGP-agendas. A very small number of Adherents have designed indicators to measure their levels of openness in an integrated manner. A notable example in this regard are Mexico’s baseline indicators on open government (Box 6.2)
Mexico’s Open Government Metrics were developed by the Centre for Economic Research and Teaching (CIDE) and were based on an initiative of the National Institute for Transparency, Access to Information and Personal Data Protection (INAI). The metrics are designed as a baseline to measure the current state of the National System of Transparency, Access to Information and Protection of Personal Data (SNT) and its open government and transparency policies. Aiming to be an “x-ray of the starting point of the open government policy of the Mexican State” at the national and subnational level, its focus goes beyond measuring the compliance with regulations, and aims to capture performance information on the outcomes of open government and transparency policies from the perspective of both government and citizens.
The metrics start with an operational definition of open government structured around two dimensions: transparency and public participation. Each dimension is approached from two perspectives: government and citizens.
The CIDE team developed an Open Government Index, consisting of measurements of transparency and participation from the perspective of both government and citizens. The construction of these indexes involved the analysis of existing regulations, a review of government websites and user simulations, including information requests. The Metrics survey included a sample of 908 governmental bodies at the national and subnational level; 754 portals were reviewed and 3 635 requests for information were sent. The resulting Open Government Index of Mexico was 0.39 (on a scale of 0 to 1). The index showed that the transparency dimension has a much higher value (0.50) than the participation dimension (0.28).
Source: INAI (2017[6]).
In order to foster the move towards better measurements of open government reforms and collect better and more relevant data, Adherents could start developing maturity models on open government. To be useful, these models need to be based on a shared understanding of what different stages of openness look like. This implies finding an answer to the questions When can a public institution be considered fully open? What does being closed imply? The OECD is currently actively supporting several Adherents, including Canada, Italy and Romania in the development of their first open government maturity models (Box 6.3).
Open Government Maturity Models can:
Set a baseline standard of what good practices in the field of open government look like.
Allow public institutions to assess their levels of openness at a given point in time and identify where they are situated in relation to national good practice.
Allow building a coherent and flexible trajectory towards high levels of maturity, adaptable to the situation of each public institution.
Show the stages of this progression and the necessary achievements that are useful at each stage and consolidate the passage into the subsequent stages.
Allow for comparison between public institutions within a defined framework.
Source: (OECD, 2022[4])
Several indices have already been developed within the OECD framework and beyond (Box 6.5) that relate to open government. As a response to the Council’s instruction, work on measuring open government reforms and developing indicators has been developed by the WPOG, including a paper titled “A Roadmap for Assessing the Impact of Open Government Reform” that establishes an “OECD Framework for Assessing the Openness of Governments” (OECD, 2020[7]). On this basis, three different types of indicators are currently under development, with the first set, the Open Government Dashboard, launched in 2021. The OECD Citizen Participation Index could be available from 2023/24 (Box 6.4) and the Results Indicators from 2024/25. In particular, the Index would be a cross-cutting indicator focusing on outputs, i.e., the actual implementation of reforms that aim to foster openness, participation and representation.
The OECD Recommendation on Open Government (OECD, 2017[8]) instructs the PGC to develop indicators on open government “to increase the availability of standardised and comparable data on inputs, processes, outputs, outcomes and impacts”. Based on the OECD Framework for Assessing the Openness of Government (OECD, 2020[7]), the OECD proposes three sets of indicators in the field of open government as part of a coherent theory of change. Among them is the OECD Citizen Participation Index which aims to assess the results (“outputs”) of open government policies.
The OECD Citizen Participation Index is one of the key deliverables under pillar 2 of the OECD Reinforcing Democracy Initiative and its Action Plan on Enhancing Representation, Participation, and Openness in Public Life (OECD/LEGAL/0484, Annex B). It will measure openness along five mutually reinforcing dimensions:
“How government informs” measuring the extent to which key information and data can be found, understood, used and re-used by citizens. It focuses on information that have a bearing on citizens’ trust in public institutions (e.g., budget information).
“How government responds” focuses on key mechanisms for citizens to trigger a response from governments, whether to request public information, demand answerability on a specific public problem, or suggest a policy priority (e.g., petitions).
“How government interacts” measuring the actual interaction between governments and citizens in terms of availability, accessibility and impact of interaction mechanisms.
“How government represents” considering how representative government is in terms of the diversity in the composition of elected bodies and the civil service.
“How government protects” evaluating the extent to which governments protect and promote fundamental civic freedoms and political rights.
The objective of the Index is to go beyond a focus on the existence of certain government mechanisms by instead capturing their implications from a citizens’ perspective and therefore the effect on openness in practice.
The index is currently under development, with a first pilot to be expected in the end of 2023. In its first instance, the Index will primarily be based on data collected through existing surveys of the OECD Public Governance Directorate (Survey on Open Government, OURdata Survey, Regulatory Indicators Surveys and others). Depending on availability and quality, data from administrative records held by OECD countries may be included. Eventually, data from other relevant parts of the OECD could be integrated into the Index.
The Rule of Law Index (World Justice Project, n.d.[9]) sub-dimension “Open Government” assesses the “extent to which a government shares information, empowers people with tools to hold the government accountable, and fosters citizen participation in public policy deliberations”.
The Civil liberties index (Pemstein et al., 2021[10])evaluates the “absence of physical violence committed by government agents and the absence of constraints of private liberties and political liberties by the government” (Coppedge et al., 2021, p. 292[11]).
The Global Right to Information Rating evaluates legal frameworks on access to information (Centre for Law and Democracy, n.d.[12]).
The Government Transparency Index (ERCAS, 2021[13]) combines elements of de jure and de facto transparency that are related to corruption.
The Open Budget Index (International Budget Partnership, 2020[14]) evaluates the central government’s income and spending.
The OECD’s OURData Index (OECD, 2020[15]) provides scores regarding Data Availability, Accessibility and Promoting Awareness and Re-Use of open government data.
The OECD Indicators of Regulatory Policy and Governance (iREG), evaluates countries stakeholder engagement during the development of laws and subordinate regulations (OECD, 2021[16]).
The Accountability Index (Lührmann, Marquardt and Mechkova, 2020[17]) assesses accountability understood as “constraints on the government’s use of political power through requirements for justification for its actions and potential sanctions” (Coppedge et al., 2021, p. 285[11]).
The OECD Digital Government Index (OECD, 2020[18]) evaluates, among others, the extent to which the open by default principle is realised, i.e., whether a country makes government data and policy-making processes available to the public.
The International Institute for Democracy and Electoral Assistance’s Global State of Democracy Indices (International IDEA, n.d.[19]) indicate democratic trends across 29 aspects of democracy. Out of the five areas that are conceptualised as contributing to a functioning democracy, especially ‘Participatory Engagement’ and ‘Fundamental Rights’ are relevant for Open Government.
Bertelsmann Stiftung’s Transformation Index (BTI, n.d.[20]) assesses the progress of developing countries and countries in transition towards democracy and a market economy.
The United Nation’s E-Participation Index (United Nations, n.d.[21]) evaluates government’s use of online services to facilitate provision of information by governments to citizens (“e-information sharing”), interaction with stakeholders (“e-consultation”) and engagement in decision-making processes (“e-decision making”).
All Adherents monitor (parts of) their open government agendas.
Data from the 2020 OECD Survey on Open Government shows that most of Respondents’ main policy documents on open government have a monitoring system attached to them. Out of 123 policy documents that were submitted, 106 (86.2%) are monitored throughout their implementation. The monitoring can be publicly followed in two-thirds of these cases (67%, 71 policy documents). These public monitoring activities can take different forms: some Adherents provide implementation reports that inform about the progress, as it is for example the case for Ireland’s Open Data Strategy 2017 - 2022. Other Adherents offer visualisations in the form of graphs and dashboards, for example such a tool is offered to track the implementation of Israel’s National Digital Program.
In addition to the monitoring conducted for their main policy documents on open government, Respondents also employ different monitoring tools and mechanisms focusing on specific policies and practices that are part of a wider open government agenda, such as access to information, open government data and public sector integrity. Notably, many Adherents are well aware of public institutions’ and/or subnational governments’ degree of compliance with existing access to information legislation (i.e., what percentage of institutions proactively publish the information that is required by the law, etc.). For example, in Canada, the Open Government Portal includes a dedicated section providing statistics on the Access to Information and Privacy Acts, including data on requests received during the identified period and requests completed or carried forward to the next period.
The system to monitor the implementation of OGP Action Plans is robust in most Adherents that are part of the OGP.
The monitoring and evaluation of the implementation of countries’ OGP Action Plans usually takes place through the OGP’s monitoring mechanisms, consisting of the independent reporting mechanism (IRM) and the self-assessment reports (OGP, n.d.[22]; OGP, n.d.[23]).While these reports provide useful inputs to the OGP-process, they do not (and do not aim to) allow for the monitoring and evaluation of the wider open government agenda (OECD, 2022[4]). Both the IRM report and the self-assessment report only focus on elements relating to the action plan (e.g., How inclusive was the co-creation process? What is the transformative potential of the commitments? etc.) (OECD, 2022[4]).
Self-assessment report: During the two-year National Action Plan (NAP) cycle, governments will produce yearly self-assessment reports. In order to minimise the administrative burden, the two self-assessment reports will have similar content to one another, differing primarily in terms of the time period covered. The mid-term self-assessment should focus on the development of the NAP, the consultation process, the relevance and ambitiousness of the commitments and progress to date. The end-of-term self-assessment should focus on the results of the reforms completed in the NAP, consultation during implementation and lessons learned. The development of the self-assessment reports must include a two-week public consultation period, as stipulated in the OGP Guidelines.
Independent reporting mechanism: The Independent Reporting Mechanism (IRM) is a key means by which all stakeholders can track OGP-progress in participating countries. The IRM produces annual independent progress reports for each country participating in the Open Government Partnership. The reports assess governments on the development and implementation of OGP Action Plans, track their progress in fulfilling open government principles and make technical recommendations for improvements. These reports are intended to stimulate dialogue and promote accountability between member governments and citizens.
Source: OGP (n.d.[22]); OGP (OGP, n.d.[23]).
While these tracking systems are instrumental for Adherents to oversee the implementation of their OGP Action Plans, in most of the cases the information that is included only allows users to ascertain whether or not an activity/process took place (e.g., Was the event organised?). The system does not involve systematic data collection to assess performance (e.g., by tracking the resources used to implement an activity or its results) or outcomes and impacts.
Usually, a range of public institutions are involved in monitoring the OGP Action Plan implementation. In 20 cases (87% of Respondents), other central/federal government institutions contribute, followed by the Prime Minister’s / President’s / Cabinet office (six, 26.1% of Respondents) and subnational governments and independent public institutions (respectively six, 26.1% of Respondents). Only rarely, institutions from other branches of power join the monitoring activities (Figure 6.1).
In some cases, the monitoring of the OGP agenda also contributes to other monitoring efforts of the government. Respectively 34.8% of Respondents (eight Adherents) report that the OGP monitoring feeds into the monitoring of the country’s Sustainable Development Goals (SDGs) or the Transparency/Anti-Corruption Strategy. The monitoring of the OGP Action Plan does not contribute to the monitoring of other policies in only 26.1% of Respondents (Figure 6.2).
Evaluating the processes and impacts of open government agendas is a relatively new area of interest among policy makers and researchers and a shared challenge across Adherents (OECD, 2019[1]). For the time being, few Adherents have conducted holistic process evaluations of their open government agendas (see Box 6.7 for an example from Canada). Respondents to the Survey on Open Government report that a process evaluation was conducted for 21.2% of 118 open government policy documents while impact evaluations were performed for only 7.8% of 116 policy documents (Figure 6.3).
Moving forward, Adherents could put a greater emphasis on evaluating their open government agendas and specific initiatives. Both process and impact evaluations could be conducted on a more frequent basis as a means to move open government agendas towards outcomes and impacts. In this regard, the evaluation of Canada’s Open Government Programme provides some interesting insights.
Canada undertook an ambitious evaluation of its open government efforts, the Evaluation of the Open Government Programme (hereafter “the Evaluation”) which was prepared by the Internal Audit and Evaluation Bureau for the Performance Measurement and Evaluation Committee of the Government of Canada. The Evaluation assessed the relevance and effectiveness of the Open Government Programme, covering fiscal years 2016-2017 to 2018-19.
The Evaluation found that Canada’s open government agenda has yielded positive results, as certain reforms at the federal level of government have “increased public access to government data and information” and “strengthened partnerships with civil society, Indigenous groups and other governments”, among other benefits. It also highlighted that “there is a need for a strong vision of open government in the Government of Canada”. As its primary action to implement this recommendation made in the Evaluation, the Treasury Board Secretariat decided to design a Federal Open Government Strategy (OGS) as the main guiding document on open government for the federal government. The OECD supported Canada in the successful design and implementation of its first whole-of-government Open Government Strategy through the OECD Open Government Scan (forthcoming).
Source: OECD (2023[24]).
The implementation of provision 5 is advancing relatively slowly. In particular:
5.1: All Adherents monitor and collect data on different open government policies and practices (e.g., on the implementation of their access to information legislations). Moreover, the systems that Adherents that are part of the OGP have established to monitor the implementation of OGP Action Plans are generally robust. However, most activities focus on monitoring processes in terms of the completion of milestones, rather than broader effects of reforms.
5.2: Adherents only conduct evaluations of their open government agenda on an ad hoc basis, if at all. In particular, impact evaluations of Adherents’ main policy documents on open government rarely take place. Over the next implementation period, Adherents could consider putting further efforts in collecting better and more robust data and evidence on the implementation of their open government policies and practices, including their broader effects, for example by designing dedicated open government indicators. In addition, Adherents could focus the efforts on moving towards evaluating outcomes and impacts of open government reforms.
Moreover, while provision 5 focuses on the development and implementation of monitoring, evaluation and learning mechanisms for open government strategies and initiatives, it is today widely recognised that open government approaches are highly relevant for the monitoring and evaluation of any government strategy or initiative. In practice, this for example means making results of monitoring reports and evaluations publicly available in a user-friendly format and involving citizens and stakeholders in monitoring activities and in preparing policy evaluations in any policy area.
References
[20] BTI (n.d.), The Transformation Index, https://bti-project.org/en/?&cb=00000.
[12] Centre for Law and Democracy (n.d.), Global Right to Information Rating, https://www.rti-rating.org/.
[11] Coppedge, M. et al. (2021), V-Dem Codebook v11.1.
[13] ERCAS (2021), Government Transparency Index, https://www.againstcorruption.eu/ercas-projects/transparencyindex/.
[6] INAI (2017), Resultados Edición 2017, http://eventos.inai.org.mx/metricasga/index.php/descargables (accessed on 11 January 2019).
[14] International Budget Partnership (2020), Open Budget Survey 2019, https://www.internationalbudget.org/open-budget-survey/open-budget-survey-2019-0.
[19] International IDEA (n.d.), Global State of Democracy Indices, https://www.idea.int/data-tools/tools/global-state-democracy-indices.
[17] Lührmann, A., K. Marquardt and V. Mechkova (2020), “Constraining Governments: New Indices of Vertical, Horizontal, and Diagonal Accountability”, American Political Science Review, Vol. 3/114, pp. 811-820.
[24] OECD (2023), Open Government Scan of Canada, Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/1290a7ef-en.
[4] OECD (2022), Open Government Review of Brazil: Towards an Integrated Open Government Agenda, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/3f9009d4-en.
[16] OECD (2021), Indicators of Regulatory Policy and Governance, https://qdd.oecd.org/subject.aspx?Subject=GOV_REG.
[7] OECD (2020), A Roadmap for Assessing the Impact of Open Government Reform, WPOG Working Paper, GOVPGCOG20205, https://one.oecd.org/document/GOV/PGC/OG(2020)5/REV1/En/pdf#:~:text=A%20Roadmap%20for%20Assessing%20the%20Impact%20of%20Open%20Government%20Reform,-This%20document%20presents&text=This%20revised%20document%20incorporates%20key,it%20will%20subsequently%20b.
[18] OECD (2020), Digital Government Index: 2019 results.
[15] OECD (2020), OECD Open, Useful and Re-usable data (OURdata) Index: 2019, https://www.oecd.org/gov/digital-government/policy-paper-ourdata-index-2019.htm.
[3] OECD (2020), OECD Survey on Open Government.
[1] OECD (2019), Open Government in Argentina, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/1988ccef-en.
[8] OECD (2017), Recommendation of the Council on Open Government, https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0438 (accessed on 23 August 2021).
[2] OECD (2016), Open Government: The Global Context and the Way Forward, OECD Publishing, Paris, https://doi.org/10.1787/9789264268104-en.
[5] OECD (2011), Quality Framework and Guidelines for OECD Statistical Activities, OECD, http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=std/qfs(2011)1&doclanguage=en.
[23] OGP (n.d.), Independent Reporting Mechanism, https://www.opengovpartnership.org/people/independent-reporting-mechanism-irm/.
[22] OGP (n.d.), Self-Assessment Process, http://www.opengovpartnership.org/how-it-works/self-assessment-process (accessed on January 2019).
[10] Pemstein, D. et al. (2021), The V-Dem Measurement Model: Latent Variable Analysis for Cross-National and Cross-Temporal Expert-Coded Data, https://ssrn.com/abstract=3395892.
[21] United Nations (n.d.), E-Participation Index, https://publicadministration.un.org/egovkb/en-us/About/Overview/E-Participation-Index.
[9] World Justice Project (n.d.), , https://worldjusticeproject.org/rule-of-law-index/global.