7. Monitoring and evaluating openness: Towards stronger impact of open government reforms
This chapter provides recommendations to support the Government of Romania in enhancing the measurements of the impact that open government reforms are having. In addition to providing recommendations on ways to strengthen ongoing monitoring exercises in different areas of open government, the chapter assists Romania in the creation of an integrated monitoring and evaluation system for the forthcoming Open Government Strategy. It proposes tools to track the implementation of the Strategy and suggests the development of an Open Government Index and/or Open Government Maturity Models.
Solid monitoring and evaluation (M&E)1 mechanisms can help to ensure that policies are achieving the intended goals, contribute to the identification of implementation barriers, and orient policy choices by building on past experiences (OECD, 2019[1]). M&E is also instrumental in initiating changes and communicating policy results in a timely and accessible manner. Given their multidimensional and cross-cutting nature, open government reforms are inherently difficult to monitor and evaluate (OECD, 2020[2]): They commonly involve initiatives in a variety of areas (e.g. citizen participation, access to information, open government data, etc.), and requires the involvement of multiple public and non-public stakeholders from the different levels of government (OECD, 2019[1]). In its consideration of the overall relevance of M&E, the Recommendation of the Council on Open Government (OECD, 2017[3]) accords substantial importance to the monitoring and evaluation of open government strategies and initiatives (Box 7.1).
The creation of more solid M&E systems and the establishment of indicators to measure the impact of open government reforms is a challenge that is shared between all OECD Member and Partner countries. Data collected through the 2020 OECD Survey on Open Government (OECD, 2021[4]) shows that – for the time being – most countries only monitor the implementation of their OGP action plans and collect limited data and evidence on the broader effects of open government initiatives (OECD, forthcoming[5]). Evaluations are mostly conducted on an ad hoc basis, if at all, and few countries have established holistic measurements to assess the outputs, outcomes, and impacts of their open government agendas (OECD, forthcoming[5]).
As part of its process to move towards a more integrated open government agenda and design the country’s first holistic and integrated Open Government Strategy, Romania also committed to foster the evidence base and develop better measurements in the field of open government. In fact, building stronger indicators and fostering the evidence base for open government is one of the overarching objectives of the ongoing collaboration between Romanian and the OECD as part of the EEA/Norway grants project (see also Chapter 3 on the Methodology).
Building upon Provision 5 of the Recommendation of the Council on Open Government (OECD, 2017[3]), this chapter assesses existing measurements in the area of open government in Romania. It finds that Romania has already developed a number of mechanisms to monitor the implementation of different open government policies and practices (e.g. through the monitoring of Law no. 544/2001 on free access to information of public interest) and that the monitoring system of the OGP action plan is relatively established. However, like most OECD Member countries, Romania still lacks a clearer understanding of the wider effects that open government reforms have. The chapter therefore provides recommendations to support the government of Romania in enhancing measurements of openness across the public sector and assist the country in the creation of a monitoring and evaluation system that is suitable to an integrated open government agenda, including by proposing the development of an Open Government Index and/or Open Government Maturity Models.
The chapter benchmarks Romania against OECD practice based on the findings of the 2020 OECD Survey on Open Government (OECD, 2021[4]), and refers to Romanian administrative data and data from the different Surveys the OECD Secretariat conducted in the country.
“Develop and implement monitoring, evaluation and learning mechanisms for open government strategies and initiatives by:
(i) Identifying institutional actors to be in charge of collecting and disseminating up-to-date and reliable information and data in an open format
(ii) Developing comparable indicators to measure processes, outputs, outcomes, and impact in collaboration with stakeholders; and
(iii) Fostering a culture of monitoring, evaluation and learning among public officials by increasing their capacity to regularly conduct exercises for these purposes in collaboration with relevant stakeholders”.
Source: OECD (2017[3]), “Recommendation of the Council on Open Government”, OECD Legal Instruments, OECD/LEGAL/0438, OECD, Paris, https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0438.
Romania shares the challenge of improving monitoring and evaluation of open government reforms with many OECD countries
While OECD Member and Partner countries have made significant progress in fostering the monitoring of open government policies and practices, holistic evaluations are still rarely conducted. Indeed, data from the 2020 OECD Survey on Open Government (OECD, 2021[4]) shows that most of the main policy documents on open government that Adherents2 to the OECD Recommendation on Open Government (hereinafter “Adherents” or “Adhering countries”) have designed have a monitoring system attached to them. Out of 123 open government policy documents that were submitted, 106 (86.2%) are monitored throughout their implementation and the monitoring can be publicly followed in two-thirds of these cases (67%, 71 policy documents).
Adhering countries also employ different monitoring tools and mechanisms focusing on specific open government principles, policies and practices, such as access to information, open government data and public sector integrity (OECD, forthcoming[5]). For example, most countries are well aware of public institutions’ and/or subnational governments’ degree of compliance with their existing access to information legislation (i.e. what percentage of institutions proactively publish the information that is required by the law etc.).
On the contrary, evaluating the impacts of open government reforms is a relatively new area of interest among policymakers and researchers (OECD, 2019[1]). Few OECD Member and Partner countries have conducted holistic assessments to understand the effects that open government reforms have had on citizens’ trust in public institutions, fighting corruption, economic growth, political efficacy, etc. (see Box 7.2 for an example of an evaluation from Canada). Respondents to the 2020 OECD Survey on Open Government report that a process evaluation was conducted for 21.5% of 121 open government policy documents, while impact evaluations were performed for only 8.47% of 118 policy documents (10) (Figure 7.1).
In 2021, the Internal Audit and Evaluation Bureau for the Performance Measurement and Evaluation Committee of the Government of Canada prepared an Evaluation of the Open Government Programme. The Evaluation assessed the relevance and effectiveness of the Open Government Programme, covering fiscal years 2016-2017 to 2018-19. Among other findings, it highlighted that “there is a need for a strong vision of open government in the Government of Canada”. As its primary action to implement these recommendations made in the Evaluation, the Treasury Board Secretariat decided to design a Federal Open Government Strategy (OGS) as the main guiding document on open government for the federal government.
Source: OECD (2023[6]), Open Government Scan of Canada: Designing and Implementing an Open Government Strategy, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/1290a7ef-en.
In Romania, like in most OECD countries, there is currently no integrated system to monitor, evaluate or collect data on open government at the level of the national government or at the level of individual public institutions. However, different tools and mechanisms monitoring specific open government principles, policies and practices, such as access to information and participatory practices, exist. In line with its institutional mandate, the General Secretariat of the Government of Romania is the key public institution when it comes to monitoring and evaluating the implementation of most open government initiatives (Box 7.3).
Taking the dimensions of the OECD Openness Spectrum as a basis (see also Chapter 3 on Methodology), this section assesses the way Romania currently monitors and evaluates the implementation of open government policies and practices. Given the parallel Civic Space Review of Romania (OECD, 2023[7]) and other ongoing OECD Policy Reviews, the section focuses on the way government informs (dimension 1), responds to (dimension 2) and interacts with (dimension 3) citizens and stakeholders, leaving out the way it represents (dimension 4) and protects and promotes civic space (dimension 5).
Source: Author’s own elaboration.
According to its mandate, the General Secretariat of the Government:
monitors and evaluates the application by public authorities and institutions of the legal provisions regarding free access to information of public interest in the public administration.
monitors and evaluates the application by public authorities and institutions of the legal provisions regarding decisional transparency in public administration.
elaborates summaries regarding the annual reports regarding the implementation by the central public administration of the legislation on free access to information of public interest, offering recommendations for improving the legislation and practices in the field;
elaborates summaries of the annual reports regarding the implementation by the central administration of the legislation on decisional transparency in public administration, offering recommendations for improving the legislation and practices in the field.
Source: Government of Romania (2020), Art. 3 point 5, Decision no.137/2020 on the organisation, functioning and attributions of the GSG.
Monitoring and evaluating Romania’s OGP action plans
The monitoring and evaluation of the implementation of countries’ Open Government Partnership (OGP) action plans usually takes place through the OGP’s monitoring mechanisms, consisting of the reports prepared by the independent reporting mechanism (IRM) and self-assessment reports prepared by the government (Box 7.4) (OECD, 2022[8]). An independent expert from the OGP’s IRM usually carries out three evaluations of each OGP action plan: the first in the design phase, the second in the middle of the implementation period and the last at the end of the implementation period.
While both the IRM reports and the self-assessment reports provide useful inputs about countries’ OGP processes, they do not (and do not aim to) allow for the monitoring and evaluation of countries’ whole open government agenda, as they only focus on elements relating to the action plan itself (e.g. How inclusive was the co-creation process? What is the transformative potential of the commitments?, etc.) (OECD, 2022[8]).
Self-assessment report(s): Self-assessment reports are a key element of the Open Government Partnership accountability mechanism. They document the overall progress of the action plan based on the information in the repository/dashboard. They must highlight opportunities and gaps to further improve the delivery of the action plan. While it is only expected for countries to develop one self-assessment report at the end of the implementation of the action plan, some countries prepare a yearly self-assessment report and invite the public to comment and provide feedback on the content of the report. The report needs to be published on the country’s OGP website and can be published on the OGP website, including the comments and how the comments were addressed.
Independent reporting mechanism: The Independent Reporting Mechanism (IRM) is a key means by which all stakeholders can track OGP progress in participating countries. The IRM produces the following for each country participating in OGP: a Co-Creation Brief; an Action Plan Review; and a Results Report. IRM products follow a regular timeline with production running parallel to the OGP action plan cycle.
Source: OGP (2022[9]), OGP NATIONAL HANDBOOK Rules + Guidance for Participants, https://www.opengovpartnership.org/wp-content/uploads/2022/03/OGP-National-Handbook-2022.pdf; OGP (n.d.[10]), INDEPENDENT REPORTING MECHANISM Products and Process, https://www.opengovpartnership.org/irm-products-and-process/ (accessed January 2023).
As a basis for drafting the self-assessment reports, the General Secretariat of the Government – as the institution co-ordinating Romania’s participation in the OGP – monitors the commitments included in the action plans. Public institutions that are responsible for commitments have to report on the implementation status every six months. The data collected by the GSG is then centralised and published on Romania’s OGP website (ogp.gov.ro). Currently, the tracking of commitments is done through an Excel sheet, which can make tracking complex for citizens and stakeholders. As done by OECD Member and Partner countries such as Argentina and Brazil (Box 7.5), Romania could take full advantage of digital tools to create a more user-friendly online system to track the implementation of commitments (e.g. a traffic light system). In addition to continuing the monitoring of the process of each of the action plan commitments through process indicators (which are included in the action plan itself and usually relate to key milestones, such as dates, for each of the commitments), Romania could further consider starting to evaluate the impact of its OGP action plans based on pre-established results indicators (which could be based on the “expected results” sections of each of the commitments).
In Brazil, the co-ordinator of each commitment included in the OGP action plan needs to prepare an Execution Status Report (CSR) every two months. In addition, monitoring meetings take place for every commitment every three months and a general meeting with all commitment co-ordinators is organised every six months. These monitoring meetings may also involve civil society stakeholders. The CGU’s Open Government Portal serves as the main mechanism to display the information collected through the monitoring meetings and the CSRs. For each commitment, the Portal provides information on associated milestones; the percentage of execution of each milestone; commitment-related information and documents; as well as the minutes of the monitoring meetings of each commitment.
In Argentina, the public institutions responsible for implementing each commitment have to report progress to the SGM’s Undersecretariat of Public Innovation and Open Government (UOG). SGM tracks progress through a dedicated dashboard and reports on a weekly basis to the Government Secretary of Modernisation for each OGP commitment. Finally, the Government Secretary of Modernisation reports to the President on the degree of progress of a number of major commitments. In addition, the UOG holds closed meetings with the institutions responsible for each commitment every two to six months. The government also holds open meetings within the framework of the National Open Government Roundtable with representatives of CSOs.
Source: OECD (forthcoming[5]), Report on the Implementation of the 2017 OECD Recommendation on Open Government.
In line with practice in OECD countries that are part of the OGP, Romania’s multi-stakeholder forum – the National Coordination Committee (CNC) of the OGP (see Chapter 5) – also plays a role in the monitoring process. During the meetings of the CNC, which take place at least four times a year, commitments-leads are asked to present the main achievements and bottleneck of each commitment. Issues are further discussed in the working groups that are dedicated to each commitment, which meet more frequently.
As in several other OECD Member and Partner countries, the monitoring of Romania’s OGP action plan contributes to other monitoring efforts conducted by the government (Figure 7.3). Notably, the monitoring of Romania’s OGP action plan is directly linked with the monitoring activities conducted for the National Anticorruption Strategy (NAS). Joint monitoring activities are facilitated by the fact that the Open Government Service of the General Secretariat of the Government has established a Technical Secretariat of the Working Group for the implementation of the National Anticorruption Strategy 2021-2025 which has the following responsibilities:
It provides guidance and support to the teams within the GSG, to the subordinate authorities of the GSG or to authorities that are under the co-ordination of the GSG, which have the responsibility to implement initiatives included in the NAS;
It performs annual monitoring of the GSG’s initiatives included in the NAS, centralising the information and reporting to the Ministry of Justice (co-ordinating authority of the NAS at the national level).
As an OECD assessment (OECD, 2022[11]) points out, Romania’s National Anti-Corruption Strategy (NAS) 2016-2020 was monitored through self-assessment reports from public institutions, thematic missions, background surveys and studies, systematic evaluation of integrity incidents and period reports issued by international observers, especially by the Group of States against Corruption (GRECO) and the European Union (EU). In particular, the on-site visits by the Ministry of Justice – so-called “thematic missions” – played an essential role in the monitoring process, as they allowed for the involvement of all relevant stakeholders. All the information gathered through these multiple channels was compiled in annual monitoring reports which served as the main tool to evaluate the main achievements and challenges that had been encountered during the implementation of the strategy.
Overall, the monitoring and evaluation of the NAS presents a mixed picture regarding its effectiveness (OECD, 2022[11]). On the one hand, while providing valuable information, the monitoring reports are not systematic enough, mostly due to the absence of action plans with pre-defined indicators and targets in the strategy. Consequently, activities could not be adapted to changing circumstances based on available evidence. On the other hand, the use of novel and “out of the box” tools for reporting and the involvement of civil society were able to compensate for an insufficiently systematic approach to monitoring. Key to these very novel tools were the thematic missions. In the format of multi-stakeholder visits, the findings of thematic missions were discussed by the co-operation platforms and are usually published on the website of the institution in question as well as on the NAS portal. In addition, the Technical Secretariat conducted reviews of thematic missions and based on these reviews developed guidance materials, including on topics such as conflict of interest and revolving doors.
Source: OECD (2022[11]), Evaluation of the Romanian National Anti-corruption Strategy 2016-2020, https://search.oecd.org/gov/ethics/evaluation-romanian-national-anti-corruption-strategy-2016-2020.pdf
The OGP action plan has also been strategically used by the Romanian government as a means to promote more holistic evaluations of open government reforms. Notably, in 2021, based on the commitment to “Coordinating the management of innovative processes for the efficiency of participation in public administration decisions” assumed by the General Secretariat of the Government in Romania’s 2020-2022 OGP action plan, the Romanian government conducted an Analysis regarding the assessment of central and local public administration practices in the decision-making process and ensuring access to information of public interest (Box 7.7). The evaluation provides a number of recommendations to foster transparency and participation in Romania many of which are aligned with those presented in this OECD Open Government Review.
The evaluation starts by noting that “although the present normative framework establishes a series of legal mechanisms capable of stimulating participation and establishing a solid collaborative relationship with civil society, most of the time, the use of these mechanisms fails to make a real contribution to improving the quality of public decisions”.
The analysis aims to present a snapshot of the moment on how civil society and the administration interact and what the prospects are based on which the quality of citizens’ participation in public life can be increased, by streamlining bureaucratic mechanisms with the help of innovative approaches in the fields of decisional transparency in public administration and access to public information. At the same time, the purpose of the analysis was to identify the ways in which the administration can increase the level of participation of citizens in public life and the capacity of institutions to take over the recommendations received from civil society, the dynamism of the administration’s activity in the field of providing public information, both on request and ex officio, by adopting a pro-active approach through innovative means.
In order to evaluate the practices of the central and local public administration in the decision-making process and ensure access to information of public interest, the questionnaire method was used, thus two questionnaires were developed, one each for public institutions and civil society, through which the issues of decision-making transparency, access to information of public interest and innovation in the public sector were addressed. The questionnaire addressed to public institutions registered a total number of 45 responses out of a possible 60 (35 out of a possible 41 responses submitted by the council counties and 10 out of 19 possible answers sent by ministries). The questionnaire addressed to the non-governmental sector recorded a total of 70 responses (49 responses from organisations operating at local level and 21 responses from organisations operating at the national level).
Source: Government of Romania (2021[12]), Evaluation of central and local public administrations’ practices in the decision-making process and in ensuring access to information of public interest, https://ogp.gov.ro/nou/panorama/coordonarea-gestionarii-proceselor-inovative-pentru-eficientizarea-participarii-la-deciziile-administratiei-publice/?psp_download=1.
Monitoring and evaluating the way government informs citizens and stakeholders in Romania
To take part in public decision-making citizens and stakeholders need to have full access to public information and data. Accordingly, the OECD Openness Spectrum (Figure 7.2) highlights citizens’ ability to see, understand and monitor the activities and decisions of the government as one of the core dimensions of open governance (“government informs”).
In Romania, relevant data on the way government informs citizens and stakeholders are currently mostly collected through the mandatory monitoring of the implementation of the country’s access to information law (Law no. 544/2001). In addition to producing a periodic report on the activities they conducted during the year (as mandated by art. 5(3) of Law no. 544/2001), public authorities have the obligation to proactively publish an annual evaluation report on the implementation of Law no. 544/2001 (as mandated by art. 27 of Government Decision no. 123/2002 which states that “(1) Each institution or public authority will annually prepare, through the information and public relations structures, a report on access to information of public interest, which will include (…)”. The form that this annual evaluation report should take was first detailed in Annex 10 of Government Decision no. 123/2002 and subsequently updated through Annex 2 of Government Decision no. 830/2022 which provided a new standardised structure for the evaluation reports submitted by public institutions.
In their annual evaluation reports, public authorities are required to provide both factual information relating to the implementation of the law (e.g. number of staff in charge of the application of the law, number of requests received, etc.) and give opinions/evaluations (e.g. if resources foreseen for the implementation of the law are sufficient, etc.) (see Table 7.1 for the reporting template).
In application of its mandate (Box 7.3), the General Secretariat of the Government annually analyses the implementation of Law no. 544/2001 on access to information of public interest. Based on the reports submitted by public institutions from all levels of government, the GSG compiles synthesis reports, which are published on its institutional website (both as a .DOC and as a .PDF) (General Secretariat of the Government, 2021[13]; General Secretariat of the Government, 2022[14]). The most recent synthesis report is based on the data submitted by 2 777 authorities, including 18 ministries (including the General Secretariat of Government), 40 institutions of the prefect, 250 authorities/institutions subordinate to ministries and 2 469 local public authorities (General Secretariat of the Government, 2022[14])3. In addition to introductory considerations (e.g. explanation of the methodology) and conclusions and recommendations, the synthesis reports follow the same structure and include sections on:
a. Information regarding the person responsible for the implementation of Law no. 544/2001 at the level of the reporting public authority/institution;
g. The total number of requests for information of public interest, broken down by: applicant, the method of addressing, areas of interest;
h. The number of requests resolved favourably, broken down by: response time, mode of communication, areas of interest;
i. The number of rejected requests, broken down by reason for rejection and areas of interest;
j. Administrative complaints to the public institution based on Law no. 544/2001 broken down by method of settlement;
k. Complaints in court against the public institution based on Law no. 544/2001, broken down according to the method of settlement;
l. Increasing the efficiency of access to information of public interest – the existence of the library virtual/information point
In a separate monitoring exercise, the GSG also assesses the de facto publication of the different classes of information that are mandatory to be proactively published according to Article 5(1) of Law no. 544/2001 and to subsequent regulations (e.g. Government Decision no. 830/2022). The compliance data for ministries, institutions subordinated/co-ordinated/under the authority of ministries, autonomous institutions, prefect institutions, county councils, as well as town halls and municipalities is available in the form of separate excel-sheets on GSG’s website. While the information is useful and provide interesting insights as to the degree of implementation of proactive disclosure obligations, the methodology for collecting the data is not explained on the website. In fact, according to the information received from the government of Romania, the GSG manually checks each public authorities’ website and informs them about information that is not displayed or updated.
Since 2021, each entity is assessed according to 25 indicators, which are extracted from Annex 4 of Government Decision no. 1 269/2021 (approving the National Anti-Corruption Strategy). Each of the 25 indicators values 4 percentage points. Accordingly, in case a public institution fully complies with the standard and displays all information required in an adequate format, it has a degree of compliance of 100% (25 x 4=100). In a positive trend, the list of indicators that is used by the GSG to assess the proactive disclosure of information by public institutions also includes an indicator relating to the accessibility of the website for people with disabilities (e.g. the GSG assesses if the webpage allows to increase or decrease the font, transform the text into an audio file; etc.).
In addition to the monitoring exercises that are conducted by the government itself, some non-public stakeholders have also monitored and evaluated the way government informs them. For example, the CSO Funky Citizens drafted a report on budget transparency in local authorities which applied the methodology of the Open Budget Partnership to the Romanian case (Box 7.8) and the Center for Public Innovation conducted a civil society assessment of the implementation of Law no. 544/2001 in 2017 (Center for Public Innovation, 2017[15]). The latter analysed decisions given over the years 2011-2016 by the Romanian Courts of Appeal regarding the use of Law no. 544/2001.
The Report looked at the process of drawing up and approving the local budgets of 109 municipalities in Romania (including the six sectors of the Capital). The Report analysed the extent to which they comply with the criteria of the legislation on local public finances related to the transparency of this process and to what extent Romanian municipalities managed to take steps towards modern international standards in public finance and to adopt measures to make their budgets open (publication format, budgets for citizens or organised public debates). Overall, the report finds that improvements can be seen compared to 2021, but that implementation of existing legal provisions remains a challenge. For example, only roughly 68% of municipalities adopted their budgets by the legal deadline. Concerning other indicators of budget openness, such as the availability of relevant budget information or the opportunity for consultations with citizens on budget drafts, the report sees improvements on a low level. While there was only one municipality that fulfilled eight out of ten criteria in 2021, there are now five: Alexandria, Brasov, Deva, Hunedoara and Pitesti. At the same time, almost one-third (30.2%) of municipalities fulfil four criteria or less in 2022. In this regard, the report particularly highlights challenges related to inaccessible and not reusable information, for example through “hidden” PDFs on the website or information being divided in such a way that it cannot be analysed easily.
Source: Funky citizens (2022[16]), Transparency of Municipal Budgets, https://funky.ong/wp-content/uploads/2022/05/2022-Analiza-Transparenta-bugetelor-de-municipii-Funky-Citizens.pdf.
The Romanian government is to be commended for regularly collecting comparable and relevant data and statistics on the implementation of the country’s transparency laws and regulations. Moving forward, Romania could make efforts to further strengthen ongoing monitoring exercises and data collections by adding additional indicators and reviewing methodologies for monitoring. For example, Romania could consider making further revisions to the template of the monitoring reports that public authorities have to submit annually to report on their implementation of Law no. 544/2001 on free access to information of public interest. For the time being, public institutions’ own monitoring focuses mostly on compliance with certain provisions of the law, but it does not assess the quality of processes, for example regarding their inclusiveness. Hence, Romania could consider adding specific questions on those dimensions to the monitoring template.
As part of the monitoring process of Law no. 544/2001, Romania could further consider asking entities to self-report on the information that they proactively publish, rather than manually scanning each institution’s website. This could, for example, be done by including indicators relating to each different type of information to be published proactively in the monitoring template, including relevant sub-indicators relating to the format of the information, the location of the information on the website, measures taken to foster accessibility and re-use; etc. Using the revised version of the monitoring template could be made mandatory by including it in an annex of a forthcoming Government Decision (e.g. the one that will approve the Open Government Strategy), as done for previous templates. Reducing the required resources for monitoring, the GSG team could periodically conduct random sample assessments to verify the information that has been submitted by public institutions, rather than scanning ministries’ websites.
Romania could further benefit from reviewing how the government reports back on the findings of its monitoring exercises, including the format and content of the synthesis report that the GSG prepares annually to report on the implementation of Law no. 544/2001. In its present form, the synthesis report does not integrate the data that GSG collects on the compliance with proactive disclosure measures (which is published in separate Excel sheets, as mentioned above). Over time, the synthesis report could be enlarged to become a monitoring reporting of Romania’s whole transparency agenda and include information on the de facto proactive disclosure of relevant types of information. The report could further elaborate on the methodological approach in more detail to ensure that the results are comprehensible.
In addition, the General Secretariat of the Government could move beyond publishing PDF/excel reports and start publicly (and graphically) displaying the results of the monitoring exercises on its website. For example, in Canada, the Open Government Portal includes a dedicated section providing statistics on the Access to Information and Privacy Acts, including data on requests received during the identified period; requests completed or carried forward to the next period; etc. which is easy to understand and access.
Finally, the government of Romania could make additional efforts to collect data and metrics on citizens’ and stakeholders’ demand for and use of proactively and reactively disclosed information and data. This could include collecting and analysing traffic data on public institutions’ websites where information is published and offering direct feedback channels to the information disclosed, for example by asking for a quick rating when the user has downloaded information or data. Such data is essential for generating a system that is centred on the actual needs and experiences of citizens and society more broadly.
Monitoring and evaluating the way government responds to citizens and stakeholders in Romania
Accountability is a key part of an open government and is recognised by the OECD definition as one of the open government principles. In line with the OECD Openness Spectrum, an open government has to be reactive to citizens’ and stakeholders’ requests, feedback, complaints and demands. Accordingly, the dimension “how government responds” focuses on key mechanisms for citizens to trigger a response from governments, by requesting public information, demanding answerability on a specific public problem, or suggesting a policy priority (e.g. petitions).
As in most OECD Member and Partner countries, in Romania, data on the way government responds to citizens and stakeholders is collected in a scattered and non-exhaustive manner. There are currently no in-depth centralised analyses of the practices in the field, mostly because there are only few legal provisions in this regard. It is only in the case of access to information requests that the legal framework requires the centralisation at the national level of activity reports. Most complaint and feedback mechanisms (such as petitions) do not provide for such a mechanism.
Romania collects extensive data and information on reactive disclosure of information both at the central and subnational levels through the annual monitoring reports that public institutions submit to report on the implementation of Law no. 544/2001. The synthesis report prepared by GSG (see above) includes data on:
The total number of requests for information of public interest (broken down by: applicant, the method of addressing, areas of interest);
The number of requests resolved favourably (broken down by: response time, mode of communication, areas of interest);
The number of rejected requests, broken down by reason for rejection and areas of interest;
Administrative complaints to the public institution based on Law no. 544/2001 broken down by method of settlement; as well as
Complaints in court against the public institution based on Law no. 544/2001 (broken down according to the method of settlement).
The report shows relatively high levels of compliance with existing legislation. For example, the report highlights that most requests are resolved favourably within 10 days and finds that both pending and rejected court cases have a declining trend.
In terms of petitions (as foreseen by Art. 51 of the Constitution and Government Order no. 27/2002), there is currently limited data that is available, as standardised reporting on the implementation of the Government Ordinance no. 27/2002 on the regulation of the activity of solving petitions to a central authority is not mandatory. Although the Government Decision stipulates that public authorities and institutions should analyse their own activity of resolving petitions every six months and prepare a report, there is no obligation as to the model and the publication of these reports, leading to a non-uniform practice across the state. For example, the Ministry of Health has not published any report on its website, while the Constitutional Court publishes annual reports (for example, (Constitutional Court of Romania, 2022[17])), and the Ministry of Education publishes reports for each county in the form of a scanned PDF on the county’s school inspectorate website every six months (School Inspectorate of the County Botosani, 2021[18])4.
Finally, another relevant source of data and information on the way Romania responds to citizens and stakeholders is the Annual report of the People’s Advocate. The report, which is presented to the joint sitting of the two Chambers of the Parliament, provides ample data on complaint and feedback mechanisms. For 2021, the People’s Advocate reports the following volume of activities: 1835 audiences, 12692 petition, 199 recommendations, 8392 phone calls, 193 inquiries, 1984 ex officio referrals, 4 special reports, 3 points of view sent to the Constitutional Court, 4 actions in administrative litigation, 1 objection of unconstitutionality, 2 appeals in the interest of the law, 79 MNP visits, and 10 direct exceptions of unconstitutionality.
The People’s Advocate’s annual activity report is divided into ten chapters, each of which reports on the specific activities of the institution:
Chapter 1: The field of human rights, equal opportunities between men and women, religious cults and minorities
Chapter 2: The field of family, youth rights, pensioners, people with disabilities
Chapter 3: Defence, protection and promotion of children’s rights
Chapter 5: Property, labour, social protection, taxes and fees
Chapter 6: The field of prevention of torture in places of detention
Chapter 7: The activity of the territorial offices of the institution of the people’s advocate
Chapter 8: Constitutional litigation service, appeal in the interest of the law, administrative and legal litigation, analysis of normative acts, external relations and communication
Source: People’s Advocate (2022[19]), Annual Activity Report 2021, https://avp.ro/wp-content/uploads/2022/01/RAPORT-2021.pdf.
Romania already collects some relevant data on the way the government responds to citizens’ and stakeholders’ requests and demands but the country lacks a strategic approach to the subject, resulting in a lack of data on the effectiveness of existing practices. Moving forward and to build a better basis for an eventual Open Government Index and/or Maturity model (see below), the country could aim to conduct new and strengthen ongoing monitoring processes. For example, as regards the monitoring of reactive disclosure of information, the GSG is dependent on the self-reporting of public authorities. While the GSG has taken important steps to standardise reporting across the administration, this process is time-consuming and potentially prone to errors. For example, depending on the (digital) infrastructure in place of a given public institution, numbers may need to be compiled and transferred manually. To counter this challenge, Romania could consider diversifying its data collection approach. Similar to current practice for proactive disclosure, the GSG (potentially in collaboration with an independent public institution or a civil society organisation) could simulate citizens’ requests to access to information to test the procedure in practice and gather relevant data.
In addition, the reports currently compiled by public institutions focus largely on the minimum aspects of the ATI process that are easily quantifiable, such as the number of requests, subjects requested, response time, etc. – see Table 7.1 above. While these data points can represent important process indicators, they do not tap into other relevant factors of the interaction between public authority and the requester that are less easily quantifiable, for example, whether plain language was employed; whether the format of the response was adequate; etc.
Moreover, Romania could consider adding relevant indicators on the implementation of the provision of Government Decision no. 830/2022 to proactively publish information that is frequently requested. For example, these indicators could capture how many times a certain information needs to be requested before it is included in the institution’s proactive disclosure scheme. Lastly, Romania could consider starting to collect data on citizens’ satisfaction with the process and the responses received to access to information requests. One example in this regard is Brazil, which displays user satisfaction data in its publicly accessible access to information dashboard (panel). Cross-cutting these recommendations is the issue of feasibility, given the limited resources that the GSG as the institution overseeing the monitoring has. As has been noted in Chapter 5, there is currently no central portal for access to information in Romania. In addition to improving accessibility and transparency for citizens, such a portal could substantially facilitate the collection of relevant information for the GSG. For example, it could make the reporting by public institutions on delays in responding to requests unnecessary, if response time is tracked automatically.
In addition, the government of Romania could collect better data on the implementation of the right to petition. The related Government Ordinance no. 27/2002 is one of the few legally mandated accountability channels that apply to all public authorities in Romania. However, it is currently difficult to meaningfully monitor whether this direct feedback channel is leading to tangible results. Improving the monitoring of petitions may require reviewing Government Ordinance no. 27/2002 to mandate the publication of monitoring reports by public institutions in a standardised format and as well as their centralisation to be able to produce a comparative report. Ideally, the burden of data collection would be reduced by using available electronic means that can collect data automatically. For the moment, most public authorities receive petitions via a designated email address. Incorporating standardised contact forms on public authorities’ websites and adding a section for petitions on the E-consultare platform (see Chapter 4) could be ways forward to streamline the data collection process while reducing the risks of inaccurate data. Similar to the monitoring of Law no. 544/2001 and Law no. 52/2003, the General Secretariat of the Government through its Open Government Service could be the entity overseeing and analysing these monitoring efforts by public institutions.
Monitoring and evaluating the way the government interacts with citizens and stakeholders in Romania
The participation of citizens and stakeholders in public decision-making is at the heart of the concept of open government. Accordingly, the OECD Openness Spectrum stipulates that “government interacts” when citizens and stakeholders are able to influence its activities and decisions. This requires a government that is open to hearing the views, perspectives and inputs of citizens.
In Romania, the General Secretariat of the Government currently monitors the implementation of Law no. 52/2003 on decisional transparency in public administration on an annual basis. According to Art. 13 of Law no. 52/2003, all public authorities are obliged to compile and make public an annual report on the implementation of the law. As mandated by the law and by Government Decision no. 478/2016, the annual reports have to be published on the authority’s own website, displayed at its headquarters in a space accessible to the public or presented in a public meeting.
The data from public institutions at all levels is collected by GSG and compiled into in an internal (not publicly available) Excel file. At the time of writing in Fall 2022, no annual synthesis report on the implementation of Law no. 52/2003 had yet been compiled, an exercise that was conducted for the first in January 2023 when a summary report on the application of Law no. 52/2003 was published5 presenting the situation of the year 2021. To further standardise reporting practices, the Open Governance Service at GSG recently created an updated model annual report on decision-making transparency for public institutions which was also included in Government Decision no. 831/2022 (updating Annex 10 of Government Decision no. 478/2016). Among other elements, Government Decision no. 831/2022 aims at standardising and optimising the data collection procedures to facilitate the synthesis analysis. Notably, it mandates that public institutions have to submit their annual reports through the E-Consultare platform (30 April of each year) (see also Chapter 4). In general terms, the progressive use of the E-Consultare platform will potentially provide the government with more in-depth data and information on participatory practices.
Thanks to the existence of Law no. 52/2003, Romania regularly collects data on the implementation of participatory processes at the central/federal level of government, which should be highlighted as a good practice. Moving forward, Romania could further review the monitoring of the implementation of Law no. 52/2003, a process that started with the adoption of Government Decision no. 831/2022 mentioned above, to collect broader and deeper information and data. This may include further reviewing the standard model for monitoring the implementation of Law no. 52/2003 to request that public institutions provide additional types of information. For example, institutions could be asked to provide information on the length of consultations of draft normative acts. Other valuable information could relate to the degree of inclusiveness, for example by tapping into the type of stakeholders that participated and measures that public institutions undertook to foster inclusive participation.
Based on the individual reports submitted by public institutions on an annual basis, Romania could produce an aggregated annual report on the implementation of Law no. 52/2003 to inform both public and non-public stakeholders about the government’s wider participation agenda. Making full use of the power of digital technologies, the government could consider stimulating peer pressure by publishing (and graphically displaying) the results of the monitoring of the implementation of Law no. 52/2003 on the Open Government Portal (beyond publishing them in separate Excel files). This could also be done as part of the Open Government Index and/or Maturity Models that are discussed below.
Moreover, diversifying the data collection method could substantially improve the data’s accuracy and validity. To reduce error-prone and resource-intensive self-reporting, some of the aspects currently evaluated through the questionnaires could be integrated into the E-consultare platform, making the collection of information automatic. Other questions that are currently part of the self-reporting template may be more suitable for interviews and focus groups (e.g. categories E) and F) – see Table 7.3). Qualitative data collection methods that enable a more in-depth sharing of knowledge may provide more useful insights for the GSG.
In addition, data and information could be collected directly from the participants of the participatory processes. For consultations that are conducted online, information could be collected through feedback channels on the E-consultare platform. In addition, further items could be added to the self-reporting template, thereby requiring public institutions to start collecting such information. Since this would imply an additional level of complexity for public institutions, the GSG could provide additional guidance to them on the way the reports answering the GSG questionnaire should be compiled internally (e.g. methodological guidelines).
Finally, in the medium to long term, Romania could aim at collecting data on the wider spectrum of participatory practices implemented at all levels of government (beyond those mandated by Law no. 52/2003). This could include gathering information on participatory budgeting, deliberative processes and hackathons, among others. Questions about these practices could for example be included as voluntary in the self-reporting template that public institutions have to submit on the implementation of Law no. 52/2003.
Romania has already established numerous relevant input and process indicators in different areas of open government. In particular, the GSG regularly collects a wide range of information and data on the implementation of open government legislations and regulations. The government’s decision to move forward with the design of an Open Government Strategy will provide an opportunity to further vamp up monitoring efforts and start the move towards measurements of outcomes and impacts.
As also mandated by the Methodology for the development, implementation, monitoring, evaluation and updating of government strategies (Decision no. 379/2022), Romania has to specify provisions for monitoring and evaluation in the Open Government Strategy itself. As Art. 14 of Decision no. 379/2022 points out, “[e]ach strategy includes a chapter for monitoring and evaluating the achievement of results, which establishes criteria for measuring the evolution of indicators” and article 7 i) explains that the strategy should include a “presentation of how the implementation and impact of the strategy will be monitored and evaluated”. Hence, the Strategy will need to include a specific part dedicated to the foreseen monitoring and evaluation system.
This section discusses the ways the government of Romania could monitor and evaluate the implementation of its first holistic Open Government Strategy and presents tools, such as an Open Government Index and Open Government Maturity Models that can complement the Strategy and further strengthen the evidence base for open government in Romania.
Foreseeing a sound monitoring and evaluation system for the Open Government Strategy
The specific mechanisms for monitoring and evaluation always need to be adapted to the context of the policy issue and the strategy which is formulated in response. In the case of Romania’s Open Government Strategy, different levels of monitoring and evaluation are possible:
M&E of the whole-of-government: The Open Government Strategy represents a unique opportunity to establish a monitoring and evaluation system that encompasses all public institutions at the central government level. This system would track the implementation of all the initiatives that are planned under the umbrella of this strategy.
M&E of individual public authorities: In addition, each public institution could have its system for M&E in place that corresponds to the respective institution’s own open government agenda. Such an M&E system could be integrated into to the monitoring and evaluation of institutional strategic plans (Government of Romania, 2022[20]). Given the decentralised nature, this system could easily cover more public authorities, including Ministries’ subordinate agencies as well as subnational governments.
The 2009 Open Government Directive aims to promote openness in the form of increased transparency, participation and collaboration. It required all US executive departments and agencies to take four steps: 1) Publish Government Information Online; 2) Improve the Quality of Government Information; 3) Create and Institutionalise a Culture of Open Government; 4) Create an Enabling Policy Framework for Open Government. Besides, the Directive described the modalities for formulating and publishing institutional Open Government Plans. Open Government Plans had to be published every two years and were made available on the White House’s website. Information about the current state of each entities’ openness and public progress tracking of the respective Open Government Plan was provided.
Source: OMB (2009[21]), “Open Government Directive”, Executive Office of The President, Office of Management and Budget, https://obamawhitehouse.archives.gov/sites/default/files/omb/assets/memoranda_2010/m10-06.pdf.
Several steps in setting up the M&E process for the Open Government Strategy are indispensable (Box 7.11). First, a national standard for open government needs to be established. Establishing this standard implies asking the question of which attributes (e.g. policies and practices) indicate that an institution or government is open. Each public institution should then adapt this national standard to its own context. Ideally, the standard should also be incorporated into the government’s and institution’s broader performance management agenda.
Secondly, the established standard and intermediate outputs need to be measured and corresponding indicators need to be selected. This requires establishing indicators that can assess if a government or public institution is open and corresponds to the requirement of “the presentation of the indicators on the basis of which the evolution of the implementation of the strategy will be measured, with reference to the targeted national sustainable development indicators, as the case may be” (Article 7h, Decision no. 379/2022). Having defined indicators allows, first, for an initial assessment of the status quo, and it is indispensable for setting specific and realistic targets. The same logic of selecting indicators also applies to the implementation – or process – of the open government initiatives that are part of the strategy. The information and data on processes and results need to be collected continuously.
Develop your national open government standards and/or adapt national standards to institutional standards by:
Establish a monitoring and evaluation routine for each open government initiative (and, potentially, your open government strategy) that includes at least the following steps:
Source: Adapted from Kusek and Rist (2004[22]), Ten Steps to a Results-Based Monitoring and Evaluation System, World Bank, https://www.oecd.org/dac/peer-reviews/World%20bank%202004%2010_Steps_to_a_Results_Based_ME_System.pdf.
Establishing a logic model for the Open Government Strategy
An Open Government Strategy co-ordinates many different actors through a multiplicity of initiatives. Accordingly, the M&E framework for the Open Government Strategy, which is the result of the abovementioned steps, should reflect this complexity through a logic model which shows how each planned initiative contributes to the broader objectives of the strategy. Figure 7.4 illustrates such an approach.
For example, the OGS could include the objective that all information that is proactively published by public institutions has to be relevant to the public. In this regard, the Strategy could include an initiative relating to a capacity-building programme, which aims to increase the relevance of published information. Accordingly, all the elements of the Monitoring and Evaluation Framework of that initiative can be defined both at the whole of the government and at the level of each public institution. This includes:
The standards and objectives derived from the Open Government Strategy, such as the relevance of information published by government institutions.
The inputs needed to set up and run the initiative, such as financial and human resources to develop the training programme, measured by input indicators.
The process of implementing the initiative, such as public officials taking part in the training programme, measured by process indicators.
The outputs of the initiative, such as public officials applying the skills they have learned and collecting and analysing user data, measured by output indicators.
The outcome of the initiative, such as information being published based on user needs, measured by outcome indicators.
The impacts of the initiative, i.e. having contributed to the goal of increased relevance of published information, measured through impact indicators that have also served to establish the baseline and the target of the objective in relation to the formulated standard.
Lastly, the process is concluded by analysing the information and data that has been gathered against the objectives of the strategy and by reporting on its findings. Based on such a logic model for the initiatives included in the Open Government Strategy, multiple types of evaluation will be available to the Romanian government, namely process evaluation, outcome evaluation and impact evaluation.6 These three types of evaluation, even if named differently, are also required by the legal provisions on evaluating strategies (Art. 22, Regulation of July 14, 2005).
Source: Author’s own elaboration based on OECD (2020[23]), Improving Governance with Policy Evaluation: Lessons From Country Experiences, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/89b1577d-en.
Assigning clear responsibilities for monitoring and evaluation
Given its role as the main co-ordinating entity of Romania’s open government agenda and its experience in monitoring different open government policies and practices, the General Secretariat of the Government would be in an ideal position to co-ordinate the monitoring of the Open Government Strategy. The strategy could provide a mandate to the Open Government Service to develop an annual M&E plan for the Open Government Strategy, fulfilling the mandate of periodic assessments (Art. 21, Regulation of July 14, 2005), and to produce monitoring and evaluation reports (Art. 11i), j), Decision no. 379/2022 based on the required periodic monitoring reports by all involved institutions (Art 21, Regulation of July 14, 2005).
The National Open Government Committee, recommended in Chapter 4, could serve as an interministerial platform to follow up and discuss progress on the strategic goals in a systematic manner (Art.9 (1), Decision no. 379/2022). For example, the National Open Government Committee could have a subcommittee on monitoring and evaluation that brings together the responsible officials from the working groups in public authorities, as required by Decision no. 379/2022 (Art. 8(1)).
An approach that has proven to be successful in the Romanian context is to have so-called ‘peer-reviewers’. For example, in the framework of the National Anti-corruption Strategy, public institutions were reviewed by other institutions that had the same obligations. Such an approach does not only reduce the burden on a single monitoring body but can also increase ownership of the strategy across the public sector, steer healthy competition and foster peer exchange. Romania could consider establishing a similar peer reviewer system for the Open Government Strategy.
Fostering accountability through a collaborative approach to M&E
Systems for monitoring and evaluation can greatly benefit from the inputs of citizens, civil society, academia and other non-governmental stakeholders. Stakeholder participation can for example support the identification of the indicators to use for measuring the implementation of the open government initiative included in the strategy. Relevant legislation in Romania explicitly allows for the involvement of “private, non-governmental, academic and higher education organisations” during monitoring and evaluation activities (Art. 21, Regulation of July 14, 2005).
Further, during the implementation of the Open Government Strategy, Romania could consider integrating non-public stakeholders as members of a review board examining a particular public institution, as done through multi-stakeholder missions in the framework of the National Anti-Corruption Strategy. These on-site visits called “thematic missions” were used to gather evidence on-site and to discuss them through meetings of the mixed working groups (“co-operation platforms”). Feedback from involved stakeholders, including civil society, has been very positive regarding the value of this activity for monitoring the progress of the NAS.
To enable the broader public to participate in monitoring activities, the Strategy could further foresee the creation of a publicly available tracking system. The tracking system could be modelled on those that some OECD Member and Partner countries have established for their OGP action plans (see Box 7.5 above) and include key process and output indicators.
Providing training and guidance material to secure standards of M&E
The National Open Government Strategy will serve as an orientation for the open government agendas of all individual public institutions and subnational governments. It is therefore pivotal for its success to ensure that the process of monitoring and evaluating open government initiatives across all public institutions is consistently of a high standard.
As lined out by the Methodology (Decision no. 379/2022), the GSG’s Policy and Priorities Coordination Directorate provides methodological guidance on all stages of the strategy, including the monitoring and evaluation (Article 12). In particular, trainings on data collection and analysis for all staff involved in monitoring and evaluating the strategy will become increasingly important. Currently, the National Institute for Administration offers a course on ‘Public Policies’ that includes monitoring and evaluation as one out of eight topics and is taught to civil servants at the management level, civil servants working on contract management and execution, as well as other interested (and paying) civil servants. In addition, INA offers a course on ‘Public Institution Strategy’ which teaches related content to the same audience (National Institute for Administration, n.d.[24]).
Currently, there are different pieces of legislation regulating strategies and public policies in Romania, including their monitoring and evaluation (Decision no. 379/2022; Regulation of July 14, 2005; and Decision no. 443/2022). For example, these legislations provide a structure to the mandatory evaluation reports (Annex 2, Regulation of July 14, 2005). However, they remain relatively high-level and provide little concrete guidance on monitoring and evaluation. Moving forward, the Romanian government could consider establishing guiding materials in the forms of manuals or toolkits that are more practical, as done for example by Peru (see Box 7.12)
The Peruvian National Center for Strategic Planning (‘Centro Nacional de Planeamiento Estratégico’ – CEPLAN) as Peru’s leading body of the National Strategic Planning System guides Peruvian entities in monitoring and evaluation of their policies. In that respect, it has published:
The Guide on monitoring and evaluation introduces the concepts of monitoring and evaluation. It helps public officials understand what these processes encompass, what their value is and how to comply with standards of models for process and impact evaluation.
The Guide for the establishment of indicators helps public institutions in operationalising and measuring processes and outcomes. It defines indicators and their usage ex ante and ex post of a policy intervention and assists in developing indicators, including their verification and placement in the results chain.
Source: National Center for Strategic Planning (2021[25]), Guide for monitoring and evaluation of national policies and Sinaplan plans; National Center for Strategic Planning (2021[26]), Guide for the development of national policy indicators and strategic plans.
Building an Open Government Index to foster M&E
Romania could consider using the Strategy process and the momentum it provides to build the country’s first holistic Open Government Index (“OGI”). OECD experience from different areas of public governance shows that, by putting a publicly available lens on a specific policy, an Index can be one of the most effective ways to stimulate reforms, while at the same time ensuring higher levels of compliance with the legal and regulatory framework.
An Open Government Index would draw from the efforts of monitoring and evaluating the Open Government Strategy. For example, the Index could focus on institution-level outcomes that are part of the strategy’s monitoring and evaluation framework and rank public institutions along corresponding dimensions. As such, an Open Government Index can serve as a tool for setting baseline indicators against which the success of the Open Government Strategy is benchmarked at the end of its implementation (see Box 7.13). To amplify its impact, the Index could be coupled with the Open Government Award suggested in Chapter 5 to recognise institutions and public servants that have made significant contributions to the open government agenda.
Mexico’s Open Government Metrics were developed by the Centre for Economic Research and Teaching (CIDE), and were based on an initiative of the National Institute for Transparency, Access to Information and Personal Data Protection (INAI). The metrics are designed as a baseline to measure the current state of the National System of Transparency, Access to Information and Protection of Personal Data (SNT) and open government policies more generally. Aiming to be an “x-ray of the starting point of the open government policy of the Mexican State” at the national and subnational level, its focus goes beyond measuring compliance with regulations, and aims to capture performance information on the outcomes of open government policies from the perspective of both government and citizens.
The metrics start with an operational definition of open government structured around two dimensions: transparency and public participation. Each dimension is approached from two perspectives: government and citizens.
The CIDE team developed an Open Government Index, consisting of measurements of transparency and participation from the perspective of both government and citizens. The construction of these indexes involved the analysis of existing regulations, a review of government websites, and user simulations, including information requests. The Metrics survey included a sample of 908 governmental bodies at the national and subnational level; 754 portals were reviewed and 3 635 requests for information were sent. The resulting Open Government Index of Mexico was 0.39 (on a scale of 0 to 1). The index showed that the transparency dimension has a much higher value (0.50) than the participation dimension (0.28).
Source: INAI (2017[27]), Resultados Edición 2017, http://eventos.inai.org.mx/metricasga/index.php/descargables (accessed 11 January 2019).
Romania will require considerable human and financial resources to build and run an Open Government Index. In addition to the specific expertise that is needed to set up the Index, multiple data points over time are needed to assess openness. This requires continuous maintenance and, potentially, updates of the methodology. The institutionalisation of the Index may require the creation of a dedicated team within GSG that collects and compiles the data and makes it available. The GSG could also consider partnering with an independent public institution (e.g. the Ombudsman) or a civil society/academic institution to build and maintain the Index.
Given the multidimensionality of the concept of open government, Romania’s Open Government Index would need to be a composite indicator, provided that an overall score of openness is desired. If the Index goes hand in hand with the strategy, the definitions and objectives of the strategy can serve as the basis for the theoretical framework of the Index.7 The data for the Index could be collected in different ways. For example, Romania could use:
Administrative data collected specifically for M&E purposes (e.g. the data the GSG currently collects on the implementation of Law no. 544/2001).
Conduct mystery shopper exercises to assess the de facto implementation of open government practices (e.g. by partnering with an independent public institution or an academic/civil society organisation). This may involve submitting policy proposals and/or access to information requests to all public entities.
Already existing quantitative data, for example from academic databases such as the digiwhist (Open Government Institute, n.d.[28]) databases on public procurement contracts.
Survey data, for example from public opinion polls and participant questionnaires such as from the European Social Survey.
Public consultations with stakeholders, illustrated for example by the “What we heard” publication on the Canadian Open Government Partnership process (Treasury Board of Canada Secretariat, 2018[29]).
Insights from focus groups, such as in the early stages of the Madrid City Observatory.
In-depth interviews with experts, for example through inviting and questioning academics in the field.
The Handbook on Constructing Composite Indicators: Methodology and User Guide developed jointly by the OECD and the European Union can provide further guidance (Box 7.14).
Develop a theoretical framework for a clear understanding of the multidimensional phenomenon of open government and the selection of variables.
Select/collect data corresponding to the variables according to analytical soundness, measurability, coverage and relevance.
Impute missing data to gain a sufficiently complete dataset.
Conduct multivariate analysis to assess the suitability of the dataset and provide the groundwork for subsequent methodological choices.
Perform normalisation to render all the variables included comparable.
Weight and aggregate the variables along the theoretical framework while respecting data properties.
Conduct uncertainty and sensitivity analysis to assess the robustness of the composite indicator against previously made methodological choices.
Analyse at a granular level what is driving the results for a given public institution or dimension to develop data-driven narratives.
Perform cross-validation with similar (composite) indicators to assess linkages and validity.
Visualise the results in a clear and accurate manner to enhance interpretability.
Source: Adapted from OECD/European Union/EC-JRC (2008[30]), Handbook on Constructing Composite Indicators: Methodology and User Guide, OECD Publishing, Paris, https://doi.org/10.1787/9789264043466-en.
Building Romania’s first open government maturity models
A maturity model is a reference instrument for assessing an entity's transition towards a given objective during a given period (OECD, 2022[8]). It includes a sequence of levels or stages that draws a path from an initial, basic state to maturity. Maturity models typically imply a self-assessment of a public institution. As such, they can be a tool to mainstream the principles and objectives that are included in the Open Government Strategy in a decentralised way. They are increasingly being developed in different areas of public governance, including in the field of open government, as they can (OECD, 2022[8]):
Set a baseline standard of what good practices in the field of open government look like;
Allow public institutions to assess their levels of openness at a given point in time and identify where they are situated in relation to national good practice;
Allow building a coherent and flexible trajectory towards high levels of maturity, adaptable to the situation of each public institution;
Show the stages of this progression and the necessary achievements that at each stage are useful and consolidate the passage into the subsequent stages;
Allow for comparison between public institutions within a defined framework.
Given Romania’s already quite advanced monitoring agenda, the country could be the first country to couple its Open Government Strategy with the design of an Open Government Maturity Model. The maturity model could serve as a diagnostic tool. For example, a maturity model may aim at assessing a public institution’s readiness to foster transparency. The corresponding maturity model could enable the public institution to periodically review its strategic plan according to the objectives and criteria defined in the strategy. Besides helping the public institution to assess its development over time, it can provide immediate guidance on how to move to the next maturity stage. In addition, such a self-assessment would enable public institutions to signal progress and challenges to the GSG, which in turn can facilitate more targeted interventions through its Open Government Service.
Given its decentralised nature, this tool’s scope of application can be large. Further, different actors, including civil society, can use the maturity models if there is sufficient available data, thereby adding another level of assessment and oversight. If the public institutions allow their maturity scores to be collected, these scores can feed into a database that can allow comparisons between institutions and could feed into central monitoring and evaluation efforts. Moreover, Romania could incentivise all public institutions to publicly disclose their maturity levels/scores based on their own assessment to foster comparison and generate a “race to the top”.
As a first step, Romania would have to decide on the maturity model’s exact purpose, target group and coverage (for example, public institutions at the national level, subnational governments, certain departments etc.). It would then have to define a suitable theoretical framework, including the different maturity dimensions and stages. The theoretical framework could build on the conceptualisation and the objectives formulates in the Open Government Strategy. In the next step, the maturity stages need to be translated into verifiable criteria that allow for assessment and placement. Each of the maturity stages can be coupled with advice on improvement measures.
The effectiveness of maturity models in serving their purpose is highly dependent on their application. Therefore, user-centred design is essential: the engagement of all relevant public and non-public stakeholders can substantially improve uptake. In addition, the maturity model should be complemented with extensive guidance for users to ensure that criteria are assessed correctly. Maturity models can be made more accessible and user-friendly by designing a digital interface (e.g. an app) specifically for this purpose.
The OECD Public Sector Integrity Maturity Models allow a government (national or subnational) or a public sector organisation to assess the elements of their integrity systems, and identify where they are situated in relation to good practice across four categories: nascent, emerging, established and leading. The maturity models can be used by political and executive leadership, government officials, public sector integrity practitioners, business and civil society.
The public integrity maturity models are a complement to the Recommendation of the Council on Public Integrity and the OECD Public Integrity Handbook. They can also be used in conjunction with the forthcoming Public Integrity Indicators, to provide a qualitative, subjective assessment of public integrity.
Source: OECD (n.d.[31]), OECD Public Integrity Maturity Models, https://www.oecd.org/governance/ethics/public-integrity-maturity-models.htm (accessed on 13 November 2022).
Open government is a wide field that is inherently difficult to monitor and evaluate. For the time being, no OECD Member or Partner country has managed to design a holistic agenda to measure the impact of its open government agenda. Assessing Romania against Provision 5 of the Recommendation of the Council on Open Government, this chapter finds that the country already collects a wide range of information and data on the implementation of relevant open government laws. In particular, Romania has at its disposal a solid basis of process indicators for the implementation of its laws on free access to information and decisional transparency in public administration. In addition, Romania gathers data on the implementation of different policy documents, notably the OGP action plan.
The chapter first recommends ways for Romania to strengthen ongoing monitoring and evaluation processes. It suggests improvements to existing data collection methods, for example by reviewing reporting templates. It further highlights the need to diversify the approach to collect data for monitoring purposes. To reduce the burden of data collection, improve data quality, and provide opportunities to showcase implementation, it suggests technical solutions to update the existing E-consultare platform.
In addition, the chapter highlights the unique opportunities that come with the design of the forthcoming Open Government Strategy and recommends the development of an integrated framework for its monitoring and evaluation. In particular, it highlights the need to set measurable objectives paired with an initial benchmarking assessment to enable meaningful evaluation of progress and outcomes of the Strategy. In this regard, the chapter proposes that Romania could couple the M&E of the open government strategy with the design of an Open Government Index and/or an Open Government Maturity Model. An Open Government Index could allow the government to continuously assess and rank public institutions according to their levels of openness, while a maturity model can stimulate comparison on a voluntary basis.
1. Enhance the monitoring and evaluation of Romania’s Open Government Partnership action plan.
Take full advantage of digital tools to create a user-friendly online system to allow citizens and stakeholders to track the implementation of OGP commitments (e.g. a traffic light system).
Start evaluating systematically the impact of commitments in the OGP action plans based on a corresponding framework for monitoring and evaluation that includes outcome indicators.
2. Foster monitoring and evaluation of the way government informs citizens and stakeholders.
Strengthen ongoing monitoring exercises and data collections by adding additional indicators and reviewing methodologies for monitoring, including by making further revisions to the template of the monitoring reports that public authorities have to submit annually to report on their implementation of Law no. 544/2001 on free access to information of public interest (e.g. on quality of processes).
As part of the monitoring process of Law no. 544/2001, ask entities to self-report on the information that they proactively publish (rather than manually scanning each institution’s website), for example by including indicators relating to each different type of information to be published proactively in the monitoring template.
Make using the revised version of the monitoring template mandatory by including it in an annex of a forthcoming Government Decision (e.g. the one that will approve the Open Government Strategy).
Periodically conduct random sample assessments to verify the information that has been submitted by public institutions, rather than scanning ministries’ websites.
Review the format and content of the synthesis report that the GSG prepares annually to report on the implementation of Law no. 544/2001, including by integrating the data that GSG collects on compliance with proactive disclosure measures.
Enlarge the synthesis report for it to become a monitoring report of Romania’s whole transparency agenda and include information on the de facto proactive disclosure of relevant types of information.
Move beyond publishing PDF/excel reports and start publicly (and graphically) displaying the results of the monitoring exercises on a website.
Collect data and metrics on citizens’ and stakeholders’ demand for and use of proactively and reactively disclosed information and data, including by collecting and analysing traffic data on public institutions’ websites and by offering direct feedback channels to the information disclosed.
3. Foster monitoring and evaluation of the way government responds to citizens and stakeholders by building a strategic approach and by conducting new and strengthening ongoing monitoring processes.
Review the collection of data on the reactive disclosure of information, including the reports compiled by public institutions.
Consider diversifying the data collection approach, including simulating citizens’ requests to access to information to test the procedure in practice and gather relevant data.
Enlarge the focus beyond the minimum aspects of the ATI process to tap into other relevant factors of the interaction between public authority and the requester, for example, whether plain language was employed; whether the format of the response was adequate; etc.
Add relevant indicators on the implementation of the provision of Government Decision no. 830/2022 to proactively publish information that is frequently requested. For example, these indicators could capture how many times a certain information needs to be requested before it is included in the institution’s proactive disclosure scheme.
Start collecting data on citizens’ satisfaction with the process and the responses received to access to information requests.
Create a central portal for access to information to enhance accessibility and transparency for citizens and facilitate the collection of relevant information for the GSG.
Collect better data on the implementation of the right to petition to monitor whether this direct feedback channel is leading to tangible results.
Review Government Ordinance 27/2002 to mandate the publication of monitoring reports by public institutions in a standardised format and as well as the centralisation of corresponding data to be able to produce a comparative report.
Reduce the burden of data collection by using available electronic means, for example by incorporating standardised contact forms on public authorities’ websites and by adding a section for petitions on the E-consultare platform.
Mandate the General Secretariat of the Government through its Open Government Directorate as the entity overseeing and analysing these monitoring efforts.
4. Enhance the monitoring and evaluation of the way government interacts with citizens and stakeholders.
Further review the monitoring of the implementation of Law no. 52/2003 to collect broader and more detailed information and data.
Review the standard model for monitoring the implementation of Law no. 52/2003 to request that public institutions provide additional types of information, including on the length of consultations of draft normative acts and the degree of inclusiveness (e.g. types of stakeholders that participated).
Produce an aggregated annual report on the implementation of Law no. 52/2003 to inform both public and non-public stakeholders about the government’s implementation of participatory practices.
Stimulate peer pressure by publishing (and graphically displaying) the results of the monitoring of the implementation of Law no. 52/2003 on a future Open Government Portal.
Diversify the data collection method to improve the data’s accuracy and validity, including by reducing error-prone and resource-intensive self-reporting (e.g. by making the collection of information automatic through the E-consultare platform).
Collect information directly from the participants of participatory processes (e.g. through feedback channels on the E-consultare platform).
Provide additional guidance to public institutions on the way the reports answering the questionnaire should be compiled (e.g. methodological guidelines).
Collect data on the wider spectrum of participatory practices implemented at all levels of government (beyond those mandated by Law no. 52/2003), including information on participatory budgeting, hackathons, etc., for example by including questions about these practices in the self-reporting template that public institutions have to submit on the implementation of Law no. 52/2003.
5. Establish a robust monitoring and evaluation system for the Open Government Strategy.
Establish a national standard for open government and invite public institutions to adapt this national standard to their own context.
Create a logic model which shows how each planned initiative of the Strategy contributes to its broader objectives.
Provide a mandate to the Open Government Directorate in GSG to develop an annual M&E plan for the Open Government Strategy.
Use the National Open Government Committee, recommended in Chapter 5, as an interministerial platform to follow up and discuss progress on the strategic goals in a systematic manner, including by creating a subcommittee on monitoring and evaluation that brings together the responsible officials from the working groups in public authorities.
Establish a peer reviewer system for the Open Government Strategy (as done for example for the National Anti-corruption Strategy) to reduce the burden on a single monitoring body, increase ownership of the strategy across the public sector, steer healthy competition and foster peer exchange.
Include non-public stakeholders as members of a review board examining the implementation of the Strategy in a particular public institution (as done through thematic multi-stakeholder missions in the framework of the National Anti-Corruption Strategy).
Create a publicly available tracking system to enable the broader public to participate in monitoring activities.
Provide trainings on data collection and analysis for all staff involved in monitoring and evaluating the Strategy and establish guiding materials in the forms of practical manuals or toolkits.
Consider using the Strategy and the momentum it provides to build Romania’s first holistic Open Government Index, putting a publicly available lens on open government issues, stimulating reforms, and ensuring compliance.
Use the Index as a tool to set baseline indicators against which the success of the Open Government Strategy is benchmarked at the end of its implementation.
Use the definitions and objectives that will be included in the Strategy (see Chapter 6) as the basis for the theoretical framework of the Index.
Create a dedicated team within GSG that collects and compiles the data and makes it available.
Consider partnering with an independent public institution (e.g. the Ombudsman) or a civil society/academic institution to build and maintain the Index.
Build Romania’s first open government maturity model to allow public institutions to self-assess their levels of openness.
Decide on the maturity model’s exact purpose, target group and coverage.
Define a suitable theoretical framework, including the different maturity dimensions and stages, building on the conceptualisation and the objectives formulated in the Open Government Strategy.
Translate the maturity stages into verifiable criteria that allow for assessment and placement.
Engage all relevant public and non-public stakeholders in the design of the maturity model to increase uptake.
Complement the maturity model with extensive guidance for users to ensure that criteria are assessed correctly.
Designing a digital interface (e.g. an app) to make the maturity model more accessible and user-friendly.
Provide guidance on how to move to the next maturity stage and enable public institutions to signal progress and challenges to the GSG, which can facilitate more targeted interventions.
Once established, incentivise all public institutions to publicly disclose their maturity levels/scores based on their own assessment to foster comparison and generate a “race to the top”.
References
[15] Center for Public Innovation (2017), Public interest information obtained in court, https://www.inovarepublica.ro/informatii-de-interes-public-castigate-instanta/ (accessed on 13 November 2022).
[17] Constitutional Court of Romania (2022), Report on the activity for resolving petitions in the year 2021, https://www.ccr.ro/wp-content/uploads/2022/02/Raport-petitii-pe-anul-2021.pdf.
[16] Funky citizens (2022), Transparency of Municipal Budgets, https://funky.ong/wp-content/uploads/2022/05/2022-Analiza-Transparenta-bugetelor-de-municipii-Funky-Citizens.pdf.
[14] General Secretariat of the Government (2022), Synthesis of the annual reports on the implementation by the authorities and institutions of the central and local public administration of the legislation on free access to information of public interest in 2021, https://sgg.gov.ro/1/wp-content/uploads/2022/12/Sinteza-rapoartelor-anuale.docx (accessed on 13 November 2022).
[13] General Secretariat of the Government (2021), Synthesis of the annual reports on the implementation by the authorities and institutions of the central and local public administration of the legislation on free access to information of public interest in 2020, https://sgg.gov.ro/1/wp-content/uploads/2021/12/Sinteza-rapoartelor-anuale-privind-implementarea-de-catre-autoritati-si-institutii-ale-administratiei-publice-centrale-si-locale-a-legislatiei-privind-liberul-acces-la-informatii-de-interes-public-in.pdf.
[20] Government of Romania (2022), Background Report prepared for the OECD Open Government Review of Romania, (unpublished working paper).
[12] Government of Romania (2021), Evaluation of central and local public administartions’ practices in the decision-making process and in ensuring access to information of public interest, https://ogp.gov.ro/nou/panorama/coordonarea-gestionarii-proceselor-inovative-pentru-eficientizarea-participarii-la-deciziile-administratiei-publice/?psp_download=1 (accessed on 13 November 2022).
[27] INAI (2017), Resultados Edición 2017, http://eventos.inai.org.mx/metricasga/index.php/descargables (accessed on 11 January 2019).
[22] Kusek, J. and R. Rist (2004), Ten Steps to a Results-Based Monitoring and Evaluation System, World Bank, https://www.oecd.org/dac/peer-reviews/World%20bank%202004%2010_Steps_to_a_Results_Based_ME_System.pdf.
[25] National Center for Strategic Planning (2021), Guide for monitoring and evaluation of national policies and Sinaplan plans, https://cdn.www.gob.pe/uploads/document/file/1724649/Gu%C3%ADa%20para%20el%20seguimiento%20y%20evaluaci%C3%B3n%20de%20pol%C3%ADticas%20nacionales%20y%20planes%20para%20el%20SINAPLAN%20-%20CEPLAN.pdf?v=1615439789 (accessed on 13 November 2022).
[26] National Center for Strategic Planning (2021), Guide for the development of national policy indicators and strategic plans, https://cdn.www.gob.pe/uploads/document/file/1724645/Gu%C3%ADa%20para%20la%20elaboraci%C3%B3n%20de%20indicadores%20de%20pol%C3%ADticas%20nacionales%20y%20planes%20estrat%C3%A9gicos%20-%20CEPLAN.pdf?v=1615439085 (accessed on 13 November 2022).
[24] National Institute for Administration (n.d.), Institutional Development and Public Policies, https://ina.gov.ro/dezvoltare-institutionala-politici-publice/ (accessed on 13 November 2022).
[7] OECD (2023), Civic Space Review of Romania, OECD Public Governance Reviews, OECD publishing, Paris, https://doi.org/10.1787/f11191be-en.
[6] OECD (2023), Open Government Scan of Canada: Designing and Implementing an Open Government Strategy, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/1290a7ef-en.
[11] OECD (2022), Evaluation of the Romanian National Anti-corruption Strategy 2016-2020, OECD, Paris, https://search.oecd.org/gov/ethics/evaluation-romanian-national-anti-corruption-strategy-2016-2020.pdf.
[8] OECD (2022), Open Government Review of Brazil: Towards an Integrated Open Government Agenda, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/3f9009d4-en.
[4] OECD (2021), 2020 OECD Survey on Open Government.
[2] OECD (2020), A Roadmap for Assessing the Impact of Open Government Reform, GOV/PGC/OG(2020)5, Paper presented to the OECD Working Party on Open Government.
[23] OECD (2020), Improving Governance with Policy Evaluation: Lessons From Country Experiences, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/89b1577d-en.
[1] OECD (2019), Open Government in Argentina, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/1988ccef-en.
[3] OECD (2017), “Recommendation of the Council on Open Government”, OECD Legal Instruments, OECD/LEGAL/0438, OECD, Paris, https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0438.
[31] OECD (n.d.), “OECD Public Integrity Maturity Models”, https://www.oecd.org/governance/ethics/public-integrity-maturity-models.htm (accessed on 13 November 2022).
[5] OECD (forthcoming), Report on the Implementation of the 2017 OECD Recommendation on Open Government.
[30] OECD/European Union/EC-JRC (2008), Handbook on Constructing Composite Indicators: Methodology and User Guide, OECD Publishing, Paris, https://doi.org/10.1787/9789264043466-en.
[9] OGP (2022), OGP National Handbook: Rules + Guidance for Participants, Open Government Partnership, https://www.opengovpartnership.org/wp-content/uploads/2022/03/OGP-National-Handbook-2022.pdf.
[10] OGP (n.d.), Independent Reporting Mechanism: Products and Process, Open Government Partnership, https://www.opengovpartnership.org/irm-products-and-process/ (accessed on 13 January 2023).
[21] OMB (2009), Open Government Directive, Executive Office of The President, Office of Management and Budget, https://obamawhitehouse.archives.gov/sites/default/files/omb/assets/memoranda_2010/m10-06.pdf.
[28] Open Government Institute (n.d.), Digiwhist, https://digiwhist.eu/ (accessed on 13 November 2022).
[19] People’s Advocate (2022), Annual Activity Report 2021, https://avp.ro/wp-content/uploads/2022/01/RAPORT-2021.pdf.
[18] School Inspectorate of the County Botosani (2021), Semestrial report on the solution of petitions: Semester II (July - December 2021), https://isjbotosani.ro/wp-content/uploads/2022/01/Raport-petitii.pdf.
[29] Treasury Board of Canada Secretariat (2018), What We Heard - Summary Report: Summary report on open government consultations October 2017 – August 2018, https://open.canada.ca/en/content/what-we-heard-summary-report-0 (accessed on 13 November 2022).
Government of Romania (2022), “Decision no. 831/2022 for the approval of the methodological norms for the application of Law no. 52/2003 on decisional transparency in public administration”, https://legislatie.just.ro/Public/DetaliiDocument/256871.
Government of Romania (2022), “Decision no. 830/2022 for the modification and completion of the Methodological Norms for the application of Law no. 544/2001 regarding free access to information of public interest, approved by Government Decision no. 123/2002”, https://legislatie.just.ro/Public/DetaliiDocument/256836.
Government of Romania (2022), “Decision no. 443/2022 for approving the content of the presentation and motivation tool, the structure of the report on the implementation of normative acts, the methodological instructions for carrying out the impact assessment, as well as for the establishment of the Advisory Council for the assessment of the impact of normative acts”, https://legislatie.just.ro/Public/DetaliiDocument/253645.
Government of Romania (2022), “Decision no. 379/2022 on the Methodology for the development, implementation, monitoring, evaluation and updating of government strategies”, https://legislatie.just.ro/Public/DetaliiDocument/253158.
Government of Romania (2022), “Methodology on the development, implementation, monitoring, evaluation and updating of government strategies approved by Decision no. 379/2022”, https://legislatie.just.ro/Public/DetaliiDocument/253159.
Government of Romania (2021), “Decision no. 1 269/2021 regarding the approval of the National Anti-corruption Strategy 2021-2025 and its related documents”, https://legislatie.just.ro/Public/DetaliiDocumentAfis/249828.
Government of Romania (2020), “Decision no.137/2020 on the organisation, functioning and attributions of the GSG”, https://legislatie.just.ro/Public/DetaliiDocument/223115.
Government of Romania (2005), “Regulation of July 14, 2005, regarding the procedures for the development, monitoring and evaluation of public policies at the central level updated as of July 28”, 2016, https://legislatie.just.ro/Public/DetaliiDocumentAfis/180532.
Government of Romania (2002), “Decision no. 123/2002 for the approval of the Methodological Norms for the application of Law no. 544/2001 on free access to information of public interest”, https://legislatie.just.ro/Public/DetaliiDocument/34416.
Government of Romania (2002), “Ordinance no. 27/2002 regarding the regulation of the petition settlement activity”, https://legislatie.just.ro/Public/DetaliiDocument/33817.
Government of Romania (2000), “Ordinance no. 26/2000 regarding associations and foundations”, https://legislatie.just.ro/Public/DetaliiDocument/20740.
Parliament of Romania (2005), “Law no. 246/2005 for the approval of Government Ordinance no. 26/2000 regarding associations and foundations”, https://legislatie.just.ro/Public/DetaliiDocumentAfis/63457.
Parliament of Romania (2003), “Law no. 52/2003 on decisional transparency in public administration”, https://legislatie.just.ro/Public/DetaliiDocument/41571.
Parliament of Romania (2001), “Law no. 544/2001 on free access to information of public interest”, OFFICIAL MONITOR no. 663 of October 23, 2001, https://legislatie.just.ro/Public/DetaliiDocument/31413.
Notes
← 1. Monitoring refers to the systematic collection of performance data to assess the progress and achievement of policy objectives against set targets and to identify and lift implementation bottlenecks. Policy evaluation refers to the structured and objective assessment of the design, implementation and/or results of a future, ongoing or completed public intervention (OECD, 2020[23]). Monitoring and evaluation encompass the collection and analysis of information and data concerning a strategy’s inputs, activities, outputs, outcomes, and impact.
← 2. Countries that have adhered to the Recommendation of the Council on Open Government include all OECD Member countries, as well as Argentina, Brazil, Morocco, Tunisia and Romania.
← 3. The synthesis report for the year 2021 was published too late to be fully taken into account for the draft of this Review.
← 4. These are just indications, getting a clear overview of the practices across all Ministries and public authorities was not possible.
← 5. The report can be consulted on: https://sgg.gov.ro/1/wp-content/uploads/2023/01/SINTEZA-ANUALA_2021-LEGEA-52_2003.pdf.
← 6. When considering the ‘success’ of the open government strategy, one considers broadly three types of evaluation: process and impact evaluation. Process evaluation refers to the examination of the implementation, i.e. whether the strategy has been properly executed, and uses the previously established process indicators to provide information on how the initiative has been delivered. An outcome evaluation sheds light on the extent to which the benefits of the initiative outweigh their costs, i.e. it analyses the initiative’s efficiency. Impact evaluation, on the other hand, describes the analysis of its outcomes. This means evaluating whether the strategy has led to the desired results given its implementation (OECD, 2020[23]).
← 7. In one basic scenario, used here for illustration, the Strategy aims to foster the implementation of existing legislation and guidelines in the area of open government with open government consisting of two areas: transparency as defined by Law no. 544/2001 and participation as defined by Law no. 52/2003. Having defined compliance with those provisions as the desired outcome, the Index could assign scores to public institutions in the areas of transparency and participation depending on their practice. For example, one point could be assigned for each type of information that a public institution proactively publishes as required by Law no. 544/2001. Going beyond assessment of availability, the Index could also provide higher scores to public institutions if they provide this information in a machine-readable format and thereby improve reusability, as recommended in GSG guidelines.