Chapter 5. Monitoring and evaluation framework
Designing and implementing open government initiatives requires resources and changes in the public administration. While these initiatives are intended to improve the relationship between government and its citizens, as well as enhance transparency and accountability, this can only be ensured and confirmed, and therefore the use of funds and efforts justified, through effective monitoring and evaluation (M&E) systems. A solid M&E system is necessary for assessing if intended goals are achieved, for identifying challenges and obstacles, and for rectifying initiatives accordingly.
Monitoring refers to “a continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing […] intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds” (OECD, 2009[39]). Therefore, it aims to ensure that the initiative is on track and that it is achieving the intended results, enabling initiatives to be to modified and adapted if necessary.
Evaluation refers to “the systematic and objective assessment of an ongoing or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, […] efficiency, effectiveness, impact and sustainability” (OECD, 2009[39]). Thus, it allows for the cost-effectiveness of an initiative to be measured, provides information to those implementing policy, as well as the larger public, on whether the initiative is achieving its intended impact, and allows for future initiatives to be enhanced.
M&E systems can therefore enhance the implementation of open government initiatives, and increase their visibility and impact. This requires, however, closing the feedback loop and using the M&E results to improve ongoing and future initiatives. In line with open government principles, M&E results should be made available to the public to enhance accountability and enable citizens to scrutinise government actions. Similarly, the public should be involved in M&E systems, especially as most open government initiatives intend to affect citizens and their perception of government directly.
A large majority of OECD countries (86%) monitor open government initiatives, yet only 59% evaluate these initiatives. Almost all countries that evaluate initiatives communicate the results. However, most countries have not put in place specific M&E systems regarding open government (except those required by the OGP), but are building upon the existing M&E frameworks of their public administration (OECD, 2016[2]). The OECD Recommendation on Open Government therefore suggests that countries:
“Develop and implement monitoring, evaluation and learning mechanisms for open government strategies and initiatives by:
1. Identifying institutional actors to be in charge of collecting and disseminating up-to-date and reliable information and data in an open format.
2. Developing comparable indicators to measure processes, outputs, outcomes and impact in collaboration with stakeholders.
3. Fostering a culture of monitoring, evaluation and learning among public officials by increasing their capacity to regularly conduct exercises for these purposes in collaboration with relevant stakeholders.” (OECD, 2017[1])
One of the great challenges OECD member and partner countries currently face in the area of open government is to move the focus from processes to outcomes and impact (OECD, 2019[40]). The implementation of open government strategies usually involves initiatives in a variety of areas and requires the involvement of multiple stakeholders. Given this multidimensional and cross-cutting nature, open government initiatives are difficult to monitor and evaluate. However, solid monitoring and evaluation (M&E) mechanisms can help to ensure that policies are achieving the intended goals, contribute to the identification of policy design and implementation barriers, and orient policy choices by building on past experiences (OECD, 2019[40]). M&E is also instrumental to initiating changes and communicating policy results in a timely and accessible manner. Examples from OECD member and partner countries include:
Spain has established a dashboard for monitoring its third open government action plan. The progress made is updated every three months in all the available categories, including axis, commitment and category. Stakeholders can also provide comments through a questionnaire available for each commitment. The dashboard provides detailed information on progress, including briefing notes, outcomes, dates for each activity, and the state of implementation of each activity. A general summary is provided with the progress made on the overall plan. This dashboard provides valuable data to monitor the implementation of the plan.
Mexico’s open government metrics were developed by the Centre for Economic Research and Teaching (CIDE) and based on an initiative of the National Institute for Transparency, Access to Information and Personal Data Protection (INAI). The metrics are designed as a baseline to measure the current state of the National System of Transparency, Access to Information and Protection of Personal Data (SNT) and its open government and transparency policies. Aiming to be an “x-ray of the starting point of the open government policy of the Mexican State” at the national and subnational level, its focus goes beyond measuring compliance with regulations and aims to capture performance information on the outcomes of open government and transparency policies from the perspective of both government and citizens. The metrics start with an operational definition of open government that is structured around two dimensions: transparency and public participation. Each dimension is then approached from two perspectives: government and citizens. The metrics survey included a sample of 908 governmental bodies at the national and subnational level; 754 portals were reviewed and 3 635 requests for information were sent. The resulting Open Government Index of Mexico was 0.39 (on a scale of 0 to 1). The index showed that the transparency dimension had a much higher value (0.50) than the participation dimension (0.28).
Argentina has integrated the monitoring and evaluation of its open government initiatives into the wider framework provided by the State Modernisation Plan (adopted in 2016). Under the leadership of the Office of the Chief of Cabinet of Ministers, the Government Secretariat of Modernisation has developed an integral system to standardise planning, monitoring and evaluation in co-operation with all line ministries. This system contains several dashboards to monitor the progress on open government initiatives, namely the Results Management Dashboard (Tablero de Gestión por Resultados), the Integral Management Dashboard (Tablero de Gestión Integral) and the Strategic Monthly Report (Informe Mensual Estratégico). With these tools, the public can track the development and implementation of key open government initiatives, such as the follow-up of the third OGP Action Plan. In addition, to offer citizens the possibility of assessing efforts in implementing priority projects, a Citizen’s Dashboard was launched in 2018. The five key areas that the dashboard covers are open government, public employment, digital government, digital inclusion and connectivity. It includes information on 20 different projects and provides information on their content, impact and progress, with process and output indicators.
Sources: OECD (2019[6]), Open Government in Argentina, OECD Public Governance Reviews, https://doi.org/10.1787/1988ccef-en; OECD (2019[41]), Open Government in Biscay, OECD Public Governance Reviews, https://doi.org/10.1787/e4e1a40c-en; Datos.gob (2020[42]), “Open Data Initiative of the Government of Spain”, https://datos.gob.es/en.
Lebanon has established an M&E system in the public administration that builds on internal monitoring and M&E conducted by an external institution, namely the Central Inspection Board (CIB). The internal monitoring system is based on key performance indicators (KPIs) and sub-KPIs, as is used for the digital transformation strategy and the action plan for implementing the Right of Access to Information law. Some of these KPIs refer to open government indicators such as “inclusive citizen/customer-oriented policy making”, according to the OECD survey. Table 5.1. shows the KPIs prepared for “citizens centricity” in the framework of the draft Digital Transformation Strategy.
OMSAR is supporting the public administration in applying these KPIs and in building the capacities of public officials and inspectors. It is also automating the methodology through the development of a web-based solution, which should facilitate reporting to the CIB. OMSAR also intends to create sectoral and organisational performance planning and monitoring units in all institutions. However, the public administration still faces challenges in monitoring its activities due to insufficient expertise and resources in the field. There is equally a lack of reliable data, the publication of results could become a more common feature and engaging citizens in the process is rare. The OECD survey found that OMSAR conducted citizen satisfaction surveys on e-government services in 2013 and surveys to prioritise digital transformation projects in 2019. This is a practice other institutions could build on, and the results should be made publicly available. Given OMSAR’s role regarding open government and supporting the public administration on M&E, OMSAR could work with those ministries that are to be involved in the open government sub-committee on building an M&E culture, expertise and indicators for the open government initiatives included in the open government action plan. The existing e-learning platform (see Chapter 4) could be complemented with a course on M&E focusing in particular on the M&E of open government initiatives and stakeholder engagement in M&E.
As per legislative decree 111/1959, all directors-general should submit biannual and annual reports to central control agencies (Article 7, item 4). According to Circular 40/1963, the CIB issued instructions on the conduct of inspection processes for public sector entities. OMSAR, together with the CIB, is currently enforcing this with six pilot ministries. The OECD survey found that one of the KPIs that should be included in these reports relates to open government. In order to strengthen the M&E of open government initiatives, OMSAR and the CIB could enhance their expertise in this field and develop specific indicators that build upon the existing KPIs and that evaluate the process, output, outcome and impact of open government initiatives. Such a pilot project could focus on the open government initiatives to be included in the open government action plan.
Source: OECD (2019[6]), Open Government in Argentina, https://dx.doi.org/10.1787/1988ccef-en.
According to interviews with the CIB and other actors, vacancies among inspectors does not allow the CIB to effectively practice control mechanisms over the public administration. While it is important to reinforce their capacities, involving stakeholders in M&E could also support a more effective monitoring and evaluation of open government initiatives. In several OECD countries, civil society organisations are members of the open government steering committee, which enables them to monitor the implementation of initiatives.
Continue efforts to strengthen Lebanon’s M&E system by automating the process and providing capacity building for the public administration.
Apply open government principles to the M&E system by systematically engaging with all relevant stakeholders and publishing the results of M&E, including the results of the different institutions and the CIB.
Build an M&E culture and system regarding open government initiatives by developing indicators for the open government initiatives. These could then be included in the future open government action plan and complemented by an M&E learning course with a focus on open government and stakeholder engagement.
References
[7] Datos.gob (2020), Open Data Initiative of the Government of Spain, https://datos.gob.es/en (accessed on 29 June 2020).
[8] Government of Lebanon (2019), Lebanon Digital Transformation: Strategies to Actions.
[4] OECD (2019), Budgeting and Public Expenditures in OECD Countries 2019, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264307957-en.
[5] OECD (2019), Open Government in Argentina, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/1988ccef-en.
[6] OECD (2019), Open Government in Biscay, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/e4e1a40c-en.
[3] OECD (2017), Recommendation of the Council on Open Government, https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0438.
[2] OECD (2016), Open Government: The Global Context and the Way Forward, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264268104-en.
[1] OECD (2009), “OECD DAC Glossary”, in Guidelines for Project and Programme Evaluations, OECD, Paris, http://www.oecd.org/development/evaluation/dcdndep/47069197.pdf (accessed on 29 October 2019).