Chapter 4. Building a monitoring and evaluation framework for open government

This chapter assesses the internal capacities and current initiatives in place in the province of Biscay to build a monitoring and evaluation (M&E) system for its public policies, as a commitment included in its Open Government Action Plan 2017-19. It examines the institutional framework with a special emphasis on the actors involved in the development of the system. Furthermore, it focuses on the monitoring and evaluation of Biscay’s open government strategies and initiatives as a tool to enhance a new culture of governance.

Introduction

A robust monitoring and evaluation (M&E) system is essential to ensure that open government strategies and initiatives are achieving their intended goals. It serves to highlight the achievements, relevance and visibility of open government initiatives across government and therefore provides incentives to ensure that public policies are designed and delivered from an open government perspective. Sound M&E can also help to identify challenges and obstacles that hinder effective policy implementation, as well as show the way forward to address the challenges, based on lessons learned about what has worked (or not) in the past.

Given the strategic relevance of M&E, the OECD Recommendation of the Council on Open Government (hereafter the “OECD Recommendation”) stresses the importance of monitoring and evaluating open government strategies and initiatives. Provision 5 calls on adherents to:

“Develop and implement monitoring, evaluation and learning mechanisms for open government strategies and initiatives by:

i. identifying institutional actors to be in charge of collecting and disseminating up-to-date and reliable information and data in an open format

ii. developing comparable indicators to measure processes, outputs, outcomes, and impact in collaboration with stakeholders

iii. fostering a culture of monitoring, evaluation and learning among public officials by increasing their capacity to regularly conduct exercises for these purposes in collaboration with relevant stakeholders.” (OECD, 2017[1])

Acknowledging the importance of M&E, the Provincial Council of Biscay (Diputación Foral de Bizkaia, hereafter “Biscay”), which constitutes the government of the Province, has committed to:

1. using monitoring and evaluation as an enabler for openness, by developing a whole-of-government M&E system
2. monitoring and evaluating open government strategies and initiatives.

Therefore, the first part of this chapter assesses the efforts of the province of Biscay (legally referred as “historic territory”1) to build a monitoring and evaluation system for its public policies, as a commitment included in its Open Government Action Plan 2017-19 (OGAP). The second part of the chapter focuses on the monitoring and evaluation of open government strategies and initiatives, assessing Biscay’s efforts to collect relevant information and to develop comparable indicators as referred to in Provision 5 of the OECD Recommendation.

Towards a robust monitoring and evaluation system

Conceptualising monitoring and evaluation

Notwithstanding their complementarity, monitoring and evaluation are two different practices, with different dynamics and goals. Policy monitoring refers to a continuous function that uses systematic data collection on specific indicators to provide policy makers and stakeholders with information regarding progress and achievements of an ongoing public policy initiative and/or the use of allocated funds (OECD, 2018[2]) (OECD, 2016[3]) (OECD, 2009[4]). It contributes to planning and operational decision making, as it provides evidence to measure performance and can help to raise specific questions in order to identify implementation delays or bottlenecks. It can also strengthen accountability related to the use of resources, the efficiency of internal management processes, or the outputs of a given policy initiative (OECD, 2017[5]).

Policy evaluation refers to the structured and objective assessment of the design, implementation and/or results of a future, ongoing or completed policy initiative. The aim is to determine the relevance and fulfilment of policy objectives, as well as to assess dimensions such as public policies’ efficiency, effectiveness, impact or sustainability. As such, policy evaluation refers to the process of determining the worth or significance of a policy (OECD, 2018[2]) (OECD, 2016[3]) (OECD, 2009[4]). It serves two main purposes. It fosters learning by helping policy makers understand why and how a policy was successful or not. Consequently, it contributes to strategic decision-making, by providing insights on how to improve the links between policy decisions and outcomes. In addition, policy evaluation promotes accountability as it provides citizens and a broad range of stakeholders – such as journalists and academics - with information whether the efforts carried out by the government, including the financial resources mobilised for them, are producing the expected results (OECD, 2017[5]).

Therefore, while monitoring is descriptive and an important (but not exclusive) source of information that can be used within the context of an evaluation, policy evaluation is a different activity that seeks to analyse and understand cause-effect links between a policy intervention and its results. Table 4.1 highlights the main distinguishing traits of both functions.

Table 4.1. Comparing policy monitoring and policy evaluation
Policy monitoring	Policy evaluation
Ongoing (leading to operational decision-making)	Episodic (leading to strategic decision-making)
Monitoring systems are generally suitable for the broad issues/questions that were anticipated in the policy design	Issue-specific
Measures are developed, and data are usually gathered through routinised processes	Measures are usually customised for each policy evaluation
Attribution is generally assumed	Attribution of observed outcomes is usually a key question
Because it is ongoing, resources are usually a part of the programme or organisational infrastructure	Targeted resources are needed for each policy evaluation
Use of the information can evolve over time to reflect changing information needs and priorities	The intended purposes of policy evaluation are usually negotiated upfront
Source: Adapted from McDavid, J.C. and Hawthorn, L.R.L. (2006[6]), Program evaluation and performance measurement, an introduction to practice, Thousand Oaks, California: Sage, in OECD (2017[5]), “Towards Open Government Indicators: Framework for the Governance of Open Government (GOOG) Index and the Checklist for Open Government Impact Indicators”, Concept Note, internal document.

Developing a strategy to build a robust M&E system

Sound policy monitoring and evaluation means that M&E is part and parcel of the policy cycle; that M&E is carried out rigorously and systematically; that decision makers use its results; and that information is readily available to the public (see also Lazaro, (2015[7]). It offers policy makers the tools and evidence to detect policy challenges, to adapt or adjust public policies, to understand their viability or their failures, as well as to communicate policy results in a timely and accessible manner (OECD, 2016[3]). A robust M&E system implies the presence of:

An institutional framework for M&E that provides: (a) the legal basis to undertake M&E; (b) macro-level guidance on when and how to carry out M&E; and (c) clearly mandated institutional actors with allocated resources to oversee or carry out M&E.
An M&E culture, including - among others - the promotion of the quality and use of policy evaluations across government, through a skilled public service and appropriate stakeholder engagement mechanisms.

The OGAP’s commitment to build an M&E system

The development of an M&E system is a high priority for Biscay’s 2015-19 administration. The strategic incentive to build such a system is provided by the Open Government Action Plan (OGAP) 2017-19, which regroups the Province’s main open government initiatives. The plan describes Biscay’s open government vision (see Box 4.1) and identifies an open government as a “government that evaluates in order to make responsible and efficient use of public resources by measuring, monitoring and evaluating public policies”, among other things.

Box 4.1. Biscay’s open government vision

Biscay’s open government vision is defined by the following characteristics (or particularities):

a relational government that seeks timely and constant follow-up in order to better know its citizens’ needs and direct public policies in a more efficient way
a transparent government that uses ambitious tools to make transparency a means for easier access to information and to contribute to improving efficiency in public management
a participative government with a joint responsibility perspective, which creates new spaces and processes for participation across the entire policy cycle
an accountable government with new spaces to subject the government administration to public scrutiny, including a government strategic plan subject to annual reviews
a government that evaluates in order to make responsible and efficient use of public resources by measuring, monitoring and evaluating public policies
a collaborative government open to co-operating with other stakeholders.

Source: Provincial Council of Biscay (2017[8]), Bizkaia Irekia: Plan de Acción de Gobierno Abierto (Open Biscay: Open Government Action Plan), http://gardentasuna.bizkaia.eus/documents/1261696/1397467/Plan+de+Gobierno+Abierto.pdf/d96264cf-022e-a2c0-3919-e1778372436c.

Biscay’s OGAP lists 6 priorities/thematic areas and 14 commitments (see Table 4.2). Commitment 5 explicitly refers to the “establishment of a system allowing the monitoring of the evolution of the territory and the performance of the Provincial Council of Biscay through an internal and external evaluation management” (Provincial Council of Biscay, 2017[8]) (see Table 4.2 and Box 4.2). While the OGAP clearly includes a reference to evaluation (both in its title and narrative), exchanges with public servants during the OECD fact-finding mission indicate that the distinction between monitoring, on the one hand, and evaluation, on the other hand, is not always clear and leads to confusion.

Table 4.2. Open Government Action Plan of Biscay: Priorities and commitments
Priorities/thematic areas	Commitments
Transparency and open data	Commitment 1. Create and implement a transparency plan for the province. Commitment 2. Launch the open data service of Biscay. Commitment 3. Open provincial budgets.
Accountability	Commitment 4. Promote accountability initiatives in the management of the Provincial Council of Biscay. Commitment 5. Define a policy evaluation system for the Provincial Council of Biscay.
Citizen participation	Commitment 6. Develop a model for citizen participation. Commitment 7. Foster social participation in gender equality policies.
Technological development and innovation	Commitment 8. Develop technological tools for the municipalities in Biscay. Commitment 9. Set up a new comprehensive model to provide citizen services. Commitment 10. Reduce bureaucracy.
Promoting collaboration between the public and private sectors	Commitment 11. Generate economic value through collaborative initiatives with businesses within the territory. Commitment 12. Launch projects that bring together the public and private sectors to generate social value. Commitment 13. Launch projects that bring together the public and private sectors and involve young people to generate social value.
Strengthening public integrity	Commitment 14. Create and develop a provincial law for integrity, conflict of interest and incompatibilities.
Source: Provincial Council of Biscay (2017[8]), Bizkaia Irekia: Plan de Acción de Gobierno Abierto (Open Biscay: Open Government Action Plan), http://gardentasuna.bizkaia.eus/documents/1261696/1397467/Plan+de+Gobierno+Abierto.pdf/d96264cf-022e-a2c0-3919-e1778372436c.

Box 4.2. Commitment 5 of the Biscay Open Government Action Plan

This commitment, under the responsibility of the Observatory of Biscay, aims to define a policy evaluation system for the Provincial Council of Biscay, allowing for the assessment of the socio-economic situation of the territory and measuring the performance of the administration through internal and external evaluations. Based on the values enacted by the Open Government Partnership (OGP), this commitment envisages to provide elements for the articulation of an efficient and transparent accountability system. In order to do so, the commitment aims to deliver the following tools:

Social Barometer of Biscay (Barómetro Social de Bizkaia): A tool for quantitative evaluation comprised of social perception indicators.
Neurbi: A tool for qualitative evaluation comprised of perception-based indicators of the Council’s performance.
Service charters (Cartas de servicio): A tool to improve service delivery and a methodology of institutional communication with the public.
A dashboard of context indicators for the territory.
A dashboard of management indicators of the strategic plans of the Council’s departments.

As outlined in Box 4.2, the government’s commitment to developing an M&E system is expressed in the OGAP and is outlined in some specific deliverables, such as the creation of evaluation tools and indicators. However, building specific tools for data collection and carrying out quantitative and qualitative evaluations successfully also requires an underlying institutional framework, including the allocation of the necessary financial and human resources to carry out these tasks, as well as a strategy (roadmap, timeline and specific actions) to gradually develop an M&E system. A clear institutional framework supports the implementation of M&E activities and can provide guidance to anticipate potential legal, administrative and/or financial challenges.

Designing an institutional framework for policy M&E

An M&E institutional framework consists of specific legal and/or policy provisions and can include, among others:

the institutional set-up for policy M&E, allocating responsibilities, resources and objectives to different institutional actors (ministries, departments, etc.)
the identification of policies to be monitored and evaluated, based on criteria such as thematic considerations (government’s priorities) or budgetary thresholds (policies exceeding certain costs)
the definition of courses of action for commissioning M&E exercises.

These provisions can be embedded in the Constitution, primary legislation (laws or equivalent), secondary/subordinate legislation or administrative acts such as government strategic plans. The formal anchorage of such provisions in legal and policy documents can facilitate their enforcement. While Biscay currently does not have a government-wide legal or policy framework guiding M&E across government, there are some legal provisions that identify key actors with shared M&E responsibilities:

Observatory of Biscay (Observatorio de Bizkaia, hereafter “Behatokia”): Created by the Provincial Decree 17 of 2016, it is part of the Deputy General Cabinet.2 According to its mandate, Behatokia is the main body responsible for the M&E agenda in the Provincial Council of Biscay. Article 6 of the Provincial Decree 17 of 2016 states that Behatokia should “propose a system/model of planning, organisation, administration, oversight and evaluation of common plans and policy actions of the departments of the Provincial Council of Biscay (or provincial ministries, hereafter “Departments”), except in those cases assigned to the Cabinet of Modernisation, Good Governance and Transparency” (Boletin Oficial de Bizkaia, 2016[9]). Moreover, it is stipulated that in terms of the M&E agenda, Behatokia is responsible for assisting the departments in applying the monitoring system/model (once in place), for encouraging the creation of a data repository, as well as for monitoring all plans and policy actions of the Biscay government. Furthermore, as explained below, Behatokia is responsible for Commitment°5 of the OGAP 2017-19 that refers to the development and establishment of an evaluation system of public policies.3
Cabinet of Modernisation, Good Governance and Transparency (Gabinete de Modernización, Buen Gobierno y Transparencia): Created by the Provincial Decree 86 of 2016, it is one of the bodies of the Department of Public Administration and Institutional Relations.4 It is composed of a Strategic Planning Office (Jefatura de Gabinete) and two directorates: General Directorate of Public Administration Modernisation (Dirección General de Modernización de la Administración) and the General Directorate of Good Governance and Transparency (Dirección General de Buen Gobierno y Transparencia). According to its mandate, it is the leading body in charge of “formulating strategic guidelines and priority areas of action to guide planning of the Provincial Council, its Departments and the entities of the Provincial public sector” (Boletin Oficial de Bizkaia, 2016[10]). In relation to Biscay’s M&E agenda, the General Directorate of Good Governance and Transparency is mandated, within the Cabinet, to “define and launch a model of evaluation and impact of public policies, Provincial services and programmes” (Boletin Oficial de Bizkaia, 2016[10]).
Lantik: A state-owned enterprise owned by the Provincial Council of Biscay, its objective is “to provide to the Provincial Council, its subsidiary organisms and institutions as well as municipalities of Biscay through BiscayTIK, data-processing systems and their management” (Bizkaia, 2018[11]). More specifically, this entity is in charge of Biscay’s data collection platform. This platform is a tool used by the General Directorate of Modernisation to monitor, through progress indicators, plans and actions carried out by the different departments of the Provincial Council.

While the Directorate General of Judicial Regime and Civil Service (Dirección General del Régimen Jurídico y Función Pública) does not play a formal role in the M&E process, it is worth mentioning. The directorate is part of the Department of Public Administration and Institutional Relations. According to its mandate, it is the body responsible for the “promotion and elaboration of training programmes and the professional development of public officials” (Provincial Decree 86, Article 20). The General Directorate also developed an ethical code for public officials that they must adhere to when employed by the government. While both the ethical code and the training modules do not have a particular M&E dimension for the time being, this could be considered for the future.

Table 4.3 lists the responsibilities of the bodies in charge of M&E in Biscay, using a list of common tasks performed by institutions responsible for M&E across government in OECD countries as a framework.5

Table 4.3. Responsibilities for monitoring and evaluation within Biscay’s public entities
Functions	Behatokia	General Directorate of Good Governance and Transparency	General Directorate of Modernisation	Civil service
Defining and updating the evaluation policy	X	X
Developing guideline(s) for policy evaluation		X
Providing incentives for carrying out policy evaluations
Undertaking policy evaluations	X	X
Requiring government institutions to undertake specific policy evaluations
Defining course of action for commissioning evaluations
Developing skills, competences and/or qualifications of evaluators				X*
Developing standards for ethical conduct				X*
Design data collection tools supporting policy monitoring		X	X
Ensuring quality standards of evaluations
Ensuring the quality of monitoring data
Promoting stakeholder engagement in policy monitoring and evaluation	X
Overseeing the monitoring and evaluation calendar and reporting	X
Reporting monitoring results	X
Following up on evaluation reports
Serving as a knowledge centre and providing a platform for exchange on M&E	X	X
Use of data collected by policy monitoring
Promoting the use of evaluation findings into policy making
Note: *the Civil Service has two functions: developing skills, competences and/or qualifications of evaluators as well as developing standards for ethical conduct. However, these are not linked to M&E.
Behatokia stands for General Directorate of the Observatory of Biscay and Civil service stands for Directorate-General of the Legal Regime and Civil Service.
Source: Author.

Table 4.3 shows that Behatokia and the Directorate General of Good Governance and Transparency (hereafter the “Directorate”) share responsibilities in the areas of: 1) defining and updating the evaluation policy; 2) undertaking policy evaluations; and 3) serving as a knowledge centre and providing a platform for exchange. While sharing responsibilities in the area of M&E does not represent a challenge per se, it can generate overlaps if these institutions do not always work in a co-ordinated way. A certain degree of overlap was mentioned during the OECD fact-finding mission, as several stakeholders referred to some confusion regarding the division of labour between the two bodies, more specifically concerning the responsibility for providing guidance on M&E across government. Furthermore, stakeholders expressed the need to clarify which institution - Behatokia or the Directorate - would lead the development of a model for policy evaluation, as both of them seem to be responsible for this function. Moreover, and as can be observed in Table 4.3, the current institutional mandates also show some gaps regarding responsibilities for defining the course of action for commissioning evaluations; developing skills, competences and/or qualifications of evaluators; or ensuring quality standards on M&E. All of these are potentially important factors for the development of a robust M&E system.

There is no one-size-fits-all model for setting up an institutional framework for M&E (see Box 4.3). In some countries, M&E across government is the responsibility of different institutions. For instance, while the management of a government-wide monitoring system can be the responsibility of the centre of government (Prime Minister Office, Cabinet Office, or equivalent), actual policy evaluation can fall under a line ministry or an independent agency. However, a clear division of labour - when several institutions are involved - is essential.

Box 4.3. Examples of institutions in charge of an M&E system in OECD countries

Centre of government

Finland

The centre of government of Finland, composed of the Ministry of Finance, the Ministry of Justice and the Prime Minister’s Office, exercises the competences related to policy evaluation. In order to enhance the use of evidence, the government established in 2014 the Policy Analysis Unit under the Prime Minister’s Office. The unit has the mandate to commission research projects and present evidence to support the government’s decisions on future strategic and economic policy. The unit exercises strategic oversight to co-ordinate ministerial collaboration to implement the government’s agenda, promoting a culture of self-reporting and partnership and decentralising the development and implementation to the ministries. The unit functions with independent funding and a fixed number of human resources in charge of co-ordinating research procurement, organising the process with ministries and systematically publishing the results for public consultation.

Autonomous agency

Mexico

The National Council of Social Development Policy Evaluation (Consejo Nacional de la Política de Desarrollo Social, CONEVAL), was created in 2004 as a decentralised body with budgetary, technical and management autonomy. It has the mandate (embedded in the Constitution in 2014) to set the standards and co-ordinate the evaluation exercises of the National Social Development Policy and its subsidiary actions and provide the guidelines to define, identify and measure poverty. The agency carries out or commissions the evaluation exercises of the social policies developed by the Mexican government.

Ministry of Finance

Chile

The Budgets Directorate (Dirección de Presupuestos), a dependent body of the Ministry of Finance (Ministerio de Hacienda), is the technical body in charge of ensuring the efficient allocation and use of the public funds. In order to do so, the Budgets Directorate carries out ex ante, impact and value for money evaluations of different government policies and programmes. Moreover, it monitors the implementation of the government programmes to collect performance information that is then introduced in the budgetary process and communicated to stakeholders.

Source: Knowledge Sector Initiative (2017[12]), “Global evidence units – Finland: Government Policy Analysis Unit”, http://www.ksi-indonesia.org/file_upload/Evidence-Policy-Unit-in-Finland-the-Government-Po-14Jun2017163532.pdf; Secretaría de Desarrollo Social (2005[13]), “Decree for which the Council of Social Development Policy Evaluation is regulated”, Decree 24/08/2005, DOF, https://www.coneval.org.mx/quienessomos/Conocenos/Paginas/Funciones.aspx, Direccion de Presupuestos, Gogierno de Chile (n.d.[14]), “Documentos en vedette”, http://www.dipres.cl/598/w3-channel.html, accessed 10 December 2018.

Considering the elements presented above, Biscay could invest in the further development of a coherent monitoring and evaluation system, supported by a sound institutional framework. More specifically, and as Biscay’s monitoring and evaluation system is currently under development, the government could consider:

Centralising responsibilities regarding M&E into a single institution in charge of 1) developing a whole-of-government M&E system; and 2) promoting the use and quality of monitoring and evaluation across government. Such an institution could be responsible for conducting M&E across government and/or developing guidelines, training courses and quality checks for other departments wishing to carry out their own M&E. In cases where other institutions have the lead (e.g. Department of Public Administration and Institutional Relations for the area of training), the co-ordinating institution could ensure a coherent and collaborative approach across institutions. In addition, the co-ordinating institution could promote the use of evidence resulting from M&E efforts, as evidence “needs to be delivered to those who actually make the decisions, at the right time and in the right format” (OECD, 2017[1]).
Establishing a co-ordination instrument, such as an M&E council or committee, to co-ordinate M&E across government at an operational level. The case of the Committee for Monitoring and Evaluation of Federal Public Policies (CMAP) in Brazil presents an interesting example of such a co-ordination instrument (see Box 4.4).

Box 4.4. Committee for Monitoring and Evaluation of Federal Public Policies (CMAP) in Brazil

The Committee for Monitoring and Evaluation Federal Public Policies (CMAP under the co-ordination of the Ministry of Planning) was created in 2016 with the objective of improving the actions, programmes and public policies of the federal executive branch, as well as to improve the allocation of resources and the quality of public spending.

The committee regroups representatives of the Ministries of Planning, Budget and Management, Ministry of Finance, the Civil House of the Presidency of the Republic and the Office of the Comptroller General of the Union with the special participation of members of invited institutions, public or private.

Its role is to define the policies, programmes and actions that will be monitored and evaluated, and propose guidelines to improve them using thematic committees. Moreover, the committee makes recommendations to policy makers on the adoption of adjustments and improvements anchored in the principles of transparency and accountability.

Source: Diário oficial da Uniao (2016[15]), “Portaria interministerial nº 102”, 7 April 2016.

Regardless of the institutional set-up chosen, body/ies in charge of M&E benefit from having a clear mandate that sets specific tasks and responsibilities regarding the M&E agenda. Such a clear mandate could subsequently facilitate the development of a full-fledged M&E strategy, as illustrated in Box 4.5.

Box 4.5. Monitoring and evaluation strategies

Several governments have developed strategies to structure the design, implementation and strengthening of their monitoring and evaluation systems, both from a government-wide perspective and from the sector level.

South Africa 2015-2020 Strategic Plan

The Department of Planning, Monitoring and Evaluation (DPME) of the Presidency of South Africa launched a 2015-2020 Strategic Plan, which is based on the National Development Plan (NDP) 2030. The Strategic Plan aims to set the DPME strategic priorities for the medium term, including on how to foster the co-ordination and monitoring of the implementation of the NDP. The plan outlines approaches to further strengthen institutional performance across government through regular monitoring, evaluation and support.

The Strategic Plan includes:

A situational analysis (assessment of the performance, organisational and financial environment).
The definition of the strategic, outcome-oriented goals as well as five key strategic areas:
- administration
- outcomes monitoring and evaluation
- institutional performance monitoring and evaluation
- national planning
- a national youth development programme.
Each of these areas includes an objective statement as well as baseline information, explicit links with the NDP, expected outputs, indicators, and five-year targets.

UK Department for Transport

The UK Department for Transport (DfT) launched a Monitoring and Evaluation Strategy in 2013. According to the DfT website, “the strategy sets out a framework for generating good quality monitoring and evaluation evidence in order to inform decision making.” The strategy defines three objectives:

establish a proportionate Monitoring and Evaluation Programme
ensure a robust governance framework for monitoring and evaluation activity
embed a culture of monitoring and evaluation.

The DfT also published a Monitoring and Evaluation Programme in 2013, which identified and reported the department’s main activities on M&E. An updated version of this programme was published in 2016.

Source: Republic of South Africa, The Presidency (n.d.[16]), “Department: Planning, Monitoring and Evaluation: Strategic Plan 2015-2020”, https://www.dpme.gov.za/publications/Strategic%20Plan%20and%20Annual%20Reports/DPME%20Strategic%20Plan%202015-2020.pdf; GOV.UK (2013[17]), “Department for Transport monitoring and evaluation strategy”, Department for Transport, https://www.gov.uk/government/publications/monitoring-and-evaluation-strategy; Department for Transport (2013[18]), “Monitoring and Evaluation Programme”, DfT, London, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/249242/monitoring-evaluation-programme.pdf.

Fostering an M&E culture

The development of a sound institutional framework that ensures systematic policy M&E does not constitute an end in itself. For instance, poor quality M&E data could hamper learning, accountability, decision making and policy design. Likewise, high-quality M&E may be completely ignored due to a lack of incentives or skills for using it in policy making.

Fostering an M&E culture among public officials and stakeholders can be a concrete action to promote the quality and uptake of M&E results. Robustness and validity of M&E data will be enhanced if public servants feel better equipped (in terms of knowledge and skills) in how to collect and/or manage M&E information and how to ensure both quality control (deliverable-oriented) and quality assurance (process-oriented, i.e. doing the right thing, the right way).

One of the challenges that Biscay is facing today is the lack of capacity-building activities (training courses) and tools (guidelines) to build skills development in M&E. Therefore, Biscay could consider offering specific training courses on policy M&E for public officials, as part of the menu of training courses that the Directorate-General of the Legal Regime and Civil Service offers, and in parallel, developing guidelines to conduct M&E. Training and guidelines could address, among other things, the following aspects:

design of M&E strategies and plans
course of action for commissioning evaluations
identification of human and financial resources for M&E
design of data collection methods
quality standards of M&E
independence of M&E
ethical conduct for M&E activities
stakeholder engagement in the M&E process
reporting on M&E results
use of evidence collected by M&E

A particular point of attention regards stakeholder engagement throughout the M&E process. As Biscay is taking its first steps towards the development of an M&E system, the Province does not yet function with dedicated stakeholder engagement tools for M&E. However, Biscay has in place a collaborative platform for social policies called Civil Dialogue Table (Mesa de Diálogo Civil) (see Box 4.6) that was created to discuss, design and implement policies related to the ‘third social sector’6 as a collegiate body. This Civil Dialogue Table, described in more detail in Chapter 5, is chaired by the President of Biscay and includes the participation of civil society representatives and public servants from the Department of Social Action. Currently, Behatokia is not a member.

Box 4.6. Civil Dialogue Table of Biscay

Created by the Provincial Decree 154/2016, the Civil Dialogue Table is the highest body of civil dialogue and co-operation between the Provincial Council of Biscay and the group of institutions of the third social sector. Its purpose is to:

promote social policies, participation, social cohesion and stakeholder engagement
provide a channel for the beneficiaries of social programmes to express their opinions and participate in the design, implementation and evaluation of public policies.

This collegiate organ chaired by the President of Biscay operates with the participation of representatives from 14 institutions of the third social sector and representatives from the Department of Social Action and Employment. Its members meet three times a year to discuss, plan and analyse the activities and programmes that the organ conducts.

Source: Provincial Council of Biscay (2016[19]), “Foral decree formalising and regulating the Mesa de diálogo civil de Bizkaia”, Foral decree 154 of 2016, BOB 203.

Given the importance of M&E for social policies, as the delivery of social policies has an immediate impact on citizens, this Civil Dialogue Table could constitute a key platform to identify key stakeholders and engage with them on the monitoring and evaluation of social issues. Biscay could consider including Behatokia in this Table. This would allow Behatokia to engage with key stakeholders from the social sector in a systematic and structured way. In particular, Biscay could use this platform to promote the establishment of a – formal or informal - network of policy monitoring and evaluation practitioners, which could include representatives of academia as well as other stakeholders.

Monitoring and evaluation of open government strategies and initiatives in Biscay

Provision 5 of the OECD Recommendation calls upon countries to “develop and implement monitoring, evaluation and learning mechanisms for open government strategies and initiatives.” In this regard, the current mandates of Behatokia and the Directorate on M&E, as well as their roles in advancing the open government agenda, provide an opportunity to strengthen these areas of work in an integrated way. This would imply the systematic application of M&E to open government strategies and initiatives. In addition, these institutions can play a key role in ensuring that the training courses and guidelines (as recommended above) reach the open government community and, in particular, those public officials working on the design and implementation of the open government agenda.

The OECD Recommendation also calls for the collection of up-to-date information and the development of “comparable indicators to measure processes, outputs, outcomes, and impact in collaboration with stakeholders.” This section will thus assess Biscay’s current efforts in building and collecting information and indicators for the monitoring and evaluation of open government strategies and initiatives.

Collecting reliable and up-to-date information to accurately monitor the open government strategy

As stated in the OECD Recommendation, identifying institutional actors in charge of collecting and disseminating up-to-date and reliable information and data in an open format is key to developing and implementing monitoring, evaluation and learning mechanisms for open government strategies and initiatives. Biscay has decided to monitor the implementation of the OGAP by collecting information from citizens and its departments on the progress made for each commitment or priority axis on a yearly basis and reporting this information through intermediate self-evaluation reports.

In order to collect the information for the 2018 intermediate self-evaluation report of the OGAP (Provincial Council of Biscay, 2018[20]), Biscay developed a perception-based methodology that includes both citizens and public servants’ views on the implementation of the commitments and priority axes of the OGAP. The applied methodology consisted of two phases to assess the degree of fulfilment of the OGAP.

The first phase was composed of: 1) a citizens’ survey; and 2) a public servants’ survey for those involved in the implementation of the OGAP initiatives. The second phase was composed of interviews and questionnaires with public servants responsible for the delivery of each of the commitments in each of the departments.

During the first phase, the surveys aimed to assess the perception of progress made for each of the OGAP priority axes (Provincial Council of Biscay, 2018[20]). Participants were asked to state the degree of progress made for each of the priority axes of the plan, using a four-option scoring system (from “very low” to “high”). In addition, public servants were asked to state - using a scoring system from one to four - the degree of progress made in overcoming the challenges identified by the OGAP.7 By way of illustration, the questionnaire addressed to public servants asked the following questions:

“How much do you consider [the Provincial Council] has advanced on each of the challenges identified during the consultation phase of the design of the Open Government Action Plan 2017–2019?”
“How much do you consider [the Provincial Council] has advanced on each of the action axes structuring the Open Government Action Plan 2017-2019?”

Engaging citizens and public servants in the assessment of the progress made on the OGAP may have potentially fostered a sense of ownership of the open government agenda, as well as promoted its principles both inside and outside the public institutions. Nevertheless, data collected in this way is not necessarily completely reliable, as perception-based measurements of the efficacy of public institutions are usually biased by external factors (e.g. positive or negative perception of the government due to a period of economic prosperity or economic crisis) (Kurtz, Marcus J; Schrank, 2007[21]). This is partly because the information is collected through the voluntary participation of citizens, who usually participate because they have strong feelings (either good or bad) regarding the topic. Moreover, concerning the survey addressed to public servants, it is difficult to ensure that their answers reflect an overall assessment, rather than their individual contributions, given that respondents are also responsible for implementing the activities. Also, perceived progress does not necessarily accurately reflect actual progress.

During the second phase, information was collected through tailored questionnaires and bilateral interviews with those public servants responsible for the delivery of each of the commitments. This phase looked at three objectives: 1) obtaining a detailed view of the degree of progress made on each commitment in order to have an overview of the achievement of the plan as a whole; 2) identifying deviations and challenges in the implementation; and 3) identifying good practices (Provincial Council of Biscay, 2018[20]).

The collection of information through the questionnaire and the bilateral interviews was useful to identify good practices and lessons learned, as well to identify some challenges and opportunities for improvement. However, this perception-based methodology does not generate an objective assessment of progress made on each commitment. Also, while the methodology focuses on (perceived) achievements, it is not particularly helpful in unpacking the “why” of strong or weak performance (e.g. lack of skills, lack of incentives, etc.).

In sum, the current data collection process only provides a snapshot of the situation in a specific moment, completely driven by perception-based analysis. Moreover, the approach is very resource-intensive (e.g. in situ interviews), compared to the information yielded (i.e. implementation progress). Therefore, Biscay could consider developing a monitoring mechanism to collect regular, up-to-date quantitative data on the implementation of the different open government initiatives. This could be operationalised, for instance, through the creation of a dashboard embedded in Biscay’s data collection platform (described above). Such a monitoring mechanism would provide systematic data to assess performance, contributing to better planning and, hence, to better decision making and accountability. As an example of tools that improve the monitoring of the open government agenda, Box 4.7 presents Mexico and Spain’s Open Government Dashboards.

Box 4.7. A dashboard for monitoring open government strategies

Mexico

In its report on Mexico’s first Open Government Partnership (OGP) Action Plan, the Independent Reporting Mechanism (IRM) stated that Mexico should aim to strengthen the monitoring and evaluation of the commitments included in its Action Plan. In response, Mexico developed its own methodology to monitor and evaluate its OGP commitments and ensure subsequent communication. Mexico’s “Open Government Dashboard” identifies specific actions, deadlines and clear responsibilities, both for civil servants and for civil society; the data are public and include a control board powered by real-time information on the progress of each commitment. The Open Government Dashboard visualizes the advances or remaining challenges of each of the commitments, allows citizens to track the progress made so far on each open government commitment, and offers links to the government bodies in charge of the implementation to obtain further information and points of contact. In addition, the dashboard offers detailed explanations of the concrete actions that have been taken so far to fulfil the commitment.

Commitment 9 of Mexico’s second OGP Action Plan, for example, aims to enhance transparency and accountability at the national and sub-national level in state expenditures for public construction projects. In the respective part of the website, representatives from the Secretariat of Finance and Public Credit (SHCP) and the Mexican Institute for Competition (IMCO), which are responsible for the fulfilment of the commitment, comment on the progress made and the remaining challenges. The website provides additional information for interested citizens and other stakeholders.

Spain

Spain established a dashboard for monitoring its third open government action plan. The progress made is updated every three months in all the available categories, including axis, commitment and category. Also, stakeholders can provide comments through a questionnaire available for each commitment. The dashboard provides detailed information on the progress, including briefing notes, outcomes, dates for each activities, and the state of implementation of each activity, among other. A general summary is provided with the progress made on the overall plan. This dashboard provides valuable data to monitor the implementation of the plan.

Source: OECD (2016[22]), Open Government Data Review of Mexico: Data Reuse for Public Sector Impact and Innovation, OECD Digital Government Studies, OECD Publishing, Paris, https://doi.org/10.1787/9789264259270-en; Mexico’s response to OECD (2015[23]), “OECD Survey on Open Government Co-ordination and Citizen Participation in the Policy Cycle”, OECD, Paris; Transparency Portal of Spain (n.d.[24]) “Compromisos del III Plan de Gobierno Abierto”, https://transparencia.gob.es/transparencia/transparencia_Home/index/Gobierno-abierto/iiiPlanAccion/CompromisosIIIPGA.html.

Developing robust indicators to monitor the initiatives of the open government strategy

Indicators serve as a tool to measure the degree of success in achieving an objective or to check whether an action has been implemented, enabling the reorientation of implementation when an action goes awry (Vági, Péter; Rimkute, 2018[25]). According to Behatokia, the degree of advancement of each of the 14 commitments is determined by the achievement or not of a milestone activity with a verifiable deliverable (hitos medibles y verificables para alcanzar el compromiso).8 These milestones include, for example:

the elaboration of an annual transparency report (Commitment 1)
an online dataset (Commitment 2)
a participation map (Commitment 6)
training courses on citizen participation (Commitment 6).

These milestones (hitos) represent important stepping-stones to achieve OGAP’s commitments. Some of them refer to a process or an activity (e.g. drafting a report, providing training courses), while others focus on deliverable (e.g. a dataset) (see Box 4.8 for an illustrative typology of open government indicators). As mentioned in Box 4.2, the milestones to assess the degree of implementation of Commitment 5 (creation of a policy evaluation system) are the creation of: a social barometer; a tool for quantitative evaluation; a tool for qualitative evaluation; a methodology for institutional communication; a dashboard of context indicators and a dashboard of management indicators.

While these milestones suggest what should be done under Commitment°5, they only provide partial guidance for assessing actual progress and achievements in more detail. For instance, the following aspects are not (fully) addressed:

Input: Resources invested such as staff, money, time, and equipment, e.g. the human and financial resources used for the establishment of the evaluation system.
Output: The products delivered by the policy implemented; examples are provided as part of the identified milestones.
Outcomes: The immediate changes produced by the policy implemented, e.g. more high-quality evaluations performed in the public sector.
Impact: The long-term change produced by the policy implemented, e.g. better policies and service delivery.

Box 4.8. Typology of open government indicators

A classic typology of open government indicators discerns between:

Context indicators, when considering the public sector as an open system, can monitor external factors such as socio-economic trends, but can also include policy measures by other governments or supranational organisations (Van Dooren, W., Bouckaert, G., & Halligan, 2010[26]). Ideally, a comprehensive M&E system should include indicators to monitor the existence and development of environmental/context factors that can influence the governance of open government strategies and initiatives.
Input indicators measure resources in the broad sense, i.e. human and financial resources, logistics, devoted to a particular open government strategy or initiative. In the context of the governance of open government, input indicators could include the number of staff working in the office in charge of open government or the budget allocated for a given open government initiative.
Process indicators refer to the link between input and output, i.e. activities that use resources and lead to an output. In the context of the governance of open government strategies and initiatives, these indicators could include the duration for creating an office in charge of the co-ordination of the open government strategies and initiatives or the time allocated to their design.
Output indicators refer to the quantity, type and quality of outputs that result from the inputs allocated. Output indicators refer to operational goals or objectives. For instance, in the context of this policy area, it can refer to the existence of a law on access to information or the existence of training courses for public officials on the implementation of open government principles.
Outcome/impact indicators refer to the (strategic) objectives of policy intervention. In a public policy context, intended effects often relate to a target group or region, but they can also relate to the internal functioning of an administration. Effects can occur or be expected with varying time gaps after the policy intervention. Outcome and impact are often the terms used together to refer to them. The difference is based on the chronological order: outcome usually refers to shorter-term effects, while impact refers to longer-term effects. In this field, these indicators could be the share of public servants aware of the open government strategy or the number of citizens’ complaints about public policy decisions.

Source: OECD (2017[27]), “Governing Better Through Evidence-Informed Policy Making”, Draft proposal, internal document.

Furthermore, measuring output indicators only provides information on whether the products of the policy/plan are delivered or not. However, this does not account for the challenges encountered throughout the implementation process, therefore limiting the possibility to address them promptly. A specific set of indicators at different levels (input, process, output, outcome, impact) could help Biscay measure the performance of its open government initiatives more accurately, as well as analyse how these initiatives contribute to Biscay’s wider open government strategic goals.

While acknowledging that the development of governance indicators is a complex endeavour for public administrations, Biscay could undertake specific initiatives to advance towards this goal gradually. Adopting a theory of change approach (as illustrated in Figure 4.1) could be instrumental in ensuring that each initiative carried out within the open government strategy pursues a specific objective (outcome and impact) related to the improvement of public governance and/or service delivery. Furthermore, this methodology will help Biscay ensure that each open government initiative contributes to the fulfilment of the broader open government strategic objectives.

Figure 4.1. Example of indicators associated with an OGAP commitment

High-quality indicators (see Box 4.9) ensure the provision of robust information on the implementation pace for each objective. Such indicators allow for collecting relevant evidence to construct an informed benchmark of the overall degree of implementation of a plan or policy. In this regard, Biscay could consider adopting standards, like the ones outlined in Box 4.8, to create robust indicators. Also, Biscay could consider engaging with key actors on the development of indicators through a quality assurance process. Behatokia and/or the institution responsible for the development of these indicators could involve experts both from the government, as well as from academia and civil society.

Box 4.9. Criteria for quality indicators

According to the OECD (2016[3]), a high-quality indicator to monitor a policy or plan should respect the following criteria:

Relevant: An accurate measure of a practice, task or the expected goal.
Reliable: Consistently measurable over time, in the same way by different observers.
Precise: Operationally defined in clear terms.
Realistic: It should not be too difficult or too expensive to collect the information.
Measurable: Quantifiable using available tools and methods.
Validity: It measures what it intends to measure.
Timely: Provides a measurement at relevant time intervals, appropriate in terms of goals and activities.

Source: OECD (2016[3]), Open Government: The Global Context and the Way Forward, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264268104-en.

The way forward: towards the evaluation of open government initiatives

The evaluation of open government initiatives is a relatively new – but key - area of interest of policy makers and is a shared challenge across OECD countries. OECD countries face challenges in designing appropriate evaluation approaches to open government, in particular in assessing the outcomes and impact of open government initiatives. While a large majority of OECD countries (86%) confirms that their government monitors open government initiatives, merely half of them evaluate their impact (OECD, 2016[3]).

The gradual development of an M&E system will positively influence Biscay’s capacity to evaluate the achievements of open government initiatives. Moreover, in the long term, the government could also make use of the development of a government-wide M&E system to explore conducting pilot evaluations of the openness of sectoral policies, in order to assess how open government approaches for policy making improves outcomes and impacts. The Transparency for Development Program, developed by Results for Development,9 is an interesting example of a specific evaluation that explores whether well-designed transparency and accountability interventions improve health outcomes (see Box 4.10).

Box 4.10. Transparency for Development Program

The Harvard Kennedy School in partnership with Results for Development launched the research project “Transparency for Development” that looks to disentangle whether, why and in what context community-led transparency and accountability activities improve social development programmes’ outcomes. Working with local civil society, the project carried out an intervention in Indonesia and Tanzania where first, the implementers carried out a group of surveys to collect information on health infrastructure and new-born children and mothers. Secondly, the community was asked to discuss the information collected to identify the barriers preventing the improvement on the provision of public services for mothers and new-born children and come up with an action plan to overcome these barriers. After the implementation of the action plan, they will evaluate the impact of transparency and accountability on the intervention’s results using a Randomised Controlled Trial (RCT) methodology. Finally, after analysing the results of the evaluation, the project will look to replicate the intervention in other areas to build a comprehensive view of different contexts.

Source: Ash Center for Democratic Governance and Innovation (2018[28]), “Transparency for Development”, https://ash.harvard.edu/transparency-development; Results for Development (2018[29]), “Transparency for Development: Do Transparency and Accountability Interventions Improve Health Outcomes?”, https://www.r4d.org/projects/transparency-development-transparency-accountability-interventions-improve-health-outcomes/.

In this respect, the existing Civil Dialogue Table plays an active role in the design, implementation and evaluation of social policies. Once Biscay moves forward with the implementation of its M&E agenda, this platform can become a relevant space to discuss specific pilot evaluations.

Recommendations and proposals for action

Biscay could invest in the further development of a coherent monitoring and evaluation system, supported by a sound institutional framework.

More specifically, and as Biscay’s monitoring and evaluation system currently is in its initial phases, the government could consider:

Centralise responsibilities regarding M&E into a single institution in charge of 1) developing a whole-of-government M&E system; and 2) promoting the use and quality of monitoring and evaluation across government. Such an institution could be responsible for conducting M&E across government and/or developing guidelines, training courses, and quality checks for other departments wishing to carry out their own M&E. It could also promote the use of evidence resulting from M&E efforts.
Provide a clear mandate that sets out specific tasks and responsibilities regarding the M&E agenda appropriate to the institutional set-up chosen.
Establish a co-ordination instrument, such as an M&E council or committee, to co-ordinate M&E across government.

Fostering an M&E culture among public officials and stakeholders, Biscay could consider:

Include a particular M&E dimension in the ethical code and training modules.
Offer specific training courses on policy M&E for public officials, as part of the menu of training courses that the Directorate-General of the Legal Regime and Civil Service offers, and in parallel, developing guidelines to conduct M&E.
Include Behatokia in the Civil Dialogue Table, which would allow Behatokia to engage with key stakeholders from the social sector in a systematic and more structured way. In particular, Biscay could use this platform to promote the establishment of a – formal or informal - network of policy monitoring and evaluation practitioners, which could include representatives of academia as well as other stakeholders.

Monitoring and evaluating open government strategies and initiatives in Biscay by:

Develop a monitoring mechanism to collect regular, up-to-date quantitative data on the implementation of the different open government initiatives. This could be operationalised, for instance, through the creation of a dashboard embedded in Biscay’s data collection platform.
Pursue specific initiatives to advance towards the development of governance indicators gradually. Adopting a theory of change approach could be instrumental in ensuring that each initiative carried out within the open government strategy pursues a specific objective (outcome and impact) related to the improvement of public governance and/or service delivery.
Consider, in addition to standards to create robust indicators, engaging with key actors on indicator development through a quality assurance process. Behatokia and/or the institution responsible for the development of these indicators could involve experts both from the government as well as from academia and civil society, so as to receive their feedback on the quality of the indicators and specific recommendations on how to improve them.
Make use, in the long term, of the development of a government-wide M&E system to explore conducting pilot evaluations of the openness of sectoral policies, in order to assess how open government approaches to policy making improves outcomes and impacts.

References

[28] Ash Center for Democratic Governance and Innovation (2018), Transparency for Development, https://ash.harvard.edu/transparency-development.

[11] Bizkaia (2018), Diputación integra en su estructura a las personas con discapacidad intelectual con un proceso de selección “ad hoc” e innovador, http://web.bizkaia.eus/es/web/area-de-prensa/noticias/-/news/detailView/18746.

[10] Boletin Oficial de Bizkaia (2016), 10791 BOB num. 95.

[9] Boletin Oficial de Bizkaia (2016), 3209 BOB num. 32.

[18] Department for Transport (2013), Monitoring and Evaluation Programme, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/249242/monitoring-evaluation-programme.pdf.

[15] Diário oficial da Uniao (2016), “Portaria interministerial nº 102”.

[14] Dirección de Presupuestos, G. (n.d.), Documentos en vedette, http://www.dipres.gob.cl/598/w3-channel.html.

[17] GOV.UK (2013), Department for Transport monitoring and evaluation strategy, https://www.gov.uk/government/publications/monitoring-and-evaluation-strategy.

[12] Knowledge Sector Initiative (2017), Global evidence units – Finland: Government Policy Analysis Unit, http://www.ksi-indonesia.org/file_upload/Evidence-Policy-Unit-in-Finland-the-Government-Po-14Jun2017163532.pdf.

[21] Kurtz, Marcus J; Schrank, A. (2007), “Growth and Governance: Models, Measures, and Mechanisms”, The Journal of Politics, Vol. 69/No. 2, May, pp. 538-554.

[7] Lazaro, B. (2015), Comparative Study on the Institutionalization of Evaluation in Europe and Latin America, Euroscial, http://sia.eurosocial-ii.eu/files/docs/1456851768-E_15_ENfin.pdf.

[13] Ministry for Social Development (2005), Decree for which the Council of Social Development Policy Evaluation is regulated, [Decree 24/08/2005], https://www.coneval.org.mx/quienessomos/Conocenos/Paginas/Funciones.aspx.

[2] OECD (2018), “OECD Surveys on Open Government in Argentina”, in Internal document.

[27] OECD (2017), Governing Better Through Evidence-informed Policy Making, Internal document.

[1] OECD (2017), Recommendation of the Council on Open Government, https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0438 (accessed on 30 July 2018).

[5] OECD (2017), “Towards Open Government Indicators: Framework for the Governance of Open Government (GOOG) Index and the Checklist for Open Government Impact Indicators”, Concept Note.

[22] OECD (2016), Open Government Data Review of Mexico: Data Reuse for Public Sector Impact and Innovation, OECD Digital Government Studies, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264259270-en.

[3] OECD (2016), Open Government: The Global Context and the Way Forward, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264268104-en.

[23] OECD (2015), OECD Survey on Open Government Co-ordination and Citizen Participation in the Policy Cycle.

[4] OECD (2009), OECD DAC Glossary, Guidelines for Project and Programme Evaluations, http://www.oecd.org/dac/dac-glossary.htm.

[20] Provincial Council of Biscay (2018), Intermediate self-evaluation report of 30/05/2018 of the Open Government Action Plan 2017-2019.

[8] Provincial Council of Biscay (2017), Bizkaia Irekia: Plan de Acción de Gobierno Abierto (Open Biscay: Open Government Action Plan), http://gardentasuna.bizkaia.eus/documents/1261696/1397467/Plan+de+Gobierno+Abierto.pdf/d96264cf-022e-a2c0-3919-e1778372436c (accessed on 1 August 2018).

[19] Provincial Council of Biscay (2016), Foral decree formalising and regulating the Mesa de diálogo civil de Bizkaia, [Foral decree 154 of 2016], BOB 203.

[16] Republic of South Africa, T. (n.d.), Department: Planning, Monitoring and Evaluation: Strategic Plan 2015-2020, https://www.dpme.gov.za/publications/Strategic%20Plan%20and%20Annual%20Reports/DPME%20Strategic%20Plan%202015-2020.pdf.

[29] Results for Development (2018), Transparency for Development: Do Transparency and Accountability Interventions Improve Health Outcomes?, https://www.r4d.org/projects/transparency-development-transparency-accountability-interventions-improve-health-outcomes/.

[6] Thousand Oaks (ed.) (2006), Program evaluation and performance measurement, an introduction to practice.

[24] Transparency Portal of Spain (n.d.), Compromisos del III Plan de Gobierno Abierto, http://transparencia.gob.es/transparencia/transparencia_Home/index/Gobierno-abierto/iiiPlanAccion/CompromisosIIIPGA.html (accessed on 3 April 2019).

[25] Vági, Péter; Rimkute, E. (2018), “Toolkit for the preparation, implementation, monitoring, reporting and evaluation of public administration reform and sector strategies: Guidance for SIGMA partners”, in SIGMA Papers, OECD Publishing, Paris.

[26] Van Dooren, W., Bouckaert, G., & Halligan, J. (2010), Performance management in the public sector.

Notes

← 1. The Province of Biscay is legally referred to as a “historic territory”, a term exclusive to the Basque Country that describes the political and administrative system of its three provinces. In view of facilitating the understanding of the term for readers not familiar with the concept and of shortening the term repeated throughout the Chapters, the Review will refer to the historic territory of Biscay as “Province of Biscay”.

← 2. The Cabinet is composed of the General Directorate of Communication, the General Directorate of International Action and the General Directorate of the Observatory of Biscay (Behatokia).

← 3. The commitment envisages “defin[ing] a policy evaluation system for the Provincial Council of Biscay” (Provincial Council of Biscay, 2017[8]).

← 4. The Cabinet of Modernisation, Good Governance and Transparency, the Directorate General of Judicial Regime and Civil Service and the Directorate General of Services, Municipal Relations and Emergencies compose the Department.

← 5. OECD analysis based on desk research.

← 6. The third social sector is composed of social initiative organisations whose main purpose is to promote, through activities of social intervention, social inclusion, development co-operation and the effective exercise of the rights of individuals, families, groups, groups or communities that face situations of vulnerability or exclusion, inequality, vulnerability, disability or dependence (Law 6/2016, Article 2).

← 7. For more information, see Chapter 2.

← 8. For more information, see https://www.opengovpartnership.org/resources/ogp-process-step-2-develop-action-plan.

← 9. For more information, see https://www.r4d.org/about/our-history/.

End of the section – Back to iLibrary publication page