2. Rationale and frameworks for assessing city-to-city partnerships
Monitoring and evaluation (M&E) mechanisms can enable partners involved in city-to-city partnerships to carry out dialogues on what works and identify opportunities for improvement. Yet, assessing city-to-city partnerships remains a challenge and often lacks a quantitative component. The United Nations (UN) Sustainable Development Goals (SDGs) provide a tool to improve the M&E of city-to-city partnership projects and their contribution to achieving global goals. This chapter takes stock of existing checklists and self-assessment tools of relevance to city-to-city partnerships and identifies persistent gaps. It also provides the rationale for a complementary framework to gauge the effectiveness of partnerships at the subnational level and in particular measure progress towards achieving the SDGs and their compliance with the G20 High-level Principles on city-to-city partnerships to localise the SDGs.
The SDG framework can provide a tool to improve the monitoring and evaluation (M&E) of city-to-city partnership projects and their contribution to achieving the global goals. The adoption of SDGs in 2015 has provided cities, regions, and countries with a framework that can help develop and implement city-to-city partnerships (OECD, 2018[1]). The OECD estimates that although the SDGs were not designed by and for local and regional governments, at least 105 of the 169 SDG targets will not be reached without engagement and co-ordination with local and regional governments (OECD, 2020[2]). Many cities around the world have started to use the SDGs as a framework to rethink their local and regional development strategies, plans and actions, improve multi-level governance and vertical co-ordination as well as engage territorial stakeholders in policy making (OECD, 2020[2]). Furthermore, the SDGs can also help monitor and evaluate the contribution of local governments’ partnerships to sustainable development and improve transparency and accountability on how effectively resources have been used and what impact the partnerships have had in the different partner countries and cities (UN-Habitat/UCLG, 2020[3]). Using the SDGs in M&E frameworks of city-to-city partnerships could help embrace a broader view of their outcomes beyond project specifics. Through its harmonised and standardised targets and respective indicators, the SDGs can also allow for internationally comparable measurements across different policy sectors and local contexts.
M&E mechanisms can enable partners involved in city-to-city partnerships to carry out dialogue on what works and identify opportunities for improvement. The information gathered through M&E offers a valuable learning mechanism that helps decision-makers put in place preventive and corrective actions where needed, learn from past experiences and ensure accountability toward relevant stakeholders (EC, 2022[4]). Dialogue on M&E among stakeholders can promote sharing of best practices on how to reduce costs and better incentivise reporting mechanisms (OECD, 2019[5]). The availability of solid data on the partnerships compiled through M&E mechanisms can also improve transparency and accountability since they allow tracking of how resources have been spent effectively. However, there is relatively little culture of M&E of the outcomes of city-to-city partnerships overall, as opposed to the results of individual projects, which constitutes a challenge in decentralised development co-operation (OECD, 2018[1]).
Better M&E of city-to-city partnerships are needed to achieve SDG impact (OECD, 2018[1]). Devolution of expenditure responsibilities to subnational governments in developing countries creates greater demand for partnerships that deliver financing, capacities and expertise at the subnational level (OECD, 2019[5]). However, transparency and accountability of financing remain longstanding barriers to effective decentralised development co-operation (DDC). Tracking how effectively resources have been spent and their impact on development outcomes is another challenge for city-to-city partnerships and DDC more broadly. For example, while 10 out of 232 SDG indicators (roughly 4% of total indicators) rely on official development assistance (ODA) data to monitor progress toward the goals, roughly a third of OECD Development Assistance Committee (DAC) members report ODA data on DDC (11 out of 30 DAC members). Sufficient accountability and transparency of financing are the first steps to ensure that DDC resources are used rationally, reliably, consistently and with high-quality standards. The small size of projects and a large number of decentralised actors active in development co-operation, alongside a lack of incentives, also impede the collection and reporting of data at the subnational level (OECD, 2018[1]; 2019[5]). To that end, this section first provides an overview of the concept of M&E with a focus on city-to-city partnerships and the SDGs. It explains the importance of data and indicators for M&E frameworks. In addition, it provides key messages from the analysis of the 16 partnerships supported by the European Commission (EC) DG INTPA and an overview of existing checklists and self-assessment tools that inspired the development of the proposed systemic M&E framework for city-to-city partnerships to localise the SDGs in Chapter 3.
The concepts of monitoring and evaluation
Monitoring and evaluation are distinct but complementary components that can be used to assess the progress and success of city-to-city partnerships. They are distinct as monitoring is a constant process, while evaluation is undertaken periodically. Monitoring provides information about the current status of partnerships, while evaluation provides recommendations and lessons learnt for future partnerships. Monitoring studies the present data and experiences of partnerships while evaluation studies past experiences of partnership performance. At the same time, the two concepts are complementary in the sense that monitoring is necessary for providing the underlying data and information for an evaluation.
Monitoring
Monitoring is a continuous activity over time to keep projects on track and measure progress based on predefined objectives, mostly through indicators (Potluka, 2020[6]). Monitoring activity is usually undertaken internally within the organisation or among project partners involved and focuses on inputs (financial, human and material resources used for the development intervention), activities (actions taken that mobilise inputs, such as funds, technical assistance and other types of resources, to produce specific outputs) and outputs (the products, capital goods and services which result from a development intervention) (OECD, 2002[7]). Monitoring frameworks involve multiple data points and frequent reporting. Thus, monitoring studies the information on the current status of a project or partnership and compares the actual progress with the planned progress (UN-Habitat/UCLG, 2020[3]).
Stakeholder engagement plays a key role in the successful monitoring of city-to-city partnerships. Subnational authorities often lack political commitment to monitor city-to-city partnerships and effective frameworks to assess the outcomes (OECD, 2018[1]). One important driver to implementing a transparent monitoring scheme is to engage stakeholders in the monitoring process and share monitoring results and data. For example, cities can hold regular meetings that provide a platform for citizens, experts and stakeholders to voice and exchange their opinions on how the partnership is achieving its intended goals. For example, cities work closely with civil society organisations (CSOs). The share of DDC channelled to and through CSOs is 19%, which is 4% higher than the average for total bilateral aid (15%) (OECD, 2019[5]). The private sector can also be an important partner. Strengthening engagement with the private sector, for example, can help to upgrade the technology used to track assets, payments and administrative processes to bring accountability and trust into investments (OECD, 2019[5]). Engaging such stakeholders in city-to-city partnerships can also help better reflect their specific needs and priorities in the evaluation process. To improve transparency, cities could share monitoring data with the public through government communication channels such as bulletins or electronic platforms including dashboards or interactive and informative websites (UN-Habitat/UCLG, 2020[3]). For example, the province of Córdoba in Argentina has launched an Open Government Roundtable (Mesa de Gobierno Abierto) composed of CSOs and representatives from academia to work on the institutionalisation of open government principles in the province. The creation of this roundtable led to the development of an Open Management Portal, through which the general public can directly access information and data related to the three axes of governmental actions aligned with the SDGs (OECD, 2021[8]). The municipality of Kópavogur in Iceland has developed an innovative management and information system where all local databases are integrated into one data warehouse to monitor and link performance indicators on municipal tasks with prioritised targets and the SDGs (OECD, 2020[9]).
Community-based monitoring can constitute a tool for enhancing dialogue with local stakeholders. It is an organised way through which the local community can monitor and collect ongoing or recurring information on specific partnership goals and objectives. It can help improve the sense of ownership and increase the likelihood that both policy makers and stakeholders at all levels will use the results of the monitoring for planning, budgeting and the implementation of local development programmes, as well as for their M&E (UCLG/CES/OIDP, 2014[10]). Community-based monitoring can also encourage the collection of additional data that may rely on diversified tools to generate and compile the necessary data for the assessment of projects and evidence-based policy making. Such tools may include community scorecards, surveys, dialogues or other feedback mechanisms (UCLG/CES/OIDP, 2014[10]). Examples of such tools for local monitoring in collaboration with stakeholders include the OECD Water Governance Indicator Framework, which helps governments to assess the state of play of water governance policy frameworks, institutions and instruments, and their needed improvements over time (OECD, 2018[11]) and the self-assessment tool of the Policy Coherence for Sustainable Development Toolkit, which enables policy makers, practitioners and stakeholders to review their institutional mechanisms, organisational structures and policy-making processes against internationally recognised good practices on policy coherence for sustainable development (OECD, 2016[12]).
Evaluation
Evaluation describes a systemic and objective examination of projects and policies and an assessment of their impact, outcomes and outputs (UN-Habitat/UCLG, 2020[3]). Evaluation processes usually include the collection of qualitative and quantitative data. They aim to assess whether the project achieved its intended impact. Impact refers to the positive, negative, primary and secondary long-term effects created by a development intervention (OECD, 2002[7]). Programme evaluations usually pursue two main objectives: i) ensuring accountability and providing learning opportunities to project partners and stakeholders; and ii) informing key decisions about the programme’s future (Aghumian, 2014[13]). Evaluation is therefore an important tool to assess the impact of partnership programmes such as city-to-city partnerships. Evaluations also aim to assess outputs – the products, goods and services resulting from a development intervention and outcomes, which are defined as the short-term and medium-term effects of an intervention’s output (OECD, 2002[7]). In addition, the term result summarises the output, outcome and impact of a development intervention (OECD, 2002[7]).
When setting the scope and design of a partnership evaluation, policy makers need to define the purpose of their evaluation. This can be done for instance through the elaboration of terms of reference to clarify expectations, roles and responsibilities for the evaluation, including why the evaluation is being undertaken (objectives), what will be examined (scope), how (methods) and when and for how long (timeframe) (UN-Habitat/UCLG, 2020[3]). In particular, project partners need to determine the target audience for their M&E (de Losada Passols, 2017[14]). For instance, an evaluation for accountability purposes requires more independence and broader coverage than one focused on learning in a specific area, for example urban resilience (Aghumian, 2014[13]). To ensure quality evaluations, sufficient resources need to be made available. According to an assessment of the World Bank’s Independent Evaluation Group, the cost of M&E for development projects usually ranges between 1% and 3% of a given programme’s annual expenditures (IEG, 2011[15]). Other common challenges that undermine the quality of evaluations include unclear terms of reference, insufficient time, weak M&E frameworks for the programmes, and lax evaluation methodology and tools (IEG, 2011[15]). Therefore, the adoption of agreed common standards to evaluate the extent to which city-to-city partnerships contribute to sustainable development would improve the quality, utilisation and impact of such evaluations.
Evaluations can be undertaken by an individual or a team, ideally comprised of local stakeholders from the public sector, private sector and civil society. The leading person or team should have the convening power to gather stakeholders and thoughtfully plan and manage the evaluation process and adapt the necessary tools (OECD, 2020[16]). For accountability and credibility purposes, those who commission and receive a project evaluation should not influence the evaluation at any stage (Aghumian, 2014[13]). However, key stakeholders should be consulted and the evaluation process should be transparent. In sum, the evaluation of city-to-city partnerships should be based on a clear understanding of the purpose of the evaluation, the availability of sufficient resources and data to assess outcomes and impacts, and the development of an evaluation work plan.
The OECD approach to the evaluation of development projects
The OECD (2018[1]) promotes a learning-focused approach to M&E with a focus on generating evidence and insights to improve effectiveness and impact. To that end and to guide evaluations, the OECD DAC Network on Development Evaluation (EvalNet) has defined six evaluation criteria. The criteria were first developed in 1991 and last updated in 2019 (Figure 2.1).
Source: OECD (2021[17]), Evaluation Criteria, https://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm (accessed on 7 March 2022).
Beyond being simple evaluation criteria, they describe the desired attributes of development interventions, which can help define what is meant by a successful city-to-city partnership for the SDGs. They are also linked to evaluation questions, which should provide a different perspective on the intervention, its implementation and results. The first criterion is relevance (is the intervention doing the right thing?) and assesses to which extent the intervention objectives and design respond to the intended beneficiaries and needs of the partners in institutions. The second criterion of coherence (how well does the intervention fit?) analyses the compatibility of the intervention with other interventions in a country, sector or institution. The third criterion of effectiveness (is the intervention achieving its objectives?) investigates to which extent the intervention has achieved or is expected to achieve its objectives and results. The fourth criterion of efficiency (how well are resources used?) measures the extent to which the intervention delivers results in an economic or timely way. The fifth criterion of impact (what difference is the intervention making?) assesses to which extent the intervention has generated or is expected to generate significant positive or negative higher-level effects. Lastly, the sixth criterion on sustainability (will the benefits last?) should measure to which extent the net benefits of the intervention continue or are likely to continue (OECD, 2020[18]).
The usage of the OECD evaluation criteria should be guided by two conditions. First, there needs to be a thoughtful application of the criteria to support high-quality and useful evaluation. This means that the criteria should be contextualised based on the circumstances of the individual evaluation and stakeholders involved, i.e. the evaluation questions and what the answers will be used for, which should inform how the criteria are interpreted and analysed (OECD, 2021[17]). Second, the application of the criteria needs to be based on the purpose of the evaluation, i.e. the criteria should be covered according to the needs of relevant stakeholders and the context of the evaluation (OECD, 2020[18]). A different amount of time and resources may be devoted to the evaluative analysis for each criterion depending on the purpose of the evaluation. Data availability, resource constraints, timing and methodological considerations may also influence whether and to what extent a particular criterion is covered (OECD, 2021[17]). Therefore, the criteria should be understood within a broader context and read in conjunction with other guidance on how to evaluate development projects (OECD, 2020[18]).
Policy alignment across levels of government, local ownership, dialogue and reciprocity among partners are critical for the effective M&E of city-to-city partnerships. In the context of city-to-city partnerships, M&E frameworks should assess: i) the level of alignment between the actions of city-to-city partnerships and the local and national policies in partner countries; ii) the degree of ownership by local partners in partner countries; iii) the degree of dialogue and participation that partner organisations plan in the project; as well as iv) the potential for reciprocity among partners. In addition, an M&E framework should help actors generate evidence-based information to capture the value-added of the partnership over alternative approaches put in place by central governments and other development actors. It should also feed into a learning process that uses the information gathered to inform future political decision making, including through the internalisation of lessons learnt, the production of strategic knowledge and the communication of results for better accountability (OECD, 2018[1]). Development effectiveness principles should provide guidance for both national and subnational actors to improve their policy environment and strengthen their partnerships with diverse actors at the local and national levels to deliver on agreed development results. The Global Partnership for Effective Development Co-operation (GPEDC) monitoring, led by national governments, aims to support the effectiveness of DDC by linking subnational and central actors for policy coherence. Hence, the GPEDC can also help to strengthen the voice of cities and regions within dialogues on development effectiveness including at high-level meetings such as the 2022 Effective Development Co-operation Summit.
Benefits and success factors of M&E frameworks
Successful M&E systems can generate various benefits for policy makers. First, they facilitate evidence-based political decision making and public spending. Being able to demonstrate the (positive) outcome of a policy or a partnership at large can make it easier to garner support from across levels of government or other funding sources (SDSN, 2016[19]). M&E also improve organisational learning and the development capacities of governments and other actors involved in the partnership. In the context of city partnerships for sustainable development, M&E help assess whether and to what extent the cities involved in the partnership are making progress towards the achievement of the SDGs and towards the key objectives of the EC Partnerships for Sustainable Cities programme (hereinafter the EC Partnerships programme) such as improved urban governance, social inclusion, resilience, prosperity and innovation. This is particularly the case if the M&E results are subsequently used to improve ongoing and future partnership initiatives (Lamhauge, Lanzi and Agrawala, 2012[20]). The United Cities and Local Governments (UCLG) therefore suggests the application of a simple M&E framework for DDC projects that local project partners are confident working with (UCLG, 2019[21]). M&E also provide researchers and other professionals the opportunity to develop research, policy recommendations and other innovative solutions that can contribute to strengthening local governance (SDSN, 2016[19]). In doing so, M&E systems can enhance the implementation of city-to-city partnerships and increase their visibility and impact.
The availability of quality data is an important success factor in M&E frameworks. These frameworks can improve the collection of data as they rely on the availability of quantitative information for policy assessment (OECD/UNDP, 2016[22]). Increased demand for data can drive up the quantity of information available, which can feed into evidence-based policy planning. ODA, for example, is a key data source to strengthen the M&E of financing for city-to-city partnerships (OECD, 2019[5]). Improved quality and quantity of data can contribute to increased transparency e.g. through annual or monthly communication on ongoing projects and sharing project outcomes, thereby improving accountability. Consistent and ongoing M&E can also improve co-ordination across governmental levels and stakeholders to guide public action through information sharing and continuous data collection (OECD, 2018[1]). The publication of the outcomes of city-to-city partnerships can improve their visibility and, thereby, potentially increase public support for and recognition of these partnerships. In addition, it enhances the efficiency of projects through consultation and networking. One example in France is the Atlas of Decentralised Co-operation, which maps the international action conducted by all French local and regional authorities and provides a platform for inter-ministerial and multi-stakeholder information sharing (OECD, 2018[1]).
Successful M&E frameworks integrate multi-level governance aspects and engage stakeholders. According to the UCLG, the most efficient M&E frameworks build on joint and co-ordinated efforts across various levels of government (i.e. local, regional, territorial) and local stakeholders such as the private sector, CSOs, academia and citizens (UCLG, 2019[21]). In Spain, for example, most regions and municipalities active in DDC have set up multi-stakeholder platforms or councils that advise them on development co-operation projects. They follow a participatory approach to engage a broader range of actors in the implementation and evaluation of DDC projects (OECD, 2018[1]). In Madrid, the city council established the Foro Madrid Solidario, a multi-stakeholder platform for consultation and dialogue that aims to allow for a participative approach to the design, implementation and evaluation of Madrid’s local development co-operation policies (de Losada Passols, 2017[14]). Another example is the region of Flanders (Belgium), which has developed guidelines for the implementation and impact evaluation of its DDC projects. While evaluations are usually developed by external evaluators, monitoring is conducted by the region and local partners. Mid-term reviews are one of the main evaluation modalities in its partner countries, which have to follow the evaluation guidelines. The region also links financial audits to the projects, which are stored in the international M&E database that represent the basis for Flanders’ ODA reporting (OECD, 2018[1]). The UN Sustainable Development Solutions Network (SDSN) (2016[19]) characterises the capacity for information flows across levels of government as another success factor of local M&E systems since it facilitates the analysis of disaggregated statistics to evaluate local performance. Additionally, successful frameworks map out the project content during the conceptualisation phase to ensure that SDG targets are identified and that accurate and exploitable data are collected from the beginning (UCLG, 2021[23]).
Knowledge-based institutions can facilitate the development of M&E frameworks. Universities and research centres are often actively involved in city-to-city partnerships (OECD, 2018[1]). Through their knowledge, capacities and tools to generate specific data, they also carry a strong potential to improve the evidence base and evaluation of city-to-city partnerships. They can thus be a critical player in data collection and drafting of evaluation reports. The UCLG Capacity and Institution Building (CIB) Working Group suggests that experienced project partners such as universities can help train local stakeholders in using M&E systems and potentially align them with the different partner cities’ overall performance management system and the SDGs, and train the other partners accordingly on how to use them (UCLG, 2019[21]). One example where a partnership between universities provides opportunities for shared learning and development is the Phoenix Project between Cardiff University in Wales (United Kingdom) and the University of Namibia. In this partnership, the two universities are engaged in training, mutual resources and the provision of information technology support among other things (de Losada Passols, 2017[14]). Multilateral platforms can also help to ensure that capacity building and technical assistance provide results. For instance, Tax Inspectors Without Borders (TIWB) deploys experienced tax auditors to work with countries’ revenue authorities, including at the subnational level: the Tax Administration Diagnostic Assessment Tool (TADAT), particularly, has witnessed growing demand from subnational entities (OECD, 2019[5]).
International co-operation can promote an M&E culture that fosters the involvement of a variety of actors. The GPEDC, co-led by the United Nations Development Programme (UNDP) and the OECD, is an example of an international partnership contributing to a more inclusive M&E culture. Through its monitoring exercise to track progress on the implementation of internationally-agreed development effectiveness principles, it allows multiple stakeholders, including local and regional actors from both provider and partner countries, to participate in the monitoring process (OECD, 2018[1]). In 2016, more than 81 low- and middle-income countries led the reporting exercise, with the participation of more than 120 countries, 74 development organisations and several hundred CSOs, private sector representatives, trade unions, foundations, parliamentarians and local governments (OECD, 2018[1]).
In sum, the success of M&E frameworks depends on a range of factors. First, it is important to foster a wider uptake of the M&E culture, e.g. through international co-operation to improve organisational learning and the development capacities of local governments and other actors involved in partnerships. Second, data collection and the availability of quality data contribute to the delivery of better and more precise M&E results. Through their knowledge, capacities and tools to generate specific data, universities and other academic institutions carry a strong potential to improve the evidence base and evaluation of city-to-city partnerships. Third, the most efficient M&E frameworks incorporate efforts across various levels of government and integrate local stakeholders such as the private sector, CSOs, academia and citizens.
Using data and indicators in monitoring and evaluation
Indicators play a central role in M&E processes. By creating regular and objective feedback on the progress of partnerships towards their objectives, indicators offer a quantitative assessment tool. As such, they can be used to examine the effects of partnerships’ objectives and outputs (Figueiredo, Honiden and Schumann, 2018[24]). They can provide essential information for cities to assess the effectiveness of their policies and objectives and to adjust them if needed. Well-designed indicators have the benefit of providing easily comprehensible information and supporting judgment based on facts that can inform political decisions (Figueiredo, Honiden and Schumann, 2018[24]). Indicators can also facilitate learning by contributing to a better understanding of what types of policies are effective or ineffective. Additionally, indicators can further contribute to transparency and accountability (Figueiredo, Honiden and Schumann, 2018[24]).
Well-defined indicators allow M&E frameworks to quantify project or partnership outcomes and achievements. Local indicators provide tangible data that can guide actions and policies relevant to local competencies, for example, administrative and operational data (OECD, 2020[2]). To be of use for M&E frameworks, such indicators should be specific (measuring the intended results), measurable (being reliable and objective), attainable (being practical and obtainable at reasonable cost), relevant (being aligned with the objectives of the partnership) and time-bound (being expected to be achieved within a certain time frame). Defining indicators that fulfil these characteristics provides the opportunity to establish quantifiable targets and assess the project’s achievement against those target values and in comparison, to the baseline situation (UN-Habitat/UCLG, 2020[3]). The selection of indicators can be theoretical and science-driven (i.e. being based on theoretical models) or value-driven (i.e. reflecting ongoing social debates and priorities), with the latter being more common for policy-making purposes (Zinkernagel, Evans and Neij, 2018[25]). However, different partners might have differing measures and criteria for success (The Partnering Initiative/UN DESA, 2020[26]). It is important to agree on monitoring criteria and indicators that fulfil the needs of all partners to avoid any overload of indicators and reports (OECD, 2018[1]).
SDG data can provide M&E frameworks with a holistic perspective on development co-operation projects. The SDGs have supported a shift towards a multi-sectoral approach to development co-operation, incentivising municipalities to collaborate internationally. The SDGs as an internationally standardised framework is another means to improve M&E frameworks through their common foundation and language. In particular, they can help leverage opportunities to engage cities and regions in monitoring and data collection (OECD, 2020[2]). An example of using the SDGs to improve M&E culture is the OECD programme A Territorial Approach to the SDGs. The programme supports cities and regions on their pathway to enhancing accountability through better monitoring of sustainable development projects and outcomes (OECD, 2020[2]). City-to-city partnerships are a major component of the localisation of the SDGs (OECD, 2018[1]). The localisation of the SDGs requires collecting and analysing context-specific data to allow for a reliable assessment of the local needs and living conditions of different communities. Without systematic M&E, it is difficult to separate project outcomes that work from those that do not (Figueiredo, Honiden and Schumann, 2018[24]).
However, the collection of local and context-specific data can be challenging for city-to-city partnerships and the assessment of their progress towards achieving the SDGs, particularly in developing countries. Local governments in the least developed territories and countries often lack the capacity and resources for data collection (de Losada Passols, 2017[14]). The PARIS21 Engagement Strategy, therefore, highlights the increasing demand from local governments for disaggregated data and the need for subnational components of national strategies for the development of statistics (NSDS) to monitor development activities and leave no one behind (OECD, 2019[5]). Due to the limited availability of granular data, 59% of local government associations and 72% of local and regional governments in countries reporting to the UN High-level Political Forum have yet to develop any form of system to track progress on the localisation of the SDGs in 2021 (UCLG, 2021[23]). Moreover, the capacity of cities and municipalities in terms of budget and staff varies, impeding the collection and reporting of data at the subnational level (OECD, 2019[5]).
Voluntary Local Reviews (VLRs) can help facilitate the creation of M&E frameworks. Tying into existing frameworks such as the reporting of progress in VLRs and Voluntary National and Local Reviews (VNRs) can simplify the introduction of M&E frameworks locally (The Partnering Initiative/UN DESA, 2020[26]). An increasing number of local authorities from all continents are publishing VLRs to assess their progress on the SDGs, including Cape Town (South Africa), Helsinki (Finland), Kitakyushu (Japan), Melbourne (Australia), New York (United States), São Paulo (Brazil) and Surabaya (Indonesia) among others (OECD, 2020[2]). These VLRs can inspire local policy makers, notably for the development of commonly agreed indicators for M&E frameworks to assess the progress of their city partnerships towards the SDGs. Furthermore, the integration of city-to-city partnerships assessments into VNRs and VLRs can provide an opportunity for cities and regions to strengthen their data on SDG implementation and thus feed into an improved M&E framework.
Monitoring and evaluation in the EC Partnerships programme – Evidence from 16 partnerships
The EC Partnerships programme uses logical frameworks including activity matrices to assess each of its city-to-city partnerships. Logical frameworks, strategic planning and management methodology to design, monitor and evaluate international development projects are the most commonly used tools to assess EC external actions or projects. The logical framework or log frame aims to measure the fulfilment of the objectives of an action, e.g. a city-to-city partnership. The log frame includes the overall objectives of the partnership (impact), specific objectives (outcome), outputs and different indicators used to measure the extent to which the different objectives have been achieved (EC, 2021[27]). For example, in the Madrid-Praia partnership Adapting Local Integrated Urban Development Plans to the SDGs, the overall objective (impact) of the partnership is fostering a territorial approach to integrated local development for more sustainable and inclusive cities. In addition, the partnership aims to empower the partners to adapt local strategies aligned with the SDGs through designing a multi-stakeholder ecosystem to finance the implementation of an SDG action plan. The indicator to measure impact is the number of inhabitants in Praia benefitting from the project activities. One of the specific objectives (outcomes) is to strengthen urban governance, which is measured by the indicator number of local strategic plans. The desired output is a local strategic plan for sustainable development and the SDGs, an action plan and an investment plan (output), which is measured by the number of local plans for development (indicators to measure output). The log frame includes an activity matrix, which identifies and describes partnerships’ key activities to be carried out to produce the intended outputs and the necessary means such as political, technical, financial, human and material resources required to implement these activities, e.g. staff, equipment, supplies and operational facilities.
The logical framework of the EC Partnerships programme includes several indicators and data sources. Indicators used in the logical framework of the EC Partnerships programme include indicators for different points in time, i.e. a baseline value (value and reference year), a current value (reference year), a target value (value and reference year) as well as sources of data and assumptions. Indicators are quantitative and/or qualitative variables that allow for simple and reliable measurement of the achievements of the partnership. The baseline refers to the value of the indicator before the action to assess and compare it against future progress. The current value refers to the latest known value at the time of reporting. Lastly, the target value refers to the intended final value. The assumptions refer to any external conditions that may have affected the action that is outside of the management’s control. The log frame is thus helpful as a reporting tool on the achievement of results of the city-to-city partnerships during the implementation phase.
The activity matrix identifies and describes partnerships’ key activities to produce the intended outputs. These main activities include the necessary steps to obtain the desired result, for example the recruitment and assignment of staff or the creation of project governance committees. The main activities are described by explaining means, costs and assumptions. The means refer to the political, technical, human and material resources that are necessary to implement the activities. The costs refer to the amount of money necessary for the action and how they are classified. Assumptions describe the external, necessary and positive conditions for implementing the intervention that is outside of its management’s control. The activity matrix is used to complement the log frame as it forces project initiators to reflect on the necessary steps for achieving their impacts, outcomes and outputs. It can also help partnerships establish clear goals and outcomes as the activity matrix can be used as a guide to assess feasibility (EC, 2021[27]).
The data and indicators used in the evaluation of EC Partnerships are not harmonised, which makes the comparison of project outcomes challenging. While all partnerships of the EC Partnerships programme use log frames and activity matrices to monitor and evaluate the outcome of its partnerships, each partnership decides on its overall (impact) and specific objectives (outcome) (Table 2.1). The partnerships also determine their indicators to measure progress in achieving these objectives. Most partnerships have one overall objective (impact) and several specific objectives (outcomes), while others pursue two or more overall objectives measured by different indicators, although the guidelines for filling the log frames suggest pursuing a single long-term objective. They also consider having only one specific objective: a good practice. However, there are no guidelines on the number of indicators required to monitor and evaluate the partnerships’ progress in achieving their objectives. Consequently, there is a large variety and diversity in the number and scope of indicators used to measure impact, outcome and outputs (as shown in Table 2.1)
Individual partnerships measure similar objectives through different indicators and there are instances where similar objectives are measured through very different indicators. For example, the overall objective (impact) of promoting sustainable urban development policies and inclusivity is measured by the indicators “number of municipalities participating in the project” (e.g. in the AccessCities partnership, a co-operation between cities in Italy, Spain and Brazil), the “number of inhabitants benefitting from the project activities” (e.g. in the partnership on Adapting Local Integrated Urban Development Plans to the SDGs), a “localised economic growth index” and the “number of tons of CO2 emitted annually” (e.g. Territorial Alliance for Sustainability and Competitiveness in Eastern Antioqueño-Colombia, between Rionegro, Colombia, Itaugua, Paraguay and Barcelona, Spain). This is the case since they are often placed at very different result levels, which highlights the missing distinction in the M&E frameworks between output (e.g. the number of municipalities participating in the project) and impact level (e.g. the number of tons of CO2 emitted annually). Additionally, the type of indicators differs, some being quantitative and others qualitative. The log frames also include sources and means of verification that differ in their level of specificity (i.e. cities’ websites and local newspapers compared to employment growth statistics of the central bureau of statistics). The level of accuracy within the log frame is hence often determined by the partnerships and their available resources.
Checklists and self-assessment tools for monitoring and evaluation
The following section provides an overview of a selection of existing M&E frameworks related to sustainable development, notably self-assessment frameworks and checklists (Table 2.2). The purpose of this overview is to draw insights into the development of an OECD evaluation framework for city-to-city partnerships to localise the SDGs. This section also identifies several gaps or missing parts and points out the need for a complementary M&E framework that allows cities to track the implementation of their city-to-city partnerships in a more comprehensive way.
The OECD Water Governance Indicator Framework
The OECD Water Governance Indicator Framework is a self-assessment tool for dialogue on the performance of water governance systems. Its framework is intended to be used as a multi-stakeholder voluntary and self-assessment tool to engage in multi-stakeholder dialogues on the performance of water governance systems involving governments, public, private and non-profit sectors. It aims to support the implementation of the OECD Principles on Water Governance and assess the state of play of water governance policy frameworks, institutions and instruments. It consists of a traffic light system of 36 water governance indicators for the 12 dimensions of the OECD Principles on Water Governance (OECD, 2015[30]). Data are collected through multi-stakeholder workshops through a five-scale self-assessment and a “not applicable option”. The scale ranges from “in place, functioning” (5 points) if the water governance dimension is complete and relevant in all aspects, to “not in place” (1 point) if the governance dimension does not exist and there are no plans or actions taken for implementing it. Stakeholders are also invited to identify the expected trend for the next three years (improvement, stable situation, decrease). The indicators are complemented by a checklist containing more than 100 questions that underline that the implementation of each principle should not be limited to the 3 indicators per principle and requires a more in-depth reflection on additional governance conditions. The last step of the self-assessment process is the creation of an action plan, which should determine which collective actions can be taken over the short, medium and long terms to improve the different dimensions of the water governance system (OECD, 2018[11]). Recently, the indicator framework was complemented by a how-to Guide to Assess Water Governance, which contains a background section to understand the OECD principles and indicators on water governance and indicator framework and further explains the ten-step methodology used to implement and carry out a multi-stakeholder assessment of water governance (OECD, 2022[31]). It also provides a range of examples and lessons learnt from the implementation of past experiences using the methodology.
The OECD Policy Coherence for Sustainable Development Toolkit
The OECD Policy Coherence for Sustainable Development Toolkit aims to analyse, enhance and track progress on policy coherence in the implementation of the SDGs. The toolkit provides practical guidance, self-assessment checklists and good practice examples. Its objective is to guide policy makers in the analysis to improve the understanding of synergies and trade-offs among SDGs. It also aims to strengthen existing institutional mechanisms to support a more coherent and integrated implementation of the SDGs. Furthermore, it helps identify options for monitoring, assessing and reporting policy progress on policy coherence for sustainable development. The self-assessment tool provides policy makers and stakeholders with the option to review their institutional mechanisms, organisational structures and policy-making processes in comparison to internationally recognised good practices. As such, it is designed to stimulate dialogue by helping users identify strengths, gaps and weaknesses regarding their implementation of the SDGs. The self-assessment tool consists of 24 questions structured across 8 building blocks (political commitment; strategic long-term vision; policy integration; policy co-ordination; local and regional involvement; stakeholder engagement; policy and financing impacts; monitoring, reporting and evaluation). The rating scheme follows the same approach as the OECD Water Governance Indicator Framework and ranges from 5 (in place, functioning) to 1 (not in place) with an additional “not applicable” option. The self-assessment should ideally be conducted by the body responsible for co-ordinating the implementation of the SDGs, for example the lead governmental institution. To improve the representativeness of the results, it could be organised through a workshop involving different territorial stakeholders (OECD, 2016[12]).
The OECD Scoreboard on the Governance of the Circular Economy (self-assessment tool)
The OECD Scoreboard on the Governance of the Circular Economy is a self-assessment tool based on 12 key governance dimensions to help create a circular economy system. The OECD scoreboard follows a 1 to 6 scoring system with 1 being “planned” and 6 being “in place, objectives achieved”. In doing so, the OECD scoreboard offers cities and regions conducting the assessment an overview of the current governance situation, allowing them to take decisions based on the assessment’s results. Additionally, it guides improvement by identifying policy areas where action is required to enable and facilitate the circular transition. The self-assessment is also a tool for dialogue as it is based on a multi-stakeholder participatory process, allowing various stakeholders to help improve the policies and tools by providing feedback during stakeholder dialogues that may promote collective thinking or foster learning. The self-assessment is based on 12 governance dimensions: roles and responsibilities, strategic vision, awareness and transparency, co-ordination, policy coherence, stakeholder engagement, appropriate scale, regulation, financing, capacity building, innovation, and data and assessment. The self-assessment allows for empirical learning that can help cities and regions rethink their overarching governance and economic models toward a circular economy. This exchange of practices across different governmental levels can help investigate and overcome barriers while generating adequate data and information and stimulating innovation (OECD, 2020[16]).
The Global Partnership Monitoring Framework for 2030
The Global Partnership Monitoring Framework for 2030 is a monitoring framework under the Global Partnership for Effective Development Co-operation (GPEDC) that aims to improve development effectiveness. The GPEDC is a multi-stakeholder platform that aims to deliver long-lasting results and contribute to the SDGs. The GPEDC is housed at the OECD and is managed jointly with the UNDP (GPEDC, 2020[32]; OECD, 2018[1]). Its framework allows local and regional actors from both donor and partner countries to participate in the monitoring process (OECD, 2018[1]). Through indicators that are assessed based on scoreboards, it incorporates evidence of how effective development co-operation aligns with supporting countries’ policies and practices to achieve the 2030 Agenda and deliver results for the people and the planet (GPEDC, 2017[33]). The framework is based on a voluntary and country-led process to strengthen multi-stakeholder dialogue at the country, regional and global levels to drive change in the way development co-operation is provided by collecting country-generated data (OECD/UNDP, 2019[34]). The framework analyses data on ten indicators across four categories: i) ownership of development priorities by developing countries; ii) a focus on results; iii) inclusive development partnerships; and iv) transparency and mutual accountability. It also assesses the extent to which civil society is engaged. The GPEDC created a virtual dashboard to visualise the progress of countries on development co-operation data. This dashboard tool allows users to view country-specific data that can be compared across countries and regions. Additionally, it allows for the measurement of progress and trends over time (GPEDC, 2017[33]).
The City Prosperity Initiative
The United Nations Human Settlements Programme (UN-Habitat) has developed the City Prosperity Initiative (CPI) to measure and monitor sustainable urban development. It provides city authorities, and local and national stakeholders with a composite index, which entails six dimensions (productivity; infrastructure; quality of life; equity and inclusion; environmental sustainability; governance and legislation) to define targets and goals supporting the formulation of evidence-based policies (UN-Habitat, 2021[28]). It allows cities in developed and developing countries the opportunity to create indicators and baseline information while providing a global monitoring mechanism adaptable to the local level. In particular, UN-Habitat’s CPI represents a global monitoring framework for SDG 11, allowing cities to track progress on the implementation of the 2030 Agenda and encouraging them to monitor and report in a more systematic manner (UN-Habitat, 2021[28]). A benefit of this global framework is a systemic approach to establishing and understanding inter-relations of different dimensions of city development. Furthermore, it provides a single score of the current state of the city through a composite index and establishes benchmarks and baselines for local monitoring. It also features a global platform for comparability, provides evidence for better policy making and accountability, and creates local monitoring mechanisms.
The SDG Partnership Guidebook
The SDG Partnership Guidebook is a tool for monitoring and evaluating partnerships’ objectives. To assess the fulfilment of a partnership’s objectives, the Partnering Initiative and the United Nations Department of Economic and Social Affairs (UN DESA) propose to set up a regular formal review workshop taking place once every 6 or 12 months. In these workshops, project partners shall be invited to provide their opinions on the partnership across different qualitative indicators regarding various aspects of the partnership. These include the fundamentals of the partnership (e.g. if the partnership has identified collaborative advantages), the partnership relation (e.g. if partners are accountable to each other for delivering on their commitment), the structuring and setup (e.g. if the partnership’s governance structure is fit for purpose), resources (e.g. if sufficient personnel is available), management (e.g. if information sharing and decision making is effective) and the enabling environment (e.g. if the partners advocate for more collaborative approaches to the SDGs) (The Partnering Initiative/UN DESA, 2020[26]).
The guidebook evaluates partnerships through the criteria of efficiency, effectiveness and value-added. The efficiency criterion is looking at how well the partnership is designed and implemented. Effectiveness assesses the work that the partnership is delivering and progress towards planned outputs, outcomes and impacts. Value-added refers to the extent to which the partnership adds value by achieving outcomes that cannot be done independently. According to the Partnering Initiative and UN DESA (2020[26]), there are some key factors to consider when monitoring and evaluating partnerships. These include ensuring that it informs the future, ensuring clarification of what data are required and by whom, and trying to take a participatory approach to its design and implementation. Furthermore, it is important to be realistic about available resources, be creative in using already existing information, try to tie into existing frameworks where possible, use the theory of change not only for partnership design but also as a management tool to support reflection on processes and be transparent about the findings and results.
An overview of selected indicator frameworks for the localisation of the SDGs
An important prerequisite for measuring the progress of city-to-city partnerships towards sustainable development is the availability of an adequate indicator framework. While the UN’s SDG indicator framework contains several city-specific indicators for SDG 11, all SDGs will require support for implementation by cities. Currently, the indicators used by the SDG framework are largely targeted towards the national level. Yet, national averages can misrepresent realities in regions and cities and tend to mask large territorial disparities, compromising the SDGs’ premise of leaving no one behind (OECD, 2020[2]). Therefore, various organisations have made strong efforts to develop localised measurement and indicator frameworks and measure cities’ and regions’ progress on the SDGs over the past few years. The following section provides an overview of existing localised measurement frameworks and points out data gaps. The four frameworks and indicators presented are the OECD localised indicator framework for the SDGs, the UN-Habitat Global Urban Monitoring Framework, the European Handbook for SDG Voluntary Local Reviews and a stocktaking on indicators for European cities to assess and monitor the SDGs by the European Topic Centre on Urban Land and Soil Systems.
The OECD localised indicator framework for the SDGs
The OECD localised indicator framework for the SDGs provides indicators to measure the distance of regions and cities to each of the 17 SDGs. The consensual, comparable and standardised framework allows for benchmarking performances within countries and across regions and cities to support public action across levels of government. Through an extensive literature review and expert consultations, the OECD has classified the 169 SDG targets from the UN indicator framework by their level of relevance for subnational levels of government (place-relevant) and advanced economies (OECD-relevant). This classification led to a subset of SDG targets based on their applicability to the context and specificities of OECD countries (OECD-relevance). While all SDGs are relevant for cities and regions, 159 of the SDG targets appear to have a strong subnational component. However, only 105 of them are also very important in the context of OECD countries. While for example, 90% of the targets for sustainable cities apply to OECD regions and cities, only around 30% of the targets of SDG 17 “Partnerships and enablers for the SDGs” and 40% of the targets for SDG 14 “Life below water” appear as a priority to be measured at the subnational level in OECD countries. The OECD measures the achievement of those targets by a selection of 135 indicators for OECD regions and cities (OECD, 2020[2]).
To evaluate the achievements of a city or region on the SDGs, the OECD framework is based on identified end values to shed light on aspirational trends in OECD regions and cities toward the SDGs. Often, these end values are directly derived from the UN framework, i.e. the targets set by the UN. When they were not inferable from the UN framework, the OECD defined end values for indicators based on international guidelines (e.g. World Health Organization [WHO] air quality guidelines), the knowledge of experts in the field or based on the best performance of regions and cities in that indicator. Such end values have the objective of providing technical guidance for governments to advance local development plans and sustain evidence-based policies. End values, which represent ambitious targets to be achieved by 2030, enable regions and cities to assess where they stand today and understand the distance they have to travel by 2030. The OECD localised indicator framework attributes end values to 88% of its indicators, of which 65% are defined using the criteria of “best performers”. The framework also normalises the SDG indicators from 0 to 100 – where 100 is the suggested end value of an indicator to be achieved by 2030 – and aggregates headline indicators that belong to the same SDG to provide an index score for each of the 17 SDGs (Figure 2.2). The distance to the target or goal is the number of units the index needs to travel to reach the maximum score of 100 (OECD, 2020[2]).
Source: OECD (OECD, 2023[35]), Measuring the Distance to the SDGs in cities and regions, http://www.oecd-local-sdgs.org
With its 135 indicators, the OECD localised framework covers at least 1 aspect of each of the 17 SDGs for both regions and cities. The OECD localised framework covers data for more than 600 cities and regions in OECD and partner countries and allows to compare them with their national average, peer cities and regions, including through a dedicated online visualisation tool.1 Although the set of 135 indicators aims to cover the broad spectrum of all 17 SDGs, the coverage in terms of indicators also varies widely across SDGs and is higher for regions than for cities (OECD, 2020[2]). While the framework measures achievements in five of the targets of SDG 11 through seven indicators for cities2 (defined as functional urban areas) and seven indicators for regions,3 the coverage for SDG 17 is expandable. Currently, the measurement for SDG 17 encompasses an indicator of Internet access, relating to how regions and cities can communicate and co-operate to build a partnership for sustainable development and one indicator of international co-patents reflecting how knowledge sharing between regions can enhance access to innovation and foster sustainable development. The indicators do not yet capture components of public capacity (e.g. subnational finance and decentralisation) and development co-operation (e.g. ODA of SDG 17) due to the lack of available data. To advance the statistical agenda on these two components, the OECD keeps developing its work on subnational finance statistics (see OECD/UCLG (2019[36])) – including pilot projects at the regional and municipal levels, as well as on measures of DDC (see OECD (2018[1]; 2019[5]; 2023[37])).
The UN-Habitat Global Urban Monitoring Framework
The UN-Habitat Global Urban Monitoring Framework (UMF) (2022[29]) aims to harmonise existing urban indices and tools to monitor the transformation towards a more sustainable, inclusive, safe and resilient urban area. As an agreed universal framework, it has been developed by taking into consideration existing urban indices and monitoring tools and the need for the development of an urban monitoring framework. As such, it aims to track the performance of the New Urban Agenda and the SDGs, and is meant to encourage data sharing and improve comparability at the global level. Furthermore, it pursues the objective to ensure a thematic integration and interlinkages between different dimensions of development, to promote working at different scales and functional urban areas and allow for comparability across cities. It also serves as a monitoring tool for UN-Habitat’s SDG Cities flagship programme and facilitates VLR reporting. In particular, the UMF allows stakeholders to evaluate the progress of their cities and urban areas through a harmonised indicator set (Figure 2.3).
Source: UN-Habitat (2022[29]), Urban Monitoring Framework, https://data.unhabitat.org/pages/urban-monitoring-framework (accessed on 31 January 2022).
The framework pursues four main objectives of safe and peaceful, inclusive, resilient and sustainable cities. It includes more than 70 indicators across 5 domains: i) society; ii) economy; iii) environment; iv) culture; and v) governance and implementation (UN-Habitat, 2022[29]). The indicators are derived from multiple sources, including the official UN SDG indicators, the United Nations Educational, Scientific and Cultural Organization (UNESCO), the United Nations Children's Fund (UNICEF), the City Prosperity Index, the OECD localised indicator framework for measuring the distance to the SDGs and the New Urban Agenda. Underlying principles for the selection of the indicators are people-centricity (e.g. indicators that could monitor whether cities are becoming more liveable for all, regardless of gender, age or ethnicity, etc.), city-centricity (the framework is intended to be distinct from other UN monitoring frameworks in that it is city-focused, rather than regional or national), usability and thematic interlinkages amongst others (UN-Habitat, 2022[29]).
The European Handbook for SDG Voluntary Local Reviews
In 2020, the EC Joint Research Centre (JRC), the EC’s science and knowledge service, published the European Handbook for SDG Voluntary Local Reviews, a framework for policy makers, researchers and practitioners to set up VLRs (Siragusa et al., 2020[38]). A second edition was released in June 2022 (Siragusa et al., 2022[39]). It builds on the first version and includes an updated analysis of the VLRs published globally and in Europe. As part of the second edition of the handbook, the JRC has compiled a range of 72 indicators covering each of the 17 SDGs. The indicators were selected based on their relevance seven years after the approval of the 2030 Agenda, and also in the context of the COVID-19 recovery as well as their relevance to the European context and the local scale. Overall, the JRC framework uses four types of indicators: i) official indicators, harmonised at the European level; ii) experimental indicators, harmonised and available for a significant number of European cities; iii) official indicators, not harmonised, that were collected by countries or local entities; and iv) experimental local indicators, which are produced by research and other institutions with innovative methodologies. Among the 72 indicators, 53 of the JRC indicators are official indicators while 19 are experimental ones.
The JRC handbook’s objective is to help cities and regions implement effective local SDG monitoring systems. The framework should allow cities and regions to assess quantitatively how they contribute to sustainable development. The SDG handbook points out challenges linked to SDG monitoring at the local scale such as data collection and indicator analysis and provides an overview of existing SDG monitoring systems. The 2020 edition furthermore offers suggestions for the integration of SDG monitoring into city strategy plans and provides considerations regarding several issues on local SDG monitoring such as desired trends, distance to targets, the selection of baseline years and frequency and scale of measurement as well as the boundaries for SDG monitoring.
The European Topic Centre on Urban Land and Soil Systems
In 2020, the European Topic Centre on Urban Land and Soil System of the European Environment Agency released its report Indicators for European Cities to Assess and Monitor the UN SDGs (EEA, 2020[40]). The report aims to provide local governments across Europe with a wide range of indicators to assess their performance towards achieving the 17 SDGs and their respective targets. It is based on the analysis of more than 2 000 indicators across 30 indicator sets published by various global, European and national institutions, all dedicated to the local and urban levels, though often without any reference to the SDGs. The report allocated these indicators to 1 or several of the SDGs and their 169 targets to generate a compilation of existing indicators aligned with the 17 SDGs that target the local level. As such, it serves as a pool of potential indicators local governments can choose from when they are starting to develop their own SDG monitoring system (EEA, 2020[40]).
The report distinguishes different aspects for each of the targets where indicators are available. For example, for SDG Target 11.1 to ensure access for all to adequate, safe, affordable housing and basic service and upgrade slums, the report lists indicators for the aspects of: i) living space; ii) informal housing; iii) adequate housing; iv) evictions and homeless people; v) housing costs; vi) social housing and subsidies; and vii) basic services. Overall, a total of 389 different indicators are allocated to SDG 11. SDG 17 is represented by 82 indicators with a predominant coverage of SDG Target 17.17 to encourage and promote effective public, public-private and civil society partnerships.
The ability of local governments to borrow funds or choose contracts for projects without permission from national levels of government.
Public assistance to international mutual aid actions (budget for international mutual aid actions).
An assessment if the city is leading by example in (integrated) environmental behaviour by using green public procurement guidelines, co-operating with other authorities and organisations on environmental topics and enabling employees to develop their skills on environmental topics (yes/no).
Various indicators on policy coherence for sustainable development.
The share of public expenditures allocated for local development co-operation (%).
Various indicators of stakeholder engagement in local policies.
The availability of a full-time employee directly responsible for the co-ordination of sustainability issues across departments.
The co-operation with other authorities at different levels or other organisations on sustainability issues.
The presence of disaggregated data collection and analysis for different social and income groups in the city.
While some of the 17 SDG indicators presented in the report are quantifiable, many of them can only be measured by yes and no answers or a unit of measurement is missing, which is a challenge that remains to be solved.
The need for a complementary monitoring and evaluation framework
Monitoring and evaluating city-to-city partnerships remains a challenge, despite existing frameworks. Current M&E frameworks for partnerships lack an assessment of how the very process of partnerships can contribute to their effectiveness and usefulness for identified policy objectives and SDG implementation (Stott, 2019[41]). The OECD report Reshaping Decentralised Development Co-operation: The Key Role of Cities and Regions for the 2030 Agenda illustrates that most efforts to monitor and evaluate DDC and city-to-city partnerships projects have so far mainly focused on project results and limited output results, while rarely considering the outcomes or impact of each action (OECD, 2018[1]). In addition, it is often challenging to isolate the value-added or impact of city-to-city partnerships. The logical frameworks that govern interventions also often focus on monitoring and reporting at the project level rather than assessing the impact of the partnerships on the broader development process of a place, which emphasises the importance of designing and aligning them with global agenda objectives, in particular the SDGs. Consequently, most of the value-added assigned to city-to-city partnerships is often speculative and based on aggregated project results. Evaluations should therefore include information on the impact of city-to-city partnerships’ activities on development outcomes and their contribution to sustainable development and citizen well-being (OECD, 2018[1]). They should also be oriented towards assessing results through the critical analysis of information and internalising the lessons learnt. Furthermore, partnerships should be aligned and aim to contribute to impact results, which M&E systems can capture.
M&E frameworks often lack a quantitative component. Some national governments have established evaluation mechanisms to assess the impact, costs and benefits of city-to-city partnerships, mostly through reports. For example, in France, an evaluation report is systematically presented at the deliberative assembly of local authorities to report on the impacts of project spending. In the Netherlands, log frames and the theory of change are often used as M&E systems. However, more quantitative methods, such as surveys and indicator systems, are less commonly used to assess the impact of city-to-city partnerships interventions (OECD, 2018[1]). Moreover, indicator availability for M&E purposes at the subnational level remains insufficient, preventing the creation of reliable analysis of SDG achievements at the local level (UCLG, 2021[23]). Thus, there is a need to assess the extent to which partnerships are achieving their goals and fostering sustainable development at the local level. In its 2018 report on reshaping DDC, the OECD encouraged a culture of M&E of city-to-city partnerships projects by moving towards a common M&E framework for all projects (OECD, 2018[1]). To that effect, the OECD proposed to develop tools such as indicators, report-back templates and tools to monitor progress, to ensure that all data and information gathered for city-to-city partnerships projects are consistent, comparable and harmonised. In particular, the OECD recommended increasing the use of data collected through the national decentralised co-operation portals, where they exist (OECD, 2018[1]).
Gaps identified from the analysis of existing toolkits and self-assessment tools
The stocktaking of checklists and self-assessment tools reveals a number of gaps in terms of simultaneously measuring the progress of cities engaged in city-to-city partnerships towards the SDGs and engaging stakeholders to increase ownership of the assessment. Although some frameworks aim to assess partnerships (SDG Partnership Guidebook) and the extent to which cities are achieving urban development (Reference Framework for Sustainable Cities, RFSC, and City Prosperity Initiative, CPI), none of them measures how cities engaged in city-to-city partnerships are making progress towards reaching the SDGs and assess the coherence with sustainability principles in the implementation process (see below). A complementary framework allowing for more systemic M&E could lead to more effective policy making and improved accountability. The SDG guidebook underlines important factors to consider when creating partnerships that could be integrated into such a complementary framework, including regular formal reviews, with the opportunity for learning and feedback on the framework and process. Similarly, the OECD Water Governance Indicator Framework suggests a ten-step multi-stakeholder process to build consensus on what works or not, and facilitate cross-fertilisation of knowledge and learning amongst partners. A complementary framework for city-to-city partnerships could seek to facilitate a multi-stakeholder approach to integrate peer-to-peer learning and increase the scope and stakeholder ownership of the assessment. The G20 High-level Principles on city-to-city partnerships for localising the SDGs, developed under the Italian G20 presidency in 2021 (see Chapter 1), provide a common framework of good practices and framework conditions for policy makers at the international, national, regional and local levels to initiate, develop and monitor city-to-city partnerships for the SDGs (G20 Development Working Group, 2021[42]) Consequently, they provide a valuable reference framework to measure to which extent cities are aligned with such principles in their efforts to engage partnerships to localise the SDGs.
Gaps identified from existing indicator frameworks on the localisation of the SDGs
The overview of indicator frameworks for the localisation of the SDGs revealed the potential to expand the scope and comparability of indicators as well as their geographical outreach. Although the OECD localised indicator framework for the SDGs can assess the SDGs at the local level, there is still scope to extend the number of indicators available, notably regarding SDG 17. The JRC European Handbook for SDG Voluntary Local Reviews (Siragusa et al., 2022[39]) contains a list of more than 70 SDG indicators that cover all 17 SDGs, however with a particular focus on European cities. Consequently, its applicability in the context of city-to-city partnerships that cover both cities from developed and developing countries is limited. The European Topic Centre report on indicators for European cities to assess and monitor the UN SDGs provides a mapping of more than 2 000 SDG indicators. Its coverage thus goes beyond what individual cities or institutions can afford in terms of measurement and management capacities. UN-Habitat’s UMF highlights some advantages of having a universally agreed framework for measuring territorial development, allowing for comparability across cities, but is not applied to city-to-city partnerships and DDC specifically.
In the EC Partnerships programme, the use of customised and project-specific indicators in log frames makes the comparability across partnerships challenging. The programme allows partnerships to establish their own indicators for the overall objectives and the specific objectives of individual partnerships. As a consequence, each partnership uses different project-specific indicators to assess its outcome, making it challenging to compare the outcomes of different partnerships and evaluate their success in comparison to other partnerships. This can create challenges when attempting to compare the results of city-to-city partnerships. First, because different types of indicators may have been chosen to measure the same objectives and impact. For example, good urban governance is a common objective of several city-to-city partnerships. For example, the MUEVE partnership between San José, Costa Rica, and San Sebastian, Spain, uses the UN-Habitat City Prosperity Index to measure good governance, while the Futureproof Banjul (Ostend, Belgium and Banjul, Gambia) partnership measures the same objective through the level of satisfaction of city actors regarding urban governance.
In addition, the type of indicators (i.e. qualitative or quantitative) used in the log frames differs between partnerships. Some partnerships use quantitative indicators. The Smart Change – Strengthening Urban Governance, Prosperity and Innovation in Jakarta partnership (Berlin, Germany and Jakarta, Indonesia) uses the indicator of level/quality of “rule of law” practices as a quantitative and numerical baseline to assess good governance. Others, such as the Partnership for Strengthening Governance, Resilience and Greening of Jinja Municipal Council (Soria, Spain and Jinja, Uganda), measure the contribution to urban governance through citizen service delivery satisfaction surveys, adopting a more qualitative approach. On the other hand, the partnership Strengthening the Governance of the Metropolitan Area of San Salvador, for the Adoption and Implementation of the Objectives of Sustainable Development (San Salvador, El Salvador and Barcelona, Spain) measures the objective of improved urban governance through the establishment of a legal and regulatory framework that takes the SDGs into account, thus using qualitative indicators. The application of different types of indicators (qualitative vs. quantitative) challenges the comparability of impacts and project success across partnerships – in the above example, the objective of good urban governance.
A homogeneous M&E framework for city-to-city partnerships could help gauge the effectiveness of partnerships at the subnational level. Setting up a dedicated M&E framework for the entire supply chain of city-to-city partnerships could help better assess the effectiveness of partnerships against their objectives such as the objectives mentioned in the EC Partnerships programme (i.e. to strengthen urban governance, ensure social inclusiveness of cities, improve resilience and greening of cities and improve prosperity and innovation in cities); or their compliance and alignment with global recommendations such as the ten G20 Principles on city-to-city partnerships, or the progress of cities involved in city-to-city partnerships towards achieving the SDGs specifically. The systemic M&E framework for city-to-city partnerships presented in Chapter 3 aims to bridge that gap by combining the assessment of the implementation of the G20 Principles with measuring the progress towards the SDGs of cities engaged in city-to-city partnerships.
References
[13] Aghumian, A. (2014), “Partnering for evaluations: Seven guiding principles for evaluating partnership programs, An IEG note for the DAC”, World Bank Group, https://www.oecd.org/dac/evaluation/BKG-IEG-Partnering-for-Evaluation.pdf.
[14] de Losada Passols, A. (2017), Shaping a New Generation of Decentralised Cooperation - For Enhanced Effectiveness and Accountability, Platforma, CPMR, https://cpmr.org/wpdm-package/cpmrplatforma-study-shaping-a-new-generation-of-decentralised-cooperation-for-enhanced-effectiveness-and-accountability/?wpdmdl=15687&ind=1512992269841.
[4] EC (2022), Monitoring and Evaluation, European Commission, https://ec.europa.eu/international-partnerships/our-impact/monitoring-and-evaluation_en (accessed on 21 January 2022).
[27] EC (2021), Contract Procedures for European Union External Action, European Commission, https://intpa-econtent-public.s3.eu-west-1.amazonaws.com/ePrag/2021.0/ePRAG_public_full_en.pdf (accessed on 9 September 2022).
[40] EEA (2020), Indicators for European Cities to Assess and Monitor the UN Sustainable Development Goals (SDGs), European Environment Agency, https://www.eionet.europa.eu/etcs/etc-uls/products/etc-uls-report-2020-08-indicators-for-european-cities-to-assess-and-monitor-the-un-sustainable-development-goals-sdgs (accessed on 9 July 2021).
[24] Figueiredo, L., T. Honiden and A. Schumann (2018), “Indicators for Resilient Cities”, OECD Regional Development Working Papers, No. 2018/02, OECD Publishing, Paris, https://doi.org/10.1787/6f1f6065-en.
[42] G20 Development Working Group (2021), Territorial Development and SDGs Localisation, Final DWG Outcome Document on Territorial Development and SDGs Localisation, https://dwgg20.org/app/uploads/2021/10/Territorial-Development-SDGs-Localisation.pdf.
[32] GPEDC (2020), GPEDC at a Glance, Global Partnership for Effective Development Co-operation, https://www.effectivecooperation.org/content/gpedc-glance-0 (accessed on 11 March 2022).
[33] GPEDC (2017), The Global Partnership Monitoring Framework for 2030: Concept Note, Global Partnership for Effective Development Co-operation, https://www.effectivecooperation.org/system/files/2020-06/20170510%20Monitoring%20Framework%20Review%20Strategy%20%2801%20June%202017%29.pdf.
[15] IEG (2011), The World Bank’s Involvement in Global and Regional Partnership Programs, An Independent Assessment, Independent Evaluation Group, https://www.oecd.org/derec/worldbankgroup/48296274.pdf.
[20] Lamhauge, N., E. Lanzi and S. Agrawala (2012), “Monitoring and Evaluation for Adaptation: Lessons from Development Co-operation Agencies”, OECD Environment Working Papers, No. 38, OECD Publishing, Paris, https://doi.org/10.1787/5kg20mj6c2bw-en.
[35] OECD (2023), Measuring the Distance to the SDGs in Regions and Cities (visualisation tool), OECD, Paris, http://www.oecd-local-sdgs.org (accessed on 30 January 2023).
[37] OECD (2023), Reshaping Decentralised Development Co-operation in Cities and Regions: Lessons from Germany for a Global Policy Toolkit, OECD, Paris.
[31] OECD (2022), How to Assess Water Governance - A Methodology Based on the OECD Principles on Water Governance, OECD, Paris, https://www.oecd.org/cfe/cities/HowToAssessWaterGovernance.pdf.
[8] OECD (2021), “A Territorial Approach to the SDGs in Córdoba, Argentina”, OECD Regional Development Papers, No. 16, OECD Publishing, Paris, https://doi.org/10.1787/f11a65c2-en.
[17] OECD (2021), Evaluation Criteria, OECD, Paris, https://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm (accessed on 7 March 2022).
[9] OECD (2020), “A territorial approach to the Sustainable Development Goals in Kópavogur, Iceland”, OECD Regional Development Working Papers, No. 2020/05, OECD Publishing, Paris, https://doi.org/10.1787/e0f3c1d6-en.
[2] OECD (2020), A Territorial Approach to the Sustainable Development Goals: Synthesis report, OECD Urban Policy Reviews, OECD Publishing, Paris, https://doi.org/10.1787/e86fa715-en.
[18] OECD (2020), Better Criteria for Better Evaluation - Revised and Updated Evaluation Criteria, OECD, Paris, https://www.oecd.org/dac/evaluation/evaluation-criteria-flyer-2020.pdf.
[16] OECD (2020), The Circular Economy in Cities and Regions: Synthesis Report, OECD Urban Studies, OECD Publishing, Paris, https://doi.org/10.1787/10ac6ae4-en.
[5] OECD (2019), “Decentralised development co-operation: Unlocking the potential of cities and regions”, OECD Development Policy Papers, No. 22, OECD Publishing, Paris, https://doi.org/10.1787/e9703003-en.
[11] OECD (2018), OECD Water Governance Indicator Framework, OECD, Paris, https://www.oecd.org/regional/OECD-Water-Governance-Indicator-Framework.pdf.
[1] OECD (2018), Reshaping Decentralised Development Co-operation: The Key Role of Cities and Regions for the 2030 Agenda, OECD Publishing, Paris, https://doi.org/10.1787/9789264302914-en.
[12] OECD (2016), Policy Coherence for Sustainable Development Toolkit, OECD, Paris, https://www.oecd.org/governance/pcsd/toolkit/ (accessed on 7 March 2022).
[30] OECD (2015), OECD Principles on Water Governance, OECD, Paris, https://www.oecd.org/cfe/regionaldevelopment/OECD-Principles-on-Water-Governance-en.pdf.
[7] OECD (2002), Glossary of Key Terms in Evaluation and Results Based Management, OECD, Paris, https://unsdg.un.org/sites/default/files/OECD-Glossary-of-Key-Terms-in-Evaluation-and-Results-based-Management-Terminology.pdf.
[36] OECD/UCLG (2019), 2019 Report of the World Observatory on Subnational Government Finance and Investment - Country Profiles, http://www.sng-wofi.org/publications/SNGWOFI_2019_report_country_profiles.pdf.
[34] OECD/UNDP (2019), Making Development Co-operation More Effective: How Development Partners Are Promoting Effective, Country-led Partnerships - Part II of the Global Partnership 2019 Progress Report, Organisation for Economic Co-operation and Development/United Nations Development Programme, https://www.oecd.org/dac/effectiveness/Part-II-of-the-Global-Partnership-Progress-Report.pdf.
[22] OECD/UNDP (2016), Making Development Co-operation More Effective: 2016 Progress Report, OECD Publishing, Paris, https://doi.org/10.1787/9789264266261-en.
[6] Potluka, O. (2020), “A conceptual overview of how and why to evaluate partnership”, in Transitioning to Strong Partnerships for the Sustainable Development Goals, https://doi.org/10.3390/books978-3-03897-883-1-8.
[19] SDSN (2016), Getting Started with the SDGs in Cities: A Guide for Stakeholders, United Nations Sustainable Development Solutions Network, https://irp-cdn.multiscreensite.com/be6d1d56/files/uploaded/9.1.8.-Cities-SDG-Guide.pdf (accessed on 13 July 2021).
[39] Siragusa, A. et al. (2022), European Handbook for SDG Voluntary Local Reviews - 2022 Edition, Publications Office of the European Union, Luxembourg, https://doi.org/10.2760/218321.
[38] Siragusa, A. et al. (2020), European Handbook for SDG Voluntary Local Reviews, EUR 30067 EN, Publications Office of the European Union, Luxembourg, https://doi.org/10.2760/670387.
[41] Stott, L. (2019), “Partnerships for sustainable development: The monitoring and evaluation challenge”, https://ecdpm.org/great-insights/civil-society-business-same-direction/partnerships-sustainable-development-monitoring-evaluation-challenge/.
[26] The Partnering Initiative/UN DESA (2020), The SDG Partnership Guidebook: A Practical Guide to Building High Impact Multi-stakeholder Partnerships for the Sustainable Development Goals, United Nations Department of Economic and Social Affairs and The Partnering Initiative, https://sustainabledevelopment.un.org/content/documents/26627SDG_Partnership_Guidebook_0.95_web.pdf.
[23] UCLG (2021), Towards the Localization of the SDGs - Sustainable and Resilient Recovery Driven by Cities and Territories, United Cities and Local Governments, https://gold.uclg.org/sites/default/files/hlpf_2021.pdf.
[21] UCLG (2019), Learning Module 4: Localising the SDGs Through Decentralized Cooperation, United Cities and Local Governments Capacity and Institution Building Working Group, https://www.cglu.org/sites/default/files/module4.pdf.
[10] UCLG/CES/OIDP (2014), Basic Principles of Community-based Monitoring, United Cities and Local Governments/Centro de Estudos Sociais/Observatoire International de la Démocratie Participative, https://issuu.com/uclgcglu/docs/community-based_monitoring.
[29] UN-Habitat (2022), Urban Monitoring Framework, United Nations Human Settlements Programme, https://data.unhabitat.org/pages/urban-monitoring-framework (accessed on 31 January 2022).
[28] UN-Habitat (2021), City Prosperity Initiative, United Nations Human Settlements Programme, https://unhabitat.org/programme/city-prosperity-initiative (accessed on 16 August 2021).
[3] UN-Habitat/UCLG (2020), Monitoring and Evaluation National Urban Policy: A Guide, United Nations Human Settlements Programme and United Cities and Local Governments, https://unhabitat.org/sites/default/files/2020/05/monitoring-and-evaluating-national-urban-policy_a-guide_web.pdf.
[25] Zinkernagel, R., J. Evans and L. Neij (2018), “Applying the SDGs to cities: Business as usual or a new dawn?”, Sustainability, Vol. 10/3201, https://doi.org/10.3390/su10093201.
Notes
← 1. For more information, see https://www.oecd-local-sdgs.org/index.html.
← 2. Percentage of households expenses dedicated to housing costs; percentage of population satisfied with affordability of housing; percentage of population satisfied with the quality of public transportation systems; difference between built-up area growth rate and population growth rate (percentage points); exposure to PM2.5 in µg/m³, population weighted (micrograms per cubic metre); percentage of population satisfied with quality of air; percentage of people exposed to more than 10 µg/m³ (micrograms per cubic metre) of PM2.5.
← 3. Performance of public transport network, ratio between accessibility and proximity to hospitals; performance of car transport network, ratio between accessibility and proximity to hospitals; difference between built-up area growth rate and population growth rate (percentage points); exposure to PM2.5 in µg/m³, population weighted (micrograms per cubic metre); percentage of people exposed to more than 10 µg/m³ (micrograms per cubic metre) of PM2.5; percentage of population with access to at least 1 hectare of green urban areas (parks) and forests within 15 minutes of walking; percentage of population with access to at least one recreational opportunity (theatres, museums, cinemas, stadiums or cultural attractions) within 15 minutes of cycling.