5. Data, monitoring and evaluation of NUP
This chapter explores how countries collect data and use monitoring and evaluation to improve the national urban policy (NUP) process. It examines data sources used in the NUP process, types of NUP monitoring and evaluation frameworks, as well as how the results are used. The chapter also discusses effective design of NUP monitoring and evaluation frameworks, including aligning NUP targets and indicators with the Sustainable Development Goals and other global indicators. Finally, the chapter presents selected internatioanl initiatives to address demand for urban-scale data and develop a more robust and internationally comparable framework.
Countries primarily use data obtained or produced at national level by the statistical office (66 out of 86 countries, or 77%), by relevant ministries or agencies (54 countries, 63%) or by the lead NUP ministry or agency (53 countries, 62%). A majority (48 countries, 58%) use data from research institutes, signalling an important synergy between policymakers and academia.
Bottom-up processes for data collection could be improved, with only 34 countries (40%) using data from sub-national governments and 29 countries (34%) from the private sector.
Countries use diverse frameworks to monitor and evaluate their NUP. Policy dialogues and workshops are the most common mechanism (40 countries, 59%), followed by progress reports (33 countries, 48%) and specific targets and indicators (26 countries, 38%).
NUP monitoring and evaluation can help countries revisit their NUP (38 countries, 56%), improve multi-level co-ordination (32 countries, 47%) and allocate budget and invest in support of NUP implementation (27 countries, 40%). 21 countries (31%) indicate that NUP monitoring and evaluation are part of their SDG monitoring and evaluation frameworks.
To address demand for urban-scale data, international organisations are developing a more robust and internationally comparable framework to measure the degree of urbanisation, also relevant to NUP monitoring and evaluation.
Data is essential to drive evidence-based policy making, and the national urban policy (NUP) process is no exception. This chapter investigates how countries use data, and which types of data, in the NUP process (Figure 5.1). Unsurprisingly, most data are obtained from the national government: 66 countries (77%) report using national census data or data produced by the national statistical office, 54 (63%) use data obtained from relevant ministries and agencies, 53 (62%) use data produced by the lead NUP ministries and agencies.
Forty-eight countries (56%) use data from academia and research centres, highlighting an important area of synergy between policymakers and academic experts. Additionally, geographic information systems (GIS) and maps are used in 47 countries (55%), reflecting the importance of understanding context-specific factors that facilitate place-based policy making.
Bottom-up processes involving the collection of data from either sub-national governments or from the private sector and other stakeholders (e.g. market surveys, community-led data) are addressed respectively by 34 (40%) and 29 (34%) countries. While direct use of bottom-up data could be scaled up in NUP processes in general, some countries do have sophisticated national-level data collection systems that incorporate data from bottom-up processes. These report the data as issuing from the national statistical office, which may explain the relatively low share of data use from sub-national governments and private sector/other stakeholders. In either case, lead NUP ministries should engage in regular dialogue with local governments to ensure the latest data and information is reflected, especially when formulating/revising NUP processes. Data from social media is used by only 10 countries (12%), which could reflect the fact that such data may not always be relevant or that there may be privacy and credibility concerns surrounding its use.
Countries use varied frameworks to monitor and evaluate their NUP (Figure 5.2). Most frequent were “dialogues, workshops and conferences to discuss NUP performance with sub-national governments and stakeholders”, reported by 40 (59%) of the 68 countries with NUP in or beyond the formulation stage. This was followed by “publication of progress reports” (33 countries, 48%), “reporting to legislative assemblies” (27 countries, 40%) and “development and use of targets and indicators” (26 countries, 38%). Over a quarter of countries (21 countries, 31%) indicate that NUP monitoring and evaluation frameworks are part of their Sustainable Development Goal (SDG) monitoring and evaluation frameworks (e.g. voluntary national review), while only 15 countries (22%) use third-party monitoring and evaluation (e.g. from a court of audit or academic review). Given the expected role of NUP in achieving the 2030 Agenda for Sustainable Development and the New Urban Agenda (Chapter 6), countries could make more effort to align NUP monitoring and evaluation frameworks with global agendas.
In France, the National Observatory on Urban Policy monitors and assesses inequalities and development gaps in priority neighbourhoods. The observatory also contributes to the evaluation of policies in the priority neighbourhoods. It prepares an annual report for parliament and government. The 2018 report focused on local employment and economic development.
In Turkey, the national government uses biannual municipal surveys containing selected indicators to measure progress on NUP at the local level. The government produces biannual activity reports to measure progress at the national level. These are submitted to the Monitoring and Steering Committee. and published online for the public.
Note: Note: data are drawn from the OECD/UN-Habitat/Cities Alliance National Urban Policy Country Survey 2020. The survey question aimed at NUPs in the “formulation stage” or beyond (i.e. not in “diagnosis” or “feasibility” stages), hence the results cover 68 countries instead of 86.
Successful formulation, implementation and review of NUP depends on the quality and credibility of the monitoring and evaluation systems in place. The NUP country survey examined how countries use or intend to use the results of monitoring and evaluation of their NUP (Figure 5.3). Many countries employ the results of monitoring and evaluation to revise their current NUP (38 NUPs, 56%) or to improve multi-governance co-ordination (32 countries, 47%). Thirty countries (44%) disseminate the results to relevant legislative bodies, sub-national governments and stakeholders, while 27 countries (40%) use the results to improve budget allocation and investments supporting NUP implementation.
The results illustrate potential to use NUP monitoring and evaluation for wider purposes, including multi-level dialogues, awareness raising, budget allocation and investment decisions. In addition, although only 19 countries (28%) use the results of monitoring and evaluation to contribute to SDG monitoring frameworks, such as voluntary national reviews, aligning NUP monitoring and evaluation frameworks with SDGs and other domestic policy frameworks would be key for effectiveness (Chapter 6).
As discussed, more than a third of countries surveyed (26 out of 68, or 38%) use a goals, targets and corresponding indicators within their NUP monitoring and evaluation frameworks (Figure 5.2). However, data and evidence at the urban scale is not always available, which makes quantitative assessment of NUP impacts difficult. The NUP country survey shows few countries reporting that their NUPs are grounded in “robust urban-scale data” (Chapter 3). The lack of urban-scale data is also highlighted in monitoring the SDGs, for which several indicators should be collected from cities or urban areas. To address demand for urban-scale data, international organisations are developing a more robust and internationally comparable framework to measure the degree of urbanisation, also relevant to NUP monitoring and evaluation.
Note: Note: data are drawn from the OECD/UN-Habitat/Cities Alliance National Urban Policy Country Survey 2020. The survey question aimed at NUPs in the “formulation stage” or beyond (i.e. not in “diagnosis” or “feasibility” stages), hence the results cover 68 countries instead of 86.
Applying a harmonised definition of cities, rural and urban areas
Monitoring global agendas requires internationally harmonised indicators for cities, urban and rural areas. For example, the Global Monitoring Framework of the SDGs includes several indicators that should be collected for cities or rural and urban areas. However, in the absence of an international harmonised framework, indicators rely on national definitions that vary and thus limit international comparability (United Nations Statistical Commission, 2020[1]).
Since 2016, six international organisations, namely the European Commission (EC), the Food and Agriculture Organization of the United Nations (FAO), the International Labour Organization (ILO), the OECD, UN-Habitat and the World Bank, work together to develop a new method to delineate cities, metropolitan, urban and rural areas for international statistical comparison. In March 2020, the United Nations Statistical Commission endorsed the new method, called the Degree of Urbanisation. The Degree of Urbanisation classifies the territory of a country into three groups along the urban-rural continuum: (1) cities, (2) towns and semi-dense areas, and (3) rural areas (Box 5.1). Applying the method will provide a new perspective on the urban system of a country and allow comparisons of urban data across countries.
In March 2020, the 51st session of the United Nations Statistical Commission endorsed a new method to delineate cities, urban and rural areas, called the Degree of Urbanisation. It includes two concepts for defining cities and their surroundings: the Degree of Urbanisation and the Functional Urban Area.
The Degree of Urbanisation classifies the territory of a country into three groups – (1) cities, (2) towns and semi-dense areas, and (3) rural areas – based on population density, population size and contiguity using 1 km² grid cells. To delineate cities, first an urban centre (or a high-density cluster) is identified. An urban centre consists of contiguous grid cells with a density of at least 1,500 inhabitants per km² and with a population of at least 50,000. Local units (e.g. municipalities) classified as cities have at least 50% of their population in urban centres.
Applying the Degree of Urbanisation reveals that, as of 2015, almost half the world’s population (48%) live in cities, a quarter (24%) in rural areas, and the remainder (28%) in towns and semi-dense areas. The population living in cities is projected to reach 55% in 2050. The newly defined category “towns and semi-dense areas” avoids the urban-rural dichotomy and captures the urban-rural continuum.
The new method also delineates functional urban areas (FUA), or metropolitan areas, by identifying commuting zones around each city. This makes it possible to gauge cities beyond their administrative boundaries in a comparable way across countries and assess their performance according to a set of economic, social, environmental and institutional indicators. While the OECD in collaboration with the European Commission introduced the concept of FUA in 2010 and applied it to OECD countries, the new method extends it on a global scale.
The Degree of Urbanisation complements rather than replaces definitions used by national statistical institutes and ministries. It is expected to help examine and monitor SDGs and other urban policy agendas globally through internationally comparable data and statistics.
Source: OECD/European Commission (2020), Cities in the World: A New Perspective on Urbanisation, https://dx.doi.org/10.1787/d0efcbda-en; United Nations Statistical Commission (2020), A recommendation on the method to delineate cities, urban and rural areas for international statistical comparisons,https://unstats.un.org/unsd/statcom/51st-session/documents/BG-Item3j-Recommendation-E.pdf.
Selecting appropriate geographical scale for measuring policy impacts
Measuring policy impacts at the urban level requires selecting the appropriate scale of analysis. Depending on the phenomenon under consideration, monitoring indicators at some geographical scales can be more appropriate than others, encouraging different levels of granularity. While sub-national authorities measure outcomes within the boundaries of their jurisdictions, it is often important from the NUP perspective to consider the economic dynamic of contiguous local authorities that function as an integrated urban area. Applying the concept of FUAs allows consideration of commuting flows, and thus reflects the full extent of labour markets and other functional linkages (Box 5.1). The OECD Principles of Urban Policy, launched in 2019, also highlight the importance of applying the appropriate scale in designing and implementing urban policy (Box 5.2).
Building on 20 years of urban policy work, the OECD Principles on Urban Policy, adopted by all OECD countries in March 2019, offer a framework to guide national and sub-national policymakers in building smart, sustainable and inclusive cities in responsibility shared between the public, private and non-profit sectors. The 11 principles are organised around the “3S” framework of scale, strategy and stakeholders:
Adapt to the scale where people live and work in real life, beyond administrative perimeters drawn on a map.
Align all policy sectors that play a key role in cities – from economic development and education to housing, transport and land use – into a coherent strategy.
Engage stakeholders from all segments of society to put people at the centre of urban policy.
Under ‘scale’, three principles present the importance of targeting effective ambits for policy action:
Maximise the potential of cities of all sizes to advance national and global prosperity and well-being over time.
Adapt policy action to the place where people live and work.
Support interdependencies and co-operation between urban and rural areas.
The Principles were developed with diverse stakeholders, including international organisations (e.g. UN-Habitat, UNEP), development banks (e.g. World Bank, Inter-American Development Bank), networks of cities and local governments (e.g. United Cities and Local Governments, ICLEI), research institutes and academia (e.g. Lincoln Institute of Land Policy), and the private sector.
The OECD is developing an implementation toolkit including an urban repository and an assessment framework.
Source: OECD (2019), OECD Principles on Urban Policy, http://www.oecd.org/cfe/Brochure-OECD-Principles-Urban-Policy.pdf.
Defining localised yet comparable indicator frameworks
To go beyond national averages and shed light on territorial disparities, many regions and cities define place-specific NUP indicators. Still missing, however, is a consensual, comparable and standardised framework to measure performance across regions and cities from an international comparative perspective. To that end, the OECD’s localised SDGs indicator framework presents a useful approach. It bridges this gap with consistent definitions and comparable indicators across regions and cities of OECD and selected partner countries (Box 5.3 and Chapter 6).
The OECD developed a framework to localise SDG targets and indicators and measure the progress of regions and cities towards each of the 17 SDGs. This consensual, comparable and standardised framework benchmarks performance within countries, and across regions and cities to support public action across levels of government.
In the context of OECD countries, at least 105 of the 169 underlying SDG targets were identified as relevant for regions and cities. Indicators were then selected based on their applicability to the context and specificities of OECD countries. The result is 105 SDG targets and 135 indicators for OECD regions and cities (also referred as the Sub-national SDG Targets).
With its 135 indicators, the OECD’s localised SDG indicator framework allows 601 regions and 649 cities (above 250 000 inhabitants) in OECD and partner countries to measure progress on the SDGs. A web-based visualisation tool (www.oecd-local-sdgs.org) fosters peer learning and policy dialogues across similar regions and cities. The framework also defines target “End Values” for 2030 (derived from either the United Nations framework, based on expert assessments, or the best-performing cities and regions per OECD country), which allow regions and cities to assess where they stand and how much distance remains.
Analysis of the 649 cities shows that cities are performing relatively well regarding SDG 2 (food security and agriculture), as 78 out of 111 cities (30%) with available data already reached the end value for 2030. In contrast, SDG 13 (climate action) appears as the most challenging goal: only 11 out of 543 cities (2%) reached the end value (Figure 5.5). Localised indicators show that OECD cities’ path towards the SDGs remains challenging, with 70% of cities or more not yet at end values for 2030 in 15 of the 17 SDGs.
Source: OECD (2020[2]), Measuring the Distance to the SDGs in Regions and Cities, https://www.oecd-local-sdgs.org/index.html.
The analysis in this chapter indicates that countries primarily use national-level data for their NUP process (77%) alongside other valuable sources, including academia and research centres (58%), sub-national governments (40%) and the private sector or other stakeholders (34%). There is room for ministries engaged in NUP to diversify their data sources, in particular from local governments, to ensure NUP processes reflect the latest data and information.
The chapter reveals that countries should consider scaling up the use of targets and indicators for more robust monitoring and evaluation of NUP. Whereas countries monitor and evaluate NUP performance primarily through policy dialogues, workshops and conferences (40 countries, 59%), a relatively low share (26 countries, 38%) report developing and using targets and indicators for NUP monitoring and evaluation. Recent initiatives by international organisations to develop a more robust and internationally comparable framework, for instance to measure the degree of urbanisation, are relevant and applicable to NUP monitoring and evaluation.
Finally, this chapter highlighted that, beyond revising NUPs, countries use the results of NUP monitoring and evaluation to improve co-ordination between ministries and across levels of government (32 countries, 47%). This illustrates the role of NUP in co-ordination within a country and confirms that NUP should not be seen as an end in itself, but as a means to improve governance for sustainable urban development.
Going forward, countries should continue to invest in robust urban-scale data to design NUPs that address place-based concerns, foster monitoring and evaluation, and facilitate evidence-based stakeholder dialogue, accountability and integrity throughout the NUP process.
References
[6] OECD (2020), A Territorial Approach to the Sustainable Development Goals: Synthesis report, OECD Urban Policy Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/e86fa715-en.
[2] OECD (2020), Measuring the Distance to the SDGs in Regions and Cities, OECD, Paris, https://www.oecd-local-sdgs.org/index.html.
[5] OECD (2019), OECD Principles on Urban Policy, OECD Centre for Entrepreneurship, SMEs, Regions and Cities, Paris, http://www.oecd.org/cfe/Brochure-OECD-Principles-Urban-Policy.pdf.
[4] OECD/European Commission (2020), Cities in the World: A New Perspective on Urbanisation, OECD Urban Studies, OECD Publishing, Paris, https://dx.doi.org/10.1787/d0efcbda-en.
[3] OECD/UN-Habitat/Cities Alliance (2020), National Urban Policy Country Survey 2020.
[1] United Nations Statistical Commission (2020), “A recommendation on the method to delineate cities, urban and rural areas for international statistical comparisons, background document for the 51st session, prepared by the European Commission, ILO, FAO, OECD, UN-Habitat and World Bank”, https://unstats.un.org/unsd/statcom/51st-session/documents/BG-Item3j-Recommendation-E.pdf.