copy the linklink copied!Overview: Using the SDGs as a shared framework for results
The international development community still does not understand enough about how, where and why the best results happen. Can using the SDG indicators to measure results begin to fill this knowledge gap? This Chapter outlines the issues and reviews progress to-date in integrating SDG indicators into national planning and development co-operation practices. Case studies in the sectors of education, sanitation and energy access reveal that both governments and providers of development co-operation are increasingly using SDG indicators to guide their efforts. A closer examination of three large recipient countries (Ethiopia, Kenya and Myanmar) suggests that providers are facing four interrelated challenges. First, the cost of using specific SDG indicators varies in relation to indicator complexity. Second, providers that synchronise their country-level results planning with partner countries’ own cycles find it easier to align to and measure SDG indicators. Third, reliance on joint monitoring approaches helps providers reduce the cost of SDG monitoring. Finally, while disaggregating SDG data by gender and by urban/rural dimensions is common, other data disaggregation that could help to leave no one behind is rare.
We lack the results data we need, and we are not taking advantage of synergies to get that data. The OECD-DAC Results Community conducted three case studies to generate evidence, analysis and good practice examples. These case studies document how development co-operation providers and partners can use the SDG framework as an entry point for co-ordinating, investing in, and using country-led results frameworks and data that are aligned to the SDGs from a technical, organisational and political perspective.
This report summarises emerging opportunities and obstacles for alignment, measurement and use of SDG indicators as a shared framework for results. It also discusses emerging findings on four cross-cutting issues that will require collective thinking and action to unlock the potential of the SDGs as a shared framework for results at all levels.
Each case study looks closely at specific SDG indicators related to education, sanitation and energy. Figure 1 summarises the research approach used to identify technical, political and organisational drivers and obstacles in using SDG indicators at global and partner country levels.1
The case studies show that:
-
1. The Sustainable Development Goals framework has been significantly strengthened and providers and partner countries are actively and increasingly applying SDG indicators in their results frameworks.
-
2. Each indicator presents a varying cost of alignment, related to indicator quality and intrinsic complexity, measurement inertias, and other contextual factors affecting its measurement and use.
-
3. Providers that synchronise their results planning cycle with partner countries’ own cycle are more successful in applying, measuring and using SDG indicators in synergy with partner governments and other providers.
-
4. Sector- and country-wide monitoring approaches help providers reduce the cost of SDG monitoring. The lack of results data against many indicators suggests that more consistent and coherent efforts are needed to strengthen partner countries’ statistical and monitoring systems and ensure their sustainability.
-
5. While gender and urban/rural data disaggregation is becoming more common in SDG indicator measurement, other locally relevant dimensions are rare, limiting the ability to capture results related to populations left behind in complex societies.
copy the linklink copied!SDG indicator adoption at global level: Opportunities and state of play
Opportunities increase with the strengthening of the SDG indicator framework
The past two years have witnessed a significant strengthening of the SDG indicator framework. The number of SDG indicators with an internationally established methodology surged from 138 (2016) to 208 (December 2019), meaning that 90% of all SDG indicators were ready to use.2 Out of these, a majority of countries are regularly collecting data for 116 indicators (50% of total). At present, the Inter-Agency Expert Group on SDGs is carrying out a comprehensive review to develop the SDG indicator framework in full by end 2020.3
The SDG targets and SDG indicators present a series of opportunities for development co-operation. The internationally agreed framework is gaining political traction at country level as a shared framework for results (see Figure 2) and as a roadmap to guide provider results at country level (OECD, 2018[1]). While prioritising amongst the broad number of targets and indicators and managing their interconnected nature across corporate and country-level results frameworks are distinctive challenges linked to SDG alignment, the substantive focus of the targets and indicators has become more relevant to partner country priorities and provider country programming, reflecting a greater focus on quality and sustainability concerns. The three case studies show that, in general, the three SDG indicators under review were a “better fit” than previous provider indicators being used to track results in the respective sectors.
Partner countries are using the SDGs in their national strategies
A recent assessment of 90 partner countries indicates that 70% of countries are orienting their national strategies towards the 2030 Agenda for Sustainable Development, with close to half of all national results frameworks already using SDG indicators (OECD/UNDP, 2019[2]). Trends indicate that, as the remaining countries move to the next planning cycle,4 most of them will have domesticated SDG indicators in the next three to four years.
The three country cases also reflect the varying degrees of SDG domestication by partner countries:
-
The Government of Kenya included SDG indicators in its 2018-22 medium-term plan and specific sector frameworks, after an extensive mapping and consultative exercise. The government is already aligned to and measuring the SDG indicators (6.2.1 and 7.1.1) reviewed in the case studies that include Kenya.
-
The Government of Ethiopia is currently updating the national strategy to address the SDGs. Still, sector plans match the two SDG indicators (4.1.1 and 7.1.1) reviewed in the country; national monitoring for both indicators exists, although with some issues of coverage and quality.
-
The Government of Myanmar is finalising a national sustainable development plan that will be fully aligned to SDG 6.2.1 and partially aligned to SDG 4.1.1. Current indicators and measurement systems do not allow for alignment and use of these two SDGs under review in this country.
Source: Adapted from GPEDC (2019[3]). https://doi.org/10.1787/26f2638f-en
Increasingly, providers are also aligning their results frameworks to the SDGs
At corporate level, a number of providers are progressively aligning their indicator sets with the SDGs. For example, the European Union (EU) updated its EU results framework indicators in 2018 to reflect the SDGs at the three levels of results (OECD, 2018[4]). Similarly, the corporate results framework of the Asian Development Bank for 2017-20 is now aligned to the SDGs, and links between projects and programmes. The SDGs have been tracked since 2016 (OECD/UNDP, 2019[2]).
Yet, when considering the sample of the 3 SDG indicators under review, many indicators in the corporate results frameworks of the 17 providers assessed in this study do not match the SDG equivalent (Figure 3).
Note: Providers are using 34 outcome/output indicators for education, 26 for sanitation, and 14 for electricity.
Source: Authors’ estimates. Data from 17 major providers with standard corporate indicators, listed in Figure 1
At country level, providers are increasingly embracing elements of the 2030 Agenda and the SDG indicator framework. Data from the 2018/19 monitoring round of the GPEDC indicates that the use of the SDGs to guide the design of provider country strategies is already widespread. About three-quarters of country strategies include SDG goals to define priority areas or sectors. Furthermore, close to 60% of country strategy documents approved in 2018 apply SDG indicators in their results frameworks. (see Figure 4).
Source: (OECD/UNDP, 2019[2]) Making Development Co-operation More Effective: 2019 Progress Report https://doi.org/10.1787/26f2638f-en. Assessment of 1 556 country strategies that development co-operation providers approved since 2014 in 90 partner countries, by year of approval.
The progressive improvement of the global SDG framework, coupled with the growing SDG alignment of the results frameworks of providers and partner countries, underlines the need to identify opportunities and to address obstacles that may affect the use of the SDGs as an effective framework for shared results.
The three case studies provide a timely contribution to inform this discussion.
The remainder of this chapter discusses four cross-cutting issues emerging from a comparative review of the three case studies. Specifically:
-
1. Selecting SDG indicators, taking into consideration indicator readiness and intrinsic complexity, measurement approaches, and contextual fit.
-
2. Managing different planning cycles in aligning to the SDGs.
-
3. Measuring SDG indicators using sustainable and effective approaches.
-
4. Mainstreaming leaving no one behind in SDG alignment and measurement.
copy the linklink copied!Key issues
Selecting SDG indicators at global and country level
While providers are increasingly applying the SDGs in their corporate and country-level results frameworks, the relative cost and quality of alignment to SDG indicators varies considerably across indicators. The case studies point to a number of indicator-specific characteristics that affect the cost of aligning with each of the three assessed SDG indicators. These include: 1) whether the indicator methodology has gained international agreement and data are being regularly produced in most countries; 2) whether the indicator design is relatively simple and suitable for results communication; 3) whether the indicator departs from existing, well-established sector measurements and/or its Millennium Development Goal (MDG) equivalent; and 4) contextual factors such as partner country alignment to the indicator, and the type of arrangements for monitoring and service provision. Table 1 summarises the key findings for the three SDG indicators under review.
The three case studies show that SDG indicators that are relatively similar to prior, well-established indicators for sector results tend to be more widely adopted. For example, Indicator 7.1.1 (i.e. “Percentage of population with access to electricity”) has been measured since 1990 and, while it was not included in the MDG framework, it is a simple, well-established measure for sector performance. Both Ethiopia and Kenya apply this indicator in their sector results frameworks, with an exact match with the SDG indicator definition. Providers apply this indicator in their corporate results frameworks in line with the SDG definition at a higher rate than for the other two SDG indicators (see subsequent chapters). However, possibly because of the need to respond to their domestic agendas, providers tend to use specific definitions (e.g. only tracking electricity access from sustainable sources) and donor-focused measurement approaches that do not produce usable data for broader SDG follow-up at country level, or globally.
In comparison, SDGs 6.2.1 and 4.1.1 are multi-layered indicators, with two and three sub-indicators respectively, and multiple internal dimensions. SDG 6.2.1 combines a sub-indicator (i.e. “Proportion of population using safely managed sanitation services”) that is a refinement of MDG 7.9 and well-grounded in sector performance measurement practices, with a sub-indicator (i.e. “Proportion of population with basic handwashing facilities on premises”) that is being rolled out in 70 countries since 2009 in household surveys, but which is less frequently applied in country-level results frameworks. Nevertheless, unified monitoring practices and a good level of partner country uptake are leading to greater indicator use over time.
SDG 4.1.1 focuses on learning outcomes during primary and lower secondary education, a concern that has been increasingly present in sector strategies but represents a departure from the traditional focus on education access under MDG 2. This multi-layered indicator relies on three different cross-country/national assessments, measuring two areas of learning (i.e. reading and maths) over time: “Proportion of children and young people: (a) in grades 2/3; (b) at the end of primary; and (c) at the end of lower secondary achieving at least a minimum proficiency level in (i) reading and (ii) maths, by sex”. In comparison with the two other indicators, communicating results for SDG 4.1.1 is more difficult. The case studies show that, despite the increased relevance of learning outcomes versus school attendance in most countries, the degree of uptake of this SDG indicator is comparatively weaker across countries and providers, which continue to rely on MDG 2 indicators to track sector results.
Nevertheless, the case studies also note the increased attention to development concerns that the SDGs prioritise, e.g. learning outcomes in the education sector. This evolution in sector priorities, coupled with the rising number of countries adapting their planning tools to the 2030 Agenda, creates favourable policy and political conditions for the adoption of the related SDG indicators by all actors at country level. Ethiopia’s five-donor pooled fund supporting government efforts to finance and measure the overall SDG Target 4.1 is an example of good practice.
Managing different planning cycles at country level
Most providers and partner countries are currently involved in processes to incorporate the SDGs. Figure 3 and Figure 4 evidenced efforts of providers and partner countries to adapt their results frameworks to the SDGs. While many providers accommodate their planning cycles at country level to their partners’ cycles, these processes are not systematically synchronised. This disconnect makes alignment around results more difficult Figure 5.
Source: Authors’ elaboration based on data from the three country case studies and GPEDC (2019) data.
Across the three case studies, early involvement in the design and implementation of national and sector results – with the partner government and within donor co-ordination structures – helps to harmonise efforts. In Ethiopia, the government and providers engaged in developing sector-wide approaches to electricity access. The National Electrification Programme was informed by a World Bank multi-tier energy access survey, and ambitious electricity access targets (using on-grid and off-grid solutions) were agreed with each provider involved in the sector co-ordination mechanism.
Several elements prevent greater harmonisation around results planning. As the case studies illustrate, providers need to manage several pulls that affect the degree of alignment to partner country results frameworks or SDG indicators. These include:
-
outdated frameworks that lag behind partner countries’ SDG-aligned or new results frameworks (or opposite situations, applying SDG indicators in partner countries that have not initiated the domestication process)
-
a mismatch between selected results indicators prioritised by partner countries and/or at corporate level, and the SDG framework
-
lack of harmonisation with other providers’ results frameworks working in the same country/sector, often due to lack of transparency of these results frameworks, which results in incompatible needs for results data measurement
-
weak or inadequate country-level arrangements for mutual accountability around results, that reduces the need for harmonisation and for adopting joint measurement approaches.
Box 1 summarises some opportunities to enhance harmonisation around results indicators.
-
To promote internal coherence, where corporate standard indicators are in place, ensure these are included in all sector or country-level results frameworks.
-
Some provider country-level or sector indicators are not publicly available on line. Making results frameworks publicly available would increase opportunities for harmonisation.
-
Make all results data against indicators publicly available.
-
New indicators should not be introduced unless data can and will be collected against these indicators. Instead, provider results frameworks should include indicators linked to the Sustainable Development Goals (SDGs) and to partner countries’ national and sector results frameworks; and provide capacity-building support to partner countries to increase their ability to monitor the SDGs and collect data against national development plan indicators.
-
Standardise levels of disaggregation in alignment with indicators drawn from the SDG framework and those drawn from partner country national and sector results frameworks.
Measuring SDG indicators
Overall, most providers do not get the results data they need. The three case studies show that data against many of the indicators collected by development co-operation providers are missing. For instance, in Kenya no data are available to report progress against 23 providers’ indicators related to SDG 6.2.1. Similarly, as regards SDG 4.1.1 on education, results data are not available for most national development and sector plan indicators in Ethiopia, with the exception of enrolment rates and related access measures.
Providers are struggling to adopt country-led, joined-up approaches to measure the SDGs at country level. In all three case studies, there are sector working groups fostering donor co-ordination, although the regularity and effectiveness of these mechanisms is uneven. In practice, measurement of sector and SDG indicators tends to include some form of joint monitoring approach (or be articulated around a singular measurement initiative) which involves the partner government and a subset of providers; but these efforts often coexist with many parallel and project-specific monitoring arrangements for most providers that work in these sectors. Table 2 summarises the opportunities, constraints and challenges to jointly measure the SDGs as identified in the three case studies.
Providers’ partnering strategies and support modalities play a major role in mitigating fragmentation in monitoring approaches at sector level. Fragmentation is lower for SDG measurement when providers pool support for sector-wide programmatic approaches, when a single donor is dominant, or when partner country ownership of existing monitoring mechanisms and assessments is high and well-established. For example, the World Bank, Finland, Italy, Norway, the United Kingdom, USAID and the Global Partnership for Education jointly support a single, large education programme in Ethiopia which is fully aligned with the SDG indicator included in the national results framework. The associated joint monitoring approach is helping to increase efficiency and harmonisation, strengthen national capacities in proficiency assessments, and increase the likelihood of sustainability of SDG results monitoring after the programme sunsets.
More transparency around results would help providers take greater advantage of synergies and improve harmonisation. The review found that in all three countries and SDG areas, many provider results frameworks are not publicly available. Their results data are also seldom made public. During the field workshops, stakeholders were often supportive of introducing joint sector review mechanisms as a way to institutionalise data sharing around results, e.g. in the water sector in Kenya. Greater disclosure of project results frameworks and results data represents a low-hanging fruit that can seed the conditions for more joined-up approaches in most country contexts.5
Lastly, and with some exceptions, there is untapped potential to use new technologies in monitoring SDG implementation. In the three case studies, most SDG measurement approaches rely on traditional top-down instruments, such as household surveys and administrative data. In all three cases, limitations in terms of coverage and quality of national surveys and administrative data will require greater provider support for country-led systems and data-gathering methods, but technology-driven innovations in monitoring approaches (e.g. satellite imagery to measure geo-referenced luminosity across the whole territory) can also help overcome existing gaps in terms of data disaggregation, sample representativeness and, ultimately, the ability to mainstream “leave no one behind” concerns in SDG implementation.
Mainstreaming leaving no one behind in SDG alignment and measurement
The UN Statistical Commission requires that SDG data are disaggregated according to a variety of socio-demographic and geographical dimensions. Disaggregation of SDG data is particularly critical to address cross-cutting inequalities (e.g. gender, diversity), to reflect locally relevant disparities (e.g. territorial, ethnic or socio-economic), and when government service delivery capacity across the territory is uneven.
Providers have identified data disaggregation as the main operational challenge to mainstream the “leave not one behind” agenda (OECD, 2018[5]). Across the three case studies, many providers in country measure SDG indicators, or similar indicators, applying some level of disaggregation. Sex-disaggregated indicators are more prevalent in education, while urban/rural disaggregation is more prevalent in electricity and sanitation. Some providers use both levels of disaggregation. This largely is in line with the official SDG methodologies. Some providers target specific geographic areas with “leave no one behind” in mind. However, the review also found that other locally relevant criteria for disaggregation are rarely monitored by providers or partner countries, limiting the usefulness of data for policy making, course correction and adaptation, including with regard to the “leave no one behind” agenda.
As mentioned earlier, providers and countries are operationalising SDG measurement generally relying on top-down traditional methods for data collection. Official assessments and household surveys have limitations related to coverage, social norms (e.g. who is the household respondent) and interpretability in various local languages, among others, which may prevent a proper inclusion of social minorities or isolated regions that are poorly reflected in official statistics. Complementary techniques can ensure that SDG programming is designed and monitored efficiently and with leaving no one behind in mind (Box 2).
Mainstreaming leave no one behind in the measurement and implementation of the SDGs at country level faces some political challenge (OECD, 2018, p. 220[5]) It requires engaging with partner countries in sensitive dialogue in order to include groups and people left behind, supported by a prior understanding of the political economy underpinning exclusion within a partner country; and approaching the necessary dialogue around data collection and results targeting with political sensitivity.
The three case studies show that national assessments and household surveys regularly provide good disaggregated data for SDG monitoring, particularly sex-disaggregated data and along urban/rural divides. However, issues related to coverage, sample size and implementation quality prevent greater disaggregation of national data, which creates “blind spots” for SDG implementation on particular issues for some areas or for some social groups. Some measurement approaches can help compensate for these limitations.
In Ethiopia, the provider-supported National Electrification Programme and its extension to rural areas was informed by remote sensing technology that allowed better planning around the type of on-grid and off-grid solutions (e.g. solar panels) that needed to be prioritised in order to reach to all the population across the country’s territory (World Bank, 2017[6]). Using this approach allowed planning and tailoring the intervention around household locations and regional needs.
Similarly, Kenya has experienced remarkable progress towards universal electricity access through on grid and off-grid solutions during recent years, reaching 75% by 2018. Similar remote sensing techniques helped identify quick wins on how to accelerate the expansion of electricity access in rural areas of western Kenya, where electricity coverage was around 5% in 2014 (Figure 6 and 7). In turn, access to electricity (SDG 7.1.1) in rural areas in Kenya supported improved learning outcomes (SDG 4.1.1) in those areas (Ye, 2017[7]).
Source: Lee, K. et al. (2016[8]), “Electrification for ‘under grid’ households in rural Kenya”, https://doi.org/10.1016/j.deveng.2015.12.001
Sources: (IEA et al., 2018[9]), Tracking SDG 7: The Energy Progress Report, https://trackingsdg7.esmap.org/data/files/download-documents/tracking_sdg7-the_energy_progress_report_full_report.pdf.
Next phase (2019-20)
In light of the potential offered by this case study approach combining quantitative and qualitative background research and fieldwork in partner countries, and recognising the limitations inherent to relying on a set of only three case studies, it is proposed to expand the work to strengthen the evidence base. Phase 2 of this project (from mid-2019 to 2020) has been defined in collaboration with the OECD-DAC Results Community. The main goal will be to provide convincing practical advice and tailored guidance to providers and partner countries on how to best use the SDGs in their country-level results frameworks.
Specifically, Phase 2 will:
-
map out and compare different results-based approaches used to incorporate the SDGs at country level, including by them it to various country contexts
-
assess the effectiveness of these approaches in favouring alignment to, measurement and use of the SDGs, with a view to produce guidance and identify good practice
-
explore how different results-based approaches co-existing at country level interact by reinforcing/deterring collective SDG alignment, measurement and use
-
identify effective uses of qualitative information to complement SDG quantitative approaches – at strategic planning, implementation, monitoring and reporting stages.
Limitations: A number of limitations inherent to the design and scope of the three case studies make it difficult to turn these messages into robust, evidence-based policy guidance at this stage. The case studies only address 3 of the 232 indicators and look at only 3 country contexts. In addition, the choice of indicators, while offering a variety of measurement challenges and covering different sectors involving various stakeholders, present limitations. In some instances, only a limited number of bilateral donors were active in the specific sectors (e.g. Myanmar’s sanitation sector). The case study looked at service delivery type of indicators, excluding cross-sectional issues or non-people centric indicators. It does not consider a variety of countries in terms of income levels, aid dependency or level of domestication of the SDGs. These potential limitations are being fully addressed in the 2019-20 phase of this research project.
References
[3] GPEDC (2019), Assessment of national strategies and results frameworks in 90 partner countries, Global Partnership for Effective Development Co-operation.
[11] IAEG-SDGs (2019), IAEG-SDGs: Tier Classification for Global SDG Indicators, https://unstats.un.org/sdgs/iaeg-sdgs/tier-classification/ (accessed on 2019).
[9] IEA et al. (2018), Tracking SDG 7: The Energy Progress Report, World Bank, Washington, D.C., https://trackingsdg7.esmap.org/data/files/download-documents/tracking_sdg7-the_energy_progress_report_full_report.pdf.
[8] Lee, K. et al. (2016), “Electrification for ’under grid’ households in rural Kenya”, Development Engineering, Vol. 1, pp. 26-35, https://doi.org/10.1016/j.deveng.2015.12.001.
[5] OECD (2018), Development Co-operation Report 2018: Joining Forces to Leave No One Behind, OECD Publishing, Paris, https://doi.org/10.1787/20747721.
[4] OECD (2018), OECD Development Co-operation Peer Reviews: European Union 2018, OECD Publishing, Paris, https://doi.org/10.1787/9789264309494-en.
[1] OECD (2018), “Using the SDGs as a shared framework for results: demonstrating good practice”, Discussion paper for the OECD/DAC Results Community Workshop, OECD, Paris, http://www.oecd.org/dac/results-development/docs/OECD-SDG-Case-studies-discussion-paper-FINAL.pdf.
[10] OECD (2018), “Using the SDGs as a shared framework for results: demonstrating good practice”, Discussion paper for the OECD/DAC Results Community Workshop, OECD, Paris, http://www.oecd.org/dac/results-development/docs/OECD-SDG-Case-studies-discussion-paper-FINAL.pdf.
[2] OECD/UNDP (2019), Making Development Co-operation More Effective: 2019 Progress Report, OECD Publishing, Paris, https://doi.org/10.1787/26f2638f-en.
[6] World Bank (2017), Kenya: Off-grid Solar Access Project for Underserved Counties, The World Bank, Washington, DC, http://projects.worldbank.org/P160009?lang=en (accessed on 6 December 2018).
[7] Ye, G. (2017), The Impact of Electricity Access in on Education in Kenya, University of Illinois at Urbana-Champaign.
Notes
← 1. The three case studies rely on a three-pronged methodology: 1) a systematic desk review of all available evidence, policies and planning documents of 17 major providers and 3 partner countries; 2) quantitative analysis of all results data and indicators used by providers and partner countries; and 3) in-country fieldwork in Ethiopia, Kenya and Myanmar. See OECD (2018[10]) for more details on the methodological approach and criteria used for case-study selection.
← 2. Source: https://unstats.un.org/sdgs/iaeg-sdgs/tier-classification. The UN classifies SDG indicators as Tier I and II when they have clear definitions, metadata and data-collection methods. Tier I indicators also meet an additional requirement related to widespread data availability collected on a regular basis.
← 3. See (IAEG-SDGs, 2019[11]). This comprehensive review will include the replacement, deletion, refinement or adjustment of indicators that have not succeeded in establishing a widely agreed upon methodology. For an updated list: https://unstats.un.org/sdgs/iaeg-sdgs/tier-classification/.
← 4. Twenty-six partner countries with national strategies that have no reference to the SDGs in any form. However, 22 of them (85%) mention an ongoing process in the country to align existing planning tools to the SDGs.
← 5. Preliminary GPEDC data for 2018 reveal that out of 3 454 major projects and programmes approved by providers in 2017, only 37% had publicly available project documents. Similarly, only 577 (34%) out of the 1 673 active provider country strategies had publicly available strategy documents (OECD/UNDP, 2019[2]).
Metadata, Legal and Rights
https://doi.org/10.1787/368cf8b4-en
© OECD 2019
The use of this work, whether digital or print, is governed by the Terms and Conditions to be found at http://www.oecd.org/termsandconditions.
