6. Germany’s results, evaluation and learning
This chapter considers the extent to which Germany assesses the results of its development co-operation; uses the findings of evaluations to feed into decision making, accountability and learning; and assists its partner countries to do the same. The chapter begins with a look at Germany’s system for managing development results and specifically whether the objectives of its development co-operation policies and programmes can be measured and assessed from output to impact. It then reviews the alignment of Germany’s evaluation system to the DAC evaluation principles, looking specifically at whether an evaluation policy is in place, whether roles and responsibilities are clear, and whether the process is impartial and independent. Finally, it explores whether there is systematic and transparent dissemination of results, evaluation findings and lessons and whether Germany learns from both failure and success and communicates what it has achieved and learned.
Germany recognises the centrality of achieving the 2030 Agenda and the Sustainable Development Goals (SDG) and aligns its development co-operation with partner country priorities. However, Germany’s overall objectives are not articulated in ways that can be measured and assessed. While German implementing organisations have long experience with results-based management at project level, linking project outcomes to portfolio impacts remains a work in progress following the recent joint procedural reform, which introduced welcome intervention logic to bilateral programmes.
Further work is required to improve results-based management and embed a results culture within German development co-operation. The Federal Ministry for Economic Cooperation and Development (BMZ) does not systematically articulate the results it aims for in special initiatives and in country, regional and global programmes. Enhancement of its integrated data management system, and a broader set of quantitative and qualitative indicators, will help Germany better implement results-based management. Results information is used for accountability and communication, but not yet sufficiently for strategic direction and learning.
Germany has strong and respected evaluation capability. It strengthened its contribution to evaluation within the international community, and the approach to evaluation across Germany’s development co-operation system continues to improve. Nevertheless, given the focus of the German Institute for Development Evaluation (DEval) on strategic evaluations and the focus of Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH and KfW Development Bank (KfW) on projects and portfolios, consideration could be given to how to best allocate resources for evaluation. Additional considerations could enhance the quality of evaluations.
Independent evaluation functions exist, supported by plans and budgets. Decisions about topics for evaluations are made independently of policy making and delivery functions within the German development co-operation system. GIZ and the KfW development bank could improve their approach to evaluation in partner countries. DEval and GIZ continue to invest in strengthening evaluation capacity.
Networks exist for knowledge sharing and learning in sectoral and thematic areas across Germany’s development co-operation system. However, knowledge management is challenging in the complex German system. Results information, evaluation findings and lessons are not yet disseminated systematically. BMZ and Deval are piloting a process to ensure stricter standards for implementing and mainstreaming implementation of recommendations from evaluations.
Germany contributes to the 2030 Agenda and aligns with partner countries’ priorities
German development policy and the BMZ 2030 reform strategy recognise the importance of achieving the 2030 Agenda and the SDGs (BMZ, 2020[1]). The reform strategy clearly points out the contribution that the five core areas and ten initiative areas are expected to make to achieve the SDGs. In addition, Germany’s refocusing of development co-operation with Africa aims to contribute to the African Union’s Agenda 2063.
Partner country priorities inform German development co-operation. Country programmes outline the contribution German interventions make to priorities articulated in partners’ national development strategies and plans.
Targets in strategies will focus as closely as possible to the SDGs. Recently approved guidelines on strategies for core and initiative themes, and country strategies requires alignment with the SDGs. As developing countries increasingly apply SDG indicators in their results frameworks, the SDGs will increasingly form a common framework for results (OECD, 2019[2]).
Results management is being extended from projects to portfolios
German implementing organisations have long experience with results-based management at project level. Results matrices — generally logical frameworks — are used by KfW and GIZ in developing projects and programmes, monitoring their progress, and steering and reporting on them (OECD, 2015[3]). This is particularly useful for projects and in field settings. The Development Effectiveness Rating (DERa) system,1 introduced in 2017 by KfW subsidiary, Deutsche Investitions- und Entwicklungsgesellschaft mbH (DEG), is used throughout the project cycle to measure contributions to development based on five outcome areas (DEG, n.d.[4]). It is being taken up by other development finance institutions such as the Development Bank of Austria (OECD, 2020[5]).
The joint procedural reform introduced intervention logic to bilateral programmes and links project outcomes to portfolio impacts. All programmes and projects are now required to use a results matrix and intervention logic, enabling implementing agencies to better design projects as part of bilateral portfolios and facilitating adaptation to challenges during implementation. This will contribute to BMZ’s ability to manage country and sector portfolios. While special initiative interventions that form part of a country portfolio apply this approach, it is not yet applied comprehensively to all special initiative activities (Federal Government, 2020[6]).
Further work is required to improve results-based management and embed a results culture within German development co-operation
Germany’s overall objectives are not always articulated in ways that can be measured and assessed. In its latest development strategy paper, Development Policy 2030:/ New Challenges, New Solutions, BMZ describes the significant global challenges facing the world and undertakes to respond to these (BMZ, 2018[7]). However, the strategy is largely input-based, describing actions Germany and its partners will take rather than articulating the results to which these efforts will contribute.
BMZ does not systematically articulate the results it aims for in special initiatives and country, regional and global programmes. The ministry’s description of the Special Initiative on Training and Job Creation2 outlines the results Germany aims to achieve, as it does in the Strategy for Interlinkages between Water, the Environment and Climate Change (BMZ, 2018[8]). However, BMZ is yet to do this systematically for the majority of its strategies and special initiatives.3 The BMZ 2030 reform strategy envisages a more systematic approach. As it develops thematic, sectoral, country, regional and global strategies and updates or creates special initiatives, BMZ should clearly articulate the results Germany seeks to achieve and how it will contribute to the SDGs in line with the DAC Guiding Principles on managing for sustainable development results (OECD DAC, 2019[9]). Staff will require clear guidance on how to do this.
Enhancement of its integrated data management system, and a broader set of quantitative and qualitative indicators, will help Germany better implement results-based management. Current work within BMZ is focused on complementing binding quality standards with standard indicators, and the ministry plans to enhance its data management system (Federal Government, 2020[6]). While development of quantitative indicators is progressing, introducing qualitative indicators that track progress — including with quality criteria such as human rights, gender, environment and climate — is proving more difficult. Given that its staff are mostly generalists, BMZ might consider recruiting staff with specialist skills in results-based management to help roll out a culture of results, investing more in training staff to manage for results and putting the necessary incentives in place to achieve this.
Results information is used by BMZ for accountability and communication, but not for strategic direction and learning. Aggregated results reporting enables Germany to record the results of engagement by GIZ and KfW in ten areas.4 Since 2017, this information has been used to improve communication and accountability, though using it for steering and learning would greatly benefit German development co-operation.
Germany’s evaluation capability is strong and respected
Germany has strengthened its contribution to evaluation within the international community. BMZ and DEval are active participants in the DAC Network on Development Evaluation — DEval’s Director is one of two vice-chairs — and played an active role in the revision of the DAC evaluation criteria. In addition, DEval is represented in the core management group of the COVID-19 Global Evaluation Coalition and is active in European and international research and evaluation networks.
The approach to evaluation across Germany’s development co-operation system continues to improve. BMZ is currently developing guidelines for evaluating development co-operation. GIZ and KfW continue to build upon their strong in-house evaluation capability. DEval’s focus is on development policies, objectives and instruments. Its institutional capacity has grown since the 2015 peer review, and its standing within the German system has improved. However, it continues to face a number of challenges, including staff retention. DEval’s long-term ambition is to establish itself as a knowledge institute for German development co-operation.
Significant resources are invested by GIZ and KfW in evaluating development projects and programmes. Each year GIZ’s evaluation unit steers evaluations of a sample of BMZ-funded activities. Corporate strategic evaluations, cross-section evaluations and evaluative studies are also undertaken along with evaluations of co-financing and those contracted by external or internal commissioning parties. KfW conducts ex-post evaluations of projects, including those funded by BMZ and BMU, cross-sectional evaluations and is increasingly undertaking thematic evaluations. Both implementing organisations publish reports summing up learning (GIZ, 2020[10]) (KfW Group, n.d.[11]).
There is room to consider how best to allocate resources for evaluation across the German development co-operation system. Mechanisms exist for co-ordination and collaboration amongst the entities engaged in evaluation. Nevertheless gaps remain. There is a need for systematic evaluation of special initiatives and country, regional and global programmes. There is also need to determine whether thematic and sectoral approaches, across all German development actors, are achieving more than the sum of the parts. Noting that portfolio management frequently focuses on individual priority areas that are often planned and implemented independently and without systematic consideration of interactions between them, DEval has proposed addressing this issue by undertaking country portfolio reviews. While a good start, systematic evaluation of country (and other) programmes is/ more likely to generate evidence for Germany to use in improving its development co-operation programmes (Hartmann and Vorwerk, 2019[12]).
Additional considerations could enhance the quality of evaluations. DEval policy briefs, in addition to sharing methods and standards used in its own evaluations (German Institute for Development Evaluation, 2018[13]), offer valuable suggestions for Germany and others to improve the quality of development evaluations as do the evaluation units of implementing organisations.5
The evaluation function is independent, but management responses to evaluations often lack specificity
Decisions about topics for evaluations are independent of policy-making and delivery functions within the German development co-operation system. In GIZ and KfW, the evaluation functions remain independent of programming departments, reporting directly to the management of both organisations. GIZ reformed its approach to evaluation in 2018 with a view to supporting evidence-based decisions, ensuring transparency and accountability, and contributing to organisation learning (GIZ Evaluation Unit, 2018[14]). KfW’s evaluation unit focuses on ex-post evaluations of the impact of projects and programmes; it conducts the evaluations itself, and also assigns either KfW employees who have never been involved in the project and/or programme or external evaluators.6
Concerns about DEval’s independence have proved unwarranted. While BMZ provides DEval’s steadily increasing budget and approves its annual programme of evaluations, DEval is ultimately responsible for the programme. Internal and external proposals for strategic evaluation topics are collated by DEval and ranked using specific criteria. A draft programme is sent to DEval’s Advisory Board and to BMZ and this is either adopted or declined in full; to date, the programme has never been declined.
More detailed management responses would enhance the value of evaluations for German development co-operation. BMZ’s responses to findings and recommendations of DEval evaluations are quite generic, welcoming findings and describing actions that may be taken by way of response. These published management responses would be more useful if they also included specific action to be taken in response to each recommendation, who will take the action and the deadline for doing so, as is the case for more detailed implementation plans.7
GIZ and KfW could improve their approach to evaluation in partner countries. Both organisations could do more to facilitate local participation in the evaluation process, including through the use of participatory processes in evaluation.
Evaluation capacity is being strengthened in partner countries
DEval and GIZ invest in strengthening evaluation capacity. BMZ and DEval participate in the Global Evaluation Initiative.8 DEval invests in evaluation capacity development in Latin America (Box 6.1) but might consider expanding this to other regions given Germany’s much stronger presence in Asia and Africa. GIZ’s evaluation teams include international and national evaluators. However, it could contribute better to building institutional evaluation capacity in partner countries, as it has done in Latin America by facilitating greater participation in its evaluations by counterparts.
Networks exist for knowledge sharing and learning in sectoral and thematic areas
Germany benefits from strong research capacity. The German Development Institute draws on domestic and international expertise in development research and is committed to working with emerging economies from the global South. It initiated a T20 Africa Standing Group as part of Think T20 during Germany’s G20 presidency in 2017.9
Germany has systems in place for managing sectoral and thematic knowledge and learning across its development co-operation system. Thematic issues are fed in to GIZ’s decades-old sector network communities of practice and disseminated to partners, civil society institutions and project managers. GIZ sector specialists play a key role in contributing learning during the design of new initiatives. KfW and GIZ have competence centres and exchange platforms that staff draw on and that enable knowledge sharing, including within regions, as seen in Tunisia (Annex C). Frequent reflection sessions are held with BMZ’s regional and sector units, including on evaluation findings. However, more could be done to broaden these, including by using digital means of communication. KfW recently rolled out an application enabling its database to be searched by country and sector. This allows staff to directly access all lessons learnt from past evaluations when designing new interventions, tailored to the specific case.
Knowledge management is challenging in the complex German system
Results information, evaluation findings and lessons are not yet disseminated systematically within the German development co-operation system. Evaluation reports are published online by GIZ, KfW and DEval, offering the opportunity for findings and lessons to be drawn upon within and across institutions. Improvements in results-based management, particularly at portfolio, thematic and country levels, should enable Germany to better capture, store and disseminate results information; improvements in information management, data and information technology systems will be key to achieving this. Nevertheless, extending this from individual institutions to the system as a whole remains a challenge. BMZ and DEval are piloting a process to monitor the implementation of evaluation recommendations, specifically how intended steps for transferring recommendations into specific actions have actually been implemented.
A 2015 study of ten countries in Latin America showed that the use of national monitoring and evaluation systems was a work in progress and uneven across countries. Institutionalising monitoring and evaluation had been a decades-long effort with notable advances and systematic application in some countries and a mix of politics and occasional impact evaluations in others (Peréz-Yarahuán, 2020[15]). When decisions are made, political agendas may be more influential than evidence arising from evaluations. Other factors at play included the difficulty of understanding evaluation results; the results were insufficiently tailored to decision-making; shortcomings in quality meant results were not credible enough or did not attract attention. Where there is willingness to use evaluations, countries lack structures and qualifications for implementation. Efforts to build capacity in evaluation were often fragmented and targeted at single measures aimed at individual stakeholders rather than targeting the evaluation system as a whole.
The German Institute for Development Evaluation worked with the Latin American and Caribbean Network of Monitoring, Evaluation and Systematization to develop evaluation standards for Latin America, taking the cultural and ethnic specificities of the region into account (Krapp and Klier, 2016[16]). Building on a first phase project implemented by GIZ, DEval implemented a systematic evaluation capacity development approach in Costa Rica as part of a BMZ-funded project, Fomento de Capacidades en Evaluación (FOCEVAL), and introduced some elements in Ecuador and Guatemala. The initiative was implemented in co-operation with Government institutions, universities, evaluation societies, civil society and parliaments (Krapp and Geuder-Jilg, 2018[17]).
DEval’s systematic approach to building evaluation capacity involves developing individual capacity, institutional capacity, and a conducive environment and facilitating collaboration between the three elements.1 DEval worked with the Costa Rican Ministry of Planning to design a four-year national evaluation agenda and provided advice and support to develop a national evaluation policy. Training is increasingly offered by local providers. Methodology, personnel and finance were offered for various types of evaluations: a participatory evaluation tested the involvement of civil society; an evaluation of the poverty reduction strategy analysed multiple dimensions of poverty; and an evaluation of municipal services provided a learning opportunity for junior staff. Responsible institutions and actors were involved early in the process.
Strengths of the systematic approach to evaluation capacity development include promotion of evidence-based decisions; increased ownership and collaboration by different actors; and sustainability of the system when governments change. But there are limitations. It is difficult to implement the approach in authoritarian systems and fragile contexts as personal safety and the existence of critical organisations may be jeopardised. In stable and democratic countries, there is a need to set up processes and structures, publicise evaluation results and feed them into political decision making. Actors need to be willing to develop evaluation systems together and to request and use evaluations. As responsible government bodies largely determine the choice of evaluation topics, the evaluation function can lack structural independence, and it can be difficult to attract finance for training. Over time, however, systematic approaches to evaluation capacity development offer the opportunity to develop evidence-based policy in Latin America and beyond.
1 Figure 2 in DEval Policy Brief 7/2018 depicts the interplay between the three different elements, https://www.deval.org/files/content/Dateien/Evaluierung/Policy_Briefs/DEval_Policy%20Brief_7.18_Foceval_EN_web.pdf.
Source: Krapp and Geuder-Jilg (2018[17]), Evaluation Capacity Development: A systematic project approach by DEval in Latin America, https://www.deval.org/files/content/Dateien/Evaluierung/Policy_Briefs/DEval_Policy%20Brief_7.18_Foceval_EN_web.pdf.
References
[19] BMZ (2021), “Initiative area One Health in development cooperation”, BMZ Strategies Paper, No. 1, Federal Ministry for Economic Cooperation and Development (BMZ), Bonn, https://www.bmz.de/en/publications/topics/health/Strategiepapier550_one_health_en.pdf (accessed on 8 February 2021).
[1] BMZ (2020), BMZ 2030 Reform Strategy. New Thinking, New Direction, Federal Ministry for Economic Cooperation and Development (BMZ), Bonn, http://www.bmz.de/en/publications/type_of_publication/information_flyer/information_brochures/Materilie520_reform_strategy.pdf.
[20] BMZ (2020), Helping Refugees Build a Future: The Special Initiative on Displacement, Federal Ministry for Economic Cooperation and Development (BMZ), Bonn, https://www.bmz.de/en/publications/type_of_publication/information_flyer/information_brochures/Materialie315_flucht.pdf.
[8] BMZ (2018), BMZ Strategy for Interlinkages Between Water, the Environment and Climate Change: Synergies and Conflicts Between Goals, Federal Ministry for Economic Cooperation and Development (BMZ), Bonn, http://www.bmz.de/en/zentrales_downloadarchiv/web-apps/wasser/Strategiepapier435_01_2018.pdf (accessed on 8 February 2021).
[7] BMZ (2018), Development Policy 2030: New Challenges, New Solutions, Federal Ministry for Economic Cooperation and Development (BMZ), Bonn, https://www.bmz.de/en/publications/type_of_publication/strategies/Strategiepapier452_10_2018.PDF (accessed on 3 February 2021).
[4] DEG (n.d.), Monitoring Development: Development Effectiveness Rating (DERa) - Brief Description, Deutsche Investitions- und Entwicklungsgesellschaft mbH (DEG), Cologne, http://www.deginvest.de/DEG-Documents-in-English/About-us/What-is-our-impact/Policy-brief_EN.pdf (accessed on 9 February 2021).
[6] Federal Government (2020), Memorandum for the DAC peer review of Germany 2020/2021 (unpublished).
[13] German Institute for Development Evaluation (2018), “DEval Methods and Standards 2018: Standards for DEval Evaluations”, DEval Policy Brief, No. 6/2018, https://www.deval.org/files/content/Dateien/Evaluierung/Policy_Briefs/DEval_Policy%20Brief_6.18_Public_Opinion_Agenda2030_EN.pdf (accessed on 9 February 2021).
[10] GIZ (2020), Evaluation Report 2020: Using Knowledge, Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ), Bonn, https://www.giz.de/en/downloads/giz2021_en_GIZ-Evaluation%20report%202020_USING%20KNOWLEDGE_.pdf (accessed on 30 March 2021).
[14] GIZ Evaluation Unit (2018), GIZ’s Evaluation System: General Description, Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ), Bonn, https://www.giz.de/en/downloads/GIZ_EVAL_EN_general%20description.pdf (accessed on 9 February 2021).
[18] Hartmann, C., F. Gaisbauer and K. Vorwerk (2017), Evaluation of the DeveloPPP.de Programme, German Institute for Development Evaluation, Bonn, http://www.deval.org/files/content/Dateien/Evaluierung/Berichte/DEval_develoPPP_Bericht_EN_web_final.pdf (accessed on 23 February 2021).
[12] Hartmann, C. and K. Vorwerk (2019), “Country Portfolio Reviews: A Tool for Strategic Portfolio Analysis in German Development Cooperation”, DEval Policy Brief, No. 4/2019, German Institute for Development Evaluation, Bonn, https://www.deval.org/files/content/Dateien/Evaluierung/Policy_Briefs/2019/DEval_Policy%20Brief_4.19_CPR.pdf (accessed on 9 February 2021).
[11] KfW Group (n.d.), 15th Evaulation Report, 2017-2018: For greater impact in small towns and cities, KfW Group, https://www.kfw-entwicklungsbank.de/PDF/Download-Center/Dokumente-Evaluierung/15_Englisch.pdf.
[17] Krapp, S. and E. Geuder-Jilg (2018), “Evaluation capacity development: A systematic project approach by DEval in Latin America”, DEval Policy Brief, No. 7/2018, German Institute for Development Evaluation, Bonn, https://www.deval.org/files/content/Dateien/Evaluierung/Policy_Briefs/DEval_Policy%20Brief_7.18_Foceval_EN_web.pdf (accessed on 14 February 2021).
[16] Krapp, S. and S. Klier (2016), “Evaluation standards for Latin America and the Caribbean”, DEval Policy Brief, No. 06/2016, German Institute for Development Evaluation, Bonn, https://www.deval.org/files/content/Dateien/Evaluierung/Policy_Briefs/DEval_Policybrief_6-2016_2-seiter_EN_WEB.pdf (accessed on 14 February 2021).
[5] OECD (2020), OECD Development Co-operation Peer Reviews: Austria 2020, OECD Publishing, Paris, https://doi.org/doi.org/10.1787/03b626d5-en.
[2] OECD (2019), Sustainable Results in Development: Using the SDGs for Shared Results and Impact, OECD Publishing, Paris, https://doi.org/dx.doi.org/10.1787/368cf8b4-en.
[3] OECD (2015), OECD Development Co-operation Peer Reviews: Germany 2015, OECD Development Co-operation Peer Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264246133-en.
[9] OECD DAC (2019), Managing for Sustainable Development Results: Guiding Principles, OECD Publishing, Paris, https://doi.org/10.1787/44a288bc-en.
[15] Peréz-Yarahuán, G. (2020), National Monitoring and Evaluation Systems: Experiences from Latin America, Center for Learning on Evaluation and Results, Latin America and the Caribbean (CLEAR) and Centro de Investigación y Docencia Económicas (CIDE), https://clear-lac.org/3d-flip-book/national-monitoring-and-evaluation-systems/.
Notes
← 1. DEG uses a theory of change that tracks client activity, client outputs, the desired development effects or societal outcome, and societal impacts. The DERa tool uses five outcome categories: decent jobs, local income, market and sector development, environmental stewardship, and community benefits. The first three categories assess what was achieved; the other two assess how these effects were achieved.
← 2. The Special Initiative on Training and Job Creation aims to create 100 000 jobs and 30 000 training places in Africa, improve local working conditions, and promote sustainable economic growth. For further information, see www.bmz.de/en/issues/sonderinitiative_ausbildung_beschaeftigung/index.html.
← 3. The One Health initiative contains four output indicators (e.g. number of instances where Germany has supported development of One Health strategies and programmes, number of people trained and number of people reached) and outlines action areas (BMZ, 2021[19]). The Special Initiative on Displacement reports on results (BMZ, 2020[20]) and the Strategy on Transitional Development Assistance requires results matrices for funded projects; however, neither articulates expected results.
← 4. The areas covered are drinking water supply; primary and secondary education and vocational training; basic health; energy; reducing the causes of flight and assisting refugees; annual savings in greenhouse gas emissions and adaptation to climate change; employment; rural development, agriculture and food security; social security; and political participation and public administration services.
← 5. DEval policy briefs examine a range of considerations, including how to evaluate sustainability and the use of causal mechanisms, geodata impact evaluation, and text mining. For more information, see www.deval.org/en/policy-briefs.html. The KfW evaluation unit is working with Agence Français de Développement on the use of geodata for evaluation in development co-operation. See www.mapme-initiative.org.
← 6. For details of KfW’s financial co-operation Evaluation Department, see www.kfw-entwicklungsbank.de/International-financing/KfW-Development-Bank/Evaluations/Principles/. Assigning staff from programming departments is a key knowledge management tool for KfW and ensures feedback of lessons learnt into those departments.
← 7. By way of example, the DEval evaluation of the develoPPP.de programme, authored by Hartmann, Gaisbauer and Vorwerk (2017[18]) contained 36 recommendations; it is available at www.deval.org/files/content/Dateien/Evaluierung/Berichte/DEval_develoPPP_Bericht_EN_web_final.pdf. BMZ’s response was three pages long: https://www.bmz.de/en/zentrales_downloadarchiv/erfolg/BMZ_response_to_the_DEval_evaluation_developpp.pdf. BMZ publishes its responses to DEval evaluations at https://www.bmz.de/en/ministry/evaluation/Evaluation/evaluierungsberichte-stellungnahmen/index.html.
← 8. The Global Evaluation Initiative aims to develop “country-owned, sustainable monitoring and evaluation frameworks and capacities to promote the use of evidence in public decision-making, enhance accountability, and achieve better results”. For additional information, see www.globalevaluationinitiative.org/about-gei.
← 9. For details about the T20 Africa Standing Group, see https://www.die-gdi.de/en/t20africastandinggroup/. For additional information about the German Development Institute, see www.die-gdi.de/en/.