6. Ireland’s results, evaluation and learning

Abstract

This chapter considers the extent to which Ireland assesses the results of its development co-operation; uses the findings of evaluations to feed into decision making, accountability and learning; and assists its partner countries to do the same.

It begins with a look at Ireland’s system for managing development results, i.e. whether the objectives of its development co-operation policies and programmes can be measured and assessed from output to impact. The chapter then reviews the alignment of Ireland’s evaluation system to the DAC evaluation principles, looking specifically at whether an evaluation policy is in place, whether roles and responsibilities are clear, and whether the process is impartial and independent. Finally, it explores whether there is systematic and transparent dissemination of results, evaluation findings and lessons and whether Ireland learns from both failure and success and communicates what it has achieved and learned.

    

In brief

Ireland is developing an Accountability Framework to track progress on implementing commitments undertaken in A Better World. Focusing mainly on the accountability function, the Framework will not capture corporate results information. However, Ireland plans to develop a more comprehensive result-based management approach in the future. Its commitment to results-based management remains strong, with the monitoring system having significantly improved since the last peer review and results information being used to adapt programmes’ design and implementation. Performance measurement frameworks at embassy level are also a helpful tool to track results, although missions vary in their approach and capacity for monitoring and evaluation.

The Evaluation and Audit Unit (EAU), responsible for the evaluation function, is independent and reports directly to the Secretary General of the Department of Foreign Affairs and Trade. A dedicated Evaluation Policy is forthcoming. Centralised evaluations focus mainly on corporate and thematic issues, covering a strategic selection of topics and responding to specific needs. A recent staffing increase allows the EAU to expand its engagement and start supporting business units and partners, particularly to strengthen their evaluative capacity. However, high staff turnover in the EAU remains a concern for evaluation planning.

Ireland is committed to learning. It uses results information for learning and to inform corrective actions. It also disseminates evaluation findings systematically. The EAU tracks implementation of the recommendations made in its evaluations. However, learning is often limited to the team that generated it. There is also a risk that knowledge is lost, especially due to the high staff turnover. If Ireland moves ahead with plans to increase its ODA programme, knowledge management will need to improve.

Management for development results

Ireland is developing a top-level Accountability Framework for A Better World but will not adopt a more comprehensive results-based management approach until afterwards

Ireland is developing an Accountability Framework that will reflect whole-of-government efforts to track progress on implementing A Better World. The new framework is intended to identify responsibilities and time frames for the implementation of commitments set out in the new international development policy (Chapter 2). Moreover, the framework will support accountability and reporting across the government – within the Department of Foreign Affairs and Trade (DFAT), between government departments, and to the Cabinet – and it will help the expanded Inter-departmental Committee (IDC) on development co-operation in its monitoring functions (Chapter 4).

The Accountability Framework will focus exclusively on monitoring. It will track progress on the implementation of 20-35 key commitments contained in A Better World including the launch of 15 new initiatives as well as new development strategies, resource commitments and changes in the way of doing things.1 Also the proposed indicators are input-driven rather than outcome and impact level indicators (e.g. the amount of funding provided to education or approval of a strategy). As a result, the Accountability Framework will not have management nor learning functions.

Ireland plans to develop a comprehensive, result-based management approach in the future. This will ideally establish a link with the 2030 Agenda at target and indicator level (whereas A Better World provides a link only at goal level) and spell out the chain of expected results from output to impact. Building the results chain will be a necessary step to fill in the missing middle in the Irish results system and ensure that results tracked at programme, country, thematic and sector levels inform strategic decision making and learning, increase transparency, and facilitate communications at different levels.

At embassy level, performance measurement frameworks are a key tool to track results. In the past, country strategies included extensive results frameworks that were published alongside the country paper; recent mission strategies are more succinct and do not always include a published results framework. The recent performance measurement framework for Tanzania (Embassy of Ireland in Tanzania, 2019[1]) provides a good basis for monitoring and accountability at outcome as well as output level, and it assesses the effectiveness of management systems to support delivery of planned results. This may be a model to replicate in future mission strategies.

Ireland regularly monitors results at programme level and uses these to adapt programme design and implementation

Ireland has a strong commitment to results-based management and has shown progress in this regard since the last peer review. It has further institutionalised results approaches in several business processes, including business planning, budgeting and grant management. Under One World, One Future, the Framework for Action allowed clarity on key results areas, influencing programming and policy guidance. In 2015, the Development Cooperation and Africa Division (DCAD) organised a dedicated workshop to update its approach to results-based management. The annual budget submitted to the IDC is more focused on results; it identifies outputs and outcomes, the Sustainable Development Goals addressed, and the targeted top policy priorities for every budget line. The government’s 2018 annual report on its official development assistance (Government of Ireland, 2019[2]) presents examples of results from country programmes, humanitarian assistance, civil society funding and Ireland’s multilateral engagement, including core funding.

The monitoring system has significantly improved, thanks in part to the introduction of a Standard Approach to Grant Management (SAGM). In particular, the SAGM has improved the documentation of grant-level performance, allowing formalised and evidence-based decisions. Grant managers are tasked to record results that will inform performance assessment and/or lessons learned. This allows needed data to be generated in a timely manner, easing the burden on implementing partners (Department of Foreign Affairs and Trade, Ireland, 2017[3]).2 In addition, partnership arrangements, such as the Programme Grant II with Irish civil society organisations, give significant importance to applicants’ approach to results and learning (Department of Foreign Affairs and Trade, Ireland, 2017[4]). Ireland also regularly communicates with multilaterals to collect results information of its core contributions.

Staff use results information at programme level to adapt design and implementation, but there is room to increase adaptability. Although missions vary in their approach and specialised capacity for monitoring,3 programmes undergo regular reviews (every six months, annually and at mid-term), with the mid-term reviews used to inform possible programme management changes. Ireland thus ensures a certain degree of flexibility in programme implementation. Nevertheless, it plans to move further in the direction of adaptive management by enhancing the flexibility already built into programmes and shifting the focus of grant approval away from “what DCAD will do” and to “how decisions will be managed” (Government of Ireland, 2019[5]).

Ireland occasionally supports partner countries in building their statistical capacity but is decreasing its use of partner countries’ data

Ireland channels funding through United Nations (UN) agencies to strengthen the capacities of national statistic bodies. In Tanzania, for example, it supports UN Women and the UN Children's Fund to strengthen capacity of the national statistic bodies, including the National Bureau of Statistics, on generation and use of data from social institutions to inform evidence-based policies on gender equality and women’s empowerment and on nutrition.

However, Ireland is decreasing its use of partner country governments’ data and statistics to monitor its own strategies and programmes. Ireland’s use of results indicators that are monitored using government sources and monitoring systems decreased from 50% in 2016 to 45% in 2018 (OECD/UNDP, 2019[6]).

Evaluation system

Ireland uses its independent evaluation function strategically

The Evaluation and Audit Unit is independent and reports directly to the Secretary General of DFAT. Its work plan and products are reviewed by an independent Audit Committee, which advises the Secretary General and whose reports are provided to the Comptroller and Auditor General. The EAU is a member of the Irish Government Economic and Evaluation Services network (IGEES) and as such, benefits from additional support from the economists of the network. Ireland considers the combination of evaluation and audit functions in the same unit as a strength. Through this, DFAT brings development affinity to the audit function, which otherwise might become too risk-averse. Nevertheless, there is a separation of roles between evaluation and auditing staff.

The EAU conducts and commissions centralised evaluations and promotes findings. A three-year Evaluation Strategic Plan guides the functioning of the EAU (Department of Foreign Affairs and Trade, Ireland, 2017[7]) and a dedicated Evaluation Policy is under development. As a unit serving the whole of DFAT, the EAU also supports other business units in the department. However, two thirds of its evaluations focus on development co-operation.

Selection of centralised evaluations ensures that they cover a strategic selection of topics and respond to specific information needs. The EAU selects evaluations among proposals received by all business units on the basis of a clear set of criteria, including among others utility, relevance, innovativeness, feasibility and timeliness. Evaluations are selected prioritising those areas of DFAT’s expenditure where evaluation coverage is weaker and then narrowing down the specific focus through consultations. To ensure responding to longer term needs, a three-year evaluation plan is then endorsed by the Senior Management Group and approved by the DFAT Secretary General. Evaluations have a dedicated budget line, although the business units involved may contribute to costs.

The EAU’s evaluative focus is on corporate and thematic evaluations. Every mission strategy is evaluated about one and a half years before expiration to inform internal financial controls and reflections on the new strategy (Chapter 2). This exercise has a dual purpose of accountability and learning. The EAU also carries out or co-ordinates evaluations on themes and a few selected programmes, although few of these were conducted in recent years due to limited capacity and more are due according to the current three-year plan.

Other business units and missions can undertake or commission decentralised evaluations based on their learning and accountability needs. According to the SAGM, grant evaluations are optional depending on the quality and content of progress reports. Ireland does not publish decentralised evaluations. The unit director or grant manager is responsible for disseminating the findings among relevant stakeholders.

A recent staff increase could enable the EAU to support missions and partners regarding the quality and use of evaluations and to scale up its engagement

Insufficient human resources impacted Ireland’s evaluation activities in the aftermath of the financial crisis. In 2014, the DFAT evaluation function had a staff equivalent to four and a half, dropping in 2015 to a minimum of one. The EAU again reached sufficient capacity in September 2019, with a staff of seven people dedicated to evaluation. Before this, the EAU was limited in its capacity to provide advisory services to missions and partners or to carry out joint evaluations. The EAU had also begun to only contract evaluations to external consultants, but started in 2017 to undertake evaluations using mixed teams of external consultants and EAU staff to ensure consistent levels of quality.

The staff increase allows the EAU to expand its engagement and start supporting business units, particularly to strengthen their evaluative capacity. Ireland actively participates in international networks, such as the DAC Evaluation Network, European Union networks and Nordic+, and is a member of the Multilateral Organisation Performance Assessment Network. Recently, it started to work with Finland and Switzerland to undertake a mutual review of their respective evaluation functions, the strategic fit of these functions in the member ministry, and their good practices and areas for improvement. This will help to improve Ireland’s evaluation system. The EAU has also started to rebuild and restructure itself. This process includes revising its internal quality assurance systems, developing new research methodologies, ensuring adequate training provision and developing new policy and guidelines.

However, staff turnover in the EAU remains a risk that affects evaluation planning. The competencies required of good evaluators are developed over time. As Ireland transfers staff from other roles in DFAT into the EAU, an assignment to the EAU comes with a steep learning curve and should thus be of sufficient duration both to allow officers time to acquire these skills and to enable DFAT to capture a return of its training investment.

Institutional learning

Ireland is committed to learning, but knowledge is not shared across teams, units and missions in a systematic way

Ireland is committed to learning and a culture of self-reflection is widespread. The formulation of every new mission strategy builds on extensive reflections, external analysis and an evaluation of the previous strategy (Chapter 2). The EAU is frequently involved in conducting reviews before management decisions are taken. Internal and external consultations also often inform decision making. Grants are regularly scrutinised and monitored and, as discussed, Ireland uses results information for learning purposes as well as to inform corrective actions.

The EAU systematically disseminates centralised evaluation findings and tracks implementation of recommendations. Findings are presented to key internal stakeholders, the partner being evaluated, the DCAD Senior Management Group (and/or Management Board), the Audit Committee, and the partner government in the case of mission strategy evaluations. DFAT also publishes reports of centralised evaluations on its website. All evaluations and reviews require a management response and the EAU now tracks the follow-up to these responses. Tracking was re-introduced in 2019 after it was suspended due to low staffing levels beforehand.

However, learning is often limited to the strategy or programme in question, as knowledge remains by and large within the team that created it. For example, even though grant managers identify and document lessons through the SAGM, learning is not captured by a broader knowledge management system in a way that could inform decisions in other parts of DFAT. Exchanges of information across teams often rely on informal staff collaboration and co-ordination.

High staff turnover and plans to grow the development programme raise the risk that knowledge is lost, making better knowledge management all the more important. Ireland is committed to improve institutional knowledge management and could start with scaling up existing good practices that build on the strong networking culture. These include structured mechanisms to facilitate exchanges among staff, such as regional thematic workshops, and formal communities of practice. Ireland could also make information on specific knowledge and experience available so that staff can identify whom to contact (i.e. transactive memory). Improved information technology infrastructure and e-based processes could dramatically facilitate knowledge exchange between missions and headquarters and better enable institutional learning. Ireland also recognises the need to scale up policy-relevant research – the review of the Research Strategy 2015-2019 will inform an expanded, multi-stream approach with clear operational guidance for research, evidence, knowledge and learning (Chapter 2).

Ireland could draw lessons on knowledge management from the experience of other DAC members. The Japan International Cooperation Agency, for example, has built up a comprehensive and public database on lessons learned from evaluations to inform project design. It also runs 22 communities of practice for knowledge exchange across the agency. The thematic sections of the Australian Department of Foreign Affairs and Trade synthesise lessons from Australia’s co-operation and research and adapt them for diverse users. They use digital collaboration platforms to link thematic expertise with knowledge generated in the field, facilitating networking, disseminating performance stories as well as key messages for media opportunities.

References

[4] Department of Foreign Affairs and Trade, Ireland (2017), Appraisers’ Guide for the Programme Grant II (2017-2022) and the Humanitarian Programme Plan (2017-2018) (internal document).

[7] Department of Foreign Affairs and Trade, Ireland (2017), Evaluation Strategic Plan 2017-2019 (internal document).

[3] Department of Foreign Affairs and Trade, Ireland (2017), Standard Approach to Grant Management in DCAD.

[1] Embassy of Ireland in Tanzania (2019), Mission Strategy Performance Measurement Framework, Government of Ireland, Dublin.

[5] Government of Ireland (2019), DAC Peer Review 2020: Memorandum of Ireland.

[2] Government of Ireland (2019), Official Development Assistance: Annual Report 2018, https://www.irishaid.ie/media/irishaid/latestnews/Irish-Aid-2018-Annual-Report.pdf.

[6] OECD/UNDP (2019), Making Development Co-operation More Effective: 2019 Progress Report, OECD Publishing, Paris, https://doi.org/10.1787/26f2638f-en.

Notes

← 1. One of the new initiatives is “creating a funding initiative for women’s economic empowerment with a focus on agriculture”; an example of a resource commitment in the policy document is to “scale up funding to education especially for girls in emergencies, committing to spending at least EUR 250 million over the next five years”.

← 2. According to the SAGM, implementing partners are not required to report against the corporate results framework but should provide regular progress updates on the basis of an agreed set of results.

← 3. The peer review team noted that additional monitoring and evaluation expertise at the embassy level, together with support from headquarters, would be helpful.