7. Evaluation of return and reintegration policies
There are different ways in which reintegration assistance is evaluated. The primary indicator is the outcome of the beneficiary. A framework developed by IOM covering different dimensions provides a detailed overview of some aspects. The many different objectives of reintegration assistance programmes require additional forms of evaluation, including cost-benefit analysis and measuring the impact on communities.
OECD Member States implementing AVRR programmes must demonstrate their effectiveness in achieving the intended objectives and the efficiency with which they can do so. Clearly articulating the desired results is the first step to achieving sustainable reintegration. The next steps are long-term, systematic and comprehensive data-collection and monitoring and evaluation schemes to help assess the relevance, effectiveness, and impact of voluntary return and reintegration assistance.
From the perspective of policy makers, evaluations must assess the extent to which planned results were achieved, as well as the extent to which the resources/inputs (funds, expertise, time) were justified, given the effects that have been achieved. More global evaluations of the sustainability of AVRR programmes are yet to be developed, but are crucial amidst concerns raised as to programmes’ effectiveness, cost-efficiency and potential to produce side-effects (e.g. repeat migration or return shopping).
Individual projects and programmes providing AVRR have their own evaluation goals, generally in line with the working definitions of sustainability – explicit or implied – within the project. More broadly, there are examples of evaluation grids designed to be used across projects and platforms established to monitor AVRR.
The two main frameworks for evaluating AVRR activities are beneficiary based (looking at inputs and outcomes at the individual level) and community based (examining the impact on institutions and systems outside of the individual participants). Reflecting the objective of most host-country initiatives, the principal evaluation approach is beneficiary based.
In programmes where individuals receive benefits, monitoring and evaluation of individual experience and outcomes is usually built into the programme. Many evaluations also follow up with beneficiaries with questionnaires. In addition, outside evaluations target beneficiaries with batteries of varying dimensions.
One example of beneficiary follow-up is the German Federal Ministry Office for Migration and Refugees Research Central collaboration with IOM to evaluate the programme StarthilfePlus (Schmitt, Bitterwolf and Baraulina, 2019[1]). The evaluation is designed as a longitudinal study, covering as many as 1 740 returnees, and is part of a broader long-term research initiative on return. The survey aims to shed light on reintegration processes and capture further mobility of returnees.
Within its Mediterranean Sustainable Integration (MEASURE) project, funded by DFID, IOM launched a study conducted by Samuel Hall which proposed a new scale of indicators to capture reintegration sustainability both on different levels (individual and community) and along dimensions (economic, social and psychosocial). It includes both qualitative and quantitative indicators, which feed into a reintegration scoring that seeks to allow for an aggregated and standardised understanding of returnees’ reintegration outcomes (Samuel Hall and IOM, 2017[2]). The IOM reintegration sustainability survey includes 15 indicators, which were developed and tested in the field, measured by 30 elements of reintegration at an individual level. The measurement elements are based on the returnees’ own perceptions of their environment or self-evaluation of their situation. The resulting AVRR “Reintegration Score” comprises three-dimensional scores (measuring reintegration in the economic, social and psychosocial dimensions) and one composite reintegration score that provides a numerical measure of overall reintegration across dimensions (Box 7.1).
The IOM-Samuel Hall “Sustainable Reintegration Score” and the Reintegration Sustainability Survey
IOM, in collaboration with Samuel Hall, developed a Reintegration Sustainability Survey and related scoring system, which seeks to provide an aggregated and standardised understanding of an AVRR beneficiary’s level of reintegration and to answer the question of the extent to which they have achieved a level of sustainable reintegration in communities to which they return.
The scoring system covers 15 indicators covering the economic, social and psychosocial dimensions. It is built based on individual-level indicators based on the returnees’ own perceptions of their environment or self-evaluation of their own situation, as well as community-level indicators. By example, the five indicators for the economic dimension are shown in Table 7.1.
In addition to providing a score of each of the three dimensions, a “Reintegration Sustainability Score” is calculated, ranging from 0 to 1, based on all 15 indicators. The score is calculated from each dimension, applying weights. It is possible to calculate a composite reintegration score, or scores for each of the three dimensions in order to identify areas where a returnee is particularly vulnerable (Majidi and Nozarian, 2019[3]).
The “Reintegration Sustainability Score” is divided into a global score, which potentially allows for a comparison of trends in beneficiaries’ reintegration across country contexts and over time and can be used by case managers.
The Reintegration Sustainability Survey and related scoring system has been institutionalised was operationalised in the field globally by IOM since early 2018. One example is the ORION (Operationalizing an Integrated Approach to Reintegration project) project, in three African countries used the survey, first as a baseline to select those with the lowest reintegration scores for a mentoring approach; then for case management of these beneficiaries (at three-month intervals); and finally for comparative analysis, to compare scores of beneficiaries who received reintegration assistance through different approaches and in different contexts. The scores are also implemented across EU-IOM Actions. The system is incorporated in IOM’s institutional information management system. It has also been published through the 2019 Reintegration Handbook as a tool for monitoring reintegration assistance.
Collecting the information requires an interview with the participant which lasts approximately 40 minutes. Some of the questions are personal and require the respondent, for example, to rate their social network in terms of the support it is providing, or to categorise the frequency of situations of conflict. The respondent is given the option of not answering.
Source: (Samuel Hall and IOM, 2017[2]; IOM, 2019[4]).
In 2016, the European Migration Network (EMN) released guidelines for the monitoring and evaluation of AVR(R) programmes that provide a list of questions and indicators to be included in post-return monitoring activities (European Migration Network, 2016[5]). The overall aim of these guidelines is to identify a common methodology for monitoring and evaluation that Member States can apply on a voluntary basis. Specifically, the guidelines propose a common set of core indicators for monitoring and evaluation, which – if applied consistently in all EU Member States – are meant to enable the analysis of EU-level aggregate data on AVR(R) programmes. Further, use of a common methodology is believed to better enable Member States to design and implement joint reintegration projects.
The EMN guidelines cover beneficiary-based evaluation. They respond to the objectives of increasing uptake of AVR(R), successful case management within programmes, and measuring individual outcomes for beneficiaries. The long list of different evaluation questions is matched to corresponding indicators, although collecting the information requires not only a long beneficiary survey following return but also in-depth information from programme operators. Overall, most beneficiaries of return and reintegration assistance – as most returns from Europe – are men. The need to better cover women has been raised (Samuel Hall, 2018[6]; Strand et al., 2016[7]) especially as results suggest that their requirements, outcome and experience are distinct from that of male returnees.
Among the important evaluation questions included in the EMN guidelines is “Are AVR(R) programmes targeting the most relevant beneficiaries (or are they targeting those who would have returned anyway without assistance)”. In this case, response is drawn from self-reported motivations for return. To better understand those “who would have returned anyway”, it is necessary to also look at other returns (spontaneous, unassisted but also forced) and non-returnees who are part of the target population. Comparative evaluation on these different groups is complex and expensive. One ongoing attempt to compare with a control group is by Sweden’s DELMI which has begun surveying assisted and non-voluntary unassisted returnees in the origin country; the output however is expected to inform the assistance programme rather than identify the role that assistance played.
When potential returnees are randomly assigned to different partners for counselling and support, variations in return rates – “conversions” – might reflect the role of information. Just as individual counsellors have varying success, so do different organisations. Many programmes, however, triage potential returnees for referral to specialised partners. Without an understanding of the relative difficulty of conversion for different categories, it is difficult to set a benchmark. A low rate may reflect factors outside of the programme and services, depending on the gap between conditions in the origin and host country and on the risk of removal.
Fedasil in Belgium developed the digital Transition to Reintegration Assistance Tool (RIAT) to support its own work (see Box 3.3). RIAT has been further developed in cooperation with the European Commission and the European Return and Reintegration Network (ERRIN). RIAT is an online tool to manage and follow-up cases of persons that return with reintegration support, which can be adapted to different national settings. It is supposed to facilitate the exchange of information on individual reintegration projects throughout application (before departure), reintegration plan (after arrival) and final reporting.
When timely and low-cost departure is the main objective of offering programmes, it is important to understand the reference cost and to calculate total programme costs. For example, Canada’s Border Services Agency ran a pilot reintegration assistance programme starting in 2012, aimed at failed asylum seekers and using IOM as the service provider. The objective was to accelerate departures and reduce costs. An evaluation after the first two years found that the programme did not reduce the time for departure (appeals continued as previously) and examined the relative cost per removal with benchmark low-risk and high risk removals; the staff cost in terms of paperwork per case was higher than for low risk removals but 29% the cost of high risk removals – in line with “Other countries with voluntary removal programs [where] per-return costs [which] cost about one third the cost of a high-risk removal.” (Canada Border Services Agency (CBSA), 2014[8]). However, since the programme involved many applicants who dropped out after paperwork was filed, the programme had additional staff costs. This example is relevant, since many assisted reintegration programmes see dropout throughout the procedure, sometimes right up to the moment of departure.
The above are examples of indicators used and the approaches to measure them. The feasibility of collecting information through these different indicators is not always addressed. Open questions include how to track returnees, who may be highly mobile and hard to reach, and how to structure incentives for them to provide information, especially when they are not receiving support. This is a challenge not limited to the European AVRR context. A United States GAO report on an IOM-implemented programme to support reintegration in the Northern Triangle noted that “determining the effectiveness of reintegration efforts is challenging because of the difficulties of tracking migrants once they return to their communities and of accounting for the various external factors that influence an individual’s decision to migrate again” (United States Government Accountability Office, 2018[9]).
Sharing data among partners is complex due to data protection and consent requirements, but it is further complicated when origin country institutions are participants in reintegration assistance programmes, providing services and collecting data. Authorities in origin countries partnering in implementation of programmes collect case information on their own nationals but may be unable or unwilling to report this information. In some origin countries, data protection is a question of sovereignty: there can be no obligation to report on nationals to foreign states, including donors. As involvement of origin country institutions is a long-term objective for some reintegration programmes, the limit on data sharing is a potential barrier to case management and to programme monitoring and evaluation, especially when donors are involved. Even in the case of an official agreement to share certain information, public officials co-ordinating cases and programmes may resent an obligation to report on their nationals to foreign authorities. Other questions revolve around ethical issues when collecting data at the individual level, as the collection and storage of data on a marginalised population is a sensitive matter.
Furthermore, most monitoring tools in use have a limited time horizon (usually 12 months), corresponding to the duration of reintegration support. This horizon may be too short or arbitrary, but there is little evidence on the appropriate time horizon for reintegration monitoring. Indeed, each individual case is different. While there is evidence that the reintegration process is not linear (i.e. returnees do not start from zero and steadily become more reintegrated), but rather contains key “up” and “down” moments over time, this aspect must be better incorporated into evaluation and measurement (Samuel Hall and IOM, 2017[2]).
Reintegration support involves provision of services and financial resources and requires reporting on expenditures and activities, including reporting on the beneficiary. The individual outcome of the beneficiary of reintegration support is the simplest and most obvious measure by which programmes are monitored. As a result, of the focus of monitoring and evaluation has been on beneficiary monitoring, i.e. the sustainability of reintegration for the individual returning migrant.
The EMN guidelines do not address programme implementation arrangements in the origin country, such as services provided by the origin country or by the implementing partner, nor do they address questions related to the community around the beneficiary.
IOM’s 2019 results-monitoring framework gives as an example the outcome of community-based reintegration measures and possible associated indicators (IOM, 2019[4]). It also mentions the outcome of measures aimed at capacity-building and awareness raising among different stakeholders.
Compared with individual outcomes, less attention has been given to the evaluation of sustainability of reintegration programmes from a technical, but also a financial and political point of view. Positive individual reintegration outcomes for beneficiaries are worthwhile goals, but for most of the host country institutions funding the programme, a positive individual reintegration is not the objective. Rather, the positive individual outcome is interesting to the extent that it supports the entire migration and asylum framework, bolstering integrity of the system and facilitating actors involved in compliance. The success of individual returnees is celebrated, but only to the extent that it contributes to other objectives. Among the different objectives of financing reintegration assistance are: increasing or accelerating return, at a lower financial, political and social cost than alternatives such as forced return or regularisation; reducing remigration; improving relations with origin countries; allowing countries to demonstrate a humanitarian concern in migration enforcement; meeting international commitments such as those contained in the Global Compact on Migration; and reinforcement of policy coherence between migration and development objectives. These objectives are not mutually exclusive.
Assessment of the effectiveness of programmes in accelerating and increasing returns, at lower costs, requires additional data relative to most current platforms. It is important to quantify the delay to departure, and whether this occurs before or after a final asylum decision. On the cost side, for persons in reception centres or receiving other services, the duration of stay is an important indicator of the cost of non-return; even accelerating return decisions by a few months can represent a significant costs savings. Accounting for changes in the time to departure can be as significant as an increase in total returns.
Similarly, triage of returnees according to the relative complexity of their cases and the difficulty of alternative solutions is important. Family units from countries which do not easily cooperate in readmission, for example, represent a greater success in terms of return than the same number of single people from countries with which readmission is straightforward.
In this light, the main interest in the sustainable reintegration of beneficiaries may lie for some programmes in the case it makes for other beneficiaries to take up the return offer. Being able to tell success stories is a key campaign point for some programmes. In other cases, information circulating within migrant and diaspora communities brings reports of success to the host country and raises interest.
The political cost of returns is also to be considered: forced returns of some categories of potential beneficiary may be negatively perceived. Evaluations should try to take into account the impact of reintegration programmes on public perception of the migration system as fair and humane. As these programmes are little known, and perceived primarily in their utility as a means to favour return.
Improved relations with origin countries can take many forms, some of which may go beyond the return domain into broader cooperation on migration management and even security. It is simpler to assess the level of cooperation in public domains than those in which cooperation is invisible, but at least the facility of collaboration on returns should be considered.
References
[8] Canada Border Services Agency (CBSA) (2014), Evaluation of the Assisted Voluntary Return and Reintegration Pilot Program - Final Report, Canada Border Services Agency (CBSA), Ottawa, https://www.cbsa-asfc.gc.ca/agency-agence/reports-rapports/ae-ve/2014/avrrpp-pparvr-eng.html (accessed on 9 August 2020).
[5] European Migration Network (2016), Guidelines for Monitoring and Evaluation of AVR(R) Programmes, https://ec.europa.eu/home-affairs/sites/homeaffairs/files/what-we-do/networks/european_migration_network/reports/docs/emn-studies/guidelines_for_monitoring_and_evaluation_final_jan2016.pdf (accessed on 23 December 2019).
[4] IOM (2019), Reintegration Handbook: Practical guidance on the design, implement ation and monitoring of reintegration assistance, International Organization for Migration, Geneva.
[3] Majidi, N. and N. Nozarian (2019), “Measuring sustainable”, Migration Policy Practice, Vol. 9/1, pp. 30-39, https://publications.iom.int/system/files/pdf/mpp_37.pdf.
[6] Samuel Hall (2018), Supporting post-return interventions in Afghanistan-GIZ’s Programme for Migration and Development (PME).
[2] Samuel Hall and IOM (2017), Setting standards for an integrated approach to reintegration.
[1] Schmitt, M., M. Bitterwolf and T. Baraulina (2019), Geforderte Rückkehr aus Deutschland: Motive und Reintegration. Eine Begleitstudie zum Bundesprogramm StarthilfePlus (Forced return from Germany: Motives and reintegration. Study accompanying the federal program StarthilfePlus), Bundesamt für Migration und Flüchtlinge (BAMF), https://www.bamf.de/SharedDocs/Anlagen/DE/Forschung/Forschungsberichte/fb34-evaluation-starthilfeplus.pdf (accessed on 7 August 2020).
[7] Strand, A. et al. (2016), “Programmes for assisted return to Afghanistan, Iraqi Kurdistan, Ethiopia and Kosovo: A comparative evaluation of effectiveness and outcomes”, CMI Report R 2016:2, Chr, Michelsen Institute.
[9] United States Government Accountability Office (2018), USAID Assists Migrants Returning to their Home Countries, but Effectiveness of Reintegration Efforts Remains to Be Determined.