Annex A. Methodology

In 2016 the OECD partnered with Ground Truth Solutions (GTS), an organisation that specialises in getting feedback from affected populations in crisis contexts.

Six contexts were selected, presenting different types of crises. Haiti was recovering from Hurricane Matthew in 2013. Lebanon is a middle-income country that for more than seven years has been hosting the world’s biggest refugee population per capita. Iraq, Somalia and Afghanistan are all experiencing protracted crises of different kinds, but have both emergency and recovery needs for their displaced and resident populations. Uganda is a low-income country with its own development needs, and is hosting a large refugee population.

A first round of surveys was conducted in 2016 with affected people in these six countries. The survey used a questionnaire designed in consultation with the Grand Bargain facilitation group and developed to broadly follow the Grand Bargain structure. Two years after the World Humanitarian Summit, the OECD and Ground Truth Solutions conducted another round of surveys in the same countries, as well as in Bangladesh – a lower middle-income country that has been hosting a refugee population since 2017, and where the international humanitarian response still in the initial phase. In total over the two rounds of surveys, 12.137 affected by crises and humanitarian workers were interviewed in the seven countries.

The surveys were conducted by GTS, which was responsible for overseeing data collection and ensuring that ethical and methodological standards are met. The design of the questionnaire in the second round of the survey was adapted after analysis of results from round 1, and to better reflect realities on the ground, in light of significant changes in the locations of affected people in the different countries.

Sampling methodology

The surveys look at the perspectives of 8,666 affected people and 3,471 humanitarian field staff. These two types of surveys were designed and conducted separately. The sampling strategies for the affected people survey were designed using the most recent figures on refugees, returnees, IDPs and residents affected by crises, that were retrieved from the websites or provided directly from the departments of UN agencies (UNHCR, OCHA and IOM). A balanced gender split amongst respondents was sought across all regions in each country.

The risk of oversampled affected groups skewing the results was evaluated by calculating weighted means based on the proportion of each region in the target population. These weighted means did not differ from the raw means by more than one decimal point, suggesting that any bias introduced by the oversampling was negligible across all questions in all seven countries (with the exception of two questions in Iraq, mentioned in the country report). As such, this methodology allowed for maximum reliability of between-group comparisons, region-specific means, as well as among the affected population at large. Due to the lack of reliable, up-to-date population demographics for Pakistani refugees in Afghanistan, we did not mean weight the results for Pakistani refugees.

When designing the sampling strategies for the humanitarian field staff survey, the selected organisations were approached and asked to participate in the surveys. The surveys are distributed by the participating organisations online among a sample of their staff.

Question formulation

Questions were formulated using the Grand Bargain commitments as a framework. The focus is on the extent to which humanitarian aid is becoming more responsive to the people it sets out to serve. People’s views were probed on whether they see progress beyond meeting their basic needs, towards creating self-reliance and opportunity.

Data disaggregation

In the affected people survey, the data were disaggregated by geographical region, type of accommodation, gender, age, status of person interviewed, gender of head of household, household size, number of dependents under the age of 18 years and disability. In specific contexts, country of origin, date of arrival and year of registration were taken into account. To identify groups of people with disabilities within the sample, a staff member of the NGO Handicap International was consulted and participants were asked a series of questions.

In the humanitarian field staff survey, the data were disaggregated by type of organisation, gender, age, time working in the local context and target beneficiary type. Nevertheless, the survey in Bangladesh did not include the data by type of organisation, role of staff or time working in Bangladesh as the sample size was too small to draw conclusions.

The analysis includes any major difference in the perceptions of different demographic groups. It does not, however, show the full breakdown of responses according to these categories.

Language of the survey

Across seven countries, the surveys were conducted in local language(s) for affected people and in local language(s) and English for humanitarian field staff. In Bangladesh, the enumerators received Rohingya language training from the NGO Translator without Borders.

Data collection

GTS staff, independent data collection companies, consultants and UN partners of GTS conducted the affected people surveys. Respondents were the beneficiaries of aid programmes from a wide variety of aid agencies, and were approached face-to-face, except for Somalia where interviews were conducted via phone. They were selected for the interview based on two sampling filters: the respondent had to be willing to participate in the survey in addition to having received aid in the past eighteen months.

Responses from humanitarian field staff were collected in 2018 from humanitarian staff members working for UN agencies, international NGOs and local organisations. The surveys were distributed online by each participating organisation.

Challenges and limitations

Affected people survey

Expectations of respondents

While enumerators were briefed and trained on managing expectations and clearly communicating the aims of the research, they reported instances of affected people expecting humanitarian assistance or mistaking them for representatives of aid agencies or the government.

Perceptual data

The perceptual data alone might be insufficient to evaluate the state of the humanitarian system and should therefore not be seen in isolation, but as complementary to other research, monitoring and data evaluation approaches.

Humanitarian field staff survey

Low response rate

In some countries, responses from participants were initially low. Feedback from international organisations suggests that staff members are experiencing survey fatigue as the result of the increasing number of surveys they are required to complete.

Self-selection bias

Self-section bias is applicable to any kind of social science research where participation is voluntary. Hence, the realised sample for this project is limited to humanitarian staff working in these seven countries who received the survey link and who consented to partake in the surveys.

Scoring in 2018 compared to 2017

Scores in 2018 are higher for participation and feedback than in 2017. This could be due in part to the fact that some of the questions were formulated differently this year.