Annex A. Methodological annex
The OECD fragility framework considers fragility to be multidimensional, measurable on a spectrum of intensity and expressed in different ways across five dimensions. It uses robust quantitative approaches to measure the magnitude of fragility, and it compares and contrasts different types of fragility descriptively. This mixed approach allows the analysis to extract the best value from the quantitative methods while also addressing the limitations of these methods through qualitative descriptions.
The methodology is based on a two-stage process that first examines contexts in each of five dimensions and then aggregates this information to obtain an overall picture of fragility. For each dimension, principal components analysis (PCA) is used to combine the risk and coping capacity indicators into two statistically derived components (Figure A A.1). Deriving two measures per dimension has distinct advantages over creating one composite index. First, using two measures allows for greater understanding of the differences among contexts that would score equally when a single measure is used. Second, using the first two principal components allows contexts to be broadly grouped based on their similarities in all of the input variables. Third, each indicator is weighted by the amount of new information it brings to the data, rather than on a set of normative judgements on their relative importance. With the components of each dimension calculated, contexts are then grouped on similarities and classified descriptively. Thus this mix of both quantitative and qualitative methods offers a more flexible approach to describing the diversity of fragility.
Once contexts are classified into groups within each dimension, the second part of the methodology aggregates this information to arrive at an overall picture of fragile contexts. To do so, the components of each dimension provide inputs to a second aggregate PCA that is then used to produce the 58 fragile contexts in the fragility framework.
The methodology is ambitious in its objectives but has limitations. By using PCA, the range of indicators can be reduced to two core components, thereby explaining most of the variance in the original data. However, in doing so information invariably is lost. The second stage of PCA (PCA Stage 2) exacerbates this loss of information. In short, the results of this approach are a summary of the initial indicators that is then interpreted in terms of fragility. Despite these limitations, this summary is both more informative and less arbitrary than any composite index based on the initial indicators.
Aside from the technical limitations, there are also certain practical limitations to what can be captured in any quantitative approach. The unit of analysis for the OECD fragility framework is country level. As a consequence, the framework is unable to capture macro-level drivers of fragility – drivers that spill over borders – or micro-level drivers that lead to localised pockets of fragility within states. Going forward, it would be useful to find ways to draw on subnational data and to link up regional and global data. Further, while data on governance are widely available, data on informal institutions are less so. While every effort has been made to include indicators of both of these, at this point the lack of quality data is a limiting factor for the model. Finally, the calculations exclude 27 countries and territories where there was insufficient data to feed into the analysis (Box A.1).
Data availability is a key issue in calculating the OECD fragility framework. As the unit of analysis is the state or territory, it is important to select indicators that are comparable across those states and territories. While statistical imputation methods can be used to fill data gaps, such an approach is best used sparingly; preference should always be given to real-world data, even if it means dropping indicators or countries and territories that otherwise would have been included. The fragility framework methodology aims to strike a balance between the number of indicators, the contexts covered and the amount of imputation that would be required to build a complete data set. A criterion for inclusion in the OECD framework was at least 70% of the required data had to be available for a country or context. As a result, only 172 contexts could be included in the calculations.
This does not mean that the excluded contexts are not fragile. Indeed, many of those excluded are small island developing states that face unique challenges. The final list also excludes two territories with UN peacekeeping missions (Kosovo and Western Sahara) and several Pacific Island countries whose high levels of interpersonal violence are well known.
Indicator coverage and missing data
The choice of indicators has been driven by selection criteria in line with the OECD’s fragility concept of high risk and low coping capacity. Normal technical criteria for selecting good indicators were used, but with a particular and added emphasis on selecting indicators based on their relationship to fragility – that is, are they a cause of fragility or an outcome of fragility? Indicators that represent outcomes of fragile contexts do not offer clear guidance as to policies that can reduce fragility. For example, infant mortality is an indicator used in several fragility measures. However, infant mortality is arguably more of an outcome of a fragile health context than a cause of it. In selecting indicators, then, the following factors were considered, in keeping with OECD concept of fragility:
-
Risk. Do the indicators alter either likelihood or impact ex-ante?
-
Coping capacity. What indicators would stop the risk cascading ex-post if the risk occurred
Using these criteria does not eliminate the challenge of separating some indicators into the category either of a risk or a coping capacity. For example, the level of armed personnel can be considered a coping capacity for dealing with insurgencies. It can also be considered a contributor to the risk of violence. Methodological decisions have been made to account for this challenge to produce the best approximation, given the limitations.
Further, some coping capacity indicators have been used in more than one dimension, introducing an unintended issue in aggregating the dimensions to provide the final 58 contexts used in analysing flows by fragility. Using indicators more than once essentially weights these indicators more than others. Statistical measures have been used to minimise this effect, and this method ultimately was chosen as the preferred alternative to dividing some of these indicators into one, and only one, dimension. For example, government effectiveness and rule of law are important not only for the security dimension but also for the environmental dimension. Forcing them into a single dimension arbitrarily creates an incomplete picture of the interconnectedness of coping capacities. Therefore, the de facto weighting effect was considered justifiable, if not ideal, given cross-dimensional importance. Table A A.2 lists the indicators used in the 2018 fragility framework.
Age of data
The 43 indicators selected do not cover all contexts and imputation techniques have been used to fill in data gaps. Lack of data is the primary reason why a context may not be included. At least 70% of data for a context had to be available for it to be included in the OECD fragility framework. In 2018, this yields a list of 172 contexts. It is possible to assume that contexts missing from the dataset have a certain value for some indicators. For example, those missing from the datasets for battle deaths and deaths by non-state actors can be assumed to have a value of 0. Where no reasonable assumption could be made, data are imputed using k-nearest neighbour (KNN) imputation that uses statistical inference to fill in missing data from the k most similar contexts. In the OECD fragility framework, this has been done using the 15 most similar contexts for each missing data point (Table A A.3).
Creating a time series
States of Fragility 2018 extends previous OECD work by creating a time series for deeper analysis of improvements and deteriorations in previously identified fragile contexts. To do this, all data have been imputed and extended to a period spanning 2016 and 2017. Using only 2016 or 2017 data, PCA models were generated for all five dimensions and aggregate scores. These models were applied across different years to create a time series comparable from year to year.
Cluster analysis
Fragile contexts can be grouped based on their similar characteristics. In order to identify frequently repeating patterns across these contexts and to group them based on their performances, a clustering algorithm was carried out using a hierarchical clustering procedure. Hierarchical clustering has emerged as a useful baselining tool in political science studies (Wolfson, Madjd-Sadjadi and James, 2004[1]). The clustering procedure provides two results. First, each context is grouped with other contexts that have the highest possible similarity with one another. Second, it defines the profile of an average context for each group. The profiles highlight the relevant attributes and distinct profile of each group, making it possible to quantitatively differentiate among groups. Six groups or clusters have been identified and named. The main attributes of each cluster are defined by their unique quantitative behaviour.
The OECD uses the clustering procedure as an indicative aid to assist in the qualitative assessment of different types of fragility. The severity of factors and/or combination of factors have been assessed by experts. Within each dimension, clusters have been ranked on a six level severity scale:
1 = Severe fragility
2 = High fragility
3 = Moderate fragility
4 = Low fragility
5 = Minor fragility
6 = Non-fragile
Once grouped, each cluster is compared to every other cluster to determine what characteristics best define its specific fragility. To do this, a Tukey ANOVA test is used (Hinton, 2014[2]). This method takes the means of all indicators within each cluster and conducts a difference in means test, which compares the mean of indicators in every other cluster. A statistical significance criterion has been developed to identify indicators whose levels are unique to any cluster when compared to the rest of the world. This criterion identifies indicators where the mean for any one cluster is significantly different at 95% confidence levels from at least four of the remaining clusters. Broadly speaking, this criterion can be interpreted as highlighting indicators in each cluster that are statistically different to at least 80% of the rest of the world.1
The next sections will show and describe the results of the cluster analyses for each of the five dimensions of the OECD’s 2018 fragility framework.
Economic dimension
The economic dimension aims to capture the vulnerability to risks stemming from the weaknesses in economic foundations and human capital including macroeconomic shocks, unequal growth, high youth unemployment, etc.
Figure A A.2 shows the economic dimension biplot with each cluster in a different colour. The results of the Tukey ANOVA test for significance suggest that the aqua-coloured cluster is distinguished by higher participation of men and women in the workforce. However, this cluster is weaker in regulatory quality, food security, education, socio-economic vulnerability and dependence on resource rent. The olive-coloured cluster shares the same weaknesses, with additional weaknesses in NEET and aid dependence.
Environmental dimension
The environmental dimension aims to capture the vulnerability to environmental climactic and health risks to citizens’ lives and livelihoods. This includes exposure to natural disasters, pollution and disease epidemics.
Figure A A.3 shows the biplot with each cluster in a different colour. The results of the Tukey ANOVA test for significance suggest that the aqua-coloured cluster is distinguished by low environmental health and high prevalence of disease and socio-economic vulnerability. The violet cluster is distinguished by high exposure to natural disaster. The grey cluster is strong in rule of law and government effectiveness and has low socio-economic vulnerability and lower levels of environmental health.
Security dimension
The security dimension aims to capture the vulnerability of citizen security emanating from social and political violence. As such it includes indicators of citizen exposure to direct political and social violence.
Figure A A.4 shows the security dimension biplot with each cluster in a different colour. The results of the Tukey ANOVA test for significance suggest that the aqua-coloured cluster is distinguished by higher deaths by non-state actors, higher battle-related deaths and low control of territory. However, this cluster has stronger coping capacities than other clusters. The olive cluster is distinguished by higher levels of interpersonal and political violence. It is weak in coping capacities relating to rule of law and government effectiveness. The grey cluster is stronger in government effectiveness, rule of law and formal alliances, which lead to lower risk of conflict and violent crime.
Political dimension
The political dimension aims to capture the vulnerability to risks inherent in political processes, events or decisions, to its political inclusiveness (incl. elites) and transparency (corruption) and to its ability to accommodate change and avoid oppression.
Figure A A.5 shows the biplot coloured by cluster. The results of the Tukey ANOVA test for significance suggest that the aqua-coloured cluster is distinguished by having higher political terror and perception of corruption. It is also weak in coping capacities relating to voice and accountability, gender physical integrity, and constraints on executive power. Conversely, the olive cluster is weak in all but one of the coping capacities but does not have the same presence of risk factors as can be seen in the aqua cluster. The brown and grey clusters have strong coping capacities coupled with low levels of risk factors.
Societal dimension
The societal dimension aims to capture the vulnerability to risks affecting societal cohesion that stem from both vertical and horizontal inequalities (inequality among culturally defined [or constructed] groups), social cleavages, etc.
Figure A A.6 shows the biplot coloured by cluster. The results of the Tukey ANOVA test for significance suggest that the aqua-coloured cluster is distinguished by having lower coping capacities in voice and accountability and access to justice. It is also faced with high risks through urbanisation growth, uprooted people and both gender and income inequality. The olive cluster is distinguished by low levels of all coping capacities and high horizontal inequality. The grey cluster is strong in coping capacities and faces lower levels of risk.
Aggregate fragility
The second part of the OECD methodology aggregates all of the information to arrive at an overall picture of combinations of fragilities. This second tier aggregate analysis generated the group of the 58 most fragile contexts, which are classified as extremely fragile and fragile.
The second tier PCA generated six fragility clusters that, are differentiated not only by their extent of fragility but also in the dominant characteristics of that fragility. This is shown in Figure A A.7. The first dimension of the PCA represents coping capacities. The second dimension represents the types of fragility. To arrive at the 58 most fragile contexts, two cut-offs have been selected. In order for a context to be classified as extremely fragile, it has to score less than -2.5 on the first principal component of the aggregate PCA shown in Figure A A.7. In order to be classified as fragile, a context must score between -1.2 and -2.5 on the first principal component.
The biplot of the aggregate can be split by contexts above and below the x-axis. Those above the x-axis are dominated by economic factors and the contexts below the x-axis are dominated by fragilities in the political, societal and/or security dimensions. Fragility in the environmental dimension can be found in contexts above and below the x-axis.
References
[2] Hinton, P. (2014), Statistics explained, Routledge, https://www.routledge.com/Statistics-Explained-3rd-Edition/Hinton/p/book/9781848723122 (accessed on 26 June 2018).
[1] Wolfson, M., Z. Madjd-Sadjadi and P. James (2004), “Identifying National Types: A Cluster Analysis of Politics, Economics, and Conflict”, Journal of Peace Research, Vol. 41/5, pp. 607-623, https://doi.org/10.1177/0022343304045975.
Note
← 1. For further information, also see (Lamb, 2017[145]), (Lamb, 2013[147]) and (Lamb and Mixon, 2014[146]).
This report includes 172 countries grouped into 6 clusters for each dimension. By conducting the Tukey ANOVA test at 95% confidence for all our indicators, we are comparing the indicators’ means of each cluster to the indicators’ means of the other clusters. If any mean is statistically different from the means of at least four of the remaining five clusters, it is considered a defining characteristic. Significance in this case can broadly be interpreted as indicators’ means for each cluster being statistically different to approximately four-fifths (80%) of the rest of the world.