Annex A. Methodology

Selecting case studies

OECD assessed several case study interventions targeting poor diets and/or physical inactivity (Table A A.1). Together, the case studies cover several OECD and non-OECD European countries.

Selected case studies represent strategic, high-priority interventions among policy makers in the OECD and EU27. A full description of the selection process is in Box A A.1.

Table A A.1. Overview of selected case study interventions
Intervention	Description	Targeted risk factors	Country*
Nutri-Score**	Front-of-pack labelling intervention to address unhealthy eating	Diet	Various European countries
Physical Activity on Prescription (PAP) **	Intervention to prescribe patients physical activity in a primary care setting	Physical activity (PA)	Sweden
Combined Lifestyle Intervention (CLI) **	Primary care intervention offered to those who are overweight or obese. Patients are guided on how to improve diet, exercise and overall health.	Diet and PA	Netherlands
Young People at a Healthy Weight (JOGG) **	Community-based intervention designed to improve diets and boost physical activity among those aged 0-19 years	Diet and PA	Netherlands
Multimodal Training Intervention (MTI) **	Physical activity and healthy eating programme targeting those aged 65 years and older.	Diet and PA (primarily PA)	Iceland, Spain and Lithuania
ToyBox	Kindergarten intervention to improve healthy eating and physical activity	Diet and PA	Various European countries
Personalised Approach to Obesity Management in Children (PAOMC)	Clinical, family-based and personalised childhood obesity programme targeting children aged 7 to 17 years	Diet and PA	Estonia
Diabetes in Europe – Prevention using Lifestyle, Physical Activity and Nutrition (DE-PLAN)	Type 2 diabetes prevention intervention aimed at improving diet and physical activity levels through a lifestyle, community-based intervention	Diet and PA	Various European countries
SI! intervention	Multidimensional school-based obesity prevention intervention, which targets lifestyle behaviour changes in 3-5 year-olds	Diet and PA	Spain
Let Food Be Your Medicine	Personalised nutrition mHealth app	Diet	Romania
Whole Grain Partnership	A front-of-pack labelling intervention to boost wholegrain consumption	Diet	Denmark
StopDia Pilot for the Somali population	Lifestyle intervention for the Somali population who are at-risk of developing type 2 diabetes (adapted from the nation-wide StopDia intervention)	Diet and PA	Finland
* The case study may operate across the country or in specific regions within that country. **Interventions evaluated using OECDs Strategic Public Health Planning for NCDs model (described in Box A A.2).

Box A A.1. Process for selecting case study interventions

Case studies were selected using the following hierarchical process:

• Case studies submitted by delegates to OECD’s Expert Group on the Economics of Public Health, which includes representatives from all 38 member countries

• Case studies involved in European Joint Actions that aim to improve health outcomes across Member States

• Case studies previously defined as “Best Practice” by member countries, such as those listed on the EU Best Practice Portal (European Commission, 2021[1]).

Assessing the performance and transferability of case studies

This section outlines two complementary frameworks used to assess case studies, both of which were developed by the OECD – the Best Practice Framework and the Transferability Framework. Limitations associated with the analysis are also discussed.

Best Practice Framework

The Best Practice Framework outlines five criteria to assess whether an intervention is “best practice” – namely Effectiveness, Efficiency, Equity, Evidence-base, and Extent of coverage (Table A A.2). A review of the academic and grey literature, existing best practice frameworks and feedback from delegates to OECD’s expert Group on the Economics of Public Health informed the selection of criteria.

Table A A.2. OECD’s Best Practice Framework – the 5 E’s
Criteria	Definition
1. Effectiveness	Extent to which intervention objectives were achieved
2. Efficiency	Extent to which inputs were used to achieve desired outcomes
3. Equity	Extent to which the intervention reduced inequalities in society
4. Evidence-base	The strength and validity of evidence used to develop or evaluate the intervention
5. Extent of coverage	Extent to which the intervention reached the target population

An intervention can be awarded a “stamp of approval” against one or multiple criteria if it performs particularly well relative to similar interventions.

Up and coming interventions (i.e. those that show promise but have not yet collected any of their own data) can be awarded a “promising best practice” stamp of approval for relevant criteria.

For a selection of case studies, effectiveness and efficiency were measured using OECD’s Strategic Public Planning for NCDs (SPHeP-NCD) microsimulation model. An overview of the model is provided in Box A A.2 with further technical information available at: http://oecdpublichealthexplorer.org/ncd-doc/.

Box A A.2. OECD’s Strategic Public Health Planning for NCDs microsimulation model

The OECD SPHeP-NCDs model is an advanced systems modelling tool for public health policy and strategic planning. It is used to predict the health and economic outcomes of the population of a country or a region up to 2050. The model consolidates previous OECD modelling work into a single platform to produce a comprehensive set of key behavioural and physiological risk factors, including obesity and physical activity, and their associated NCDs and other medical conditions. The model covers 52 countries, including OECD member countries, G20 countries, EU27 countries and OECD accession countries. For the purpose of this project, the model only covered OECD and non-OECD EU countries.

For each of the 52 countries, the model uses demographic and risk factor characteristics by age- and sex-specific population groups from international databases (see Figure A A.1). These inputs are used to generate synthetic populations, in which each individual is assigned demographic characteristics and a risk factor profile. Based on these characteristics, an individual has a certain risk of developing a disease each year. Individuals can develop 12 categories of disease, including seven directly related with alcohol (i.e. alcohol dependence, cirrhosis, injuries, cancer, depression, diabetes and CVDs). Therefore, the model takes into account the fact that individuals who do not develop an alcohol-related disease may develop other diseases that affect health care expenditure, workforce productivity and mortality. Incidence and prevalence of diseases in a specific country’s population were calibrated to match estimates from international datasets (IHME, 2017[2]; IARC, 2020[3]).

The links between risk factors and diseases are modelled through age- and sex-specific relative risks retrieved from the literature.

For each year, a cross-sectional representation of the population can be obtained, to calculate health status indicators such as life expectancy, disease prevalence and disability-adjusted life years using disability weights. Health care costs of disease treatment are estimated based on a per-case annual cost, which is extrapolated from national health-related expenditure data. The additional cost of multimorbidity is also calculated and applied. The extra cost of end-of-life care is also taken into account. In the model, people not dying from an alcohol-related disease or injury continue to consume medical care for other conditions (e.g. diabetes) and incur medical costs.

The labour market module uses relative risks to relate disease status to the risk of absenteeism, presenteeism (where sick individuals, even if physically present at work, are not fully productive), early retirement and employment. These changes in employment and productivity are estimated in number of full-time equivalent workers and costed based on a human capital approach, using national average wages.

There are two noteworthy limitations associated with using OECD’s SPHeP-NCD microsimulation model. First, microsimulation models, such as the one used in this study, are a simplified version of the population they aim to model given they are heavily constrained by data availability. Second, the model does not take into account the interconnecting relationship between different risk factors due to a lack of robust available evidence as well as the effect interventions have on risk factors other than those they directly aim to modify (e.g. an increase in physical activity may reduce pollution due to reduced use of private transport and thus the associated health issues). Due to the second limitation, it is likely the model underestimates the impact an intervention has on disease prevalence.

Figure A A.1. Schematic overview of the modules in the OECD SPHeP-NCDs model

Note: This schematic is highly simplified and focuses on the disease component – it does not reflect some other components of the model (including births, immigration, emigration, death, remission and fatality).

Source: SPHeP-NCDs Technical Documentation, http://oecdpublichealthexplorer.org/ncd-doc.

Transferability Framework

Public health interventions are complex given they involve multiple stakeholders, often target heterogeneous groups, and have outcomes affected by various direct and indirect factors. Therefore, positive outcomes achieved in one setting aren’t necessarily transferable to a different setting.

OECD has developed a Transferability Framework to assist policy makers assess whether a best practice intervention can be transferred from where it has been implemented (i.e. best practice “owner setting”) to a different country/region (i.e. the “target setting”). Specifically, whether the desired outcomes achieved in the owner setting are achievable in the target setting (Trompette et al., 2014[4]; Burchett, Umoquit and Dobrow, 2011[5]).

The Transferability Framework includes four contextual factors that affect transferability:

• Population context: covers population characteristics such as sociodemographic factors as well as broader cultural considerations

• Sector specific context: covers governance/regulation, financing, workforce, capital and access arrangements in the sector the intervention operates

• Political context: political will from key decision-makers to implement the intervention

• Economic context: the affordability of the intervention in the target setting.

In each case study, indicators to assess transferability are grouped under one of these four contextual factors. For the case studies presented in this document, countries are allocated into a group based on how far the indicator’s value is from the best practice owner setting. This method is referred to as the “distance from reference country” and is explained in Box A A.3. In addition, OECD developed a clustering methodology to group countries according to their potential to transfer a best practice intervention (Box A A.4).

Indicators were sourced from international databases to maximise coverage across OECD and non-OECD European countries (e.g. OECD Stat, Eurostat, World Bank Indicators, and the WHO). Relevant indicators were excluded if data was missing for the best practice owner setting and could not be identified through desktop research, or, if more than 50% of data was missing across countries.

By using international data, the scope of the analysis was inevitably limited – i.e. indicators from international sources are high-level and don’t cover all relevant information for assessing transferability. Therefore, each case study also includes a set of “new indicators” (i.e. those with no publically available information) policy makers should consider before transferring the intervention.

Finally, indicators to measure the risk factor level in each country (e.g. obesity rates) were not included given it is presumed all OECD and non-OECD European countries face challenges caused by growing rates of non-communicable diseases.

Box A A.3. Transferability methodology using distance from reference country

Quantitative indicators

Quantitative indicator values have been normalised using distance to a reference country, that is, the country in which the best practice intervention is currently implemented (also referred to as the best practice “owner” setting) (OECD and European Commission, 2008[6]).

The normalisation equation is below:

${NV}_{ci}= \frac{(X_{ci}- X_{oi})}{X_{oi}}$(Equation 1).

Where:

• ${NV}_{ci}$ = normalised value for target setting (country c) for indicator i

• $X_{ci}$ = original value for target setting (country c) for indicator i

• $X_{oi}$ = original value in the owner setting for indicator i.

Normalised values for equation (1) can be interpreted as percentage distance each country is from the best practice owner setting, whose value is centred on 0. Normalised values were used to allocate countries into one of five groups for each indicator, with a darker shade indicating greater transferability potential:

Value equal or greater than 0 =

Value less than 0 but greater than -25% = (+25% when a lower value indicates better transferability)

Value less than -25% but greater than -50% = (>+25% but less than <+50%)

Value less than -50% but greater than -75% = (>+50% but less than <+75%)

Value less than -75% = (>75%)

Binary indicators

For binary indicators, countries that respond “Yes” to the indicator are allocated the darkest shade () while countries that respond “No” are allocated the lightest shade ().

Categorical indicators

For categorical indicators, any country that responds at least as well as the best practice owner are allocated the darkest shade (), while the remaining countries are allocated a lighter shade based on the number of remaining categories.

Box A A.4. Transferability methodology using clustering

OECD has developed a methodology to cluster countries and to make personalised recommendations on which member states and member countries are more likely to successfully transfer a recognised best practice intervention. A high level summary of the clustering methodology is below.

Cluster analysis helps to identify countries which could successfully be transferred a best practice intervention

Cluster analysis partitions data into homogenous groups, based on similarities in the data. In this case it was used to separate countries into groups with similar characteristics, based on how well adapted or suited they are for transfer of a best practice intervention from a host country. For each cluster, specific recommendations can then be made to address potential obstacles for implementation. This can help guide decision makers and potentially lead to the smoother implementation and increased success of interventions.

K-medoids clustering was found to be the optimal methodology

To select the best methodology four different cluster methods were compared: k-means, k-medoids, hierarchical and DBSCAN (Density-Based Spatial Clustering of Applications with Noise). K-medoids using Gower distance was found to be the most effective method for clustering countries taking into account validation statistics, data characteristics, interpretability of the results and flexibility to use with other datasets. This is because it works with small, imbalanced datasets with missing data, and can accommodate categorical data as well as continuous data.

The K-Medoids Clustering Algorithm

The k-medoids algorithm is based on the medoid: this is the most central observation (country in this case) in the cluster, where the total distance between it and all the other countries in the cluster is smallest. Distance is a quantitative measure of dissimilarity, where the larger the distance between two observations, the more different they are from each other. The number of clusters (k) must be chosen prior to running the algorithm.

The k-medoids algorithm has the following steps:

• Randomly assign k countries as medoids.

• Repeat until there is no change in assignment of medoid:

Assign each country to a cluster, based on distance to the closest medoid.

For each cluster, test whether selecting another country as the medoid decreases the total distance from the medoid to all other points in the cluster. If it does, reassign this country as the new medoid.

Gower Distance is used to measure similarity between countries

Gower distance was chosen because it is able to compute the difference between both categorical and continuous variables. Gower distance is calculated from the mean of the partial pairwise distances between observations (countries). The partial pairwise distance is the difference between two observations at a single variable and is calculated differently depending on whether the variable is continuous or categorical.

Continuous Variables: The partial pairwise distance, $d_{i{i}^{\mathrm{\text{'}}}}^{\left(j\right)},$ between two observations $i$ and $i^{\text{'}}$, for variable $j$ is the difference between the two values $x_{ij}$ and $x_{i^{\text{'}}j}$, divided by the maximal range ($R_j$) of all the values for variable $j$, as follows:

${\mathit{d}}_{\mathit{i}{\mathit{i}}^{\mathrm{\text{'}}}}^{\left(\mathit{j}\right)}=\frac{\left|{\mathit{x}}_{\mathit{i}\mathit{j}}-{\mathit{x}}_{{\mathit{i}}^{\mathrm{\text{'}}}\mathit{j}}\right|}{{\mathit{R}}_{\mathit{j}}}$

Categorical Variables: If two countries have the same value for a categorical variable then the partial pairwise distance is 0 (identical). Otherwise, it is 1.

The Gower distance between two observations is then calculated as the mean of the partial pairwise distances. The partial pairwise distances can be weighted differently. Here, the variables were weighted so that each contextual factor had equal weighting and therefore equal influence on the Gower distance. The resulting value lies between 0 and 1, with values closer to 0 indicating greater similarity between countries and values closer to 1 indicating greater dissimilarity. If one or both values are missing for a given variable in a pair of countries, the partial distance for that variable will not be included in the Gower distance, meaning there is no need for data imputation. However, if a country had over 50% variables missing it led to inaccurate Gower distances and so these countries were removed.

Interpreting and comparing clusters by indicator and by contextual factor

The clusters were compared by calculating the difference between the mean of each cluster and the mean of the dataset, for each indicator. A positive difference meant a higher likelihood of successful transfer for that indicator, allowing the characteristics of each cluster to be identified. To more broadly compare clusters, identifying the contextual factors (or domains) where clusters were stronger or weaker, domain scores were created and used to compare cluster means. Domain scores were created using the following steps:

• Assign categorical variables dummy values (0 = no, 1 = yes).

• Normalise using min-max scaling.

• Aggregate by the mean of the variables in each contextual factor.

Summary of steps in Clustering process

In summary, the following steps are required:

• Remove countries where >50% variables are missing.

• Compute a Gower Distance Matrix, with each contextual factor having equal weighting.

• Determine optimal value of clusters (k) between 3 and 5.

• Run k-medoids clustering using the optimal number of clusters from step 3.

Create domain scores in order to compare cluster means with the dataset means, and identify strength and weakness of each cluster.

Further details will be made available in an upcoming Health Working Paper.

Limitations

Limitations associated with the analysis of case study interventions are summarised in Table A A.3.

Table A A.3. Assessment limitations
Limitation theme	Description
Selecting case studies	Case studies were assessed against the Best Practice Framework after they were selected. For this reason, case studies don’t necessarily perform well against the criteria. Case studies analysed using OECD’s SPHeP-NCD model were further restricted to those that recorded data on risk factors included in the model.
Diversity of case studies	Given the criteria for selecting cases studies, the majority of interventions analysed target individual lifestyle behaviours as opposed to structural interventions that address society as a whole. Again, due to the selection criteria, all case studies interventions are based in European countries.
Benchmarking performance	The performance of case studies is not benchmarked for two key reasons: 1) heterogeneity in terms of target populations and implementation setting (e.g. schools versus primary care) and 2) lack of comparable data.
Classifying case studies	This report does not classify case studies as “best practice” or not given this is ultimately up to policy makers in each country who may have different priorities. Further, it was not possible to benchmark performance. Case study analyses instead summarise evidence considered relevant for choosing which interventions to fund, scale-up and/or transfer.
Transferability data	The assessment of transferability using the Transferability Framework relied on publically available data at the national level. This poses several limitations – for example, it does not take into account differences within countries, in addition, it limits the extent of the analysis given availability of comparable data. For these reasons, findings should be considered as high-level only.

References

[5] Burchett, H., M. Umoquit and M. Dobrow (2011), How do we know when research from one setting can be useful in another? A review of external validity, applicability and transferability frameworks, https://doi.org/10.1258/jhsrp.2011.010124.

[1] European Commission (2021), Public Health Best Practice Portal, https://webgate.ec.europa.eu/dyna/bp-portal/ (accessed on 4 October 2021).

[3] IARC (2020), Global Cancer Observatory, https://gco.iarc.fr/ (accessed on 9 November 2020).

[2] IHME (2017), Epi Visualization | IHME Viz Hub, https://vizhub.healthdata.org/epi/ (accessed on 21 December 2017).

[6] OECD and European Commission (2008), Handbook on constructing composite indicators: methodology and user guide, OECD (the Statistics Directorate and the Directorate for Science, Technology and Industry) and the Applied Statistics and Econometrics Unit of the Joint Research Centre (JRC) of the European Commission, https://www.oecd.org/sdd/42495745.pdf.

[4] Trompette, J. et al. (2014), “Stakeholders’ perceptions of transferability criteria for health promotion interventions: A case study”, BMC Public Health, Vol. 14/1, pp. 1-11, https://doi.org/10.1186/1471-2458-14-1134.