Annex B. Technical Annex

Introduction

The OECD identified policy lessons from six countries that have successfully increased participation levels in adult learning in the past 15 years. This involved the mapping and analysis of recent reforms of the adult learning system that may have contributed to the observed improvements (‘case study research’). The analysis included five European countries and one non-European country.

This annex provides an overview of the methodology applied for the selection of countries and reforms included in the review. The selection of countries is based on the quantitative analysis of participation trends in the past 15 years, as well as trends in inclusiveness and alignment of the training system with labour market needs. It addition, it takes into account qualitative evidence on the existence of relevant reforms of the adult learning system and aims to include countries representing the geographic, policy and cultural diversity of (European) adult learning systems. The final selection of countries was made in consultation with the European Commission.

Country selection

Initial country selection

Based on the data analysed and qualitative evidence, the following countries were proposed to be included in the case study research (see Table A B.1). This proposal included countries that had seen significant improvements in adult learning participation and/or the inclusiveness and alignment of the adult learning system, according to the indicators considered. A detailed outline of the selection methodology is provided in the following.

Following qualitative considerations, it was decided in consultation with the European Commission that Estonia, instead of Latvia, and Singapore, instead of Canada or the United States would be included in the case study research.

Table A B.1. Ranked short-list of countries based on quantitative and qualitative evidence
#	Proposed country	Alternative
European Union (EU)
1	Hungary	Czech Republic
2	Latvia	Estonia
3	Netherlands
4	Italy	Portugal, France
5	Austria
Non-European
6	Canada	United States, Singapore

Detailed methodology – Selection of European countries

The following outlines the quantitative data sources used as basis for the country selection, as well as the selection criteria and process applied.

Data sources

There are a number of comparative data sources available that facilitate cross-country comparison on the ‘performance’ of adult learning systems over time. These include:

• The European Adult Education Survey (AES) is a survey covering persons between 25 and 64 years old and enquiring about their participation in education and training (formal, non-formal and informal) in the last 12 months. The survey is part of the EU statistics on lifelong learning and covers 35 countries, including all EU Member States, the United Kingdom, Albania, Bosnia-Herzegovina, North Macedonia, Norway, Switzerland, Serbia and Turkey. Three waves of data collection have taken place (2007, 2011, 2016).

• The European Union Labour Force Survey (LFS) is a large household survey covering people aged 15 and over. It contains questions on participation in education and training (formal and non-formal) in the last 4 weeks. The survey covers EU Member States and the United Kingdom, 4 candidate countries and 3 countries of the EFTA. Data on participation in education and training are available with an annual frequency mostly from the early 2000s.

• Continuing Vocational Training Survey (CVTS) is a long-running enterprise survey on continuing vocational training and other training in enterprises in the business economy (excluding micro-enterprises with less than 10 persons employed). The survey is part of the EU statistics on lifelong learning and covers all EU Member States, the United Kingdom and Norway. Comparable data are available for the three last waves of data collection (2005, 2010, 2015).

Limitations of these data for the purpose of this project include the non-availability of data for non-European countries and the fact that time-series data of the key sources (AES and CVTS) only cover the past 10 years and is collected in large intervals (typically 5 years).

Selection criteria

The following selection criteria were applied:

• Improved education and training participation of adults: Different data sources on adult learning participation can show different trends. Hence, both AES and LFS data on participation were used for the country selection to take into account potential inconsistencies. Trends can be reported as percentage and percentage point change. Both indicators have advantages, with percentage change taking into account initial levels of participation and percentage point change providing a direct reflection of the additional share of the population that has joined training. Again, both indicators are included in the selection for completeness. The AES data includes training participation of 25-64 year-olds in the last 12 months, while the LFS survey provides data on education and training participation of all people aged 15 plus in the past 4 weeks. To be consistent with the age group covered by the AES, the population of the LFS data was also restricted to adults aged 25-64 year olds. Although the LFS is conducted annually, trends were calculated for the same time period as AES for consistency. In both datasets, the change in training participation was calculated for two time periods, namely the past decade (2007-2016), as well as more recent years (2011-2016) to allow for the identification of potential effects of more recent policy changes.

• Inclusiveness of training participation: AES data was used to construct indicators on the inclusiveness of adult learning systems. Indicators were created that reflect the increase or decrease of the participation gap for different under-represented groups over time. Specifically, indicators included the change in the participation gap for older workers (age 55-64) compared to prime age workers (age 35-54) and between workers with low qualification levels (ISCED 0-2) and those who are highly educated (ISCED 5-8) in the time period 2007 to 2016.

• Alignment of training provision: Alignment ensures that individuals that participate in adult learning get equipped with the ‘right’ skills, i.e. skill that are in demand in the labour market. This is especially important in a context where skill needs and the world of work is rapidly changing due to megatrends. Alignment is difficult to measure directly, but some proxies from the CVTS database can shed light on trends. For example, the share of training hours that are not spent on compulsory health and safety courses, can indicate the extent of courses focusing on the development of ‘new’ skills. Similarly, the share of companies who respond to future skill needs by providing continuing vocational training to their current staff, makes it more likely that the skills of the workforce will be aligned to the labour market. The country selection includes indicators regarding the percentage point change between 2010 and 2015 for both indicators.

Selection process

The generation of a short-list of countries to include in the study was based on a step-wise approach:

1. 1. Counting the number of times a country was amongst the top-10 performers with respect to an increase in training participation, across all indicators considered. Training participation was based on two datasets, i.e. AES data on participation in the last 12 months, and LFS data on participation in the past 4 weeks. For each variable, both the percentage and percentage-point change between 2007-2016 and between 2011-2016 were measured. Countries could therefore be listed up to 8 times among the top-10 performers with respect to increasing training participation.

2. 2. Inclusiveness and alignment variables were taken into account, when the previous step leads to inconclusive results because two or more countries display similar performance. Again, the number of times each country was amongst the top-10 performers with respect to the 4 variables, i.e. decreasing participation-gap with respect to age, and with respect to education level, increasing hours spent on non-compulsory/health and safety courses, and increasing share of firms providing continuing vocational training as response to changing skill needs, were counted. Countries could be listed up to 4 times among the top-10 performers on these variables.

3. 3. Where the previous two steps are inconclusive, priority was given to countries where training participation increased in both time periods for both participation variables. In other words, countries where training participation increased from 2007-2016, but decreased between 2011-2016 in either AES or LFS, were given lower priority in the short-list. For example, according to the LFS, participation increased in the Czech Republic between 2016 and 2007, but decreased between 2011 and 2016.

4. 4. Countries that had opposite time trends when looking at training participation in the two different data sources, were excluded from the short-list (marked in red). This means that, for example, Luxembourg was excluded from the final short-list of countries, because even though training participation increased strongly according to the LFS survey, the trends in adult education decreased according to the AES survey.

5. 5. Countries that were never among the top-10 performers with respect to any of the eight training participation indicators (step 1), as well as non-OECD member countries, were also excluded from the short-list (marked in white).

6. 6. Countries for whom preliminary research indicates that they had implemented one or more substantial reforms to their adult learning systems in the past 10-15 years, which may have contributed to the increased participation rates were given preference.

7. 7. Finally and importantly, the short-list ensures that the geographic, policy and cultural diversity of European adult learning systems was represented in the research.

The ordered short-list of countries following this step-wise approach is shown in Table A B.2 overleaf. The short-listed European countries proposed to be included in the research are circled in black. The final selection of countries, i.e. Austria, Estonia, Hungary, Italy and the Netherlands, was made in consultation with the European Commission.

Table A B.2. Ranked list of countries based on results from the dashboard
		Number of times listed among top-10 countries		Inconsistencies
#	Country	in improving participation (out of 8)	in inclusiveness or alignment (out of 4)	Opposing trends in the different surveys	Opposing trend of the different time periods at least in one survey
1	Hungary	8
2	Latvia	6	3
3	France	6
	Turkey†	6
4	Italy	5	2
5	Czech Republic	4	2		x
	Portugal	4	2		x
6	Luxembourg	4	2	x
7	Switzerland†	4	1
8	Estonia	4	1		x
9	Austria	4
10	Greece	3	1
11	Sweden	3	1	x
12	Bulgaria	2	3	x
13	Netherlands	2	2
14	Slovenia	2	2	x
	Spain	2	2	x
15	Belgium	2	1	x
	Finland	2	1	x
	United Kingdom	2	1	x
17	Croatia*	2			x
18	Cyprus	1	2	x
19	Germany		2
	Romania*		2
20	Poland		2	x
21	Malta*		1
	Norway†		1
22	Denmark		1		x
23	Lithuania		1	x
	Slovak Republic		1	x
24	Iceland†
	Ireland
		Short-listed countries
		Listed among the top-10 countries for at least 50% of the indicators.
		Listed among the top-10 countries for 25-50% of the indicators.
		Listed at least once among the top-10 countries, but for less than 25% of the indicators.
		Should be excluded due to disparity across surveys.
		Can be considered, but has disparity across time periods in at least one survey.
		Never listed among the top-10 countries that increased participation rates in adult learning.
		* Not an OECD member country, † Not an EU member country.

Methodology – non-European countries

In total, there are nine non-European countries that are member of the OECD, i.e. Australia, Canada, Chile, Israel, Japan, Korea, Mexico, New Zealand and the United States, which were first considered for the selection of a non-European case study country. There is no comparative data-source on time trends in adult learning for these countries, however there is PIAAC data, which can at least provide comparable data on participation in formal/non-formal education and training in a given year. Except for Mexico, all countries mentioned above are included in PIAAC.

Canada, New Zealand and the United States are the top-3 non-European OECD countries with the highest average participation rates, according to PIAAC. Their participation rates are also well above the OECD average. These countries could therefore be interesting for inclusion in the study. Within Canada, participation rates are particularly high in the province Alberta, which makes it an interesting candidate for inclusion in this study.

Although Australia’s participation rates are very similar to those of Canada and the United States, we do not propose to include Australia as a potential country for the research, because other national data sources (HILDA, WRTAL) indicate that participation rates have continuously declined in the past decade.

As mentioned above, there is no publicly available longitudinal data with respect to individuals’ participation in adult learning activities for New Zealand, Canada (Alberta) and the United States. One can therefore not apply selection criteria that are similar to those used for the short-list of European countries. This means that it (currently) cannot be verified whether the participation rates reflect an increasing or decreasing trend. However, preliminary policy research indicated that all three countries / regions place significant emphasis on adult learning. For example, New Zealand introduced major investments of NZD 168 million to increase literacy, language and numeracy skills of the workforce between 2008-2012, Alberta (Canada) has a Ministry of Advanced Education that focuses specifically on academic upgrading and adult learning and introduced various policies on the topic in the past decade, and in 2011, the Obama administration in the United States introduced USD 500 million in grants to community colleges around the country for targeted training and workforce development to help economically dislocated workers who are changing careers.

Finally, in consultation with the European Commission, it was decided to also review time-series data on learning participation from Singapore. National registry data was used to determine how participation in adult education evolved over time. Based on data from the Ministry of Manpower (MOM), the participation rate increased from around 30% to close to 50% over the previous decade as a result of a moderate increase from 2011 and a stark one from 2015. The availability of time-series data, as well as the qualitative evidence on Singaporean reform efforts in the area of adult learning, led to the inclusion of Singapore in this study.

Selection of reforms

The selected countries implemented a large number of policy reforms, which may have influenced adult learning participation in the past decades. To focus the review, the second stage of the research process involved the identification of the most important adult learning reforms for the observed increase in participation. To identify these reforms, the following selection criteria were applied:

• The selected reform had the explicit aim to improve some or more aspects of the adult learning system. This implies that major reforms outside the realm of adult learning policy, e.g. of the social security system, were not taken into account.

• The mechanism by which the reform would have increased adult learning participation had to be clear and plausible. Reforms directly affecting learning participation, such as by funding additional training places or initiating new education and training programmes, were given preference of those indirectly affecting adult learning participation, such as lifelong learning strategies or initiatives related to improving the quality of adult learning overall. Exceptions to this rule were made in cases, where multiple experts at national level emphasised the importance of the reform for increasing adult learning participation, even when the mechanisms was indirect (e.g. the Estonian Lifelong Learning Strategy).

• In the absence of any causal evidence on the effect of the reform on adult learning participation, it needed to be plausible that it had contributed to the observed increased in adult learning participation. This implied that only reforms that were implemented from or after 2005 were selected, as only these could have plausibly contributed to increased adult learning participation between 2007 and 2016. Reforms also had to display large coverage, i.e. reach a large enough part of the population so that it could probably have contributed to increased learning participation.

A selection of reforms meeting these criteria were made based on desk research and interviews with national adult learning experts. Seventeen adult learning reforms were selected to be included in this review.

Table A B.3. Full data on training participation, inclusiveness and alignment of adult learning systems for European countries
Improvement in		Participation								Inclusiveness		Alignment
Indicator		Change in participation in adult education and training in the past 12 months (AES)*				Change in participation in adult education and training in the past 4 weeks (LFS)*				Older workers	Low skilled	Training other than health and safety	Continuing training of staff
Unit /		percentage		percentage point		percentage		percentage point		percentage point		percentage point
Country	Period	2016/2007	2016/2011	2016/2007	2016/2011	2016/2007	2016/2011	2016/2007	2016/2011	2016/2007	2016/2007	2015/1010	2015/2010
Austria		43.0	24.3	18.0	11.7	15.5	10.4	2.0	1.4	-2.2	2.7	-6.1	1.2
Belgium		11.6	19.9	4.7	7.5	-5.4	-5.4	-0.4	-0.4	1.5	-1.4	-2.2	4.5
Bulgaria		-32.4	-5.4	-11.8	-1.4	37.5	37.5	0.6	0.6	7.1	7.0	-15.2	5.6
Croatia		50.0		10.6		3.4	-3.2	0.1	-0.1	-8.4	-2.9	-5.6	-6.1
Cyprus		18.5	13.7	7.5	5.8	-20.7	-11.5	-1.8	-0.9	0.8	8.9	-2.0	8.1
Czech Republic		22.6	24.3	8.5	9.0	46.7	-24.1	2.8	-2.8	0.1	-2.7	12.3	29.0
Denmark		13.3	-13.8	5.9	-8.1	-4.8	-14.2	-1.4	-4.6	3.5	-2.4	-3.5	-11.2
Estonia		4.5	-11.8	1.9	-5.9	124.3	31.9	8.7	3.8	-0.1	3.8	-4.4
Finland		-1.6	-2.9	-0.9	-1.6	12.8	10.9	3.0	2.6	-1.6	8.0	-3.6	3.9
North Macedonia						-3.3	-19.4	-0.1	-0.7
France		47.0	1.6	16.4	0.8	208.2	241.8	12.7	13.3	0.4	-9.2	-14.2	-2.2
Germany		14.5	3.6	6.6	1.8	9.0	7.6	0.7	0.6	10.9	2.1	-1.5	0.7
Greece		15.2	42.7	2.2	5.0	66.7	42.9	1.6	1.2	-1.8	0.0	4.0	-0.5
Hungary		518.9	35.5	46.7	14.6	61.5	110.0	2.4	3.3	-15.9	-8.9	-5.4	-3.0
Iceland						-8.5	-6.4	-2.3	-1.7
Ireland						-17.7	-9.7	-1.4	-0.7
Italy		86.9	16.6	19.3	5.9	33.9	45.6	2.1	2.6	1.9	-7.6	-7.7	6.5
Latvia		45.3	47.1	14.8	15.2	1.4	35.2	0.1	1.9	-3.1	8.3	3.8	5.8
Lithuania		-17.7	-2.1	-6.0	-0.6	9.1	0.0	0.5	0.0	7.0		-3.4	2.3
Luxembourg			-31.4		-22.0	133.3	20.9	9.6	2.9			1.5	7.5
Malta		7.7	1.1	2.6	0.4	32.2	18.2	1.9	1.2	0.9	11.7	-13.1	3.5
Netherlands		43.7	8.1	19.5	4.8	10.6	9.9	1.8	1.7	1.8	-2.8	-4.5	5.8
Norway		9.9	0.0	5.4	0.0	6.5	5.4	1.2	1.0	0.2	3.1	-3.3	-1.4
Poland		17.0	5.4	3.7	1.3	-27.5	-15.9	-1.4	-0.7	0.2	7.0	-0.2	4.6
Portugal		74.6	3.8	19.7	1.7	118.2	-16.5	5.2	-1.9	-5.3	8.5	-3.4	6.2
Romania		-5.4	-12.5	-0.4	-1.0	-20.0	-25.0	-0.3	-0.4	-0.5	4.5	7.2	3.9
Serbia			20.0		3.3		45.7		1.6
Slovak Republic		4.8	10.8	2.1	4.5	-29.3	-29.3	-1.2	-1.2	5.7		-7.8	-3.1
Slovenia		13.5	27.3	5.5	9.9	-23.2	-27.5	-3.5	-4.4	-3.4	-1.4	4.6	30.6
Spain		40.5	15.1	12.5	5.7	-13.0	-16.1	-1.4	-1.8	-1.7	-5.9	7.1	14.2
Sweden		-13.1	-11.1	-9.6	-8.0	55.8	17.0	10.6	4.3	5.7	-0.4	-2.6
Switzerland		41.9	5.5	20.4	3.6	17.2	8.7	4.6	2.5	2.4	2.3
Turkey		48.2	17.4	6.8	3.1	222.2	70.6	4.0	2.4	-5.4	-1.7
United Kingdom		5.7	45.5	2.8	16.3	-29.8	-11.7	-6.1	-1.9	-1.3	-10.8	7.6	0.7
			among top 5 performers in the given indicator
			among top 10 performers in the given indicator