4. Beyond increasing participation

High participation in adult learning is a necessary, but not sufficient condition for a well-functioning and future-ready adult learning system. While this study has primarily focused on an increase in participation in adult learning, this is simply not enough. To enable more adults to reskill and upskill, policy-makers must also focus their attention to issues of quality and labour market impact of their reform efforts on the intended targets. This chapter reviews indicators of success of adult learning reforms beyond overall participation, including increases in training quality, labour market outcomes, alignment with individual and labour market needs, as well as increased inclusiveness.

    

Introduction

High participation in adult learning is a necessary, but not sufficient condition for a well-functioning and future-ready adult learning system. This study has so far focused on the features of reforms, which were successful in raising participation in adult learning. However, focusing on the increase in the overall quantity of adult learning is simply not enough. To enable more adults to reap the benefits of participating in learning activities, policy-makers must not only focus on participation rates, but also on training quality, participants’ labour market outcomes, inclusiveness of the policies and the alignment of programmes with individual and labour market needs.

Training quality

The number of individuals involved in training will not translate into better labour market or society-wide outcomes if the learning activities are not of good quality. Training quality is therefore fundamental for the reforms to reach desired outcomes beyond training participation, e.g. the increased employability of participants. However, quality is also a multi-dimensional concept, touching upon the content of training, the competencies of the providers and teachers, and eventually the effectiveness of training in changing the labour market or societal condition of participants. This feature, as well as the largely subjective nature of quality assessments, makes measurement and implementation of quality a relevant challenge of adult learning systems.

Quality assurance

Countries may need to set up minimum quality criteria and standards, and certification mechanisms to ensure that these are respected. In some countries, publicly funded training programmes can only be delivered by certified providers as a way to ensure quality. Monitoring and evaluation systems should also be in place, as well as mechanisms that enable the sharing of information about the quality of different providers. Capacity-building activities for staff in adult training institutions would ensure that they acquire a better understanding of what quality is and how to monitor and assess it. Special provisions should be made to adapt curricula, course organisation and teachers’ mindset and methods to working with adults, as opposed to youth (Box 4.1).

Interviewed experts highlighted that a mechanism of quality assurance is clearly missing for the Dutch Training Vouchers and for the Italian Training Funds. In the latter case, most of training activities are delivered by non-formal training providers, and in 60% of the cases (2016) by the company itself, and hence they escape a framework of quality monitoring. Each Fund has its own mechanism to assess the quality of the training proposed by a provider. The latter is usually accredited by the regional accreditation system, but some Funds also developed their own accreditation system. The training activity can lead to a certification, but here again, there is no uniform mechanism across Funds or Regions. Many training activities supported by the Funds do not yield a certification, in any case.

Teaching quality

Considering the policies in this study, provisions for quality of teaching and providers are made for one type of learning provision, namely basic education courses, which are very similar to courses provided in initial education. The Life-Long Learning Strategy in Estonia stressed motivation and quality of teachers and school management as one of five key objectives of the strategy. The Austrian Initiative for Adult Education sets strong quality criteria, be it on providers, courses, instructors or counsellors.

Counselling services and infrastructure

Education courses are also accompanied by a professional counselling and introduction phase. In the Hungarian Open Learning Centres, participants receive advice at the beginning of their learning path, and a certification for every completed course at the end – although these certifications are not necessarily recognised throughout the country.¹ In a similar vein, the Italian Adult Education Centres provide guidance to participants, at least in the first phase of the learning activity, and aim at certifying the skills acquired in it, based on the National Qualification Framework.

That said, training quality in the Italian Adult Education Centres may be affected negatively by the relatively old infrastructure in which courses often take place. Moreover, although one of the Centres’ missions is to foster advancements in the methodology and didactics for adult learners, instructors were most often trained to work with children and not with adults, according to the interviewed stakeholders. This was also mentioned by experts as an important complaint of participants to the Hungarian Basic Skill Courses. The same shortcoming (instructors’ quality and age) seem to affect provision of the Free Second Vocational Degrees in Hungary.

Box 4.1. Training quality case study – Programmes targeted at disadvantaged adults in Hungary

Two Hungarian programmes were very similar in terms of goals, target groups (disadvantaged adults) and even funding per person, however they had very different outcomes due to differences in quality:

Both teachers and participants pointed out issues related to the Basic Skill Courses, mainly because the learning material and the methods were not adapted to adults. Most of the recruited teachers did not have any experience in teaching and none of them received help in how to teach adults. Much of the teaching materials were based on exercises designed for children, and the courses took place in a classroom setting. As a result, the programme reinforced participant’s negative views of education. One participant noted: “I am 40, even if I did not finish school they should not give me the tasks of a 5 year old. This really made me angry”.

On the other hand, courses delivered at Open Learning Centres are tailored to adults. Teaching content relates to their everyday lives, teachers are experienced in working with adults and classes are delivered in a relaxed atmosphere surrounded by modern technology. As a result, centres contribute to a positive learning culture. As noted by one participant: “I am very thankful for this opportunity! I would like to continue to learn with this teacher and the group”. 70% of participants enrol in another course at the Centres.

Source: Kerülő and Nyilas (2014[1]), A közfoglalkoztatásban résztvevők képzésének andragógiai konzekvenciái; NYITOK (2019[2]), NYITOK Project, http://www.nyitok.hu/rolunk3.0.

Labour market outcomes of training programmes

Good policies that increase participation in adult learning can still fail translate into better working or living conditions for participants. This section therefore explores the labour market outcomes of individuals participating in a training programme, both looking into the specific reforms considered so far, and in the broader literature.

The assessment of the “true” effect of reforming training policies on selected individual outcomes necessarily relies on counterfactual impact evaluation analyses. The established micro-economic literature has estimated the impacts of ALMPs, including training policies, on different labour market outcomes, but not that of learning policies outside of ALMPs, e.g. of support to basic education. For the former case, Kluve (2010[3]) and Card et al. (2018[4]) perform meta-analyses of existing studies since the late ‘90s, and find that training programmes in developed countries have been generally effective in moving people from unemployment to employment, or raising workers’ salaries. These effects are usually positive but negligible in the short term, and larger in the medium to long term. They set training policies apart from other ALMPs such as job-search assistance programmes, whose effects are positive and approximately constant over time, and public sector employment subsidies, which tend to have small or even negative average impacts at all horizons.

The economic rationale is clear: participants in learning activities do not usually work (full time) while in training, which translates into worse labour market outcomes in the initial period than for an untrained comparison group. If the training is valuable, however, trained individuals slowly catch up and outperform the untrained comparison group over time. A larger effect in the long run, while true on average, seems to be mostly driven by the subgroup of young participants (Card, Kluve and Weber, 2018[4]). More recent evidence across country (Bown et al., 2019[5]) and for Germany only (Dauth, 2019[6]) confirms these results. McKenzie (2017[7]) analyses ALMPs in developing countries, and finds small positive effects of VET programmes on employment, much larger ones on formal employment, and rather insignificant ones on wages relative to wages of the control group. Training programmes can also influence other labour-market or social outcomes, such as civic participation or political involvement, mental and physical health, criminal activity, or choices regarding schooling and work for other members of the trainee’s household (Barnow and Smith, 2015[8]). These outcomes, however, have been investigated less often and less consistently across country, often due to lack of available data on them, especially in the longer run.

A number of the policies considered for the present study performed an evaluation of the reform’s impacts on outcomes of participants, as highlighted in Table 2.9 in Chapter 2. That section described the evaluations’ goals, methods, involved stakeholders, and consequences for changes in the policy design. This section focuses instead on the results of the evaluation. When available, inference is mostly based on experimental or quasi-experimental settings and on qualitative information in some cases (e.g. the Initiative for Adult Education in Austria). Broadly speaking, the existing evaluations suggest that reforms included in this study positively affected participants’ probability of employment and, though to a lesser extent, earnings.

This is certainly the case for the State-Commissioned Training Courses in Estonia, which increased the probability of participants to be employed, time in employment, and earnings, although by a different extent depending on the used estimation techniques (Leetma et al., 2015[9]). Effects were also larger for participants who were in employment, those with low education levels, younger and older age. Higher earnings and a higher probability of employment compared to the control group characterised participants in occupational training in the framework of the reformed Estonian ALMPs - see case study box and (Lauringson et al., 2011[10]).

In the Netherlands, De Groot and Van der Klaauw (2017[11]) and Van Hoof and Van den Hee (2017[12]) evaluated the effectiveness of Network Training based on a randomised experiment on unemployed individuals. They found that the policy increased participants’ chances of finding a job relative to the control group, but not their salary. Results for the Austrian ALMP reform and Paid Educational Leave are less encouraging instead. Multiple evaluations focused on different aspects of the Austrian ALMP reform, but retrieved negligible macro-level effects at the federal or regional level, and only small positive effects for selected subgroups of participants (Aumayr et al., 2009[13]). Lassnigg et al. (2011[14]) find that changes in the eligibility criteria to the Paid Educational Leave did not lead to any important economic effects on employment and wages of participants. Effects were indeed mixed: the policy positively affected labour market mobility of employed participants, and males engaging in VET. Over the entire sample, real wages seven years after participation were only marginally higher, and tenure was fewer days shorter than for the control group.

Box 4.2. Impact of training case study – ALMP reform in Estonia

A 2011 evaluation using quasi-experimental methods (propensity-score matching) found that unemployed people who had participated in labour market training in 2009 and 2010 experienced a positive effect on employment outcomes and wages. For one programme in particular, which was focused on occupational training:

• Participants who completed their training in 2009 (respectively, 2010) experienced a wage increase of EUR 50 (resp. EUR 90) per month on average, one year after participation. This was a 1/3 (respectively 1/2) larger increase than the control group. The requirement for greater labour market relevance of training from 2010 onwards likely explains the larger effect in 2010 relative to 2009.

• 36% of participants who finished training in 2009 were employed one year later, compared to 26% of individuals in the control group. For the 2010 cohort, figures reach 46% vs 34% respectively.

• Participants in the 2010 cohort had 10 fewer days of benefits than the cohort group (-13%) and for a lower amount (-12%), but differences were not statistically significant.

Some groups saw greater returns from training, including women, older participants and those that had been unemployed for shorter periods of time.

Source: Lauringson et al. (2011[15]), Impact Evaluation of Labour Market Training: the Estonian Unemployment Insurance Fund.

A number of reasons may contribute to the limited effects of training programmes on participants’ employability and earnings. Card, Kluve and Weber (2018[4]) stress the importance of considering both the short term and the medium- and long term impacts. Taking a longer perspective may be challenging, if individuals cannot be followed over time (attrition) or enter the informal sector (McKenzie, 2017[7]). Access to administrative data is scarce or fragmented across different institutions, information is self-reported and unreliable, or not collected altogether. The same authors also propose that a policy’s estimated effect may be affected by too much aggregation over groups of participants with heterogeneous outcomes (in their analysis: young vs older individuals, but also high- vs low-educated individuals), over different moments of the business cycle, or over programmes of heterogeneous duration or intensity. Inference can also change if training participants are enrolling in a learning activity for the first time, and are compared to other individuals which have already participated in training before, or never did so at all (Hidalgo, Oosterbeek and Webbink, 2014[16]).

While labour market institutions are found to play no role in Card, Kluve and Weber (2018[4]), Escudero (2018[17]) shows that ALMPs are more effective in improving labour market outcomes if they are better managed and implemented, and in particular if resources are congruous to the tasks and there is continuity in the policy. Training programmes are found to improve participants’ labour market outcomes overall, and those of the low skilled in particular through the interaction with implementation variables. Once again, evidence is scarce about the impact of the same features in learning activities which do not fall within the spectrum of ALMPs. Lastly, Bown et al. (2019[5]) highlight that training programmes are most successful when they target a specific job. This point will be further developed in the next section.

Alignment with individual and labour market needs

An individual’s skill set is fundamental for their success in the labour market. Adult learning policies aiming to strengthen individual career progression should therefore enhance the acquisition of skills that are job-relevant and aligned with labour market needs. Training programmes should therefore: ii) especially target individuals whose skill set is becoming obsolete; ii) tailor content to current as well as future skill requirements in the market, while also considering how much these overlap with the needs of the individual’s employer; and iii) guide the provision and take-up of training through a system of incentives (OECD, 2019[18]). This in turn asks for a sound understanding of changing skill needs, often acquired through Skill Assessment and Anticipation exercises (see also Chapter 2). That knowledge can then be used to design incentives and assistance to workers’ re-skilling.

Personalised training

The relevance of an adult learning activity is pinned down in the assessment of an individual’s existing skill set, needs, work- and learning history. This allows adults to focus on developing the skills they actually miss, make the most of the time spent in training and facilitate the combination of work and training. It also ensures that barriers to training are minimised, and in particular those that relate to lack of time and limited flexibility of training provision. Personal trajectories in training, however, require the ability to recognise prior learning, which must be transparent, streamlined and ensure the buy-in of all relevant stakeholders, including employers and education and training providers. To reduce misallocation of resources, personalised training should also be well aligned with market needs, as developed here further below.

Initiative for Adult Education in Austria sets a remarkable example in this case, too: it provides individualised learning, as well as complementary counselling which looks holistically at the life situation of the individuals before making a recommendation for further learning. The Open Learning Centres in Hungary advise individuals on the learning activity that best fits their needs, too. They can also tailor courses to local needs, which can therefore differ from country-wide programmes. The Italian Adult Education Centres target courses to the (adult) student’s needs, too. Students’ skills, even if of informally acquired, are screened at the beginning of the learning period, and the resulting information factored in when establishing the students’ workload. A greater focus on individualised learning is another of the five strategic objectives of the Estonian Lifelong-Learning Strategy. State-Commissioned Short Courses in the same country allow individuals to choose a course of their choice, or a choice that has been screened by counsellors in the case of employed workers. This feature has not always produced positive outcomes, as training content can be of limited labour market relevance.

Alignment to market needs

Reforms in some of the countries considered in the present analysis were successful in aligning training to labour market needs. The Estonian Lifelong-Learning Strategy sets greater “accordance of lifelong learning opportunities with the demands of the labour market” as one of its five key objectives (Ministry of Education and Research, 2014[19]). This was combined with measures that aimed to improve the labour market relevance of training, including a skill assessment and anticipation system. Singapore’s SkillsFuture series introduced short, industry-relevant training programmes that focus on emerging skills such as data analytics, cyber security or advanced manufacturing skills. The Dutch Training Vouchers for older unemployed people could be spent only if a future employer was expressing interest for it in written form, or for training that would improve the person’s probability of finding employment in an occupation in demand in a given region. The Sector Plans in the same country aimed at helping people find work or stay employable throughout the working life, by making regional and sectoral labour markets “future-ready”. In practice, the Sector Plans supported training through a crisis period. Indeed learning systems may be understood as fostering the alignment of training to market needs if they contribute to smooth the business cycle’s consequences on the labour market impacts of recessions. This was also the goal of the Estonian EUIF in the context of the ALMP reforms. Finally, the Austrian Paid Educational Leave was used to keep workers on a company’s payroll at the peak of the 2009 crisis (Lassnigg et al., 2011[14]).

Rather than responding to current or future skill needs, some policies are leveraged to provide compulsory training opportunities, such as health and safety training. While this type of training is certainly useful, it should not substitute for market-relevant training. The Italian Training Funds would be perfectly designed to target market-relevant skills, as employers can largely choose the type of training a Fund would support. In practice, in too many instances (16% of workers, 11% of plans in 2016), they co-sponsor training in areas that are already compulsory by law – hence training that would have taken place even in the absence of the Training Funds – such as in the area of health and safety at the workplace.

Many other reforms, instead, aimed at raising the competencies of adults with low skills or low qualification, irrespective of specific market needs. This is the case for Initiative for Adult Education in Austria, the Basic Skill Courses in Hungary, and the Adult Education Centres in Italy, where courses aimed to develop basic literacy, ICT and national language skills, rather than job-specific skills. While this is not what is typically understood with alignment of training with market skill requirements, it can still lead to positive labour market outcomes of participants, if it is the case that lack of basic skills is limiting employment perspectives. In Estonia, however, State-Commissioned short courses targeted those with low or obsolete skills, yet there is evidence that they do not fully reach the target, as low-skilled and other disadvantaged individuals were underrepresented amongst participants (Leetma et al., 2015[9]).

Box 4.3. Alignment case study – the Dutch Training Vouchers

Between 2013 and 2017, all unemployed people aged 50 and older could get a voucher from the Public Employment Service, which covered 100% of the cost of education and training not exceeding EUR 1 000. The learning activity had to increase their employment opportunities, i.e. be required by a prospective employer or leading to employment in a high-demand occupation in the region.

In order to access the voucher, adults could submit an agreement between themselves and their employer, where the latter committed to hiring the former after the training activity. Alternatively, the trainee could argue that a given training activity would enhance their prospects of finding a job in an occupation currently in shortage in the region. Each region produced its own list of occupations with “good employment perspectives”, itself based on the number of open vacancies in that region.

Other market considerations

Better alignment of training with employers’ skill demand should translate into better matches of workers with firms. While some degree of misalignment between the supply and demand for skills is inevitable, especially in a short-run perspective, mismatches in OECD countries are widespread. More than 40% of workers in Europe, Japan, Korea and Mexico feel their skill levels do not correspond to those required by the job, either because they think they could cope with more demanding work or because they cannot meet the demands of their present job (OECD, 2019[18]). Misallocation of labour can reduce firms’ productivity and individuals’ likelihood of employment, wages, or job satisfaction.

Training policies, however, do not always lead to improved allocation of resources for the whole economy. As mentioned in Chapter 3, support to learning may be given to individuals who would have trained in any case, even in the absence of the policy, generating deadweight losses. One such point was raised for the Dutch Training Vouchers, which subsidised training for older unemployed individuals, although this target group are usually less financially constrained than youth. Moreover, training may allow the participant to take a job that would have otherwise gone to someone else, so that the value of this match relative to its counterfactual remains unobservable. This may also lead to an overestimation of the benefits associated to the training programme, if the non-participant does not enter the policy’s comparison group (Andersson et al., 2018[20]). In the case of the Italian Training Funds, targeting of certain categories of workers was judged to generate an unjustified substitution between subsidised- and non-subsidised training participants (OECD, 2019[21]).²

The latter point highlights that the reform can give origin to a number of general equilibrium effects, i.e. affect economic agents other than those directly participating to the programme. The salary of non-participants, for example, can be affected by the policy if this: i) strengthens competition for jobs in the occupation or geographical market where non-participants are employed; ii) increases the supply of certain skills in the local labour market (Barnow and Smith, 2015[8]); or iii) raises the productivity requirements for all workers in the company where participants are employed. Furthermore, adult learning reforms can publicise adult learning and improve the culture of learning in the country as a whole, thus convincing non-participants to engage in adult learning and raising the net benefits of the programme.

Lastly, adult learning reforms can have important consequences for the (quasi-) market of training provision in a country. Several interviewees mentioned that the provision of State-Commissioned Short Courses essentially destroyed the market for short non-formal training courses in Estonia. Similarly, the reform of the Free Second Vocational Degree in Hungary made them free and reportedly displaced private providers of VET courses as a consequence. One final observation deals with the opportunity cost of public resources invested in adult learning. It cannot be excluded that tax receipts would be higher if the programme resources were invested in a different policy. This consideration, however, remains mostly theoretical, because of the inherent difficulty to estimate the actual opportunity cost of public expenditure under a vast array of alternative policies.

Inclusiveness considerations

While policy should facilitate access to adult learning for every individual interested in or in need of upskilling, some groups receive additional support to engage in training activities because of their disadvantaged position in learning (low-skilled adults), specific socio-demographic characteristics (age, gender, nationality or ethnicity), or employment situation (length in unemployment, low income, contract type). Support to these groups may be motivated by policy-makers desire to improve social- and labour-market outcomes of these categories. Moreover, the society-wide spill overs of these improved outcomes are not necessarily factored-in in the decision to train, either by the prospective trainee or by employers. This provides further justification to public support to training participation for selected target groups.

Based on data from the OECD Survey of Adult Skills (PIAAC), older and lower-skilled adults, as well as low-wage workers, are the socio-demographic categories displaying the lowest participation rates in every country (OECD, 2019[18]). Chapter 2 described how the reforms in this study pay special attention to the unemployed and to adults with low skills, and how some of the analysed policies did not consider any specific target group in the reform design (the Estonian Life-Long Learning Strategy or the Free Second Vocational Degrees in Hungary). Chapter 2 also discussed barriers to training, and how financial support is provided to participants, either by subsidising the firm- or individual- cost of training (e.g. Training Vouchers in the Netherlands, the Estonia ALMPs, and Singapore SkillsFuture Credit), or by providing altogether free access to it (e.g. Hungary’s Free Second Vocational Degrees or Open Learning Centres, Italy’s Adult Education Centres, Austria’s Initiative for Adult Education).

Policies targeting the unemployed or the low-skilled, who often have low income or fewer employment opportunities, can be leveraged to target low-income adults, too. Special provisions for low-income adults specifically are much rarer among the policies considered. By design, only the Estonian labour market training for the employed, in the context of the expansion of training-related ALMP reform, supports workers seeking occupational development but earning less than EUR 15 492 per year.

Conversely, many of the considered policies distinguish adults by age, for instance. A long literature recognises that older adults engage less frequently in training, as their shorter time to retirement decrease the cumulated returns to the investment in training (Cunha et al., 2006[22]). Moreover, older adults are on average less acquainted with digital technologies, making them more susceptible to skills obsolescence. Lastly, motivation, the way training is provided and the degree to which training addresses relevant problems at work have a direct impact on the effectiveness of training for older workers (Callahan, Kiker and Cross, 2003[23]; Zwick, 2015[24]). Special provisions for older workers characterise two of the considered reforms in the Netherlands. Furthermore, employed workers of age 50 and above are eligible for assistance under the reformed ALMPs in Estonia, while workers of age 40 and above see 90% of their training costs reimbursed in Singapore under the SkillsFuture Mid-Career Enhanced Subsidy programme. Conversely, the Hungarian Open Learning Centres targeted – among others – older workers as well as young ones entering the world of work.

Some of the policies in this study provide special conditions for migrants and minorities, too. Migrants, in particular when newly arrived, benefit from targeted adult learning support, be it to improve their proficiency in the host language or to validate and adapt their skills to the requirements of the host country’s labour market. The participation of migrants in the adult learning system was mentioned by many interviewees as one of the key factors driving demand for courses under the Initiative for Adult Education in Austria, and in the Adult Education Centres in Italy. Lastly, the only policy considered in this study which explicitly addresses a minority group is the Open Learning Centres in Hungary. Roma also represented a relatively large share of participants in the Basic Skill Courses.

Other dimensions of inclusiveness did not figure prominently in the interviews carried out for the present study. Gaps in participation to training of women and men are possibly less stark than between the opposites in other categorisations, which may explain why the design of reforms has not featured gender prominently. Conversely, take-up of learning activities is usually less frequent in small and medium enterprises (SMEs) than large firms (OECD, 2019[18]). SMEs tend to underinvest in human capital, likely because they are more resource- and capacity-constrained, less informed about existing training- or support opportunities, or simply less aware of the potential benefits of training for their workforce. Only one interviewed expert mentioned that the current design of the policy – the Italian Training Funds – was preferred to alternative forms and in particular to a tax credit scheme, because it was believed to favour take-up in SMEs.

Lessons learnt – going beyond participation

• Minimum quality criteria and standards, as well as certification mechanisms to ensure that these are respected, are needed to make sure that participants benefit from training in terms of societal or labour market outcomes. Special provisions should be made to adapt curricula, course organisation and teachers’ mind-sets and methods to working with adults, as opposed to children.

• Training programmes are found to impact an individual’s employment perspectives and, to a lesser extent, their earnings, but more so in the medium- to long-term than in the short term. Evaluations of the policies considered in this study are broadly in line with these results from the relevant economic literature.

• Skill Assessment and Anticipation exercises can help design adult learning policies, which align training to labour market requirements, including over the business cycle.

• While extremely useful to reduce the pervasiveness of mismatches in the labour market, adult learning programmes can also create misallocation of resources in the economy. These considerations did not emerge very frequently from the analysis of the policies covered by the present study.

• Special provisions exists in many countries for the participation of vulnerable groups in adult learning, and in particular for migrants, elderly workers and low-skilled workers.

References

[20] Andersson, F. et al. (2018), Does Federally-Funded Job Training Work? Nonexperimental Estimates of WIA Training Impacts Using Longitudinal Data on Workers and Firms, http://www.census.gov/ces (accessed on 26 November 2019).

[13] Aumayr, C. et al. (2009), Makroökonometrische Effekte der aktiven Arbeitsmarktpolitik in Österreich 2001-2007, https://www.sozialministerium.at/cms/site/attachments/5/2/6/CH3434/CMS1459844028422/20_makrooekonomische_effekte_der_aktiven_arbeitsmarktpolitik_in_oesterreich_2001-2007_endbericht_inkl._summary_e_f.pdf (accessed on 16 July 2019).

[8] Barnow, B. and J. Smith (2015), Employment and Training Programs, National Bureau of Economic Research, Cambridge, MA, https://doi.org/10.3386/w21659.

[5] Bown, C. et al. (2019), Working Paper 19-2: Active Labor Market Policies: Lessons from Other Countries for the United States, http://www.piie.com (accessed on 26 November 2019).

[23] Callahan, J., D. Kiker and T. Cross (2003), “Does Method Matter? A Meta-Analysis of the Effects of Training Method on Older Learner Training Performance”, Journal of Management, Vol. 29/5, pp. 663-680, https://doi.org/10.1016/S0149-2063_03_00029-1.

[4] Card, D., J. Kluve and A. Weber (2018), “What Works? A Meta Analysis of Recent Active Labor Market Program Evaluations”, Journal of the European Economic Association, Vol. 16/3, pp. 894-931, https://doi.org/10.1093/jeea/jvx028.

[22] Cunha, F. et al. (2006), “Interpreting the evidence on life-cycle skill formation”, https://doi.org/10.1016/S1574-0692(06)01012-9.

[6] Dauth, C. (2019), “Regional Discontinuities and the Effectiveness of Further Training Subsidies for Low-Skilled Employees”, ILR Review, p. 001979391988510, https://doi.org/10.1177/0019793919885109.

[11] De Groot, N. and B. Van der Klaauw (2017), De resultaten van de effectmeting Succesvol naar Werk - Eindrapport voor UWV, https://www.rijksoverheid.nl/documenten/rapporten/2017/12/19/de-resultaten-van-de-effectmeting-succesvol-naar-werk-eindrapport-voor-uwv (accessed on 2 December 2019).

[17] Escudero, V. (2018), “Are active labour market policies effective in activating and integrating low-skilled individuals? An international comparison”, IZA Journal of Labor Policy, Vol. 7/1, p. 4, https://doi.org/10.1186/s40173-018-0097-5.

[16] Hidalgo, D., H. Oosterbeek and D. Webbink (2014), “The impact of training vouchers on low-skilled workers”, Labour Economics, Vol. 31, pp. 117-128, https://doi.org/10.1016/j.labeco.2014.09.002.

[1] Kerülő, J. and O. Nyilas (2014), “A közfoglalkoztatásban résztvevők képzésének andragógiai konzekvenciái”, Oktatás és nevelés - Gyakorlat és tudomány, pp. 182-197.

[3] Kluve, J. (2010), “The effectiveness of European active labor market programs”, Labour Economics, Vol. 17/6, pp. 904-918, https://doi.org/10.1016/j.labeco.2010.02.004.

[14] Lassnigg, L. et al. (2011), Evaluierung der Bildungskarenz.

[10] Lauringson, A. et al. (2011), Impact evaluation of labour market training. Estonian Unemployment Insurance Fund, Estonian Unemployment Insurance Fund, Tallinn, https://www.tootukassa.ee/sites/tootukassa.ee/files/Impact_Evaluation_of_Labour_Market_Training.pdf (accessed on 26 June 2019).

[15] Lauringson, A. et al. (2011), Impact Evaluation of Labour Market Training: the Estonian Unemployment Insurance Fund.

[9] Leetma, R. et al. (2015), Counterfactual Impact Evaluation (CIE) of Estonian Adult Vocational Training Activity, Praxis, Tallinn, http://www.praxis.ee (accessed on 7 July 2019).

[7] McKenzie, D. (2017), “How Effective Are Active Labor Market Policies in Developing Countries? A Critical Review of Recent Evidence”, The World Bank Research Observer, Vol. 32/2, pp. 127-154, https://doi.org/10.1093/wbro/lkx001.

[19] Ministry of Education and Research (2014), The Estonian Lifelong Learning Strategy 2020, https://www.hm.ee/sites/default/files/estonian_lifelong_strategy.pdf (accessed on 7 December 2018).

[2] NYITOK (2019), NYITOK Projekt, http://www.nyitok.hu/rolunk3.0 (accessed on 5 May 2019).

[21] OECD (2019), Adult Learning in Italy: What Role for Training Funds ?, Getting Skills Right, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264311978-en.

[18] OECD (2019), Getting Skills Right: Future-Ready Adult Learning Systems, Getting Skills Right, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264311756-en.

[25] Van der Werff, S. et al. (2019), De opbrengst van de sectorplannen: Eindevaluatie Regeling Cofinanciering Sectorplannen, http://www.seo.nl (accessed on 10 February 2020).

[12] Van Hoof, E. and S. Van den Hee (2017), Inhoudelijke effectevaluatie trainingen 50plus WW: Eindrapportage Resultaten Voormeting, Nameting 1 en Nameting 2, https://www.rijksoverheid.nl/documenten/rapporten/2017/12/19/eindrapportage-inhoudelijke-effectevaluatie-trainingen-50plus-ww (accessed on 2 December 2019).

[24] Zwick, T. (2015), “Training older employees: what is effective?”, International Journal of Manpower, Vol. 36/2, pp. 136-150, https://doi.org/10.1108/IJM-09-2012-0138.