6. Impact evaluation of the Community Employment programme
This chapter presents an evaluation of the Community Employment (CE) programme, a large public works programme in Ireland. The evaluation is based on a rich set of administrative data, which include background information on jobseekers, details on their participation in CE, and individual outcomes after participation. Using a methodological approach to correct for selection into the programme, the evaluation shows that CE is effective at improving medium and long-term labour market outcomes, and additionally has beneficial effects on non-labour market outcomes. The impact of CE on labour market and social inclusion outcomes is not the same for all types of CE participants but varies markedly across subgroups of jobseekers.
Community Employment (CE) is a public work programme that has been at the heart of active labour market policies (ALMPs) in Ireland for several decades (see Chapter 3). CE is one of the most widely used ALMPs for long-term unemployed people provided by the Department of Social Protection (DSP) and has supported an average of 22 000 participants at any one point in time over the last 20 years.
CE has two major objectives. On the one hand, it aims at connecting jobseekers with the labour market to increase employment levels, and ultimately at boosting economic growth. But on the other hand, CE has a social inclusion objective, as it seeks to reduce social isolation and social barriers of jobseekers. In addition, it plays a central role in the provision of important local services to communities, via the financing of labour for voluntary and third sector service organisations.
Over time, the social aspect of CE has taken on a more formal role, to the extent that separate activation and social inclusion strands have been introduced for CE placements. This distinction makes it possible to formally differentiate roles that are mostly aimed at providing job experience and enhancing employability from roles which are important for the provision of local services and reducing social difficulties but may be less likely to be a direct pathway to private sector jobs.
This chapter provides evidence on both the activation objective and, to the extent data availability permits, the social inclusion objective of CE. By doing so, the chapter aims to provide insights into the efficiency of the policy, thereby enabling more informed discussion on what aspects of the programme might benefit from further reviewing. Such evidence is crucial as the Irish economy and CE have strongly evolved over the last years while the last counterfactual analysis of the programme dates back to the early 2000s.
One major challenge of this evaluation relates to the assessment of “social inclusion” as one of the twin aims of CE. Firstly, there is no sharp and precise definition of social inclusion in the context of the programme. Previous research categorises the social inclusion aspect of CE as the objective to reduce social exclusion and social isolation, noting that social inclusion is about assisting individuals who encounter structural obstacles to employment and who are distant from the labour market (DSP, 2021[1]; 2015[2]). While insightful, this definition remains too vague to make social inclusion quantitatively measurable. A more tractable approach to the “social inclusion” objective, i.e. in the form of a set of SMART1 objectives, would enable precise operational metrics to be formulated in the future. Second, data on people’s inclusion in society is either sparse or even entirely unavailable. Against this backdrop, further data collection, including qualitative data, would be worthwhile for further analyses.
A second challenge of the evaluation relates to ensuring reliable estimates of the effects of CE. The programme was not run as a randomised experiment, but individual jobseekers make choices on whether or not to apply for a CE vacancy. This type of selection into the programme can give rise to systematic differences between jobseekers that choose to participate and those that do not. Therefore, the report uses a methodology to ensure that participants are compared only to non-participants that are similar to them, by building typologies of participants and non-participants, to remove potential bias. The accompanying technical report (OECD/Department of Social Protection, Ireland/European Commission, Joint Research Centre, 2024[3]), provides alternatives to the baseline specification to ensure that the general narrative presented here is not overly dependent on either the analytical technique or the time period and content of the data that was utilised.
Recognising that labour market programmes do not necessarily have the same impact on all types of participants, the report also compares the results on the effectiveness of CE for different groups of jobseekers. Such comparisons are important to avoid sweeping conclusions that only apply to some, but not all types of jobseekers. Sub-group analysis is also of use for policy makers in determining whether further targeting of the programme is desirable, or where improvements to enhance effectiveness might be needed.
The chapter starts by describing the methodology that applied, including possible limitations of the approach (6.2). It then presents the main results in section 6.3, as well as results for subgroups, and discusses the findings in the context of similar studies in other countries. Section 6.4 concludes.
This section discusses the analytical challenges that evaluating a live-running programme like CE presents and outlines the methodology that is used in the report to ensure a high degree of reliability of the evaluation.
6.2.1. A world without CE is never observed for participants
To assess the impact of CE on an individual’s labour market and non-labour market outcomes, it is necessary to know what would have happened to that individual had they never participated in the programme. This “counterfactual” is impossible to observe because individuals either participate or they do not. Assessments of programme impacts therefore rely on estimation methods to construct this counterfactual scenario. The simplest estimation method would be to simply compare outcomes of participants against non-participants but, unless the programme is set up as a randomised experiment, results would be biased.
In the case of CE, jobseekers can decide whether they would like to apply for CE or not, i.e. participation is not random, and simply comparing labour market outcomes of participants after CE and of eligible non-participants would produce unreliable results. In fact, participants and eligible non-participants differ in many respects, which can be linked to the probability of taking up CE, but also to labour market outcomes later on. For example, long-term unemployed people with very low previous earnings are more likely to start CE than eligible jobseekers with higher previous earnings. This differential in past earnings may very well affect earnings trajectories in the longer run, and failing to account for this difference would not allow to estimate the true causal impact of CE, leading to wrong conclusions. In order to produce unbiased results, it is necessary to remove the “selection bias” between participants and non-participants that occurs when participation is not random and participants self-select into the programme.
For CE, several sources of selection bias may be present. For example, it could be the case that more motivated and job-ready individuals are more likely to participate in training or employment programmes and have better employment outcomes for reasons besides their participation in CE. Conversely, certain individuals who face additional barriers to employment, and therefore have worse employment outcomes, may be more likely to be directed towards ALMPs by caseworkers. Many of those who do not participate in an ALMP may not be included simply by virtue of the fact they find a job quickly and exit unemployment without support from Intreo. This latter group of individuals may have better future employment outcomes than ALMP participants by construction: if they exit unemployment quickly, they have a good chance of keeping that job, and are much more likely to be employed in several years or months than if they had remained unemployed.
6.2.2. Matching is used to compare similar CE participants and non-participants
To address such sources of bias, the approach in this report uses a matching technique. This approach is possible because the administrative data utilised are sufficiently rich and cover several important dimensions of individual characteristics and labour market histories, which is a key requirement to ensure correct causal estimates of the programme impacts (Lechner and Wunsch, 2013[4]).
The approach controls for differences in demographic characteristics (e.g. gender, education, age, etc.) and observed labour market and benefit history between CE participants and eligible non-participants. Doing so aims to produce an estimate of the “treatment effect” (the effect of participating in CE) by comparing participants (the “treatment” group) only to eligible non-participants that are very similar in their observable characteristics (the “control” group).
More specifically, the econometric approach imposes several restrictions in order to ensure the comparability of the treatment and control groups and to provide unbiased results:
Only individuals in the same calendar year quarter are compared with each other. The method compares the labour market outcomes of those who entered CE in a given quarter with those jobseekers with similar characteristics who have not (yet) entered CE in the same quarter but were eligible to enter. The application of this “dynamic selection-on-observables” methodology – initially adopted by Sianesi (2004[5]) – is explained in greater detail in Box 6.1. Because there are CE participants across the years 2013 to 2018, ensuring that individuals are compared to one another in the same quarter accounts for effects due to differences in the economic cycle.
Each participant is compared to a non-participant with a similar probability of entering CE based on their individual characteristics. Individuals are matched with similar individuals based on an estimate of the probability that they enter into CE. This approach – based on a “propensity score” – is commonly used in the literature to tackle for the difficulty of otherwise accounting for a wide array of additional personal characteristics (Card, Kluve and Weber, 2018[6]). The propensity score is a measure of the probability of participating in a programme. In this analysis, it estimates the likelihood that individuals start CE in a given quarter for all jobseekers that are eligible to start CE in that quarter. The calculations of the propensity score take into account the following factors: (i) an individual’s employment history, including employment and self-employment earnings in the calendar years prior to long-term unemployment, (ii) Live Register history (receipt of Jobseeker‘s Allowance and Jobseeker‘s Benefit in each of the three years prior to long-term unemployment. Total lifetime Live Register spells and duration are also accounted for), (iii) demographic characteristics (age, gender, marital status, location), (iv) wider benefit histories (receipt of other DSP benefits such as illness and disability payments, One-Parent Family Payment, Rent Supplement and Maternity Benefit). In addition, a sub-group analysis is conducted for those individuals for whom the Probability of Exit (PEX) questionnaire is available to control for factors like self-reported health, education and access to transport. Further details on these administrative data and PEX questions is provided in the accompanying technical report (OECD/Department of Social Protection, Ireland/European Commission, Joint Research Centre, 2024[3]).
The outcome estimates are derived using “doubly robust” analytical techniques. Utilisation of a second stage regression to estimate programme impacts allows any residual differences stemming from observables that remain between participants and their matched non-participant to be accounted for. For example, if there are remain differences between the participants and non-participants for pre-unemployment earnings, entering the latter into the outcome regressions accounts for any impact these might have on outcomes and removes this effect from the estimated “programme effect”. Doing so provides a second layer of protection to the analysis, as only one of the two techniques (matching or the second stage regression) has to correctly construct the proper counterfactual in order for the estimates to be valid. The difference in the estimates is small, providing reassurance that the matching has done a good job in comparing alike non-participants to participants.
The choice of the research design is driven by the relatively broad eligibility criteria for CE coupled with the availability of rich administrative data. Alternative approaches, e.g. regression discontinuity designs, would have been feasible, but turned out to be less well suited in this context. Instead, the research design makes use of the rich available administrative data to match similar individuals along a number of dimensions. Capturing a person’s individual situation with precision is crucial for the matching. For example, the approach takes account of duration at the precise calendar quarter an individual enters CE. If this was not done, there would be a likelihood of comparing e.g. an individual with five years of registered unemployment to an individual with only one year. These individuals may then have very different lived experience of unemployment, with the individual at five years having already had the opportunity to participate in the range of DSP employment services for a long period of time yet still being subsequently unemployed. Correcting for unemployment duration is often used in impact evaluations and was also employed in a recent OECD evaluations of ALMPs in Latvia (OECD, 2019[7]) and Lithuania (OECD, 2022[8]).
Reviewing statistics from the evaluation demonstrates that matching has been successful in pairing CE participants to similar non-participants to ensure a solid comparison of the two groups. Table 4.4 of Chapter 4 shows that after matching, the imbalance that is seen between participants and all non-participants across different observable characteristics is significantly reduced. For example, earnings three years before participation are EUR 4 100 for participants and matched non-participants, compared to EUR 4 900 for all non-participants. Similarly on age, CE participants (mean age of 43.1 years) compare favourably to the matched non-participants (mean age of 42.9 years), relative to all non-participants (mean age of 40.8 years). The technical report (OECD/Department of Social Protection, Ireland/European Commission, Joint Research Centre, 2024[3]), provides further detail on this and on the accompanying sensitivity analyses conducted. The aggregated bias across all variables in the matched non-participant group compares favourably to thresholds in the academic literature, suggesting a good balance is achieved.
When individuals begin ALMPs (are “treated”) at different times in their unemployment spells, selecting “dynamically” into such measures, the set of individuals who were never treated may not provide a suitable comparison group to those who were treated. Individuals only become available for treatment if they stay in unemployment long enough. Conversely, one of the main reasons that some individuals do not get treated is because they are able to find jobs and exit unemployment quickly. This motivates an approach that does not simply compare the ever treated with the never treated, but rather compares those who begin treatment at a given point in their unemployment spell with those who have been unemployed for a similar duration but have not yet entered an ALMP. Note that individuals in both the treatment and control groups are potentially eligible to enter treatment, but individuals in the control group who are paired with individuals in the treatment group do not enter treatment in the same quarter as the individual entering treatment (although they may subsequently enter treatment). This ensures that the probability distribution of individuals subsequently finding a job or of later joining a programme is the same as the distribution for the similar treated individuals had they decided to wait longer as well. This “dynamic selection-on-observables” method was developed by Sianesi (2004[5]).
In this report, the dynamic selection-on-observables approach is implemented in conjunction with nearest neighbour propensity score matching. This entails the following:
1. Calculating propensity scores based on a rich set of covariates.
2. Matching exactly on calendar quarter and using elapsed duration in long-term unemployment to control for differences in the duration profile of participants and non-participants.
3. Within the groups defined in the second step, conducting nearest neighbour matching – pairing individuals with similar propensity scores.
4. Estimating treatment effects separately for each time horizon of interest (, the amount of time elapsed since the start of the CE, when the employment and earnings are measured). Denoting potential labour market outcomes (such as employment or earnings) for an individual () as , where under treatment and otherwise, the average treatment effect on the treated (for each is then:
The s are the key treatment effects reported for labour market outcomes in this analysis, looking at individuals annually until eight years after entering treatment (for individuals in the treatment group) or after being matched to an individual in the treatment group (for individuals not entering treatment). Given the exact matching on calendar quarter and year and the nearest neighbour matching, any time -specific effects are differenced out by construction.
A key identifying assumption in propensity score matching is that all outcome -relevant differences between programme participants and non-participants are captured in their observed characteristics. In other words, conditional on observed covariates, the selection into the treatment can be considered random (e.g. Imbens (2000[9])). If selection into programme is governed not only by observable characteristics but also by unobservable characteristics, which are correlated with the potential outcomes, then propensity score matching cannot produce unbiased estimates of treatment effects. To address this issue, the technical report (OECD/Department of Social Protection, Ireland/European Commission, Joint Research Centre, 2024[3]) includes a sensitivity analysis which combines propensity score matching with a difference–in-differences approach (Heckman et al. (1998[10]); Smith and Todd (2005[11])). Intertemporal changes in outcomes between participants are compared to changes in outcomes for the comparison group, where changes are measured relative to a pre-programme benchmark period.
Given the large sample sizes involved in the analysis, with several thousand participants per year starting CE, the analysis uses direct matching estimators. This has the benefit of exhibiting the smallest bias for all sample sizes, even though nearest neighbour matching has the drawback of higher variance estimates (Huber, Lechner and Wunsch, 2013[12]). With large sample sizes, the superior bias properties gain in importance given that the absolute difference in precision relative to more efficient estimators diminishes as the variances tend to zero asymptotically. The technical report (OECD/Department of Social Protection, Ireland/European Commission, Joint Research Centre, 2024[3]) reports variations of the estimation approach, using both regression weight adjustment (inverse-probability weighting) and alternative matching estimators (Kernel and n-nearest neighbours).
Under this framework, the dynamic selection-on-observables approach can only be used to estimate the treatment effect of the first CE participation in which individuals participate. Everything that happens after starting participation in the first ALMP measure is effectively treated as part of an individuals’ outcome, even if that entails not working due to further CE participation.
6.2.3. The report evaluates a comprehensive set of outcomes
The effects of ALMPs on employment probability have been widely studied, as documented by a meta-analysis by Card, Kluve and Weber (2018[6]) including employment probability estimates from 111 impact evaluations of ALMPs. While directly labour market related outcomes are crucial in the context of CE, there are wider considerations for CE in light of its social inclusion objective, such as impacts on outcomes such as health and engagement in education.
In the case of Ireland, the rich and comprehensive data available enable the CE evaluation to track a set of labour market outcomes, and to a lesser extent social inclusion outcomes, for a relatively long period. The outcomes are tracked continuously up to the eight years post the start of the programme for the cohort starting in 2013. Outcomes are calculated on an annual basis and tracked over time relative to the year of the programme start date, which is defined using the quarter of entry for the CE participants (for the treatment group) or that same calendar quarter for an individual in the control group who is matched to someone in the treatment group.
The fact that outcome data are only available annually limits the extent to which granular analysis of the timing of impacts is possible. However, this is not a major problem in the analysis of CE. CE is a long programme and has long-term objectives, i.e. short-term temporal impacts are much less of an analytical concern than in the case of support like job counselling, where the treatment primarily targets the ability of jobseekers to secure work rapidly. The administrative data allow the analysis of a long time series of post-participation outcomes, and whether e.g. an individual finds work some weeks earlier or later is of limited relevance to what their employment status looks like several years later.
Several outcomes’ variables are used to capture both the activation and the social inclusion objectives of CE (Table 6.1). Employment outcomes are captured by four indicators, namely gross annual earnings from dependant employment or self-employment, the probability of working at least one day in a given year, annual weeks in (most unsupported) employment and annual weeks in unemployment. The four indicators are complementary to each other, and jointly paint a detailed picture of labour market outcomes after CE participation.
Social inclusion outcomes are difficult to evaluate using the current administrative data
As for social inclusion outcomes, while they cannot be captured as directly as activation outcomes, health outcomes can be proxied by the receipt of the Disability Allowance, and engagement in further education after being eligible for CE is captured through the receipt of the Back to Education Allowance.
Disability Allowance is a weekly allowance paid to people aged 16-66 who have a disease or disability that has lasted, or is expected to last, for at least 1 year and significantly restricts their ability to work. Whilst the receipt of Disability Allowance can provide insight into underlying health conditions, it may not cover all aspects of them. Substantial mental disability is covered by Disability Allowance, but it is unlikely that it provides much insight into moderate mental health issues of participants, which can be barriers to engagement in local communities or social networks. All of these things might reasonably be considered a part of “social inclusion”.
Similarly, the services provided to local communities, the fabric of society in these communities and the usefulness of CE to social inclusion in these respects is not currently measured or observed. This will only be remedied with a proper definition of what social inclusion means for CE, with metrics defined to measure success and data collected to evaluate progress.
This section presents the main findings on the impact of CE on labour market and social inclusion outcomes. The analysis shows that CE improves labour market prospects of participants in the medium and long run, after a short initial negative lock-in effect. Additionally, CE contributes to enhancing social inclusion, notably by reducing the dependency on disability benefits of former participants. The results differ markedly across different subgroups of participants, highlighting that the programme does not play the same role for all types of jobseekers.
6.3.1. After initial lock-in effects, CE leads to more employment and higher earnings
The impact of CE on labour market outcomes follows a U-shaped pattern, initially showing a short negative effect directly after CE participation before generating positive outcomes in the medium and long-run (Figure 6.1). The phenomenon is observable in both employment among former CE participants, as well as their employment earnings. In the first two years after starting CE, there is a discernible decline in the number of weeks worked in the open labour market and employment earnings, highlighting the initial challenges faced by individuals when transitioning from CE to employment. This pattern can be explained by a “lock-in effect” wherein individuals have temporarily restricted access to the labour market during and directly after CE, as they might be unable to carry out part-time work in parallel to CE, have less time for job search activities, and might not receive other ALMPs in addition to CE that could be beneficial.
Note: 95% confidence intervals shown on vertical bars.
Source: Calculations based on Department of Social Protection (DSP) data.
Following this initial decline, the effect of CE on employment outcomes becomes positive. In the 4th year after starting CE, a former participant is predicted to work about three weeks more per year in the regular labour market, and to have annual earnings that are about EUR 1 000 higher as a result of CE. While the effect might seem relatively small, it is important to bear in mind that many CE participants face major employment barriers and would have been expected to work little and have low earnings, i.e. relatively speaking, these levels are not negligeable. In addition, it is important to notice that the estimates reflect the effect of CE as compared to other eligible non-participants, who may or may not benefit from other ALMPs, and not just jobseekers who do not receive any ALMPs. Put differently, the estimates should be interpreted as the effect of CE versus the effect of the average support pathway of an eligible jobseeker, and not the effect of CE versus no support at all.
The cumulative number of weeks in employment (taking account of negative impact directly after the start of CE) starts turning positive after five years, and cumulative earnings after six years. That is, the negative initial impact is more than outweighed after five and six years, respectively, pointing to a positive long-term effect of CE. For example, within seven years after starting CE, former participants are estimated to have worked a total of eight weeks more as a result of participating in CE and have had additional earnings of EUR 2 500.
This positive shift becomes apparent more rapidly when considering an indicator variable based on whether a person has any employment earnings in a given year. With this metrics, the impact turns positive as early as from the second year following CE, indicating a quick assimilation of former CE participants into some form of employment. In the third year after starting CE, former participants are 7 percentage points more likely to be in employment at some point during the year, and it takes only four years for the cumulative measure to become positive. The quicker convergence of this indicator of employment than the other labour market outcomes indicators might hint to a gradual move towards employment, with some former participants taking up intermittent or part-time work first after finalising CE, before moving on to more stable jobs.
Overall, the estimated effects of CE on labour market outcomes are encouraging. They all hint to improvements of the labour market situation after CE, suggesting that the programme can reconnect jobseekers with jobs in the open labour market in the medium and long run. One way of reducing the lock-in effect and decreasing the time it takes for the positive effect of CE on employment outcomes to unfold could be to explore making working hours more flexible throughout the CE spell and introduce counselling elements towards the end of the placement. For instance, some more “job-ready” participants could begin their placement working more than 19.5 hours per week to gain more intensive experience at the start of the programme. They could then reduce working hours to put a stronger focus on job search activities and e.g. benefit from counselling sessions with DSP counsellors, to maximise the chances of finding a job rapidly. Conversely those with multiple barriers might start with fewer hours and increase their weekly hours towards the end of their placement.
There are no long-term effects on unemployment benefits
In addition to the effects on employment and earnings, there are no discernible impacts on long-run registered unemployment. There is some increase to receipt of Jobseeker’s Benefit (JB), as CE participation confers social insurance contributions towards JB entitlement. Extra annual weeks of JB receipt peak at just over two in years 3 and 4, before falling back towards 0 over the estimated period and ending as insignificant in year 8 (see Figure 6.3, Panel F). However, when looking at unemployment benefits as a whole (considering both Jobseeker’s Allowance and JB together), this extra JB entitlement makes only a statistically significant impact in years 4 and 5 (at 2 weeks and 0.9 weeks respectively) before becoming insignificant later on.
Coupled with the results on incidence of employment, this suggests that much of the impact of CE falls on inactivity, via moving individuals who would previously have entered into inactivity into employment.
6.3.2. CE reduces disability receipt and enhances take-up of education subsidies
Besides the impact on labour market outcomes, the effect of CE on other outcomes, especially social outcomes, is also crucial considering the programme’s explicit objective to enhance social inclusion. Disability Allowance receipt and take-up of the back-to-education allowance post-CE participation are indicators of the broader effects CE generates, providing insights into possible health and education effects.
Participating in CE has an immediate and persisting positive effect on disability allowance receipt, which is paid to people with long-lasting injuries and disability (Figure 6.2). Already one year after starting CE, the probability of claiming disability benefits is lower among CE participants than in the comparison group, and the difference is statistically significant. That is, fewer former CE participants than comparable non-participants are in need of income support because they have a serious disease or disability that is expected to last for at least 1 year and prevents them from working. This effect continues to grow and cumulate over time. For instance, six years after starting CE, former participants are 6 percentage points less likely to receive disability allowance than people with similar characteristics, but who did not take part in CE. The accompanying technical report (OECD/Department of Social Protection, Ireland/European Commission, Joint Research Centre, 2024[3]) details some variation on the magnitude of the impacts on Disability Allowance receipt on different sensitivity analyses. This means care should be taken on being overly precise on the point estimate of the impact, though lower receipt was common across the different specifications.
There are several reasons that could explain this result. From a methodological perspective, it cannot be excluded with certainty that there is some residual unobserved difference between CE participants and eligible non-participants that persists despite the matching approach (see 6.2). For example, individuals with underlying unobserved health conditions might be less likely to take up CE, and these health problems could subsequently lead to the need for disability allowance receipt.
However, while some unobserved variable bias cannot be excluded, the matching procedure corrects for a lot of the differences between CE participants and eligible non-participants, and it is likely that other factors drive or at least contribute to the results. For example, past receipt of Disability Allowance, past employment and earnings and past occupation are all controlled for in the analysis. These may all help to explain present health. In addition to this, the impact on Disability Allowance receipt grows over time, which is less expected if individuals were already unhealthy at the beginning of the comparison period (where an immediate gap might be expected).
There is a significant body of economic and public health literature documenting a link between physical and mental inactivity and health (for example, see (OECD, 2015[13]; 2012[14]). Inactivity can lead to mental health issues, e.g. by increasing anxiety and reducing social connections, and is also associated with physical health effects, including through unhealthy behaviours. Positive links between employment and health have also been found (OECD, 2008[15]). CE keeps participants physically and mentally active, ensures that they have regular social interactions with CE co-workers and supervisors, and also provides a sense of purpose. All of these factors can benefit health outcomes, even more so in comparison to non-participants who do not engage in another activity or ALMP.
Note: 95% confidence intervals shown on vertical bars.
Source: Calculations based on Department of Social Protection (DSP) data.
Besides likely beneficial effects on health, CE also has some impact on jobseekers’ probability of engaging in education. More specifically, CE increases the likelihood of receiving the Back to Education Allowance (BTEA) in the medium term. BTEA, which can be granted for second and third level courses with a Quality and Qualifications Ireland accreditation, is in most cases only available to participants whose highest educational attainment level increases through the course. That is, BTEA is associated with an increase in human capital.
As is in the case of employment outcomes, the small positive effect of CE on BTEA receipt is not immediate. In the first year after starting CE, annual weeks of BTEA are lower for CE participants than for comparable non-participants. This initial reduction in BTEA take-up can be explained by the fact that CE participants are not available for BTEA while they are still on the programme, and might prioritise other activities, such as job search activities, directly after CE. However, already from year two after starting CE, the impact of CE on annual weeks of BTEA turns positive, highlighting that all else equal, participating in CE increases BTEA take-up, even though the effect is small. Three and four years after the start of CE, when the effect is strongest, it increases the expected time spent receiving BTEA by about 0.1 weeks per year, which is small, but statically different from 0. After that, the effect starts ebbing away over time, slowly approaching zero. Whilst the overall effect is relatively small when averaged over all participants, it does suggest that there is potential to help a small subset of CE participants engage better with education incentives offered by DSP.
6.3.3. Impacts of CE differ dependent on the characteristics of the participant
To gain a comprehensive understanding of the impact of CE, it is useful to break down the data into different sub-groups. The analysis on these sub-groups offers valuable insights into how different segments of the population are affected by the programme. The administrative data that are available allow for disentangling the effect of CE depending on age, gender, nationality, location (urban vs. rural) and family status.
Furthermore, for a subset of the population for whom PEX information is available, the impact can be separated depending on education, access to transport, level of English, willingness to travel, and other personal characteristics. However, as the limited number of individuals with PEX scores in the sample poses a challenge in establishing solid conclusions, the report only provides a brief overview of these results.
It is important to notice that the differences in the results between different subgroups do not correct for correlation between sub-groups. For example, the effects of CE vary depending on age and on gender, and, at the same time, female participants tend to be younger than their male peers. It is impossible to disentangle which subgroup (age or gender) drives the difference in the strength of the effects. Therefore, instead of interpreting the difference in results as being causal effects of belonging to a specific subgroup, they should rather be interpreted descriptively: The effects of CE are not the same for men and women, which may be due to gender or any other difference between the pool of male and female participants.
Prime-age individuals benefit particularly strongly from the programme
The effect of CE varies markedly across age groups (Figure 6.3). Employment outcomes improve strongly for prime-age individuals and young jobseekers under 30, while participants over 50 benefit a lot less from better labour market prospects after CE. For example, 4-5 years after CE, total annual weeks worked are expected to be six weeks higher due to CE participation for people under 30 as well as 30-50 year-olds, whereas the effect is close to zero for the 50+. Similarly, annual employment earnings are predicted to increase only very moderately due to CE participation for older jobseekers, and only six years after starting CE, while increases are stronger and faster for younger participants, in particular prime-age participants who seem to benefit from rapid and sustained income increases. Prime-aged participants can expect to earn EUR 2 300 more per year four years after starting CE, and EUR 3 000 after seven years, against EUR 800 and EUR 900 for the 50+, respectively.
One likely driver of this discrepancy is the role CE plays for different ages. While the activation angle is central for most young and prime-aged jobseekers, for whom the main objective is to prepare them for the primary labour market, CE is sometimes used as a pathway into retirement in the case of older participants. For jobseekers just a few years below retirement age, the focus might lie on other aspects of CE, such as contributing to the community and increasing social inclusion, rather than connecting them with a job in the open labour market.
The supposedly larger focus of CE on outcomes other than employment for older CE participants also matches results of the social inclusion outcome indicators, most notably the receipt of Disability Allowance as a proxy for health outcomes. Indeed, the probability of receiving Disability Allowance decreases strongly due to CE participation among older jobseekers, much more so than among their younger peers. For instance, former CE participants aged 50+ face a probability of claiming Disability Allowance five years after starting CE that is 10 percentage points lower than it would have been without CE participation, pointing to substantial beneficial health effects. This effect is quantitatively large and persistent, as after eight years, the predicted probability of needing Disability Allowance is still reduced by 8 percentage points. For younger age-groups, the effect is lower, in line with a lower baseline probability of claiming Disability Allowance at younger ages. Nevertheless, the effect is negative for all age-groups, i.e. lowering the probability of claiming Disability Allowance, suggesting that CE has beneficial health effects at all ages.
There is also some evidence that CE helps young participants to invest in their human capital. Four years after starting CE, participants under 30 claim the Back-to-Education-Allowance on average 0.4 weeks more than comparable peers of the same age who did not take up CE. While small, this effect is statistically significant. For older groups of CE participants, the effect is close to zero.
EU migrants and women benefit from CE participation
In terms of labour market outcomes, most notably weeks of employment and the receipt of any positive earnings in the year, post-2004 EU accession country migrants (hereafter “EU migrants”) and women benefit more than the average CE participant (Figure 6.4). Five years after the programme, EU migrants are earning an average of around EUR 3 600 per year more than non-participants, far above the cross-group average of EUR 1 250. For women this figure is around EUR 2 300. Both groups also see decreases to receipt of Jobseeker’s Allowance, hinting to positive effects, even though this decrease is only statistically significant for women in years six and seven after participation.
In contrast to this, in the medium term, three to four years after participation, both EU migrants and women see above average increases to receipt of Jobseeker’s Benefit. These increases may represent stronger engagement with the labour market, with previous participation on CE qualifying towards Jobseeker’s Benefit eligibility. However, the impact on Jobseeker’s Benefit abates by year five, while positive employment effects persist (e.g. in terms of weeks worked), suggesting that the benefit is assisting individuals well in the transition period between CE participation and job entry.
As stated above, the larger employment effects for EU migrants and women could be due to a correlation with other subgroups rather than to the fact of being a migrant or female, respectively. For example, differences in the age structure between male and female participants, as well as EU migrants and non-migrants are possible factors contributing to these differences, and results should be interpreted cautiously.
Note: Community employment (CE); Back to Employment Allowance (BTEA).
Source: Calculations based on Department of Social Protection (DSP) data.
Participants in activation jobs and on the health and social care schemes earn more
In terms of labour market outcomes, participants in the activation strand benefit more than CE participants in social inclusion placements (Figure 6.5). For example, five years after starting CE, participants in an activation CE placement benefit from an estimated increase of EUR 2 000 in annual earnings due to the programme, whereas the effect is close to 0 for social inclusion participants. One possible explanation for this is the stronger focus on reconnecting jobseekers with the labour market in the activation strand than in the social conclusion strand. On average, participants in social inclusion placements tend to face bigger barriers to employment than their peers on activation placements, with earnings that are 7% lower in the five years prior to the scheme. However, it may be that this differential in outcomes is driven by unobserved differences between participants in the two placement types. Attempts to compare similar participants in the two schemes were not fruitful, so it is likely that innate differences do explain some of the differences in programme effectiveness. This gives further weight to considering what activation and social inclusion mean in practice and how they apply to individuals, not to jobs, as is the current policy practice. Bringing the focus back to the individual may also pave the way for case officers to have more structured conversations and engagement with their clients about their journey back to the labour market.
Source: Calculations based on Department of Social Protection (DSP) data.
As for scheme types, labour market outcomes post participation tend to be better for participants in Health and Social Care. For example, four years after participating in CE, participants of one of these specific schemes have estimated annual income increases of about EUR 4 000, much higher than the average impact for all scheme types. One of the reasons may lie in the higher degree of specialisation of these specific schemes, which may make it easier for participants to identify job vacancies in need of the job skills they acquired during CE.
PEX subgroups highlight importance of education
The sub-group analysis presented thus far has been conducted without making use of PEX information. This is due to the limited availability of PEX data, particularly for early CE cohorts, resulting in a relatively small sample size of sub-groups with PEX data and, consequently, less reliable findings.
Nonetheless, it is noteworthy that when conducting subgroup analysis with the inclusion of PEX data, there is some preliminary evidence suggesting that the program’s impact could be stronger among participants with higher levels of education. More details on these findings are included in the accompanying technical report (OECD/Department of Social Protection, Ireland/European Commission, Joint Research Centre, 2024[3]).
A higher effectiveness of the programme for people with higher educational attainment, which is in line with the greater labour market impact for participants in the activation strand rather than the social inclusion strand, highlights the important role training modules during CE can play, as a means of increasing educational attainment of participants. It provides some evidence that CE may be effective at helping to unlock latent labour market potential for individuals that have skills to succeed in the labour market, but who may be lacking confidence after a period of extended unemployment. In addition, it also underscores the need to ensure the CE has sufficient support mechanisms in place that are tailored to participants with the largest labour market barriers.
6.3.4. The positive long-term effects of CE compare favourably with estimates from studies of similar programmes in other countries
This section compares the estimated effects of CE in Ireland with those of similar studies in other countries, placing them in the context of the results of two meta-analyses. The first, conducted by Card, Kluve and Weber (2018[6]), covers a total of 49 countries and- summarises estimates from over 200 impact evaluations of ALMPs. Of these, 34 impact evaluations include point estimates of the employment effects of public works programmes comparable to the ones in Ireland. The second meta-analysis covers projects funded by the EU’s European Social Fund (ESF) and includes estimates from seven studies examining public works programmes as well as three classified as mixed interventions, combining e.g. public employment with training components (European Commission and Ismeri Europa, 2023[16]).
The discussion in this section focuses only on the results for employment. The meta-analysis by Card, Kluve and Weber (2018[6]) does not provide estimates of the effects on other outcomes analysed for Ireland, such as earnings or weeks worked. While the meta-analysis of the ESF programmes does contain some estimates on measures such as earnings, the number of estimates is insufficient to make meaningful comparisons by programme type.
Compared with the results of the meta-analyses, the estimated effects for Ireland are in the middle of the distribution for shorter-time horizons and toward the upper end of the distribution for long-time horizons (Figure 6.6). The effects in the first two years after entering the programme are very close to the median estimates of the other studies, with negative effects on employment probability ranging from negative two and negative 6 percentage points. The effects at time horizons longer than two years, by contrast, are considerably more positive in the case of Ireland. The average estimated effect for CE over time horizons longer than two years is 8 percentage points, higher than any of the estimates reported in the Card, Kluve and Weber (2018[6]) meta-analysis and at the 75th percentile of the ESF programme estimates.
One possible explanation for the relatively high positive effect in Ireland relates to the fact that CE targets long-term unemployed people (those unemployed for at least 12 months). If one focuses only on the studies (in the meta-analyses) that examine the effect of public works programmes on the long-term unemployed, the results are more favourable than the overall estimates. For example, at time horizons of between one and two years after entry into the programme, the median estimate for this subset of studies is 3.0 percentage points in the ESF programme estimates (compared to -2.5 percentage points when including also other groups of unemployed in the analysis). However, these conclusions are tentative given the relatively small number of estimates pertaining only to the long-term unemployed (9, compared to 16 for all the estimates in the ESF programme study relating to public employment programmes).
CE: Community Employment. ESF: European Social Fund.
Note: Short, medium and long-term effects respectively refer to effects up to one year, 1-2 years, and more than two years after programme completion. For Ireland, results refer to 0-12, 13-24 and 24-108 months after beginning the programme. As such, the observation periods are similar, but potentially not fully aligned. The box and whisker plots for the other studies refer to the 5th, 25th, 75th and 95th percentiles of the estimates (i.e. the black line at the bottom refers to the 5th percentile and the bottom of the blue box refers to the 25th percentile). Point estimates are included in the chart even if they are statistically insignificant. The studies presented adopt various research designs and econometric techniques – the results for Ireland use nearest-neighbour propensity score matching.
Source: Card, D., Kluve, J. and Weber, A. (2018[6]), “What Works? A Meta Analysis of Recent Active Labor Market Program Evaluations”, Journal of the European Economic Association, Vol. 16/3, pp. 894-931, https://doi.org/10.1093/jeea/jvx028; European Commission and Ismeri Europa (2023[16]), Meta-analysis of the ESF counterfactual impact evaluations – Final report, Publications Office of the European Union, https://doi.org/10.2767/580759 and calculations based on administrative data from Greece.
In interpreting the results, it is worth noting that, while the point estimates in the comparison studies are generally positive, they are not necessarily statistically significant. Figure 6.6 plots all the point estimates in the studies found in the meta-analysis by regardless of statistical significance. In fact, only a minority (3 out of 11) of the studies in the Card, Kluve and Weber (2018[6]) meta-analysis find positive and statistically significant results over the long term for public works programmes.
This chapter presents evidence of CE’s efficacy in supporting long-term unemployed people. It shows that the programme contributes to improving labour market outcomes in the medium-to-long term, especially for younger and prime-aged participants. After an initial lock-in effect during and shortly after CE, former participants benefit from significantly more employment and higher earnings a few years after starting the programme. The employment effects after two years are remarkably strong also in an international context, with the long-term effects higher than three-quarters of similar studies done in other countries. CE also contributes to improving some non-labour market related outcomes, in particular a lower likelihood of receiving Disability Allowance and a somewhat higher probability of receiving the Back to Education Allowance, pointing to possible beneficial effects on health and engagement in education. These latter outcomes can be seen as an indication that CE also contributes to improving social inclusion.
The effects of CE vary markedly across different groups of participants. While the effects of CE on labour market outcomes are generally better for younger participants, the programme is less effective at connecting older jobseekers with jobs in the open labour market. Nevertheless, jobseekers close to retirement age seem to benefit from social inclusion effects of CE. As a result, further scrutiny will be worthwhile to ensure that CE is tailored to individual needs and maximises the benefits it can achieve for different types of jobseekers. For example, more flexibility in terms of working hours, job tasks and training, could allow enhancing the match between CE placements and the profile of specific jobseekers.
While positive, the results also reveal that there is a lock-in period during and a few years after CE. During this lock-in period, CE has a negative effect on labour-market outcomes before giving way to medium and long-term beneficial effects. Therefore, it will be important to take action to shorten this lock-in period as much as possible. For example, combining CE with other types of support, e.g. counselling services by DSP, could help making participants job ready more swiftly after the end of CE. Smooth transitions into the labour market could also be facilitated by strong involvement of employers and business organisations, to provide insights on skill needs and help review the match between CE placements and training during CE and local labour market needs.
References
[11] A. Smith, J. and P. E. Todd (2005), “Does matching overcome LaLonde’s critique of nonexperimental estimators?”, Journal of Econometrics, Vol. 125/1-2, pp. 305-353, https://doi.org/10.1016/j.jeconom.2004.04.011.
[6] Card, D., J. Kluve and A. Weber (2018), “What Works? A Meta Analysis of Recent Active Labor Market Program Evaluations”, Journal of the European Economic Association, Vol. 16/3, pp. 894–931, https://doi.org/10.1093/jeea/jvx028.
[1] DSP (2021), Report of the Inter-Departmental Group to explore the most appropriate organisation arrangements, including which Department should host the Social Inclusion schemes, Department of Social Protection, Ireland, https://www.gov.ie/ga/foilsiuchan/a3572-report-of-the-inter-departmental-group-to-explore-the-most-appropriate-organisation-arrangements/.
[2] DSP (2015), An Analysis of the Community Employment Programme, Department of Social Protection, Ireland, https://assets.gov.ie/37356/14ef34b753414dcbb5ca5a8f23277f22.pdf.
[16] European Commission and Ismeri Europa (2023), Meta-analysis of the ESF counterfactual impact evaluations – Final report, Publications Office of the European Union, https://doi.org/10.2767/580759.
[10] Heckman, J. et al. (1998), “Characterizing Selection Bias Using Experimental Data”, Econometrica, Vol. 66/5, p. 1017, https://doi.org/10.2307/2999630.
[12] Huber, M., M. Lechner and C. Wunsch (2013), “The performance of estimators based on the propensity score”, Journal of Econometrics, Vol. 175/1, pp. 1-21, https://doi.org/10.1016/j.jeconom.2012.11.006.
[9] Imbens, G. (2000), “The role of the propensity score in estimating dose-response functions”, Biometrika, Vol. 87/3, pp. 706-710, https://doi.org/10.1093/biomet/87.3.706.
[4] Lechner, M. and C. Wunsch (2013), “Sensitivity of matching-based program evaluations to the availability of control variables”, Labour Economics, Vol. 21, pp. 111-121, https://doi.org/10.1016/J.LABECO.2013.01.004.
[8] OECD (2022), Impact Evaluation of Vocational Training and Employment Subsidies for the Unemployed in Lithuania, Connecting People with Jobs, OECD Publishing, Paris, https://doi.org/10.1787/c22d68b3-en.
[7] OECD (2019), Evaluating Latvia’s Active Labour Market Policies, Connecting People with Jobs, OECD Publishing, Paris, https://doi.org/10.1787/6037200a-en.
[13] OECD (2015), Fit Mind, Fit Job: From Evidence to Practice in Mental Health and Work, Mental Health and Work, OECD Publishing, Paris, https://doi.org/10.1787/9789264228283-en.
[14] OECD (2012), Sick on the Job?: Myths and Realities about Mental Health and Work, Mental Health and Work, OECD Publishing, Paris, https://doi.org/10.1787/9789264124523-en.
[15] OECD (2008), “Are All Jobs Good for Your Health? The Impact of Work Status and Working Conditions on Mental Health”, in OECD Employment Outlook 2008, OECD Publishing, Paris, https://doi.org/10.1787/empl_outlook-2008-6-en.
[3] OECD/Department of Social Protection, Ireland/European Commission, Joint Research Centre (2024), “Technical report: Impact Evaluation of Ireland’s Active Labour Market Policies”, OECD, Paris, http://www.oecd.org/els/emp/Ireland_ALMP_Technical_Report.pdf.
[5] Sianesi, B. (2004), “An Evaluation of the Swedish System of Active Labor Market Programs in the 1990s”, Review of Economics and Statistics, Vol. 86/1, pp. 133-155, https://www.jstor.org/stable/3211664.
Note
← 1. Specific; Measurable; Achievable; Relevant; Timebound.