7. Impact evaluation of Tús
This chapter presents an evaluation of Tús, a small-scale public works programme in Ireland. The evaluation is based on a rich set of administrative data, which includes background information on jobseekers, details on their participation in Tús, and individual outcomes after participation. Using a methodological approach to correct for selection into the programme, the evaluation shows that Tús has a modest positive impact on jobseekers exiting the unemployment register and a long-lasting positive effect on earnings from employment.
As a policy response to the dramatic increase in the unemployment rate in the aftermath of the Global Financial Crisis (GFC), where a large volume of jobseekers registered with Ireland’s Public employment service (PES) developed into a large number of long-term unemployment people some years later, a number of new active labour market programmes (ALMPs) were introduced along with a reconfiguration of how employment services would be delivered. Tús, a community work placement scheme offering short-term work opportunities for long-term unemployed people, was one of these measures. Tús was launched in December 2010, became operational in mid-2011 and remains in operation. Up to 5 000 work placements opportunities were initially announced, with an additional 2 500 Tús placements added as part of the 2013 Budget package, in line with the commitments set out in the Action Plan for Jobs (Department of Jobs, Enterprise and Innovation, 2012[1]) and Pathways to Work 2013 (Department of Social Protection,, 2013[2]). The counterfactual impact evaluation (CIE) presented in this chapter is the first evaluation of Tús since its introduction in mid-2011.
As is discussed earlier in Chapter 6, research in many countries has shown that despite the (generally) negative findings on public work programmes, for long-term unemployed individuals, the effects are better in some countries and cases (Card, Kluve and Weber, 2018[3]). Among estimates of over 800 separate programmes, recent findings on public employment programmes are relatively rare but the average impacts from ALMP participation for participants entering from long-term unemployment and women are larger.
The evaluation compares participants in Tús between mid-2011 and the end of 2018 with similar individuals who were eligible for selection to Tús during the same period but did not participate. The chapter outlines the choice of the analysis period, the outcomes and the outcome period and the background information available, and the methodology to model participation in Tús and labour market outcomes. After presenting the main results, as well as results for subgroups, the chapter concludes with a discussion of findings.
This section discusses the analytical approach to evaluating Tús and the methodology that is used in the report. It outlines the relevance of the selection and referral process and how the eligibility criteria for Tús influence the methodology.
7.2.1. The differences in scheme characteristics between CE and Tús underpin the variation in empirical approaches
Inverse probability weighting is the method applied in the Tús evaluation after propensity scores have been generated. Inverse probability weighting retains all units and weights the control group to resemble the treatment observations. The appropriateness in the weighting approach for Tús is that all long-term jobseekers are liable to be referred to Tús through the random referral mechanism. Accordingly, all eligible jobseekers are retained in the sample. In contrast, the matching approach adopted in the Community Employment (CE) evaluation reflects participants’ capacity to self-select into the scheme.
7.2.2. Enrolment to Tús is ongoing, with no seasonal pattern
The eligible population consists of those who, from the administrative data available, meet the eligibility criteria for Tús participation (Chapter 3) which is an eligible unemployment spell (and registered on the Live Register) of at least 12 months. Figure 7.1 depicts the number of individuals who were eligible for selection to Tús in each quarter but did not participate in Tús. The rapid decrease from late 2015 is due to referrals to JobPath as until mid-2018, people who were referred to JobPath could not participate on Tús (or CE, see Chapter 6).
Tús participants make up a small proportion of the eligible population in each quarter with just 1-5% of the eligible population in each quarter starting an episode of Tús. As noted in Chapter 3, Tús participants are defined as those whose episode meets the minimum treatment threshold of 30 days, as is the case with the evaluation of CE.
Note: Only Tús episodes meeting the minimum treatment threshold of 30 days are included in the analysis.
Source: Calculations based on Department of Social Protection (DSP) data.
7.2.3. Referral to Tús is random but participation is not
As outlined in Chapter 3, jobseekers are randomly selected from the Live Register, Ireland’s register of jobseeker claims, and referred to Tús for interview by the Implementing Body (IB). Random selection approximates the conditions for a randomised control trial (RCT), where, on average, both arms of a trial have identical values across relevant variables and so there is no specific selection into the treatment. In an RCT, the treatment variable is disconnected from variables that influence the outcome, with no other avenue for treatment to be connected to outcome. Consequently, any differences in observed outcomes between the treated and the control group can only be due to the treatment.
Even where referral to Tús is random, participation is not. After jobseekers are referred to Tús, they decide whether to participate or not. Their willingness to take up a Tús placement may be due to greater motivation (in which case participants are positively selected with regard to the other eligible jobseekers) or due to greater need of the placement (in which case participants are negatively selected with regard to other eligible jobseekers).
Indeed, the descriptive statistics in Chapter 4 illustrate some differences between Tus participants and other eligible jobseekers across key variables. Given that these differences may correspond to better or worse outcomes, an appropriate methodology is required to account for selection into Tús. This adjustment results in a treatment group (Tús participants starting in a given quarter) and a control group (jobseekers who are eligible but do not start). Comparing the treatment and control groups reveals the impact of the programme.
The “Future Treated” are included in the analysis
In any given quarter, several members of the control group may participate in Tús in the future. The choice of how to account for the future treatment is between excluding all future treated cases from the control group and allowing all future treated cases into the control group. The main results for Tús in this evaluation include the future treated in the control group. The accompanying technical report (OECD/Department of Social Protection, Ireland/European Commission, Joint Research Centre, 2024[4]) develops further sensitivity analyses in this regard and confirms that the result is not dependent on this choice.
Administrative data underpin the validity of the analysis
All of the analytical work rests upon a set of assumptions about the background data and how Tús participation is represented in the data, which are considered further in the technical report. A key feature of this OECD-EC project is the use of administrative data. Indeed, the availability of rich and comprehensive administrative data is often essential for counterfactual analysis, facilitating a detailed account of people’s previous labour market experience. The absence of education data (outside of the subset with complete values for PEX) is one notable data weakness given the objective of estimating the impact on employment outcomes.
However, the analysis proceeds on the basis that the detailed data on social insurance contributions, jobseeker claims and employment history and the long period it covers are satisfactory to model the effect of Tús participation on outcomes, even in the absence of complete coverage of education levels. The availability of longitudinal data, which allows complex histories to be documented, is critical in meeting the assumptions on which counterfactual analysis is built.
7.2.4. The probability of starting Tús is calculated by quarter
Selection and referral to Tús occurs on an ongoing basis and therefore there is no enrolment period (as there may be with education courses for example). The analytical approach is to assess eligibility for the treatment and control population on a quarterly basis capturing movements in and out of eligibility throughout the analysis period.
As with Chapter 6, the analysis of Tús applies the dynamic selection-on-observables approach to compare people who commence Tús in a quarter to people who are eligible but have not commenced Tús. Given the labour market variability over the period 2011 to 2018, the employment prospects of long-term unemployed people were perhaps different in 2011 to those facing long-term unemployed people in 2018. For this reason, each quarter is treated entirely separately for the purposes of modelling participation in Tús. Only individuals in the same calendar year quarter are compared with each other when allocating probability of participant in Tús in that quarter.
The first step is to model participation in Tús by comparing participants in each quarter between Q3 2011 and Q4 2018 to the set of non-participants in the same quarter. This generates a set of propensity scores, which is a measure of the probability of starting Tús in a given quarter. The status of beginning Tús in a given quarter is a binary outcome and the score that is generated is a continuous measure between 0 and 1, taking into account employment history, including earnings in the calendar years prior to long-term unemployment, unemployment duration, a measure of total duration in receipt of unemployment benefits and, separately, the number of unemployment spells over which this duration accrued. Demographic characteristics such as age, sex, nationality and location are also included. The output of this exercise is a set of weights generated from the propensity score. These weights are used to ensure the control group is similar to the treatment group on the basis of the probability of each individual starting Tús in a given quarter.
Having generated propensity scores, as is done in Chapter 6, inverse probability weighting is the method applied in the Tús evaluation. Rather than selecting control units to be in the comparison between treatment and control, inverse probability weighting retains all units and weights the control group to resemble the treatment observations. In this sense, the only distinction between the eligible population (described in Chapter 4) and the control group (described here) is the application of weights. The appropriateness for Tús is that, as outlined in Chapter 3, all long-term jobseekers are liable to be referred to Tús. Accordingly, all eligible jobseekers are retained in the sample. In contrast, eligible candidates self-select into Community Employment (CE).
Once each participant and eligible non-participant has been allocated a weight based on the likelihood of starting Tús in a given quarter, the second step is the outcome estimation, using weighted least squares regression. This takes the same “doubly robust” approach as Chapter 6, where the second stage regression also controls for a range of characteristics.
7.2.5. Adjusting for differences between participants and non-participants allows similar individuals to be compared to each other
The re-weighting that is conducted to account for differences in individuals is successful in ensuring that only similar treatment and control individuals are compared to each other. The effectiveness of the re-weighting using propensity scores can be examined by looking at how similar control and treatment groups are in the periods before programme eligibility – for example, by looking at how similar past earnings and unemployment histories are for the treatment and control groups. Doing this for the Tús evaluation confirms that the re-weighting procedures are successful in achieving this similarity (further details of these comparisons are provided in the accompanying technical report (OECD/Department of Social Protection, Ireland/European Commission, Joint Research Centre, 2024[4])). This has two implications. Firstly, it shows that despite random selection for referrals, there is non-random selection into participation, such that participants (the treated) differ systematically from non-participants (the controls). Second, it shows that the evaluation methodology is successfully able to adjust for this non-random selection, such that estimates for Tús impacts can be considered “as good as random” and show causal impacts of Tús participation.
7.2.6. Outcomes are labour-market focussed and, for the early years of Tús, are observed over eight years
Tús resembles in many ways CE in respect of the placement types, and eligible organisations, the increase on the weekly jobseeker payment rate, the social insurance sub-class and the 19.5 hours per week requirement. In contrast to CE, which categorises placements as “activation” or “social inclusion”, Tús does not categorise any of its placements as “social inclusion”. For that reason, the main outcomes the evaluation of Tús focusses on are labour market outcomes. Annual earnings from employment, weeks of employment and weeks on the unemployment register (the Live Register) are the primary outcomes considered in this chapter. Outcomes are observed until the end of 2021 and include earnings and receipt of jobseeker payments measured at various points after the start of Tús participation, or the point at which jobseekers are eligible for Tús but not participate and form part of the control. This allows for a longer-term view of outcomes, acknowledging that the return to employment from long-term unemployment may not be immediate and that the employment benefits of public works programmes may take time to accrue. For the earliest quarters of Tús participation, in 2011, outcomes are available up to eight years later; for later quarters, those commencing in 2018, only a shorter horizon up to 2021 is available.
This section presents the results from the counterfactual impact evaluation of Tús, looking at the impact of Tús participation on different outcome measures and across different groups of individuals.
7.3.1. Tús has a small positive and long-lasting effect on earnings
Participation on Tús has a positive impact on earnings from employment in the years following participation (Figure 7.2, Panel A). There is a moderate decline in earnings in the year following participation before the positive effect of Tús participation is reflected in earnings from employment in subsequent years. Earnings of participants continue to rise gradually in the years after Tús participation. Earnings data are available for all participants for at least three years following participation. In the 3rd year after starting Tús, a former participant has, on average, annual earnings from employment that are over EUR 1 100 greater than an eligible non-participant. While earnings information up to eight years after beginning Tús is not available for every participant, for participants from the earlier years of the analysis period, the positive effect on earnings from Tús continues for many years after participation. Former participants experience earnings of approximately EUR 1 600 higher than eligible non-participants in the 7th year after beginning Tús.
Alternative outcome measures, such as weeks in unsubsidised employment, are consistent with this finding. Compared to eligible non-participants, Tús participants spend three weeks per year more in unsubsidised employment three years after participation starts, rising to four weeks more in employment in the 6th year. The binary measure of any earnings from employment in a year tells a similar story. This is the same probability of employment outcome measure as in Chapter 6. In the 3rd year after starting Tús, former participants are 7 percentage points more likely to be in employment at any point in the year.
Note: 95% confidence intervals shown on vertical bars. The earnings data for the 8th year outcomes are not available every participant, resulting in a smaller number of observations, which is reflected in the increasing confidence intervals for the 5th-7th years. The count of Live Register weeks is the annual sum of weeks signing for credits, weeks on a casual claim or a core Live Register claim. Data in Panel C includes only primary welfare payments per year.
Source: Calculations based on Department of Social Protection (DSP) data.
7.3.2. Tús modestly lowers the likelihood of becoming unemployed
Tús has a more modest impact on returns to the unemployment register. Following participation in Tús, participants spend fewer weeks on the unemployment register. In the 4th year after participation former participants spend one week less on the Live Register than eligible non-participants (Figure 7.2, Panel B). An alternate outcome measurement is the probability of being in employment, measured as the presence or absence of earnings from employment in a given year. This shows an increase in the incidence of employment for Tús participants. This rises to 8 percentage points after four years and remains positive thereafter. This is considered further in the technical report (OECD/Department of Social Protection, Ireland/European Commission, Joint Research Centre, 2024[4]).
As illustrated in Figure 7.2, Panel C the reduced presence on unemployment register attributable to Tús is not due to receipt of other benefits provided by DSP. In the years following the start of Tús participation, participants spend fewer weeks in receipt of another welfare payment than eligible non-participants (a difference of approximately three weeks in the 3rd year after participation). Thus, the decrease in unemployment benefit claims is not associated with an increase in other (non-unemployment-related) benefits.
7.3.3. The effects of Tús vary slightly across different groups of participants
While the overall result provides an estimate of the impact of Tús, a key policy question is how the effect is distributed across certain groups. For example, a programme may have a particularly strong or weak effect on particular age groups – which should inform how the intervention should be targeted.
One caveat that should be attached to this is that interpretation is not as straightforward once results are disaggregated by sub-group. Sub-group analysis places the focus solely on the characteristic that differentiates the levels of the sub-group but any difference in outcome is not solely attributable to this. In other words, the fact of being under 30 or female is not necessarily the sole attribute driving better (or worse) results – female Tús participants may also be younger and so the sub-group analyses should not be interpreted in an exclusively causal way.
The impact of Tús varies slightly by gender, with women experiencing a slightly greater boost to their earnings in comparison with men. Figure 7.3, Panel A shows that, compared to the non-participants, women who completed Tús earned approximately EUR 1 300 more in the 3rd year, while men in the same position earned just under EUR 1 000 more than similar men who did not participate in Tús. The result for both genders combined is also included for reference.
The impact of Tús is similar when comparing people with unemployment durations of 1-3 years and those with durations greater than three years. (Figure 7.3, Panel B). When analysed by broad age groups, the youngest cohort experience the greatest boost in earnings following Tús participation. In the 4th year after starting Tús, those in the youngest cohort see a EUR 1 600 increase in earnings from employment whereas 30-50 year-olds see a EUR 1 300 increase. Earnings of participants over 50 years rise by only EUR 1 000 (Figure 7.3, Panel C). The effect is enduring, in particular for the youngest cohort, whose earnings in the 7th year after Tús are almost EUR 2 000 greater than eligible non-participants of the same age.
Source: Calculations based on Department of Social Protection (DSP) data.
The period in which participants commence Tús is also examined. These are periods of time when labour market conditions varied significantly. As Tús was rolled out, long-term unemployment was on the increase, then peaked in 2014 and declined sharply over the 2016-2018 period (Chapter 2). For this reason, each era is considered separately in Figure 7.4 which shows that the positive impact of Tús is broadly similar across very different labour market conditions. The exception is the 2011-12 period, with a much smaller sample size and when the programme was ramping up, the impact remains positive but occurs more gradually, and remains lower, than in other periods.
Source: Calculations based on Department of Social Protection (DSP) data.
This chapter presents evidence of the impact of Tús on the employment prospects of long-term unemployed people. It shows that the programme contributes to increasing the earnings of participants and this effect persists in the long-term.
Over the analysis period 2011-18, eligibility for the Tús programme was restricted to jobseekers (primarily people in receipt of Jobseeker’s Allowance) who have been unemployed for 12 months. Since 1 July 2022, people claiming Disability Allowance once they are over 18 years of age and unemployed are allowed to participate without a qualifying period in terms of the duration of benefit receipt. This analysis focusses on the impact of Tús on jobseekers and not cohorts who became eligible more recently.
The findings on Tús hold across the economic cycle. The period under consideration ranges from the peak in the number of registered unemployed (short-term and long-term together) – 470 284 in July 2011 – through to the peak in long-term unemployment in mid-2014 and onwards to the end of 2018 when the number of long-term unemployed people in receipt of unemployment benefits fell below 80 000.
While this analysis is an estimation of the difference Tús makes to the employment prospects of jobseekers, these positive results should also be contextualised by the geographical distribution of Tús schemes, which are predominantly located in less affluent areas. Accordingly, in its provision of services, Tus could have a re-distributive impact (see Chapter 4).
A prerequisite of successfully moving from being unemployed to employment is that the public employment services have a range of programmes from which to choose a suitable progression pathway for each jobseeker. As the evidence base for the effectiveness of ALMPs offered to jobseekers grows, the analysis in this chapter suggests that Tús could be considered as a programme with a positive, albeit small, impact on earnings.
References
[3] Card, D., J. Kluve and A. Weber (2018), “What Works? A Meta Analysis of Recent Active Labour Market Program Evaluations”, Journal of the European Economic Association, Vol. 16/3, pp. 894-931, https://doi.org/10.1093/jeea/jvx028.
[1] Department of Jobs, Enterprise and Innovation (2012), “Action Plan for Jobs 2012”, Action Plan for Jobs 2012, https://enterprise.gov.ie/en/publications/publication-files/action-plan-for-jobs-2012.pdf (accessed on 6 December 2023).
[2] Department of Social Protection, (2013), “Pathways to Work 2013”, Pathways to Work 2013- 50 Point Plan to Tackle Long-term Unemployment, https://merrionstreet.ie/en/wp-content/uploads/2013/07/pathways-to-work-2013-with-cover.pdf (accessed on 6 December 2023).
[4] OECD/Department of Social Protection, Ireland/European Commission, Joint Research Centre (2024), “Technical report: Impact Evaluation of Ireland’s Active Labour Market Policies”, OECD/Department of Social Protection, Ireland/European Commission, Joint Research Centre, OECD, Paris, http://www.oecd.org/els/emp/Ireland_ALMP_Technical_Report.pdf.