Executive summary
Rigorous evaluation of public policies and programmes is a key step in informing policy making. Sound evidence on what works and for whom helps governments to achieve their strategic objectives. In recent years, the increasing availability of rich administrative data and computing power to conduct statistical analysis has meant that an increasing number of OECD countries are now able to conduct impact assessments of their policies using these rich data. Canada is an exemplar in this respect and the work of its employment and social affairs ministry, Employment and Social Development Canada (ESDC) in planning and conducting evaluation in collaboration with officials from Provinces and Territories (PTs) provides a good example of how to deliver such assessments. Knowing whether a policy is effective at achieving its stated aims and offers value for money is imperative to the sound utilisation of public funds and ESDC is now able to have fully informed discussions about its active labour market policy.
Prior to the pandemic, Canada’s federal government invested around CAD 5 billion annually to deliver a range of active labour market policies (ALMP) to support individuals. The Labour Market Development Agreements (LMDA) – bilateral labour market transfers with provinces and territories – are the largest funding stream within this investment and the programmes that can be offered in it are defined in federal legislation. They span support and coaching for jobseekers to look for work, training to enhance skills, and recruitment subsidies and job creation to foster employment opportunities. There is further support for harder-to-help jobseekers financed separately through Workforce Development Agreements (WDAs), the second of the two labour market transfers to PTs. PTs may also supplement this funding with their own provincial programming. Altogether federal active labour market spending in 2019, excluding the WDAs, represented just over 0.2% of GDP, which placed Canada in the bottom half of spending among OECD countries. This share has been decreasing over time and was over 0.3% of GDP around the turn of the century. The LMDA have been flat in nominal terms since their introduction in 1996, with periodic top-ups of funding to meet demand. This goes some way to explaining the funding pressures on ALMPs over time in Canada, increasing further the importance of knowing what policies work and how effective they are, to ensure they are given adequate weighting in public finance debates.
Against this backdrop, Canada has developed a highly proficient analytical capacity within ESDC to conduct policy evaluation of ALMPs. Since the inception of the LMDA, there has been a transformation of both the process and delivery of policy evaluation. Funding was transferred to PTs with the requirement that they consult annually with labour market stakeholders in their respective jurisdictions and report back to ESDC on how those consultations inform their labour market priorities and programming. A legislative requirement was introduced for PTs to conduct regular evaluations of their policy delivery financed by the LMDA. All PTs except Quebec (which conducts its own evaluations) have chosen to discharge this responsibility through evaluations conducted jointly with ESDC. When these evaluations were first incarnated, they were delivered via external contractors and data was gathered via client surveys. This process was both costly, cumbersome and contained some inaccuracies in participants’ recall of their income. It took around ten years for the first set of evaluations to be completed, as the nature of the surveys and associated analysis meant that only two to three studies could be conducted simultaneously.
ESDC embarked upon a programme of transformation, enlisting the use of administrative micro-data on employment insurance receipt and ALMP participation linked to Canada Revenue Agency data on income, and conducting analysis in-house. With strong provisions to protect personal information, ESDC has created a data platform from these separate datasets so that it can conduct assessments, this ensures consistency across evaluations of different policies and reduces data re-work for separate projects. ESDC has now conducted assessment of a number of different funding streams with these data. Increased internal resources for evaluation, due to lower contracting-out costs, and the establishment of a Chief Data Office have allowed ESDC to formalise and embed resources and best practice. ESDC is now able to deliver rigorous assessments of ALMPs, using established analytical techniques to ensure robustness, at a significantly quicker pace. These techniques are observational in nature, meaning linked administrative data are crucial in data processes which ensure that similar participants and non-participants are compared. These data also mean participants can also be sub-categorised so that effects for a range of different participants can be studied. A range of work is undertaken to assure the quality of this work, both to ensure the data and methods selected are appropriate and to test the sensitivity of estimates to various variations in these. ESDC has built a network of relationship with PTs officials and across analytical, data and programme staff within ESDC, using this teamwork to effectively deliver cycles of evaluation work. It also conducts a full cost benefit analysis of ALMPs, so that it can fully appraise them relative to one another. The utilisation of expert peer reviewers and the communication of analysis to external forums allows ESDC to ensure analysis is validated, fit-for-purpose, has credibility, and benefits from external analytical advice and expertise.
There are a number of small improvements that would further enhance ESDC’s ability to conduct detailed impact assessment of its ALMPs.
Enhancing some socio-economic data – such as education, family and immigration data – would allow better identification of sub-group impacts on young people, parents and migrants.
Real-time access to tax records for income data would allow evaluations to be conducted two years earlier, allowing a more contemporaneous discussion of policy effectiveness.
Conducting federal co-ordination of small-scale randomised studies within PTs, would allow more evidence to be gathered on programme delivery. PTs have considerable freedom in how they deliver programmes and their precise content, which introduces questions on delivery mode and programme design. For example, whether in-house delivery is more efficient than out-sourced, what the ideal contract design of out-sourced programmes is and what are optimal programme durations and content.
Opening up ALMP data to external researchers, via Statistics Canada, would democratise and expand the range of analysis, permitting greater innovation, reducing the direct costs of analysis for ESDC and ensuring PTs can properly analyse their policy delivery.
Breaking down the analysis that is used to estimate programme impacts into discrete stages would allow more discussion of how much the estimates change, and therefore how sensitive they are to their underlying assumptions.
Considering weighting the outcomes for ALMP participants in cost-benefit analysis to recognise their low level of income on average relative to other Canadians, would allow ESDC to better position their funding requirements relative to other government ministries.
Re-embedding the face-to-face forums for knowledge sharing between PTs after COVID-19 sanitary requirements have subsidised would allow for the continued fruitful exchange of analysis and policy delivery information between them.