8. What needs to be improved in SME and entrepreneurship policy evaluation?
This chapter focuses on drawing the lessons from the review of evaluations in Part II for the conduct of evaluations and for commissioning of evaluations by policymakers. It covers the importance of: i) increasing the number of evaluations that are conducted; ii) clearly specifying the Objectives and Targets of policy at the outset to facilitate evaluations; iii) making greater programme expenditure information available to facilitate cost-effectiveness assessments; iv) exploiting new evaluation techniques and data; v) increasing evaluation co-ordination and coherence; and vi) strengthening the international exchange of information on evaluation results. The chapter includes recommendations in these areas.
This chapter acknowledges that the technical practice of SME and entrepreneurship policy evaluation has improved considerably during the 15 years or so since the first OECD SME and entrepreneurship policy evaluation framework (OECD, 2007[1]), and that there are now many more examples of high-quality evaluations, although reliable evaluation evidence is still lacking in the field.
As shown in Part II, these evaluations point to mixed evidence on the impact of policy, with less than half of our 50 reviewed evaluations showing an overwhelmingly positive impact across a range of indicators, and the remainder showing either impacts on only some of their chosen indicators or no impacts or negative impacts. As discussed in chapter 7, the divergences in outcomes may reflect a number of aspects of policy targeting, design, context and so on. Taking account of these influences will ultimately strengthen SME and entrepreneurship policy overall. However, learning from evaluation evidence on policy success factors will only become more reliable with improvements in the scale, frequency, coverage, focus and technical quality of policy evaluations in this field.
With this background, this chapter therefore sets out a forward agenda for SME and entrepreneurship policy evaluation. It stresses the importance of increasing the scale of evaluation, exploiting available techniques and data, clearly specifying Objectives and Targets, increasing information available to evaluators on programme expenditure, and developing cross-government co-ordination in evaluation.
Reliably-conducted evaluations provide crucial insights into the impact of SME and entrepreneurship policies. Ideally governments would be able to compare the effectiveness of different policies with the aim of redistributing funding away from the less effective, and towards the more effective, policy areas. Unfortunately, reliable evaluations in most countries remain the exception rather than the rule. It is important to increase the coverage of SME and entrepreneurship policy evaluation in order to assess where policy is the most effective and to balance policy across the portfolio, as well as to provide policy learning and accountability for each individual programme.
Recommendation: Every three years, all major SME and entrepreneurship programmes should be the subject of a reliable evaluation, defined as a minimum of Step V, only the very “short-lifers” being excluded.
There have been two important changes over the last 15 years which enable evaluations to be undertaken more easily and more reliably than in the past. The first is the advance in analytical techniques and the second is the arrival of “big data”. High quality evaluations can now be undertaken much more easily and more quickly both within government and/or commissioned from outsiders. Because they can be delivered more quickly this makes them more relevant for current policy decisions.
As an example, we noted in Part I the increased use of Randomised Control Trials (Roper, 2020[2]). This is deemed to be the Gold Standard for ensuring that any observed improvements in the performance of firms or individuals within a programme are legitimately attributed to the programme alone.
As another example, although initially limited to the Nordic countries, there are now several examples of other countries opening up government datasets, collected for a variety of different purposes, and enabling them to be used for evaluation purposes. For example, (Barrot et al., 2019[3]) use administrative microdata extracted from the tax files of the French Ministry of Finance for corporate tax collection purposes covering the universe of French firms. The data are not publicly available, but are available for academic research through a procedure similar to accessing Census data in the USA and the UK. (Barrot et al., 2019[3]) then link this information on firm performance to data on firms using and not using the Loan Guarantee Programme, enabling the creation of treatment and control groups.
Recommendation: Governments should investigate the use of the data they collect for tax and other purposes with a view to making it more widely available to those conducting policy evaluations.
In making this recommendation it is vital to emphasise that appropriate confidentiality safeguards have to be in place.
Part II also re-emphasised the importance of clarity in the specification of Objectives and Targets for evaluation, which was initially introduced in Part I. It showed that the Objectives (the desired outcome, e.g. in terms of increased start-ups, sales, or employment) and Targets (the scale of the change targeted, e.g. in numbers of additional start-ups, sales or employment by a given timescale) were often not adequately specified, so that an assessment of policy impact against policy intentions would be difficult or impossible.
Recalling that the 50 evaluations we documented in Part II can be considered as exemplars in many respects, it was disappointing that only 1 out of the 50 clearly specified Objectives and Targets.1 Much more typical was that policy Objectives were specified, but expressed in a form that made a careful assessment of impact difficult to conduct. In other cases, the Objectives were never specified by the policymakers themselves, leaving them to be inferred by those undertaking the evaluation.2 A final group were those where the Objectives that were specified were so numerous, but lacking in priority, that some element of the programme was almost certain to emerge as successful. This made it close to impossible to identify the circumstances in which the policy could ever be viewed as either clearly successful or clearly unsuccessful. The overriding picture is that policies frequently succeed on some objectives but not on others, as captured our frequently used description of the evidence as “mixed”.
It is therefore important that government specifies a small number of priority Objectives for their SME and entrepreneurship policy evaluations. If these core Objectives are common across policy areas, then comparison of programme performance on key government priorities is possible.
Policymakers with responsibility for SME and entrepreneurship policy are continually faced with choices. These might be between the provision of “Hard” assistance – such as loans/grants – or “Soft” assistance such as advice/training. They also face choices over the sectors where this assistance is to be provided – e.g. high-tech or personal services. A third choice might be over policies open to all enterprises or individuals (entrepreneurs or potential entrepreneurs), compared with those that select specified groups.
Evaluations can play a key role in assisting such choices, but only when the metrics upon which policy effectiveness is judged are clearly specified. Without these “common” metrics it is very difficult for the outcomes from different policies to be compared. We recommend establishing three core metrics, corresponding to key common Objectives and Targets that governments have for SME and entrepreneurship policy interventions – namely change in Sales, Employment and Survival in supported enterprises.
Recommendation: Three core metrics – Sales, Employment and Survival – should be specified and assessed in all evaluations. These can be complemented with additional measures for other objectives, where targeted, such as environmental and social benefits.
These three core metrics can be supplemented by other metrics appropriate for a specific policy area – such as patents for Innovation evaluations, or wages for Enterprise Culture and Skills or Areas of Disadvantage evaluations. However, as their name implies, these metrics should be regarded as supplementary and not as alternatives to the three core metrics.
A common set of enterprise metrics enables policymakers to compare how far different policies achieve these Objectives and Targets. So, for example, job creation in areas of disadvantage might be enhanced by a range of policies such as improving access to finance, by the provision of free business advice, or by programmes to enhance enterprise culture. By specifying the core metrics, and then undertaking evaluations, policy makers are able to assess the job-creation impact of each of the three approaches, and which is most effective.
Clearly, the priorities and preferences of governments need to be taken into account when comparing the effectiveness of different policy measures, since an acceptable cost per job created, for example, might be higher in a context of disadvantaged places or disadvantaged entrepreneurs.
Making a judgement on the cost-effectiveness of different approaches requires data on programme expenditure as well as programme impacts. However, expenditure on SME and entrepreneurship policy – both on individual programmes and in aggregate – is frequently difficult to identify. Unfortunately, in 10 out of the 50 cases reviewed in Part II, this information was unavailable. It is possible that in some cases the information was available within government and not to outsiders but, even so, this is a serious information deficiency.
Recommendation: Expenditure data should be made available to policymakers for each policy measure to facilitate cost-effectiveness assessments.
The patchy information on the cost of individual policies and programmes noted above is replicated for data on the aggregate cost of SME and entrepreneurship policy. Many parts of government undertake policies and programmes that aim to promote SMEs and entrepreneurship. However, there is commonly little overview of the total policy effort devoted to this policy area (compared for example to other countries, or the priority placed on SMEs and entrepreneurship by the government) or of the relative effectiveness of the different measures pursued that would help steer the overall policy mix.
A study by (Lundström et al., 2014[4]) examined Sweden in detail, together with more limited coverage of Poland, Austria and the Flanders region of Belgium. It confirmed the findings of an earlier UK review that this expenditure was considerable – for example exceeding that on the police or universities in some countries – yet it was not the responsibility of a single ministry/department of government. There was no part of government with the authority to assess the cost-effectiveness of policies by being able to compare, for example, the policy impact per unit of expenditure on business advice with loan guarantee programmes for supporting SME survival and growth. In part, but not exclusively, this was because virtually all ministries/departments of government had their own programmes to promote enterprise – with these programmes frequently working independently of other parts of government.
In addition to systematically collecting and sharing information on policy expenditures on SMEs and entrepreneurship across government, the objective of achieving greater coherence and effectiveness of the policy portfolio would be served by establishing a central monitoring and evaluation unit for SME and entrepreneurship policy. Such a central monitoring and evaluation unit would work with all relevant ministries/departments and government bodies to identify the objectives, targets, expenditures, activities, and impacts of SME and entrepreneurship policy measures wherever they are undertaken. It would develop a management information system to track policy and a system of focal points in different ministries and government bodies to share information.
Recommendation: Governments should establish a central monitoring and evaluation unit and a co-ordination process for the monitoring and evaluation of SME and entrepreneurship policy across government ministries and bodies.
Evaluation evidence should be a key guide to policy development in SME and entrepreneurship policy. Relevant evaluations of new policy initiatives under consideration by a particular government or government agency may already have been undertaken in many countries, and a rich base of evaluation evidence across different types of policy intervention and different contexts of application can best be built up internationally.
Recommendation: Lessons from reliable evaluations should be shared between countries, with the OECD’s Committee on SMEs and Entrepreneurship (CSMEE) being an ideal vehicle for facilitating this exchange.
The OECD’s Committee on SMEs and Entrepreneurship (CSMEE) can play a vital role in sharing information internationally from reliable SME and entrepreneurship policy evaluations. First, it can expand the international database of impact evaluation results and make it accessible to policy makers. Second, it can support countries in developing their own evaluation frameworks and in setting up processes to gather reliable impact evaluation evidence in their own contexts.
References
[3] Barrot, J. et al. (2019), “Employment Effects of Alleviating Financing Frictions: Worker-Level Evidence from a Loan Guarantee Program”, SSRN Electronic Journal, https://doi.org/10.2139/ssrn.3409349.
[4] Lundström, A. et al. (2014), “Measuring the Costs and Coverage of SME and Entrepreneurship Policy: A Pioneering Study”, Entrepreneurship: Theory and Practice, Vol. 38/4, https://doi.org/10.1111/etap.12037.
[1] OECD (2007), OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes, OECD Publishing, Paris, https://doi.org/10.1787/9789264040090-en.
[2] Roper, S. (2020), “Using RCTs as a research method for SME policy research: The UK experience”, in Handbook of Quantitative Research Methods in Entrepreneurship, https://doi.org/10.4337/9781786430960.00012.