10. Re-stating the key policy and evaluation messages
This chapter concludes Part III of the report by pulling together the key recommendations made in earlier chapters. These recommendations provide guidance on undertaking reliable evaluations of SME and entrepreneurship policy measures and using the evaluation evidence to improve the design and delivery of SME and entrepreneurship policies. A brief explanation and rationale is provided for each recommendation. The interested reader can return to earlier chapters for further details.
Recommendation 1. Governments should specify in advance the Objectives and Targets for each policy measure introduced.
A pre-condition for effective evaluation is to be able to assess impact against the intended Objectives and Targets of the policy. Governments should specify Objectives and Targets in terms of the nature and scale of impacts they seek to achieve. This should include identification of the specific groups of entrepreneurs or SMEs to be affected and a clear justification of the policy in terms of how it is expected to lead to the changes anticipated, or the problem it is expected to address. Both are important, since differences in policy effectiveness may be expected according to whether or not a policy targets important market failures and whether or not it targets potentially impactful entrepreneurship.
Recommendation 2. Three core metrics – Sales, Employment and Survival – should be specified and assessed in all evaluations. These can be complemented with additional measures for other Objectives, where targeted, such as environmental and social benefits.
It is necessary to have some common indicators in order to compare evaluation results across different types of policy interventions and policy designs. This type of evidence is needed to adjust the policy mix towards the most effective interventions for given Objectives. Growth in Sales and Employment are key Objectives of most SME and entrepreneurship programmes and information tends to be relatively well available. In addition, the Survival metric is very important in judging policy effectiveness because of the high exit rates for new and small firms. Clearly, the desired results of many SME and entrepreneurship policies interventions expand beyond these three metrics. Additional metrics should be included where other Objectives are important, for example social impacts or environmental impacts.
Recommendation 3. Expenditure data should be made available to evaluators for each policy measure to facilitate cost-effectiveness assessments.
Programme expenditure data need to be readily available to permit assessment of the cost-effectiveness of policy. This information is critical for making valid comparisons of success across different policies and programmes. The implicit threshold for success in cost-effectiveness terms could nonetheless vary according to the target populations and contexts of an initiative. For example, the acceptable cost of achieving business start-ups or additional employment through SMEs and entrepreneurship could be higher in regions that lag in economic development terms or in populations that are disadvantaged in the labour market. These are political decisions, but evaluation evidence is required to help politicians judge whether the cost-benefit ratios are acceptable.
Recommendation 4. Governments should establish a central monitoring and evaluation unit and a co-ordination process for the monitoring and evaluation of SME and entrepreneurship policy across government ministries and bodies.
A single organisation needs to be established within government to monitor and evaluate policy across all relevant government ministries and bodies against national policy Objectives, Targets and key performance indicators. It could be placed, for example, in the ministry with lead responsibility for developing the SME and entrepreneurship strategy and for co-ordinating SME and entrepreneurship policy. It should include staff in a central monitoring and evaluation unit, who reach out to focal points in different ministries and bodies to ensure the flow of monitoring and evaluation information.
Recommendation 5. Every three years, all major SME and entrepreneurship programmes should be the subject of a reliable evaluation, defined as a minimum of Step V, only the very “short-lifers” being excluded.
This Framework shows that, for most countries and for most policy areas, there are no longer either technical- or data-based reasons for not conducting evaluations or for conducting sub-optimal evaluations. We therefore recommend that, at minimum, all large-scale programmes with a duration of at least two years should be evaluated at Step V level, as presented in Part I of this Framework. Evaluation effort should take account of proportionality considerations. A share of 1% of the programme budget should be devoted to evaluation.
Recommendation 6. Governments should look carefully, using at least Step V methods, at the impact of their existing, and any new proposed, “Soft” programmes.
Although there are examples of reliable evaluations pointing to “Soft” support – advisory or training programmes – being cost-effective, these are rare. The relative effectiveness of “Hard” and “Soft” policies needs to be better understood, as well as the potential interactions between the two. More high-quality evaluation evidence is required on Soft policies and the conditions where they are effective or not effective.
Recommendation 7. Governments should review the role played by “Macro” policies.
Many SME and entrepreneurship programmes are “Micro” policy interventions, aimed at increasing capabilities and resources in specific SMEs and entrepreneurs targeted by policy. However, macro conditions may have equal or even greater impacts on SME and entrepreneurship performance, including long-established cultural traditions, legal frameworks, finance markets, knowledge generation factors etc. Macro policies aimed at these areas, but not specifically targeted on SME and entrepreneurship development, can have therefore important impacts on SMEs and entrepreneurship, including policies for infrastructure development, education, taxation, police and security etc. Indeed, the objectives that governments have for SMEs and entrepreneurship may be more easily attained through Macro actions than through dedicated Micro policies. It is important to increase understanding of the impact and cost-effectiveness of Macro policies on achieving SME and entrepreneurship objectives, and to benchmark dedicated SME and entrepreneurship policies against them.
Recommendation 8. Evaluations should provide the evidence for making decisions on the scale and nature of selective support.
Within most countries an important role is played by selective support – aimed at certain categories of SMEs or entrepreneurship, such as high-growth SMEs, gazelles, or start-ups with good survival prospects. Ideally, public support would be focused on those new and small firms most able to contribute to the economic and social welfare of the country. In practice, however, selecting for support those new and small firms most likely to survive and grow rapidly is currently problematic. This may become more reliable as data and analyses improve. A strong role needs to be played by evaluation for supporting selection approaches.
Recommendation 9. Evaluations should identify exceptional performers and the role such firms have in reaching a judgment on the overall effectiveness of a programme.
More information is needed on the extent to which different programmes succeed in generating exceptional performers, how their design enables this, and the extent to which it is the result of random, non-controllable factors. This information would help inform more selective and effective policies in the future.
Recommendation 10. Evaluations should systematically include the performance of non-surviving SMEs and start-ups in their assessments of treatment and control group performance.
Because new and small firms have high exit rates, and exit rates may be affected by policy measures, the evaluation of policy impact must take into account the impact of non-survival on estimated impacts, such as employment and sales. This requires tracking all firms and entrepreneurs in the treatment and control groups at least from the beginning of the policy intervention.
Recommendation 11. Governments should investigate the use of the data they collect for tax and other purposes with a view to making it more widely available to those conducting policy evaluations.
Data collected “automatically” by government has wide coverage and relatively high reliability. This makes it ideal for most policy evaluations. Its collection imposes no extra charges or burdens on the business and, for these reasons, it is now used in several countries. Crucially, appropriate standards of confidentiality have to be in place.
Recommendation 12. Lessons from reliable evaluations should be shared between countries, with the OECD CSMEE being an ideal vehicle for facilitating this exchange.
There are valuable lessons to be gained from reviewing the impacts of SME and entrepreneurship policy interventions across countries. This Framework has reviewed 50 evaluations from OECD countries and sought to draw lessons. However, a much larger body of reliable evaluation evidence would help to throw light on some of the outstanding questions highlighted in this review and support international learning from evaluation practice. This will require co-ordination that the OECD Committee on SMEs and Entrepreneurship (CSMEE) is in a unique position to provide.
Recommendation 13. Internationally-co-ordinated policy evaluation should be undertaken on the impact of COVID-19 SME and entrepreneurship policy responses.
Governments need to ensure that individual COVID-19 SME and entrepreneurship policy measures are reliably evaluated and share the evaluation results to support policy learning for future shocks. In addition, a broader international analysis of the impact of the COVID-19 response on subsequent SME and entrepreneurship performance should be undertaken. The OECD CSMEE is well placed to undertake this work.