3. Existing meta-evaluations

Abstract

This chapter reviews twelve meta-evaluations of different aspects of SME and entrepreneurship policies and programmes. For each of these meta-evaluations, information is provided on the sources, policy focus and key findings.

    

Since the publication of the first OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes in 2007, there have been several meta-reviews of evaluation approaches and evidence in this area.

To identify meta reviews of evaluations of entrepreneurship and SME policies published since 2007, a search was made of academic publications, working papers and policy reports online via Google.com and Scholar.google.com. The citation databases Scopus and Web of Sciences were also searched. A total of twelve meta-evaluations were identified.

A synthesis of the reviews is presented in Table 3.1. It provides information on the year when the review was published, its source, focus and key findings. These groupings, together with others, will be used in our own review of 50 evaluations covered in Chapter 4.

There are however two important areas where the reviews presented in Table 3.1 differ from our review. First, Table 3.1 includes four reviews from low- and middle-income countries. Our review includes only OECD Members, which are all either high-income or upper-middle-income countries. It should be recognised that the impact of policy may differ according to country income levels. However, the inclusion of these evaluations in Table 3.1 is justified on the grounds that high quality evaluation methods were used and also that many of the outcome indicators are common and aligned with what we recommend for OECD countries. Second, because there were only four dimensions common to all the reviews, Table 3.1 compares the reviews only across year of publication, source, focus and key findings. Our review is considerably more wide-ranging on the dimensions of the evaluations considered.

Turning now to our interpretation of Table 3.1 we see that, of the twelve reviews, nine are of the impact of direct financial assistance (“Hard” support) – albeit in different forms – three of which focus on start-up subsidies for the unemployed. There is only one review that focuses solely on the effects of “Soft” business support (advice, training etc.) and one on the effects of business incubation. So, whilst there is reasonable coverage of the impact of financial support, there are major areas of policy expenditure where it was not possible to find reviews. This re-emphasises the point made in Part I that policy assessment is patchy – a point to which we will return in our own review.

In many respects, Table 3.1 confirms our conclusion from Part I that policy impact is mixed. Dealing with the eight reviews of financial assistance, Review 1 describes the impact of interventions as mixed. Tax incentives in low-income countries, covered in Review 2, also appear to have an “inconclusive” impact on economic outcomes and Review 3 paints a similar picture on financial subsidies. Review 4, again of low- and middle-income countries, confirms the above but, most interestingly in the light of our later findings for OECD countries, suggests that training and information programmes do have a positive effect. Review 5 returns to the “mixed evidence” theme, pointing to some finance programmes enhancing some performance metrics – such as survival – but having little impact on productivity. Review 8, for low and middle-income countries, finds that financial subsidies have a positive but only modest impact on several measures of firm performance. Review 10 on financial support for the unemployed seeking to enter self-employment finds that these enhance the earnings of participants. This is confirmed in Reviews 11 and 12, both of which are of German programmes. This programme in particular appears to be consistently successful on its chosen criterion.

Of the remaining non-financial programmes the picture is again mixed. Review 6 covering R&D-related programmes reports positive outcomes on a range of performance metrics such as employment and sales. Review 7 is the most positive of all, finding incubation “greatly enhances” the performance of tenants, but Review 9 can find no statistically significant positive impact on firm performance of “Soft” assistance.

The reviews to date do point to some clear policy successes – most notably the German programmes providing finance to the unemployed to become self-employed. The single review of incubation is also positive but several of the finance reviews – despite using good data and advanced statistical techniques – are unable to clearly link programme assistance to enhanced firm performance.

Table 3.1. Meta-evaluations of SME and entrepreneurship policy
Review Number	Review Name	Year of the Review	Topic	Key Findings	Source of Evidence
1.	The theory and practice of financial instruments for small and medium-sized enterprises	2017	Finance, financial instruments, credit guarantees, soft loans	The authors conclude in their review that the existing empirical evidence concerning financial instruments is somewhat mixed. While the constraints on the credit markets for SMEs exist, the careful design of the public programme, institutional context, evaluation methods and nature of economic conditions influence findings on the overall success of the programmes.	Working paper: (Brown and Lee, 2017[1]). The theory and practice of financial instruments for small and medium-sized enterprises. EC-OECD seminar series on Designing Better Economic Development Policies for Regions and Cities. Organisation for Economic Co-operation and Development (OECD), Paris. Available at: http://www.oecd.org/cfe/regional-policy/Brown_When-to-use-financial-instruments.pdf
2.	Review of corporate tax incentives for investment in low- and middle-income countries	2018	Finance, tax incentives	The authors conclude in their review of interventions in Low- and Middle- income countries that the existing literature shows inconclusive evidence on the impact of tax incentives on investment and other economic outcomes such as employment and output. Evidence from cross-country studies using aggregate-level outcomes shows that tax incentives may affect foreign direct investment levels but not necessarily total investment, suggesting the possibility of crowding out effects. Cross-country studies, however, suffer from some methodological limitations.	Working paper: (Abramovsky, Bird and Tyskerud, 2018[2]). Review of corporate tax incentives for investment in low- and middle-income countries. Institute for Fiscal Studies Briefing note BN229. Institute for Fiscal Studies, London. Available at: https://www.ifs.org.uk/uploads/BN229.pdf
3.	Small firms, large impact? A systematic review of the SME finance literature	2017	Finance, subsidies, tax incentives, financial instruments	The authors conclude in their review of interventions in Low- and Middle- income countries that SME support has positive effects on firm performance, capital investment and employment, but insignificant effects on profitability and wages. However, they also claim that it remains unclear to what extent SME finance contributes to economic development and poverty reduction.	Academic article: (Kersten et al., 2017[3]). Small firms, large impact? A systematic review of the SME finance literature. World Development, 97, 330-348. Available at: https://doi.org/10.1016/j.worlddev.2017.04.012
4.	Do interventions targeted at micro-entrepreneurs and small and medium-sized firms create jobs? A systematic review of the evidence for low and middle-income countries.	2015	Finance, subsidies, tax incentives, financial instruments, training	The authors conclude in their review of interventions in Low- and Middle-income countries that effects of entrepreneurship and SME policies on employment are so far small. They also claim that financial programmes (i.e. subsidies and financial instruments) are less effective than training or business development services.	Academic article: (Grimm and Paffhausen, 2015[4]). Do interventions targeted at micro-entrepreneurs and small and medium-sized firms create jobs? A systematic review of the evidence for low and middle-income countries. Labour Economics, 32, 67-85. Available at: https://doi.org/10.1016/j.labeco.2015.01.003
5.	Public SME grants and firm performance in the European Union: A systematic review of empirical evidence	2019	Finance, subsidies, grants	The authors conclude in their review of interventions in European Union Member States that effects of direct subsidies and capital grants are mostly positive on firm survival, employment, tangible/fixed assets, and sales/turnover, with mixed findings for labour productivity and total factor productivity. The authors also point out that there are significant differences concerning the time-period of analysis (investigating short-term vs long-term outcomes), and importantly, the heterogeneity of effects concerning firm size and age, region, industry, and intensity of support.	Academic article: (Dvouletý, Srhoj and Pantea, 2021[5]). Public SME grants and firm performance in the European Union: A systematic review of empirical evidence. Small Business Economics. DOI: https://doi.org/10.1007/s11187-019-00306-x.
6.	Improving access to finance for young innovative enterprises with growth potential: Evidence of the impact of R&D grant schemes on firms' outputs	2019	Innovation grants, R&D grants, science and technology grants	The authors review different types of R&D programmes and analyse the wider policy implications. Overall, they report positive outcomes on employment, total sales, share of innovative sales, and companies’ innovation capacities. Moreover, the effects of R&D grants for scale-ups are larger than the effects of both generic R&D grants and R&D subsidies. In terms of policy implications, R&D grants stimulate and prepare companies for growth and targeted funding (technology-focused) delivers better results for disruptive innovations, whereas generic grants for small and medium-sized enterprises are better suited for knowledge diffusion.	Academic article: (Testa, Szkuta and Cunningham, 2019[6]). Improving access to finance for young innovative enterprises with growth potential: Evidence of the impact of R&D grant schemes on firms' outputs. Research Evaluation. Available at: https://doi.org/10.1093/reseval/rvz016
7.	Business incubation process and firm performance: an empirical review	2017	Business incubators, support of innovative ventures	The authors conclude in their review that firm performance is greatly enhanced when a firm avails itself of an incubation programme. Revenue growth, employment or job creation, venture funding, networking and alliance building are the performance indices most impacted by the business incubation process. However, tenants should not overstay their tenancy in an incubation programme as doing so reduces their chances of survival upon graduation.	Academic article: (Ayatse, Kwahar and Iyortsuun, 2017[7]). Business incubation process and firm performance: an empirical review. Journal of Global Entrepreneurship Research, 7(1), 2. Available at: https://doi.org/10.1186/s40497-016-0059-6
8.	The impact of business support services for small and medium enterprises on firm performance in low-and middle-income countries: A systematic review	2016	Finance, subsidies, tax incentives, financial instruments, training, counselling, advisory services	The authors conclude in their review of interventions in Low- and Middle-income countries conclude that the effects on the firm´s performance, employment and labour productivity are positive. But these effects are not large, and the cost effectiveness of the interventions is not known. The effects on innovation are unclear.	Academic article: (Piza et al., 2016[8]). The impact of business support services for small and medium enterprises on firm performance in low-and middle-income countries: A systematic review. Campbell Systematic Reviews, 12(1), 1-167. Available at: https://doi.org/10.4073/csr.2016.1
9.	The effects of human capital interventions on entrepreneurial performance in industrialized countries	2019	Soft business support, counselling, advisory services, business training, entrepreneurship education	The authors conclude in their review that Soft business support does not have statistically significant effects on entrepreneurial performance, with the exception of formal education, which shows positive effects on firm profits and entrepreneurial earnings.	Academic article: (Hogendoorn et al., 2019[9]). The effects of human capital interventions on entrepreneurial performance in industrialized countries. Journal of Economic Surveys. Available at: https://doi.org/10.1111/joes.12308
10.	Review of empirical studies on self-employment out of unemployment: Do self-employment policies make a positive impact?	2016	Finance, start-up subsidy for unemployed	The authors conclude in their review that self-employment programmes (i.e. start-subsidies for the unemployed) succeed in activating individuals out of unemployment as the subsidised businesses have high survival rates and there is high cost-efficiency of intervention. The reviewed studies also report positive outcomes of the programmes on the income and earnings of formerly unemployed individuals.	Academic article: (Dvouletý and Lukeš, 2016[10]). Review of empirical studies on self-employment out of unemployment: Do self-employment policies make a positive impact?. International Review of Entrepreneurship, 14(3), 361-376. Available at: https://www.senatehall.com/entrepreneurship?article=552
11.	Start-up subsidies for the unemployed: Opportunities and limitations	2016	Finance, start-up subsidy for unemployed	The authors conclude in their review that self-employment programmes succeed in activating individuals out of unemployment as the subsidised businesses have high survival rates and there is high cost-efficiency. The authors also conclude that these programmes may have greater positive effects for the disadvantaged group such as women, youth and low-educated workers.	Academic article: (Caliendo, 2016[11]). Start-up subsidies for the unemployed: Opportunities and limitations. IZA World of Labor, 200. Available at: https://wol.iza.org/uploads/articles/200/pdfs/start-up-subsidies-for-unemployed-opportunities-and-limitations.pdf?v=1
12.	An evaluation of German active labour market policies: A review of the empirical evidence	2018	Finance, start-up subsidy for unemployed	The authors conclude in their review focused on Germany that most self-employment programmes increased the prospects of the participants. In particular,, evaluations of the entrepreneurship promotion activities show high success rates as well as high cost-efficiency. The bulk of participants of entrepreneurship measures was still self-employed after several years, and nearly one-third of these businesses had at least one employee.	Academic article: (Zoellner, Fritsch and Wyrwich, 2018[12]). An evaluation of German active labour market policies: A review of the empirical evidence. Journal of Entrepreneurship and Public Policy, 7(4), 377–410. Available at: https://doi.org/10.1108/JEPP-D-18-00023

References

[2] Abramovsky, L., N. Bird and Y. Tyskerud (2018), Review of Corporate Tax Incentives For Investment in Low- and Middle-Income Countries, Institute for Fiscal Studies.

[7] Ayatse, F., N. Kwahar and A. Iyortsuun (2017), “Business incubation process and firm performance: an empirical review”, Journal of Global Entrepreneurship Research, Vol. 7/1, https://doi.org/10.1186/s40497-016-0059-6.

[1] Brown, R. and N. Lee (2017), “The theory and practice of financial instruments for small and medium-sized entreprises”, Ec & Oecd June.

[11] Caliendo, M. (2016), “Start-up subsidies for the unemployed: Opportunities and limitations”, IZA World of Labor.

[10] Dvouletý, O. and M. Lukeš (2016), “Review of Empirical Studies on Self-Employment out of Unemployment: Do Self-Employment Policies Make a Positive Impact?”, International Review of Entrepreneurship, Article #1540, Vol. 14/3.

[5] Dvouletý, O., S. Srhoj and S. Pantea (2021), “Public SME grants and firm performance in European Union: A systematic review of empirical evidence”, Small Business Economics, Vol. 57/1, https://doi.org/10.1007/s11187-019-00306-x.

[4] Grimm, M. and A. Paffhausen (2015), “Do interventions targeted at micro-entrepreneurs and small and medium-sized firms create jobs? A systematic review of the evidence for low and middle income countries”, Labour Economics, Vol. 32, https://doi.org/10.1016/j.labeco.2015.01.003.

[9] Hogendoorn, B. et al. (2019), “THE EFFECTS OF HUMAN CAPITAL INTERVENTIONS ON ENTREPRENEURIAL PERFORMANCE IN INDUSTRIALIZED COUNTRIES”, Journal of Economic Surveys, Vol. 33/3, https://doi.org/10.1111/joes.12308.

[3] Kersten, R. et al. (2017), Small Firms, large Impact? A systematic review of the SME Finance Literature, https://doi.org/10.1016/j.worlddev.2017.04.012.

[13] OECD (2008), OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes, OECD Publishing, Paris, https://doi.org/10.1787/9789264040090-en.

[8] Piza, C. et al. (2016), “The Impact of Business Support Services for Small and Medium Enterprises on Firm Performance in Low‐ and Middle‐Income Countries: A Systematic Review”, Campbell Systematic Reviews, Vol. 12/1, https://doi.org/10.4073/csr.2016.1.

[6] Testa, G., K. Szkuta and P. Cunningham (2019), “Improving access to finance for young innovative enterprises with growth potential: Evidence of impact of R&D grant schemes on firms’ outputs”, Research Evaluation, Vol. 28/4, https://doi.org/10.1093/reseval/rvz016.

[12] Zoellner, M., M. Fritsch and M. Wyrwich (2018), An evaluation of German active labour market policies: a review of the empirical evidence, https://doi.org/10.1108/JEPP-D-18-00023.