6. Why do SME and entrepreneurship policy evaluations provide mixed evidence of impact?
This chapter poses the key question: Why, even when the evidence is reliable, do evaluations often show lack of effectiveness of SME and entrepreneurship policy measures in achieving key policy objectives? Five potential explanations are explored: i) the policies and programmes are flawed; ii) the delivery of SME and entrepreneurship policy is problematic, iii) the evaluations themselves fail to appropriately take context into account; iv) policy outcomes are diverse because SMEs and entrepreneurs are diverse; and v) policy can have unintended consequences.
Part II of this Framework summarised several earlier reviews of SME and entrepreneurship policy evaluations and concluded that the findings were “mixed”, with some policies having a positive impact, others showing no positive impact and others finding impacts on some metrics but not others. What remained unclear was whether these, frequently very different, findings reflected real differences in policy impact, or whether they merely reflected the different reliability of the evaluations.
To address the problem of reliability, Part II then examined 50 evaluations that each satisfied OECD reliability criteria at the Step V or Step VI levels of the Six Steps to Heaven guidelines. The conclusion based on these cases was that, although some programmes clearly “worked”, a, generally smaller number, did not. In other cases, reaching a clear conclusion was difficult because whereas there were impacts on some indicators there were not on other (potentially more crucial) indicators. It was therefore reasonable to conclude that policy outcomes were “mixed” overall.
The diversity of findings from SME and entrepreneurship policy evaluations, even when all have used reliable data and methods, may be influenced by factors such as: the metric(s) on which the policy was judged; the context in which it was delivered; the timescale of the programme; or the target group of firms or entrepreneurs addressed. For example, a programme might be effective for larger, but not for smaller, SMEs; it might be effective in enhancing the profitability of an SME but have no effect on its employment; it might have a short-run impact, but no longer-run impact; it might be successful in taking individuals out of unemployment but unsuccessful if the objective is to create new, innovative and productive enterprises (Caliendo, Künn and Weissenberger, 2020[1]).
This chapter examines further what may explain the diversity of outcomes identified by SME and entrepreneurship policy evaluations, and how this should be taken into account by policymakers when framing, and subsequently evaluating their policies. Ideally there should be a clear link – in terms of cause and effect – between policy application and impact, but the evidence from the exemplar Part II cases shows this is often not the case. Why, then, is the link between SME and entrepreneurship policy application and impact not always clear?
Five explanations for this mixed picture can be proposed. The first repeats the arguments set out in Part I, namely that much policy in this area is flawed in principle and therefore would not be expected to “work”. The second explanation is that the diversity of outcomes reflects a diverse set of influences on how policy is delivered, and hence how effective it is. The third is limitations in the evaluation approach with respect to accounting for programme context. The fourth is that impact is diverse because the performance of SMEs and entrepreneurs themselves is so diverse. The final explanation is that, because such policies are relatively new, there will always be unintended outcomes. Each of these issues is now discussed.
Several leading academic commentators have argued that large parts of SME and entrepreneurship policy, as delivered in most OECD countries, are misguided for a range of reasons, and so the “mixed” outcome is the best to be expected.
For example, Part I referred to the work by (Acs et al., 2016[2]). Their starting position is that government intervention to support new and small firms is justified only where there is clear evidence of market failure as, for example, in public support for innovative enterprises with the motivation to grow. By implication, it is the absence of clear market failure for other forms of support that explains the “mixed” picture of impact overall.
Political scientists who have examined SME and entrepreneurship policy see policy performance as being strongly influenced by the processes through which it emerged. This approach emphasises the need, in deciding what works and what does not, to better understand how the policy evolved, since it is this which determines how it is delivered “on the ground” and, by implication, its ultimate success. For example, (Arshed, Carter and Mason, 2014[3]) describe a six-stage process through which UK enterprise policy evolved in the years 2009-10. Drawing upon interviews with those involved with policy, they point to public servants being fully aware of “how” policy is expected to evolve but acknowledging that in practice these “formal structures can never conquer the non-rational dimensions of organisational behaviour”.
The authors acknowledge that much UK policy in this area has been ineffective but attribute this, not to the policy itself, but to the inability of law-makers to negotiate its passage through government and to then ensure it is delivered as intended. In this context, the term “delivery” is wide-ranging. It includes an acceptance of an idea, but this then being modified, even hijacked, at points in time as it progresses through the legislative process by individuals or interest groups with “agendas” that are not necessarily fully in line with those underpinning the policy.
A third group of explanations for the diverse range of outcomes from evaluations of SME and entrepreneurship policy is that the mixed picture reflects the limitations of evaluation per se, even when following Step V and VI methods. So, although many evaluations of SME and entrepreneurship policies have not found significant impacts, this does not justify the policy overall as being categorised as ineffective. Instead what is required is for the assumptions and limitations of the evaluations to be highlighted, rather than the ineffectiveness of policies.
At its most extreme, it is argued that policy impact cannot be decomposed into a set of simple metrics, implying that policymakers are faced with a limited set of choices, all of which can be assessed with certainty; instead, much remains uncertain in policymaking and any evaluation approach has to highlight these uncertainties.
A more nuanced critique is the lack of recognition given to the extent to which evaluation outcomes are influenced by context – both the period of time over which the evaluation is conducted and the individual national, regional or local circumstances in which the policies are delivered. The further critique is that it is inappropriate to evaluate the role played by individual programmes or policies when these are often only one of many macro or spatial factors influencing SMEs and entrepreneurship at a national or regional level. Instead, what has to be captured is the inter-dependence of these factors upon each other and how policy influences them as a whole – often referred to as the entrepreneurial ecosystem. Each of these points are now addressed in turn.
The role of context: time dimension
The time dimension has to be taken into account in assessing the impact of SME and entrepreneurship policy, in part because the outcomes from these policies are likely to vary with the duration of the programme. All else equal, we would expect that programmes which operate only for a short period of time would be both less likely to be evaluated, and less likely to produce positive results, compared with longer-run programmes. This is because, if the programme is clearly experiencing operational difficulties – most notably low “take-up” by participants – this is likely to lead to its early closure. A second reason is that policies and programmes are frequently cut short following a change of government. This perhaps explains why, of the 50 evaluations reviewed in Part II, only 4 were of programmes having a lifespan of less than 2 years. Nevertheless, one of the reasons for some evaluations showing lack of impact on key objectives may be that the evaluation took place before enough time had been allowed for impacts to be achieved.
On the other hand, there are also reasons why evaluations of a programme over a short period of time may over-estimate results compared with evaluations undertaken over a longer period. This is because short-term evaluations cannot, by definition, fully capture the exit of firms or individual entrepreneurs from programmes – although that risk is extremely high1. There is therefore a real risk that short-term evaluations inflate the estimated impact (e.g. businesses established, jobs created) by including firms/individuals that will exit very shortly, and would have been excluded if the time period were longer. As was shown in Part II, business survival was taken into account only in 13 out of 50 evaluations. Moreover, of real concern is that accounting for exit varied by programme type – being particularly low in the finance and the local-area programmes. This could bias the comparison of the effectiveness of different types of policy intervention.
A further reason for taking full account of time is that, even where the above concerns are fully taken into account, programme impact itself can vary over time. For example, (Drews and Hart, 2015[4]) shows that a UK business advice programme had a modest immediate impact on survival. Over a two- to three-year period the sales and employment in assisted businesses rose, but productivity declined. By year seven there was no observable impact on any dimension.
It is therefore a valid criticism of evaluations that their findings are likely to be sensitive to the time period over which they, and the programme itself, are conducted. It is also valid to acknowledge that our current knowledge base is not sufficient to be clear on those performance metrics, perhaps other than survival, that are the most sensitive to time.
Our policy-related view is that short-life programmes lasting for less than 2 years should not be a priority for evaluation because their impact is very difficult to assess using reliable techniques. For the longer-duration programmes, evaluation findings become more robust with time. A useful rule of thumb is that, for most SME support programmes2 it is valid to assess the impact of “treatment” after two years, on the grounds that making assessments prior to that time is likely to under-estimate exits and hence risks inflating the impact of the programme. However, the impact of time on other metrics – employment, sales or productivity – is less clear and needs to be explored.
The role of context: macro and spatial
It is important to recognise that the same policy can have very different outcomes in benign, compared with hostile, macro-economic conditions (Sedláček and Sterk, 2017[5]). Therefore, the period of time in which the policy is applied (recession/boom) may affect policy outcomes, and this could help explain why the estimated impacts of SME and entrepreneurship interventions vary widely in evaluation.
A second group of contextual factors influencing policy outcomes are the regional and local circumstances in which policy is delivered. These are relevant for national SME and entrepreneurship policies that are delivered without an explicit local/regional differentiation. Here programme and policy take-up rates may vary markedly between regions, and this needs to be taken into account in assessing impact. Evidence of this is documented most clearly in SME finance programmes such as loan guarantees (Cowling, 1998[6])3.
However, the role of contextual factors is most relevant in explicit, place-based policies, in which SME and entrepreneurship policies seek to improve regional and local enterprise activity, including by increasing the performance of weaker regions and localities. The case for these policies is that, although the recipients are in the same country – making them subject to the same national macro-economic and institutional conditions – local conditions for SME and entrepreneurship development frequently vary widely between areas. For example, a broad, long-standing, rule of thumb is that new firm formation rates in the highest region of a country are approximately three times that of the lowest region in the same country (Reynolds, Storey and Westhead, 1994[7])4. Place-based policies aim to improve the local or regional environment for SME and entrepreneurship development. In the area of start-ups and scale-ups, the “entrepreneurial ecosystem” concept is often used to identify, and then address, local bottlenecks across a set of inter-related influences such as culture, regulation, networks, leadership, finance and talent. This entrepreneurial ecosystem approach can “deliver holistic and stakeholder-driven interventions to improve local conditions.” (Spigel, 2020[8]). However, it must also be recognised that the institutions and resources – both financial and non-financial – that SMEs and new businesses are aware of and able to draw upon are both extremely diverse and unevenly distributed spatially. These spatial differences may lead to mixed results from policy evaluations as they are applied in different parts of a country.
New and small firms are a highly diverse sub-group of the population of enterprises in a country. Although present in every major sector of the economy, their individual performance is considerably more diverse than for individual larger firms.
For example, an established rule of thumb is that each doubling of size reduces enterprise closure rates by 5% for enterprises up to 500 employees (Hart, 1998[9]). Once enterprises exceed this size, exit is unaffected. Public policies that focus on SMEs must therefore expect high closure rates – especially when the supported firms are both young and small. The problem for policy makers is that entrepreneurship policy may be seen to lack effectiveness if it supports many start-ups that subsequently have only a very short life.
A second source of diversity is that the performance – even of surviving SMEs – is highly variable over time. This implies that programme impacts will vary according to the firms and entrepreneurs targeted. Some, for example, generate much of their revenue in a single month, so requiring considerable financial skills to survive for the rest of the year (Lundmark et al., 2020[10]). Volatility of performance is associated not only with revenue and exit but also with expectations of the future. It has been shown that, again particularly amongst new and small firms, current performance is only a weak guide to future performance. Therefore, if current or prior fast growth were used as the sole basis for identifying future fast growth this would be a serious error5.
A final issue is the diversity of the motivations of the owner(s) of new and small firms. Some seek to grow, whereas others do not. Some report being motivated by a desire for financial independence, others by a desire to avoid involvement with government. In contrast, the assumption that large enterprises seek to maximise shareholder value is more widely accepted. The implication for public policy of this diversity of motivations is that it also leads to a diversity of outcomes. Public policies designed to promote growth, for example, are likely to interest only the minority of new and small firms seeking to grow.
This diversity amongst new and small firms causes problems for governments seeking to help such enterprises. Because of the need to avoid public funds being used inappropriately, clear rules for eligibility for public support have to be drawn up in order to minimise the risk of fraud and avoid favouritism or discrimination. Unfortunately, the generality of these rules inevitably fails to capture the diversity of circumstances in which new and small enterprises find themselves, meaning many appropriate enterprises are excluded and some inappropriate ones included. A second problem, particularly characteristic of publicly-funded training programmes, is that the policy measures fail to capture the highly individualistic circumstances in which individual enterprises find themselves. What is delivered is generic, rather than specific and hence less valued by the business owner.
Those formulating and delivering publicly-funded programmes to support SMEs, and new firms in particular, have to be alert to this diversity and to recognise that what might “work” with one group of firms, at one point in time, might not work consistently. Policymakers need to be keenly aware of such risks. As an example, selective policies targeted towards gazelles – the small proportion of those that grow exceptionally quickly – are risky but potentially productive. The risk comes from such firms only being clearly observable with hindsight; they are much less easily observable at start-up and also subsequently exhibit highly fluctuating rates of growth (Parker, Storey and van Witteloostuijn, 2010[11]). The benefit of such policies is, of course, their function as a “role model” and the credibility they provide for policymakers through this association.
If policymakers decide to concentrate support on such firms – referred to as selective policies – then a decision has to be made on when that support is to be provided; too early and it risks having little effect because most firms die early in life; too late and the firm no longer needs the support, which then becomes impossible to justify on market-failure grounds.6
In short, diversity amongst new and small firms is likely to be a powerful explanation for the “mixed” picture that emerged from the impact evaluations reported in Part II. A greater body of reliable evaluation evidence of the impact of selective policies and the impact of policies on different types of entrepreneur and SME will help better understand these issues and where SME and entrepreneurship policy interventions can be most effective.
A final issue is that of “unintended consequences”. For example, many countries have developed policies to encourage the unemployed to start a business and these have been the subject of evaluation. These show that, although these policies may stimulate the unemployed individual to start a business, other businesses may suffer as a result of the (subsidised) competition from the new firm. For this reason, policy impact cannot be assessed solely by examining what happens to the subsidised businesses or their owners. Instead, impact has to take account of any “displacement7”.
A second illustration is that policy may even have unintended consequences for the target group itself. For example, public subsidies to Korean manufacturing SMEs have been shown to encourage firms to keep below the SME size threshold in order to continue to be in receipt of the subsidy. This, so-called Peter Pan effect, is documented by (Choi and Lee, 2020[12]).
Policy makers and evaluators need to be aware of potential unintended consequences – positive or negative – and seek to pick them up through evaluation.
This chapter has explored a range of explanations for the mixed evaluation results on the impact of SME and entrepreneurship policy. It concludes that the most persuasive of these is the considerably greater diversity of the population of SMEs and entrepreneurs compared with large enterprises.
This implies a need to target and tailor SME and entrepreneurship policies so that they meet the highly-specific needs of different groups of SMEs and entrepreneurs. This is always likely to be problematic. Building a solid body of reliable evaluation evidence on the impacts of selective policies and programmes, in terms of the different populations they target and the different policy instruments they use, and on the impact of non-selective policies on different types of SMEs and entrepreneurs is critical in the task of increasing the effectiveness of SME and entrepreneurship policies. It will provide evidence to help refine the focus of individual schemes towards where they can have the most impacts on the populations targeted and to refine the overall policy mix towards the policies that can have the greatest impacts by targeting the most significant problems facing the most impactful enterprises and entrepreneurs.
References
[2] Acs, Z. et al. (2016), “Public policy to promote entrepreneurship: a call to arms”, Small Business Economics, Vol. 47/1, https://doi.org/10.1007/s11187-016-9712-2.
[3] Arshed, N., S. Carter and C. Mason (2014), “The ineffectiveness of entrepreneurship policy: is policy formulation to blame?”, Small Business Economics, Vol. 43/3, https://doi.org/10.1007/s11187-014-9554-8.
[13] Barrot, J. et al. (2019), “Employment Effects of Alleviating Financing Frictions: Worker-Level Evidence from a Loan Guarantee Program”, SSRN Electronic Journal, https://doi.org/10.2139/ssrn.3409349.
[1] Caliendo, M., S. Künn and M. Weissenberger (2020), “Catching up or lagging behind? The long-term business and innovation potential of subsidized start-ups out of unemployment”, Research Policy, Vol. 49/10, https://doi.org/10.1016/j.respol.2020.104053.
[12] Choi, M. and C. Lee (2020), “The Peter Pan syndrome for small and medium-sized enterprises: Evidence from Korean manufacturing firms”, Managerial and Decision Economics, Vol. 41/3, https://doi.org/10.1002/mde.3111.
[6] Cowling, M. (1998), “Regional Determinants of Small Firm Loans under the U.K. Loan Guarantee Scheme”, Small Business Economics, Vol. 11/2, https://doi.org/10.1023/A:1007956403565.
[14] Daunfeldt, S. and D. Halvarsson (2015), “Are high-growth firms one-hit wonders? Evidence from Sweden”, Small Business Economics, Vol. 44/2, https://doi.org/10.1007/s11187-014-9599-8.
[4] Drews, C. and M. Hart (2015), “Feasibility Study – Exploring the Long-Term Impact of Business Support Services”.
[17] Frankish, J. et al. (2013), “Do entrepreneurs really learn? Or do they just tell us that they do?”, Industrial and Corporate Change, Vol. 22/1, https://doi.org/10.1093/icc/dts016.
[15] Fritsch, M. and S. Kublina (2019), “Persistence and change of regional new business formation in the national league table”, Journal of Evolutionary Economics, Vol. 29/3, https://doi.org/10.1007/s00191-019-00610-5.
[9] Hart, P. (1998), “Job Creation and Destruction in the Corporate Sector: The relative importance of births, deaths and survivors”, National Institute of Economic and Social Research Discussion Papers.
[16] Kobayashi, N. (2020), “A factor analysis of business start-up rates in Japan: contemporary and historical context”, Discussion Paper Series 202, School of Economics, Kwansei Gakuin University.
[10] Lundmark, E. et al. (2020), “The Liability of Volatility and How it Changes Over Time Among New Ventures”, Entrepreneurship: Theory and Practice, Vol. 44/5, https://doi.org/10.1177/1042258719867564.
[11] Parker, S., D. Storey and A. van Witteloostuijn (2010), “What happens to gazelles? The importance of dynamic management strategy”, Small Business Economics, Vol. 35/2, https://doi.org/10.1007/s11187-009-9250-2.
[7] Reynolds, P., D. Storey and P. Westhead (1994), “Cross-national Comparisons of the Variation in New Firm Formation Rates”, Regional Studies, Vol. 28/4, https://doi.org/10.1080/00343409412331348386.
[5] Sedláček, P. and V. Sterk (2017), “The growth potential of startups over the business cycle”, American Economic Review, Vol. 107/10, https://doi.org/10.1257/aer.20141280.
[8] Spigel, B. (2020), Entrepreneurial Ecosystems, Theory, Practice and Futures.
Notes
← 1. The broad rule of thumb is that 40% of new firms cease to trade/ exit within three years (Frankish et al., 2013[17]).
← 2. Clear exceptions might include enterprise education programmes in schools where there would be little expectation of impact in less than a decade.
← 3. In France the Bpifrance loan guarantee programme, although national, is delivered differently in regions. See (Barrot et al., 2019[13]).
← 4. This may have narrowed over time. In Japan in the early 1970s the normalised start-up rates varied across 47 prefectures from highest to lowest by 4.34. By 2012-14 the ratio was 1.85 (Kobayashi, 2020[16]). For Germany, normalised start-up rates in 2006-7 varied by a ratio of 2.41 (Fritsch and Kublina, 2019[15]).
← 5. Evidence from Sweden identified 100 fast growth firms in 1999-2002. It found 2.6 – pro rata – were fast growth in the next three-year period and 0.003 in the next three year period (Daunfeldt and Halvarsson, 2015[14]).
← 6. The late Michael Anyadike-Danes, in a private note to one of the authors, examined Exceptionally Productive Job Creators (EPJCs), defined as those with less than 5 jobs when born (Year 0) but which survived for 15 years and created more than 20 jobs. At that time these EJPCs provided 40% of all jobs, having comprised only 0.5% of all firms in Year 0. The chances of government selecting an EPJC in Year 0 would therefore be extremely low. If government decided to wait until year 5 to identify the EPJCs, this would be an improvement but would only raise the target group to 1.8%. Only if the government selected at Year 5 and chose only those firms that had already created 20 jobs, would it have a large population group to target (32% of cases).
← 7. A full evaluation of such a programme would also take account of cases where the individual would have started the business, even in the absence of the subsidy – referred to as “deadweight.”