Annex D. Brief description of included evaluation methods

Evaluation Method	Description	Reference and further reading
Randomized controlled trial (RCT)	“With this methodological approach, beneficiaries are randomly selected to receive an intervention, and each has an equal chance of receiving the support. With large-enough sample sizes, the process of random assignment ensures equivalence, in both observed and unobserved characteristics, between the treatment and control groups, thereby addressing selection bias.”	(Gertler et al., 2016[79]) (p. 233; further pp. 49-81)
Simple difference-in-differences (DiD) approach	“Difference-in-differences estimates the counterfactual for the change in outcome for the treatment group by taking the change in outcome for the comparison group. This method allows us to take into account any differences between the treatment and comparison groups that are constant over time. The two differences are thus before and after, and between the treatment and comparison groups. It is also known as ‘double difference’ or ‘DD’.”	(Gertler et al., 2016[79]) (p. 230; further details on pp. 95-106)
Multivariate regression models	“Regression analysis includes any techniques for modelling and analyzing several variables, when the focus is on the relationship between a dependent variable and one or more independent variables. In impact evaluation, regression analysis helps us understand how the typical value of the outcome indicator (dependent variable) changes when the assignment to treatment or comparison group (independent variable) is varied, while the characteristics of the beneficiaries (other independent variables) are held fixed.” Preferably we estimate effects over time, benefiting from longitudinal research design. The following types of regression analyses occur in the handbook: Ordinary Least Squares (OLS) regressions, Random effects regressions, Fixed effects regressions, Quantile regressions, Generalized Method of Moments (GMM) regressions.	(Gertler et al., 2016[79]) (p. 234) See (Cameron and Trivedi, 2005[80]) for details on different types of regression techniques.
Two-stage Heckman selection model	“A statistical technique used to correct bias from non-randomly selected samples in a two-step approach. The approach begins by estimating a so-called selection equation that estimates the individual probability of each observation to be supported by the intervention. Then, taking into account the factors included in the selection equation is estimated a second equation explaining the change in the outcome variables.”	(Storey, 2017[81]), (Heckman, 1976[82])
Statistical matching techniques	“Statistical matching is a nonexperimental evaluation method that uses large data sets and heavy statistical techniques to construct the best possible comparison group for a given treatment group.” The matching is conducted based on the observable (available) characteristics of treatment and comparison (control) groups. The following types of matching techniques occur in the handbook: Propensity score matching (PSM), Radius matching (RM), Direct Covariate Matching (DCM), Nearest Neighbour Matching (NNM), Coarsened exact matching (CEM).	(Gertler et al., 2016[79]) (p. 232; further details on pp. 107-116) See (Khandker, Koolwal and Samad, 2010[1]), (Stuart, 2010[83]) or (Rässler, 2012[84]) for details on different types of matching techniques
Instrumental variables (IV) approach	“An instrumental variable approach involves finding a variable (or instrument) that is highly correlated with program placement or participation but that is not correlated with unobserved characteristics affecting outcomes. Instruments can be constructed from program design (for example, if the program of interest was randomized or if exogenous rules were used in determining eligibility for the program).”	(Khandker, Koolwal and Samad, 2010[1]) (p. 87)
Regression discontinuity design (RDD)	“Regression discontinuity design is a nonexperimental evaluation method that adequate for programs that use a continuous index to rank potential beneficiaries and that have a threshold along the index that determines whether potential beneficiaries receive the public support or not. The cutoff threshold for program eligibility provides a dividing point between the treatment and comparison groups” allowing the impact of an intervention to be quantified.	(Gertler et al., 2016[79]) (p. 234; further details on pp. 81-93)
Note: Some of the methods are often combined (e. g. multivariate regression analysis or matching techniques with difference-in-differences approach).

References

[32] Amezcua, A. (2010), Boon or Boondoggle? Business Incubation as Entrepreneurship Policy, Syracuse University ProQuest Dissertations Publishing.

[78] Amezcua, A. et al. (2013), “Organizational sponsorship and founding environments: A contingency view on the survival of business-incubated firms, 1994-2007”, Academy of Management Journal, Vol. 56/6, https://doi.org/10.5465/amj.2011.0652.

[87] Anyadike-Danes, M. and M. Hart (2018), “All grown up? The fate after 15 years of a quarter of a million UK firms born in 1998”, Journal of Evolutionary Economics, Vol. 28/1, https://doi.org/10.1007/s00191-017-0549-x.

[18] Arráiz, I., F. Henríquez and R. Stucchi (2013), “Supplier development programs and firm performance: Evidence from Chile”, Small Business Economics, Vol. 41/1, https://doi.org/10.1007/s11187-012-9428-x.

[30] Arvanitis, S., L. Donzé and N. Sydow (2010), “Impact of Swiss technology policy on firm innovation performance: An evaluation based on a matching approach”, Science and Public Policy, Vol. 37/1, https://doi.org/10.3152/030234210X491623.

[25] Autio, E. and H. Rannikko (2016), “Retaining winners: Can policy boost high-growth entrepreneurship?”, Research Policy, Vol. 45/1, https://doi.org/10.1016/j.respol.2015.06.002.

[8] Banai, Á. et al. (2017), “Impact evaluation of EU subsidies for economic development on the Hungarian SME sector.”, MNB Working Papers 8.

[45] Behrenz, L., L. Delander and J. Månsson (2016), “Is Starting a Business a Sustainable way out of Unemployment? Treatment Effects of the Swedish Start-up Subsidy”, Journal of Labor Research, Vol. 37/4, https://doi.org/10.1007/s12122-016-9233-4.

[61] Bellmann, L., M. Caliendo and S. Tübbicke (2018), “The Post-Reform Effectiveness of the New German Start-Up Subsidy for the Unemployed”, Labour, Vol. 32/3, https://doi.org/10.1111/labr.12126.

[70] Beņkovskis, K., O. Tkačevs and N. Yashiro (2019), “Importance of EU regional support programmes for firm performance”, Economic Policy, Vol. 34/98, https://doi.org/10.1093/epolic/eiz003.

[48] Bernini, C. and G. Pellegrini (2011), “How are growth and productivity in private firms affected by public subsidy? Evidence from a regional policy”, Regional Science and Urban Economics, Vol. 41/3, https://doi.org/10.1016/j.regsciurbeco.2011.01.005.

[28] Bertoni, F., J. Martí and C. Reverte (2019), “The impact of government-supported participative loans on the growth of entrepreneurial ventures”, Research Policy, Vol. 48/1, https://doi.org/10.1016/j.respol.2018.09.006.

[63] Brachert, M., E. Dettmann and M. Titze (2019), “The regional effects of a place-based policy – Causal evidence from Germany”, Regional Science and Urban Economics, Vol. 79, https://doi.org/10.1016/j.regsciurbeco.2019.103483.

[47] Brachert, M., E. Dettmann and M. Titze (2018), Public Investment Subsidies and Firm Performance-Evidence from Germany, https://doi.org/10.1515/jbnst-2017-0131.

[54] Broocks, A. and J. Van Biesebroeck (2017), “The impact of export promotion on export market entry”, Journal of International Economics, Vol. 107, https://doi.org/10.1016/j.jinteco.2017.03.009.

[16] Brown, J. and J. Earle (2017), “Finance and Growth at the Firm Level: Evidence from SBA Loans”, Journal of Finance, Vol. 72/3, https://doi.org/10.1111/jofi.12492.

[21] Bruhn, M., D. Karlan and A. Schoar (2018), “The impact of consulting services on small and medium enterprises: Evidence from a randomized trial in Mexico”, Journal of Political Economy, Vol. 126/2, https://doi.org/10.1086/696154.

[26] Bruhn, M. and D. McKenzie (2019), “Can Grants to Consortia Spur Innovation and Science-Industry Collaboration? Regression-Discontinuity Evidence from Poland”, World Bank Economic Review, Vol. 33/3, https://doi.org/10.1093/wber/lhx014.

[77] Buffart, M. et al. (2019), “Picking the Right Winners: Government Business Support Programs and Entrepreneurial Growth”, Academy of Management Proceedings, Vol. 2019/1, https://doi.org/10.5465/ambpp.2019.10868abstract.

[14] Burger, A. and M. Rojec (2018), “Impotence of Crisis-Motivated Subsidization of Firms: The Case of Slovenia”, Eastern European Economics, Vol. 56/2, https://doi.org/10.1080/00128775.2017.1416294.

[62] Caliendo, M., S. Künn and M. Weissenberger (2020), “Catching up or lagging behind? The long-term business and innovation potential of subsidized start-ups out of unemployment”, Research Policy, Vol. 49/10, https://doi.org/10.1016/j.respol.2020.104053.

[60] Caliendo, M., S. Künn and M. Weißenberger (2016), “Personality traits and the evaluation of start-up subsidies”, European Economic Review, Vol. 86, https://doi.org/10.1016/j.euroecorev.2015.11.008.

[41] Caliendo, M. and S. Tübbicke (2020), “New evidence on long-term effects of start-up subsidies: matching estimates and their robustness”, Empirical Economics, Vol. 59/4, https://doi.org/10.1007/s00181-019-01701-9.

[80] Cameron, A. and P. Trivedi (2005), Microeconometrics: methods and applications. Cambridge university press..

[46] Capelleras, J., I. Contín-Pilart and M. Larraza-Kintana (2011), “Publicly funded prestart support for new firms: Who demands it and how it affects their employment growth”, Environment and Planning C: Government and Policy, Vol. 29/5, https://doi.org/10.1068/c10110b.

[9] Caselli, S. et al. (2019), “Public Credit Guarantee Schemes and SMEs’ Profitability: Evidence from Italy”, Journal of Small Business Management, Vol. 57/S2, https://doi.org/10.1111/jsbm.12509.

[50] Cerqua, A. and G. Pellegrini (2017), “Industrial policy evaluation in the presence of spillovers”, Small Business Economics, Vol. 49/3, https://doi.org/10.1007/s11187-017-9855-9.

[49] Cerqua, A. and G. Pellegrini (2014), “Do subsidies to private capital boost firms’ growth? A multiple regression discontinuity design approach”, Journal of Public Economics, Vol. 109, https://doi.org/10.1016/j.jpubeco.2013.11.005.

[4] Chandler, V. (2012), “The economic impact of the Canada small business financing program”, Small Business Economics, Vol. 39/1, https://doi.org/10.1007/s11187-010-9302-7.

[68] Comi, S. and L. Resmini (2020), “Are export promotion programs effective in promoting the internalization of SMEs?”, Economia Politica, Vol. 37/2, https://doi.org/10.1007/s40888-019-00170-8.

[15] Cowling, M. et al. (2018), “Loan guarantee schemes in the UK: the natural experiment of the enterprise finance guarantee and the 5 year rule”, Applied Economics, Vol. 50/20, https://doi.org/10.1080/00036846.2017.1392004.

[75] Cowling, M., E. Ughetto and N. Lee (2018), “The innovation debt penalty: Cost of debt, loan default, and the effects of a public loan guarantee on high-tech firms”, Technological Forecasting and Social Change, Vol. 127, https://doi.org/10.1016/j.techfore.2017.06.016.

[51] Criscuolo, C. et al. (2019), “Some causal effects of an industrial policy”, American Economic Review, Vol. 109/1, https://doi.org/10.1257/aer.20160034.

[44] Cueto, B., M. Mayor and P. Suárez (2017), “Evaluation of the Spanish flat rate for young self-employed workers”, Small Business Economics, Vol. 49/4, https://doi.org/10.1007/s11187-017-9853-y.

[17] Cumming, D. and E. Fischer (2012), “Publicly funded business advisory services and entrepreneurial outcomes”, Research Policy, Vol. 41/2, https://doi.org/10.1016/j.respol.2011.09.004.

[3] Decramer, S. and S. Vanormelingen (2016), “The effectiveness of investment subsidies: evidence from a regression discontinuity design”, Small Business Economics, Vol. 47/4, https://doi.org/10.1007/s11187-016-9749-2.

[39] Duhautois, R., D. Redor and L. Desiage (2015), Long term effect of public subsidies on start-up survival and economic performance: An empirical study with french data, https://doi.org/10.4000/rei.6063.

[5] Dvouletý, O. and I. Blažková (2019), “Assessing the microeconomic effects of public subsidies on the performance of firms in the czech food processing industry: A counterfactual impact evaluation”, Agribusiness, Vol. 35/3, https://doi.org/10.1002/agr.21582.

[56] Dvouletý, O. and I. Blažková (2019), “The impact of public grants on firm-level productivity: Findings from the Czech food industry”, Sustainability (Switzerland), Vol. 11/2, https://doi.org/10.3390/su11020552.

[6] Dvouletý, O., J. Čadil and K. Mirošník (2019), “Do Firms Supported by Credit Guarantee Schemes Report Better Financial Results 2 Years After the End of Intervention?”, B.E. Journal of Economic Analysis and Policy, Vol. 19/1, https://doi.org/10.1515/bejeap-2018-0057.

[57] Dvouletý, O. et al. (2018), “Are publicly funded Czech incubators effective? The comparison of performance of supported and non-supported firms”, European Journal of Innovation Management, Vol. 21/4, https://doi.org/10.1108/EJIM-02-2018-0043.

[58] Einiö, E. (2014), “R&D subsidies and company performance: Evidence from geographic variation in government funding based on the ERDF population-density rule”, Review of Economics and Statistics, Vol. 96/4, https://doi.org/10.1162/REST_a_00410.

[53] European Commission (2019), Explanatory note on the paper of the services of DG Competition containing draft regional aid guidelines 2014-2020.

[52] European Commission (2014), Common methodology for state aid evaluation.

[37] Fairlie, R., D. Karlan and J. Zinman (2015), “Behind the GATE experiment: Evidence on effects of and rationales for subsidized entrepreneurship training”, American Economic Journal: Economic Policy, Vol. 7/2, https://doi.org/10.1257/pol.20120337.

[36] Georgiadis, A. and C. Pitelis (2016), “The Impact of Employees’ and Managers’ Training on the Performance of Small- and Medium-Sized Enterprises: Evidence from a Randomized Natural Experiment in the UK Service Sector”, British Journal of Industrial Relations, Vol. 54/2, https://doi.org/10.1111/bjir.12094.

[79] Gertler, P. et al. (2016), Impact Evaluation in Practice, Second Edition, https://doi.org/10.1596/978-1-4648-0779-4.

[23] Girma, S. et al. (2010), “The effect of grant receipt on start-up size: Evidence from plant level data”, Journal of International Entrepreneurship, Vol. 8/4, https://doi.org/10.1007/s10843-010-0061-y.

[7] Hartšenko, J. and A. Sauga (2013), “The role of financial support in SME and economic development in Estonia”, Business and Economic Horizons, Vol. 9/2, https://doi.org/10.15208/beh.2013.6.

[82] Heckman, J. (1976), “The Common Structure of Statistical Models of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models”, National Bureau of Economic Research.

[85] Hoffmann, A. and D. Storey (2017), “Can Governments Promote Gazelles? Evidence from Denmark”, in The SAGE Handbook of Small Business and Entrepreneurship, https://doi.org/10.4135/9781473984080.n19.

[65] Johan, S. and P. Valenzuela (2021), “Business Advisory Services and Female Employment in an Extreme Institutional Context”, British Journal of Management, Vol. 32/4, https://doi.org/10.1111/1467-8551.12429.

[1] Khandker, S., G. Koolwal and H. Samad (2010), Handbook on Impact Evaluation: Quantitative Methods and Practices.

[69] Kim, Y., I. Oh and J. Lee (2015), “Economic impact assessment of public-private matching fund programs using firm-level data”, Singapore Economic Review, Vol. 60/4, https://doi.org/10.1142/S0217590815500605.

[59] Koski, H. and M. Pajarinen (2013), “The role of business subsidies in job creation of start-ups, gazelles and incumbents”, Small Business Economics, Vol. 41/1, https://doi.org/10.1007/s11187-012-9420-5.

[13] Lopez-Acevedo, G. and M. Tinajero (2010), “Mexico impact evaluation of SME programs using panel firm data”, World Bank Policy Research Working Paper Series.

[67] Lukeš, M., M. Longo and J. Zouhar (2019), “Do business incubators really enhance entrepreneurial growth? Evidence from a large sample of innovative Italian start-ups”, Technovation, Vol. 82-83, https://doi.org/10.1016/j.technovation.2018.07.008.

[43] Mariani, M. et al. (2019), “The ambiguous effects of public assistance to youth and female start-ups between job creation and entrepreneurship enhancement”, Scienze Regionali, Vol. 18/2, https://doi.org/10.14650/93649.

[64] Marshall, C. (2010), “Is the Tax Credit for SME in Chile an Effective Policy to Boost Investment”, CID Research Fellow and Graduate Student Working Paper Series.

[38] Martínez, A., E. Puentes and J. Ruiz-Tagle (2018), “The effects of micro-entrepreneurship programs on labor market performance: Experimental evidence from Chile”, American Economic Journal: Applied Economics, Vol. 10/2, https://doi.org/10.1257/app.20150245.

[74] Martín-García, R. and J. Morán Santor (2021), “Public guarantees: a countercyclical instrument for SME growth. Evidence from the Spanish Region of Madrid”, Small Business Economics, Vol. 56/1, https://doi.org/10.1007/s11187-019-00214-0.

[22] Mole, K. et al. (2011), “Broader or deeper? exploring the most effective intervention profile for public small business support”, Environment and Planning A, Vol. 43/1, https://doi.org/10.1068/a43268.

[35] Nagel, H. et al. (2019), “The effect of a tax training program on tax compliance and business outcomes of starting entrepreneurs: Evidence from a field experiment”, Journal of Business Venturing, Vol. 34/2, https://doi.org/10.1016/j.jbusvent.2018.10.006.

[12] Namiotko, V. et al. (2019), “The impact of investment support on labour productivity in Lithuanian family farms: A propensity score matching approach”, Economics and Sociology, Vol. 12/1, https://doi.org/10.14254/2071-789X.2019/12-1/21.

[24] Navarro, L. (2018), “Entrepreneurship policy and firm performance Chile’s CORFO seed capital program”, Estudios de Economia, Vol. 45/2, https://doi.org/10.4067/S0718-52862018000200301.

[72] Nemethova, V., M. Siranova and M. Sipikal (2019), “Public support for firms in lagging regions-evaluation of innovation subsidy in Slovakia”, Science and Public Policy, Vol. 46/2, https://doi.org/10.1093/scipol/scy046.

[42] Oberschachtsiek, D. (2015), “The outcome of coaching and training for self-employment. A statistical evaluation of outside assistance support programs for unemployed business founders in Germany”, Journal for Labour Market Research, Vol. 48/1, https://doi.org/10.1007/s12651-014-0161-6.

[11] Oh, I. et al. (2009), “Evaluation of credit guarantee policy using propensity score matching”, Small Business Economics, Vol. 33/3, https://doi.org/10.1007/s11187-008-9102-5.

[10] Ono, A., I. Uesugi and Y. Yasuda (2013), “Are lending relationships beneficial or harmful for public credit guarantees? Evidence from Japan’s Emergency Credit Guarantee Program”, Journal of Financial Stability, Vol. 9/2, https://doi.org/10.1016/j.jfs.2013.01.005.

[34] Oosterbeek, H., M. van Praag and A. Ijsselstein (2010), “The impact of entrepreneurship education on entrepreneurship skills and motivation”, European Economic Review, Vol. 54/3, https://doi.org/10.1016/j.euroecorev.2009.08.002.

[66] Pellegrini, G. and T. Muccigrosso (2017), “Do subsidized new firms survive longer? Evidence from a counterfactual approach”, Regional Studies, Vol. 51/10, https://doi.org/10.1080/00343404.2016.1190814.

[55] Pelucha, M., V. Kveton and O. Potluka (2019), “Using mixed method approach in measuring effects of training in firms: Case study of the European Social Fund support”, Evaluation and Program Planning, Vol. 73, https://doi.org/10.1016/j.evalprogplan.2018.12.008.

[33] Potluka, O. et al. (2016), “Counterfactual impact evaluation on EU cohesion policy interventions in training in companies”, Ekonomicky casopis, Vol. 64/6.

[84] Rässler, S. (2012), Statistical Matching: A Frequentist Theory, Practical Applications, and Alternative Bayesian Approaches.

[86] Rässler, S. (2002), Statistical matching: A frequentist theory, practical applications, and alternative Bayesian approaches.

[19] Rotger, G., M. Gørtz and D. Storey (2012), “Assessing the effectiveness of guided preparation for new venture creation and performance: Theory and practice”, Journal of Business Venturing, Vol. 27/4, https://doi.org/10.1016/j.jbusvent.2012.01.003.

[27] Santos, A. (2019), “Do selected firms show higher performance? The case of Portugal’s innovation subsidy”, Structural Change and Economic Dynamics, Vol. 50, https://doi.org/10.1016/j.strueco.2019.04.003.

[71] Santos, A. et al. (2019), “Which projects are selected for an innovation subsidy? The Portuguese case”, Portuguese Economic Journal, Vol. 18/3, https://doi.org/10.1007/s10258-019-00159-y.

[73] Schweiger, H. (2012), “The Impact of State Aid for Restructuring on the Allocation of Resources”, SSRN Electronic Journal, https://doi.org/10.2139/ssrn.2180457.

[88] Sedláček, P. and V. Sterk (2017), “The growth potential of startups over the business cycle”, American Economic Review, Vol. 107/10, https://doi.org/10.1257/aer.20141280.

[29] Söderblom, A. et al. (2015), “Inside the black box of outcome additionality: Effects of early-stage government subsidies on resource accumulation and new venture performance”, Research Policy, Vol. 44/8, https://doi.org/10.1016/j.respol.2015.05.009.

[81] Storey, D. (2017), “Six Steps to Heaven: Evaluating the Impact of Public Policies to Support Small Businesses in Developed Economies”, in Sexton, D. and H. Landström (eds.), The Blackwell handbook of entrepreneurship.

[83] Stuart, E. (2010), Matching methods for causal inference: A review and a look forward, https://doi.org/10.1214/09-STS313.

[31] Tandogan, V. and M. Pamukcu (2011), “Evaluating effectiveness of public support to business R&D in Turkey through concepts of input and output additionality”, Economic Research Forum Working Paper Series.

[20] Ungerer, C. et al. (2019), The Effect of Business Coaching on NTBF Survival-Findings and Lessons Learned from a Randomized Controlled Trial, https://doi.org/10.1109/ICE.2019.8792604.

[40] Wolff, J., A. Nivorozhkin and S. Bernhard (2016), “You can go your own way! The long-term effectiveness of a self-employment programme for welfare recipients in Germany”, International Journal of Social Welfare, Vol. 25/2, https://doi.org/10.1111/ijsw.12176.

[2] Xiang, D. and A. Worthington (2017), “The impact of government financial assistance on the performance and financing of Australian SMEs”, Accounting Research Journal, Vol. 30/4, https://doi.org/10.1108/ARJ-04-2014-0034.

[76] Zhang, S. (2019), “Rethinking U.S. enterprise zones: The role of research design in program evaluation”, Local Economy, Vol. 34/6, https://doi.org/10.1177/0269094219871955.