Policy framework and assessment process
Introduction
The SME Policy Index is a benchmarking tool for emerging economies to monitor and evaluate progress in policies that support small and medium-sized enterprises (SMEs). It is structured around the ten principles of the Small Business Act for Europe (SBA), providing a wide range of pro-enterprise measures to guide the design and implementation of SME policies in the European Union (EU). The index was developed in 2006 by the OECD in partnership with the European Commission, the European Bank for Reconstruction and Development (EBRD), and the European Training Foundation (ETF). It has been applied 12 times, covering 33 economies in 4 regions: the Western Balkans and Turkey, Eastern partnership countries, North African and Middle East regions, and the Association of Southeast Asian Nations (ASEAN) countries.
While a number of other indices and benchmarking reports have been used to assess the business environment in the Western Balkans and Turkey (WBT), the SME Policy Index follows a holistic approach that seeks to provide policy makers with a single window through which to assess and, if necessary, readjust policies favouring the development of SMEs. This report identifies the nature and extent of their strengths and weaknesses in SME-related policy settings and their implementation, allowing comparisons to be made across economies and measuring convergence towards OECD and EU good practice.
Overview of the 2019 assessment framework
The SME Policy Index: Western Balkans and Turkey 2019 analyses and benchmarks SME policies across 12 policy areas (“dimensions”). These are further broken down into sub-dimensions and thematic blocks. Each of the 12 policy dimensions comprises up to 5 sub-dimensions that capture the critical elements of policy development in the policy dimension of which they are part. The sub-dimensions are, in turn, made up of thematic blocks that consist of concrete indicators, both quantitative and qualitative (Figure 9).
The SME Policy Index assessment framework was developed to respond to the priorities identified in the ten principles of the SBA. Table 9 shows how the policy dimensions and sub-dimensions relate to each SBA principle.
The SME Policy Index 2019 assessment framework has been adjusted since the 2016 assessment to provide a more in-depth analysis and to better respond to the needs of the economies being assessed. Over the last decade, while most of the economies have demonstrated improvements in building up their institutional frameworks and designing comprehensive policies, the implementation of SME policies has largely remained below par, with limited impact on their local SME populations. In light of this, the assessment framework has been adjusted to account for policy implementation (Box 2).
Compared to the 2016 SME Policy Index, the 2019 assessment includes new features introduced to strengthen the analysis and increase its impact. These refinements include:
-
five new sub-dimensions to include policy aspects that were not covered in the previous cycle
-
extending and amending other sub-dimensions to collect more in-depth information
-
increased focus on the economy’s performance and implementation of SME policies
-
private sector feedback to gauge the outcome of policies
-
greater involvement of national statistical offices and more extensive collection of relevant statistical data.
A detailed description of how the assessment framework has been adjusted is presented in Annex A.
Methodological approach
Policy areas are given a score from 1 to 5, with Level 1 being the weakest and Level 5 the strongest (Table 10). Reaching Levels 4 and 5 is more difficult, requiring solid policy implementation and monitoring and evaluation. Scores are determined by assigning a numerical value to each qualitative indicator. The results are then aggregated with weightings applied at the sub-dimensional and thematic block levels, taking into consideration the importance of each indicator in policy formulation or implementation. A detailed description of the allocation of scores is presented in Annex A.
By assigning scores, the index transforms qualitative information into numerical information, facilitating cross-economy comparison and allowing for systematic monitoring of policy developments.
In order to make it easier to compare performance across economies and over time, quantitative indicators are used to provide additional evidence on the performance of policy settings, processes and programmes. These include horizontal statistical data, to better understand the nature of the SME sector; and data specific to individual dimensions to assess policy outputs and outcomes relevant for implementation. Although the quantitative data are not included in the scoring, they have been taken into account in the narrative text, including the analysis of the overall SME policy setting and formulation of policy recommendations. The quantitative data were primarily collected by the OECD from national statistical offices.
The assessment process
The SME Policy Index is based on the results of two parallel assessments. The economies’ governments conducted a self-assessment, which involved completing a questionnaire and assigning a score for each policy indicator using the assessment grid, and providing an accompanying justification. The OECD and its partner organisations also conducted an independent assessment, with input from a team of local experts.
The final scores are the result of a highly collaborative and consultative consolidation of these two assessments, enhanced by further desk research by the OECD and the partner organisations, as well as consultations with government representatives. To achieve this, a series of reconciliation meetings were held in each of the assessed economies to discuss and compare the two parallel assessments, which helped to reduce bias and misjudgements (see below for more details).
Based on the information gathered in these reconciliation meetings, the OECD and its partner organisations then decided the final results. These were presented to the SBA Co-ordinators in a regional meeting at the end of the assessment process. Minor modifications of the scores were subsequently made.
Assessment methodology for Dimensions 1 and 8a
For this assessment, the methodology for the two human capital dimensions, Dimensions 1 and 8a, has been harmonised with the remaining dimensions. Previously, Dimensions 1 and 8a were assessed using 5-level qualitative indicators. Moving to the questionnaire, with its binary (yes/no), multiple-choice and open questions, has harmonised the assessment methodology across all the SBA dimensions (the other dimensions were first assessed in this way in 2016). However, comparison of the scores between the 2016 and 2019 assessments need to be made with caution because of this change.
For Dimensions 1 and 8a, the self-assessments took place with the support of an ETF local expert. Focus group meetings on each of the three sub-dimensions – entrepreneurial learning, women’s entrepreneurship and SME skills – were held in each economy, attracting between 10 and 75 stakeholders from all relevant sectors. The aim was to validate and interpret the initial assessment findings, and identify additional evidence to strengthen the analysis. The final results of the assessment have been agreed between the ETF and the national counterparts.
Assessment phases and timing of the 2019 assessment
The assessment was carried out between January 2017 and January 2019 in four main phases: 1) design; 2) data collection and evaluation; 3) consolidation; and 4) review and publication (Figure 10).
-
Design phase (January 2017-August 2017). The OECD and the partner institutions revised the assessment framework in consultation with the SBA Co-ordinators, partner organisations and the European Commission. The assessment methodology for Dimensions 1 and 8a was harmonised with the remaining dimensions.
-
Data collection and evaluation phase (September 2017-February 2018). Launch meetings were held in the seven capitals of the assessed economies to present the new assessment framework to the line ministries and public institutions which were expected to contribute to the information collection process. The two documents making up the assessment framework – the questionnaire and statistical data sheet – were explained in depth to the participants, directing particular attention to the newly added questions and indicators.
Following the launch events, the SBA Co-ordinators distributed the questionnaires to the appropriate counterparts in the ministries and government agencies, as well as the statistical sheet to the national statistical offices. The SBA Co-ordinators then compiled the data and documentation, and completed the questionnaire. Scores were assigned for each policy dimension, with an accompanying justification. The completed questionnaires and statistical data sheets were sent to the OECD team by February 2018. Meanwhile, the OECD and its partner organisations also conducted an independent assessment. This was based on input from a team of local experts who collected data and information, and conducted interviews with SME representatives as well as key public and civil society stakeholders. A detailed description of the interviews is presented in Annex C.
-
Consolidation phase (February 2018-August 2018). Reviews of the inputs by the OECD and the partner institutions revealed the need for additional information on certain elements. These were requested and the revised questionnaires were sent back during February-April 2018.
Once the data collection was complete, the OECD and EBRD organised reconciliation meetings in each of the assessed economies. These meetings were typically attended by 60-80 SME policy stakeholders, including ministries and government agencies, international donors, and representatives from civil society, academia, non-government organisations, the private sector and the EU delegations. These meetings aimed to close any remaining information gaps in the questionnaire, discussing any discrepancies between the two parallel assessments and filling in missing information. They also served as an opportunity to seek the views of a broad range of policy stakeholders on how SMEs are affected by current policies, and what else can be done across various policy areas to improve SMEs’ performance and competitiveness in the Western Balkans and Turkey.
For the two policy dimensions (entrepreneurial learning and women’s entrepreneurship, and enterprise skills) whose assessment was led by the ETF, similar meetings took place during February-March 2018, as described above.
After the reconciliation meetings, the assessment findings were consolidated. The OECD and partner organisations decided on the final scores under each policy dimension presented in this report. The preliminary findings and scores were subsequently presented to SBA Co-ordinators at a meeting in Paris on 12 July 2018.
-
Review and publication phase (September 2018-May 2019). The draft SME Policy Index chapters were made available to the WBT governments, the European Commission and the EU delegations for review during September-December 2018. Meanwhile, the draft report was also peer reviewed internally by the OECD, the EBRD and the ETF. Following the review, final comments were integrated into the report. The publication was launched on 7 May 2019 in Sarajevo.
Strengths and limitations
The SME Policy Index: Western Balkans and Turkey 2019 has a number of strengths which make it a uniquely valuable report for policy makers, citizens, researchers and international organisations. However, it also has some limitations which need to be borne in mind (Table 11).