Annex A. Methodology for the 2019 Small Business Act assessment:
This section provides a detailed overview of the assessment methodology for the 2019 SME Policy Index.
Overview of the 2019 assessment framework and scoring
The process comprises two parallel assessments, a self-assessment by the government and an independent assessment by local consultants (see the Policy Framework and Assessment Process chapter).
The assessment grid is built upon the ten principles of the SBA, divided into 34 sub-dimensions. The sub-dimensions are usually also divided into thematic blocks, each with its own set of indicators. The thematic blocks are typically broken down into three components, representing different stages of the policy cycle: planning and design, implementation, and monitoring and evaluation (Figure A A.1). In few sub-dimensions where this approach is not applicable, for example in relation to the SME definition or the availability of some financial instruments within the access to finance dimension, thematic blocks may differ.
The approach of assigning scores to reflect different stages of the policy cycle, allows governments to identify and target the stages where they have particular strengths or weaknesses. Each policy dimension, and their constituting parts, is assigned a numerical score ranging from 1 to 5 according to the level of policy development and implementation, so that performance can be compared across economies and over time. Level 1 is the weakest level, whist the level 5 is the strongest, indicating a level of development commensurate with OECD good practices (Table A A.1). The levels are determined through a participatory and analytical process, which is conducted by means of a self-assessment by the government and an independent assessment by local consultants, supplemented by private-sector interviews.
The assessment framework comprises qualitative and quantitative indicators, which are scored based on questions in the following forms:
-
Binary questions, allowing yes or no answers, such as “Does a legal definition of SMEs exist in your country?”
-
Multiple-choice questions, such as “Does a multi-year SME strategy exist?” or “Does an SME policy implementation body or equivalent exist?” These questions offer a drop-down list of answers, such as “strategy is in the process of development”, “draft strategy exists but not yet approved by the government”, “strategy exists, it has been approved by the government and is in the process of implementation”, and “there is no strategy in development”.
-
Open-ended questions, acquiring further descriptive evidence to add to the binary and multiple-choice questions. These questions are not themselves scored. Examples include “What is the budget for the SME implementation agency?”, “How many people work in the agency?” and “How many ministries are represented in the governance board?”
Whenever possible, economies are asked to support their answers with evidence, by providing source documents such as strategies and budget plans. Each questionnaire also includes definitions of some of the terms used in the questions, as well as references and examples to help respondents answer the questions.
Each of the core questions is scored equally within the thematic block. For binary questions, a “yes” is awarded full points and a “no” receives zero points. For multiple-choice questions, scores for the different options range between zero and full points, depending on the indicated level of policy development.
The scores for the core questions are then added up to provide a score for each component. Scores are initially derived as percentages (0-100) and then converted into the 1-5 scale. Scores for the thematic blocks are then aggregated to attain a score for the sub-dimension, with each thematic block being weighted based on expert consultation. In general, planning and design accounts for 30% of the score, implementation 50%, and monitoring and evaluation 20%, to emphasise the importance of policy implementation (Figure A A.2). Weights were applied at sub-dimension level and thematic block level in the same way.
Main changes to the assessment framework since the 2016 SBA assessment
Since 2016, the SBA assessment framework has been revised to provide a more in-depth analysis of SME polices by covering the cutting-edge aspects. By asking more detailed questions about some institutions and policy measures, the assessment allows the formulation of more concrete and targeted recommendations.
Five new sub-dimensions have been added to the assessment and others expanded and amended to include policy aspects that were not covered in the previous cycle. The policy dimensions and sub-dimension weightings have been adjusted to allow for these additions while maintaining as much comparability with the 2016 assessment as possible.
Greater emphasis on the implementation of policies
In the previous assessment cycle, the assessed economies had made substantial progress in establishing policy frameworks across different areas. However, solid policy designs and institutional set-ups do not always translate into effective policy implementation. In order to better assess the effectiveness of SME policies, the assessment has been revised, to place more emphasis on their implementation.
The weightings have therefore been adjusted and the implementation stage of the policy cycle is now allocated a weight of 50%, compared to 45% in the previous assessment. While the weight assigned to the planning and design stage has fallen to 30%, the weight for monitoring and evaluation has remained the same.
Private-sector feedback has been included to gauge the outcome of policies
The new assessment cycle endeavoured to measure policy outcomes more effectively by paying more attention to the insights of representatives from the private sector and civil society on the effectiveness of SME policies and support measures. In addition to shedding light on the actual state of policy implementation, their feedback also helped to uncover any unintended effects of existing policies. Standardised interviews were used for SME owners and managers and representatives of chambers of commerce, while the views of civil society representatives were captured by the local consultants when they were filling out questionnaires.
The face-to-face interviews with SME owners and managers were in two parts. In the first part, the interviewees were asked screening questions to collect information about their firms’ characteristics. In the second part, interviewees were asked questions related to SME policies, covering the various SME Policy Index dimensions. Figure A A.3 illustrates the two-part structure of interviews with examples of questions for Dimension 2: Bankruptcy and second chance for SMEs. A detailed description of the interviews is presented in Annex C.
Statistical data have been collected to allow improved comparison of performance over time
As some economies have reached high levels of convergence with the SBA principles (Level 3 or above) and advanced levels of policy implementation, information about the performance of policies became more important. The revised approach has enhanced its use of several types of statistics, which complement the information acquired from the two assessments. The approach integrates national statistics, company-level data from other international organisations, or independent company surveys to measure the performance of policies on the ground. Table A A.3 provides an overview of the types of statistics and data collected, their purpose within the SME Policy Index, and their main sources.
The enhanced data collection also led to the greater involvement of national statistical offices in this cycle.