Annex A. Methodology for the Small Business Act assessment
The SME Policy Index is a benchmarking tool for emerging economies to monitor and evaluate progress in policies that support SMEs. The tool is structured around the ten principles of the Small Business Act for Europe (SBA), providing a wide range of measures to guide the design and implementation of SME policies. The main objective of the SME Policy Index is to provide governments with a framework to assess policies targeting SME development. The Index identifies strengths and weaknesses in policy design, implementation, and monitoring, allows for comparison across countries and measures convergence towards EU SME policy standards. For more detailed information on the assessment framework and process, please refer to the chapter “Policy framework, structure of the report and assessment process”.
Based on lessons learnt from previous SBA assessments and to capture important changes and emerging trends in the business and policy environment, this cycle benefitted from an updated assessment methodology, which broadened the scope of the analysis to cover new topics, such as policies for SME digitalisation and the responses to the COVID-19 crisis.
This annex provides a detailed explanation of the methodology used to calculate the 2024 SME Policy Index. In particular, i) it outlines the guiding principles of the methodology update, ii) it describes in detail the methodology used to calculate the 2024 scores, and iii) it presents the methodology used to calculate 2020 and 2024 comparable scores to safeguard comparability with the previous assessment.
Guiding principles for the revision of the methodology
The OECD has undertaken a revision of the methodology used to calculate the SME Policy Index, guided by four key principles to enhance its effectiveness and relevance.
1. Firstly, in order to reduce the response burden on respondents, the OECD carried out a comprehensive revision of the questionnaire through which the information used for the computation of the index scores is gathered. Ambiguous questions are clarified, and duplications or obsolete items are removed to streamline the data collection process. Additionally, for the first time, the questionnaire has been delivered using an online platform to improve the overall user experience, ensuring greater efficiency and accessibility.
2. Secondly, the revised methodology seeks to capture emerging economic and policy trends and priorities. Numerous factors contribute to keeping the business environment in a constant state of flux, including technological advancements, globalisation, shifts in consumer behaviour, evolving regulatory landscapes, and unexpected crises. To capture this changing environment, new questions have been added to the questionnaire, encompassing crucial areas, including responses to the COVID-19 crisis and digitalisation. Digital technologies have spread rapidly in recent years and found new applications in many dimensions of our societies, and the COVID-19 pandemic has further accelerated these trends. Digital adoption offers SMEs a clear opportunity to introduce fundamental changes to the way they do business, experiment with new technologies and ultimately increase productivity. To achieve this, policy makers should adopt a comprehensive approach to designing and implementing policies for SME digitalisation (OECD, 2021[1]; OECD, 2021[2]). Government support to SME digitalisation has been chosen as the thematic focus of this edition of the SME Policy Index as the OECD identifies it as a strategic policy priority to modernise EaP economies, increase overall productivity, and bridge the productivity gap between SMEs and larger firms. In particular, the updated methodology underpinning this edition of the SME Policy Index aims to incorporate the progress made by EaP governments to support the digital transformation of SMEs as part of its broader assessment. To capture these policy efforts, the updated methodology includes a dedicated pillar on Digital Economy framework conditions, and six additional digitalisation-oriented sub-dimensions incorporated into the five core assessment pillars (see Box A A.1 for a list of the new digitalisation-oriented sub-dimensions).
3. Thirdly, the revised assessment methodology strengthens the link between policies and outcomes. This involves considering countries’ ability to regularly collect statistical information about outcome-oriented key performance indicators, thereby providing the evidence base for more robust assessments of policy effectiveness.
4. Lastly, while introducing these improvements, the new SME Policy Index strives to safeguard comparability with previous SBA assessments. Consistency in assessment allows for tracking progress over time and facilitates comparisons between different evaluation cycles.
By following these guiding principles, the revised methodology for the SME Policy Index aims to provide governments with a more effective and reliable framework to assess policies supporting SME development and to foster convergence towards EU SME policy standards.
Five digitalisation-oriented sub-dimensions have been added to already existing dimensions of the SBA assessment. In particular:
In the dimension Institutional and Regulatory Framework for SME Policy, the new sub-dimension Institutional Settings for SME Digitalisation analyses whether and how support to SME digitalisation is embedded in the framework for SME policy making.
In the dimension Access to Finance, the new sub-dimension Digital Financial Services investigates the development of regulatory and supervisory frameworks for digital financial services.
In the dimension Standards and Technical Regulations, the new sub-dimension Digitalisation of Standards and Technical Regulation evaluates support and training offered to SMEs on standards and technical regulations for their integration into the EU Digital Single Market; the digitalisation of processes within authorities responsible for technical regulation, national standards and accreditation bodies, metrology institutes, and market surveillance authorities; and the introduction by accreditation bodies, Conformity Assessment Bodies (CABs), and surveillance authorities of remote audit and inspection activities.
In the dimension Business Development Services, the new sub-dimension Business Support Services for the Digital Transformation of SMEs looks at government-led initiatives to support the digital transformation of SMEs. These may include informational support, trainings, financial instruments and advisory services to better understand company needs, procure digital technologies, and develop tailored digital roadmaps.
In addition to these new sub-dimensions, the new pillar Digital Economy Framework Conditions assesses the state of framework conditions for SME digitalisation in EaP countries, looking at:
Besides these newly introduced elements, a number of sub-dimensions already included in previous assessments also cover issues related to digitalisation. These include E-government services and Use of e-commerce.
The Digital Economy Framework Conditions pillar and the mentioned digitalisation-oriented sub-dimensions have been averaged to obtain a composite score for SME digitalisation policies capturing cross-cutting policy efforts to promote digitalisation among SMEs.
Process followed to reach consensus on the new methodology
To gather feedback and endorse the updated SME Policy Index methodology, the OECD organised three dedicated online workshops to present the new methodological approach with key stakeholders. These included government representatives of EaP countries, in particular the SBA co-ordinators, as well as representatives from other international organisations, such as the EU, EBRD, ETF, and UNECE. The revised questionnaire was shared in advance of each meeting to facilitate the discussion.
The three workshops took place in spring 2022:
On 30 May 2022 the OECD presented the guiding principles of the methodology update and described in detail the amendments to Pillar A (Responsive government) and Pillar B (Entrepreneurial human capital).
On 14 June 2022 the discussion focused on changes affecting Pillar C (Access to finance) and Pillar D (Access to markets).
On 21 June 2022 the conversation revolved around updates to Pillar E (Innovation and business support) and a new area of assessment covering Digital economy framework conditions.
The SME Policy Index is calculated through a process articulated in subsequent phases.
Data collection, evaluation, and verification
The information that feeds into the 2024 SME Policy Index is collected through a questionnaire containing about 1 000 questions, grouped by policy dimension and in turn organised in sub-dimensions. In addition, for each dimension a set of outcome-oriented indicators is collected to strengthen the link between policies and outcomes (see above). In turn, each sub-dimension includes a number of thematic blocks, each with their own set of questions. These thematic blocks are typically broken down into the three components or stages of the policy process (design, implementation, monitoring and evaluation).
This approach allows for better monitoring of policy progress and enhances the depth of policy recommendations, while addressing systemic policy issues in a more detailed manner. Each block contains two types of questions: 1) core questions to determine the assessment score; and 2) open questions to acquire further descriptive evidence. Core questions have a defined set of answer options.
The SBA assessment questionnaire is sent to appointed government representatives (the “SBA co-ordinators) in each EaP countries. Through this questionnaire, national governments carry out a self-assessment of their policy frameworks. In parallel, the OECD and its partners conduct an independent assessment via extensive desk research and follow-up with relevant stakeholders in order to fill information gaps and resolve inconsistencies in findings. This process is essential to ensure that questions have been correctly interpreted and that the answers are consistent across countries and, where relevant, over time. To facilitate the verification process, the questionnaire asks national authorities to provide evidence in support of the responses provided to each question. The assessment also benefits from inputs by a team of local experts.
From qualitative information to quantitative values
The qualitative information collected through the core questions of the questionnaire is converted into quantitative values. Each answer option provided to a core question is assigned a score from 0 to 1 where 1 represents alignment with best practice (see Figure A A.1).
Sometimes the answers to two or more questions can be scored jointly. An example can be seen in Figure A A.2 and relates to the two questions: “Are there monitoring mechanisms in place for the implementation of SME policy framework?”, and “If yes, is information about progress in the implementation of the SME policy framework/strategy publicly available?”. In this case, the scores are attributed to a combination of the answers to the two questions and the two questions represent a so called data point.
Final SME policy index scores
Once 0-1 scores are assigned to the core questions, a weighted average is computed to obtain scores at thematic block level. The obtained scores are then converted into the 1-5 scale. Scores for the thematic blocks are then aggregated to attain a score for the sub-dimension, with each component being assigned a weight based on expert consultation. The final step to obtain the overall 1-5 dimension score requires the aggregation of the sub-dimensions scores as well as the score assessing countries ability to collect outcome-oriented indicators. The latter is calculated as the proportion of indicators that a country collects over the total of requested indicators. Sub-dimension scores and scores assessing the ability to collect outcome-oriented indicators are finally aggregated using expert-determined weightings (based on the relative significance of each policy area) to reach an overall 1-5 dimension score. Figure A A.3 presents the structure of the SME policy index and the different levels of aggregations to obtain the final scores at dimension level.
As described at the beginning of this annex, the 2024 assessment questionnaire has undergone a major revision. The differences in the 2024 SBA questionnaire and the 2020 SBA questionnaire imply that the scores as calculated and presented in this report and in the report published in 2020 (OECD et al., 2020[3]) are not immediately comparable. However, the OECD remains committed to safeguarding comparability with previous SBA assessments. To this end, both 2024 scores and 2020 scores have also been recalculated using a comparable methodology that allows to obtain an indication of the direction and magnitude of the policy changes.
While it is true that some new questions have been added and others have been removed, there is a bulk of questions that are common to both assessments. The 2024 and 2020 comparable scores have been calculated on the basis of the sub-set of questions common to both questionnaires (see Figure A A.4).
Once identified the sub-set of common questions to the 2 SBA questionnaires, the responses provided in 2020 and in 2024 have been used to calculate comparable scores according to the same methodology described above.
References
[1] OECD (2021), Beyond COVID-19 Advancing Digital Transformation in the Eastern Partner Countries, OECD Publishing, https://www.oecd.org/eurasia/Beyond%20COVID-19%20Advancing%20Digital%20Transformation%20in%20the%20Eastern%20Partner%20Countries%20.pdf.
[2] OECD (2021), The Digital Transformation of SMEs, https://www.oecd-ilibrary.org/industry-and-services/the-digital-transformation-of-smes_bdb9256a-en.
[3] OECD et al. (2020), SME Policy Index: Eastern Partner Countries 2020: Assessing the Implementation of the Small Business Act for Europe, SME Policy Index, OECD Publishing, Paris/European Union, Brussels, https://doi.org/10.1787/8b45614b-en.