Annex A. Methodology for the 2018 assessment
This section provides a detailed overview of the assessment methodology of the 2018 ASEAN SME Policy Index (ASPI), including an overview of how the SAP SMED 2016-2025 has been integrated into its assessment framework.
For the purpose of this publication and due to varied definitions of micro, small- and medium-sized enterprises across the region, the authors of this report typically use the collective terms “MSME” and “SME” synonymously, unless specified otherwise.
Overview of the 2018 assessment framework and scoring
The 2018 ASPI assessment grid comprises eight dimensions and 25 sub-dimensions. The sub-dimensions are further divided into thematic blocks, each with its own set of indicators. The thematic blocks are typically broken down into three components, representing different stages of the policy cycle: Planning and Design, Implementation and Monitoring and Evaluation. In a few sub-dimensions where this approach is not applicable, for example in relation to the SME definition or the availability of some financial instruments within the access to finance dimension, thematic blocks may differ (Figure A.1).
This approach – slicing scores to reflect different stages of the policy cycle – allows governments to identify and target stages where they face notable strengths or weaknesses.
The assessment framework comprises qualitative and quantitative indicators, which take the following forms:
-
Standard indicators: These are indicators that determine the assessment score and are either binary or multiple choice indicators on qualitative policy measures
-
Core indicators: Like standard indicators, these could be qualitative or quantitative, but due to their importance they are assigned greater weights
-
Open questions: Open questions are included after the standard and core indicators for each sub-dimension in the assessment questionnaire. Open questions are not scored, but help to assess the overall policy context, thus informing the final score.
In the 2018 assessment, countries received graduated scores for each indicator to reflect the depth of policy development, implementation and monitoring. This was necessitated by significant regional disparities in institutional capacity (Box A.1). Scores were calculated based on weighted medians at the thematic block, sub-dimension (at times, also sub-sub-dimension) and dimension levels.
1. Given wide disparities in institutional capacity across ASEAN, the scoring system was honed to reflect the intensity of SME policies and programmes. Thus, instead of simply scoring a yes/no question as 1-yes and 0-no, the following levels were adopted for each indicator:
-
0.0 = no activity
-
0.33 = very little activity, significant limitations faced
-
0.66 = an adequate level of activity, yet limitations remain
-
1.0 = substantial activity, well-run, a case of good practice.
2. The scoring system introduces core indicators with higher weights in areas deemed crucial to enhance the SME policy environment in general or to realise the objectives of the SAP SMED in particular.
3. Overall results run on a scale of 1 to 6, compared to 0 to 5 in other regions.
4. A median is calculated at the regional level for each dimension and sub-dimension, rather than the mean, as is typical in other regions. This is to address substantial regional disparity.
Weights were applied at sub-dimension level and thematic block level in the same way for all AMS. Sub-dimension weights were assigned through a focus-group meeting in October 2017 of some 50 stakeholders, including policy makers from all ten AMS and representatives of academia, the private sector, ERIA and OECD. During this discussion, it was agreed how to assign the weights based on the importance of specific sub-dimensions for countries and the region. The most common thematic blocks – Planning and Design, Implementation and Monitoring and Evaluation – were assigned respective weights of 35%, 45% and 20% in order to emphasise the importance of policy implementation. This is supported by SME Policy Indices developed for other countries.
Development of the 2018 assessment framework
The development of the 2018 assessment framework was informed by two instruments: the generic OECD SME Policy Index methodology and the ASEAN SAP SMED 2016-2025. Pre-existing data collected by the OECD, ERIA and other international organisations have also been incorporated as a thematic block in a number of cases. The framework was also developed in reference to the ERIA-OECD ASEAN SME Policy Index 2014.
The methodology for the report development was agreed upon by the ASEAN Coordinating Committee on Micro Small and Medium Enterprises (ACCMSME) and experts from the region. The methodology for this report was developed with the support of the ASEAN Member States (Brunei Darussalam, Cambodia, Indonesia, Lao PDR, Malaysia, Myanmar, the Philippines, Singapore, Thailand and Viet Nam).
The OECD SME Policy Index methodology
The SME Policy Index is an analytical tool developed by the OECD, in co-operation with international partners, to map SME policies and programmes and to assess alignment with good practice over time. The index was developed for application in non-OECD partner countries within the context of the organisation’s external partner programme. Since its first application in 2007, it has been applied in 32 economies and four regions worldwide.
The main objective of the index is to gather a comprehensive body of information on the policy inputs in each country, to harmonise this information and to transform what are largely qualitative inputs into quantitative indices that can be compared across time and across different economies and regions. By regularly repeating the assessment, typically every three to four years, participating economies can assess their progress in aligning to internationally-recognised good practice, in responding to the needs of their SME population and in converging towards a common set of objectives outlined at regional level.
All SME Policy Index assessments share a common methodology. For each regional application, however, the methodology is adapted to reflect the regional priorities of the economies in question in order to anchor the assessment to the regional policy debate. The assessment is primarily conducted through a questionnaire (the “assessment grid”), which is developed by the OECD and partner organisations, and informed by expert and stakeholder feedback. The framework is comprised of “dimensions” (policy areas), which are further broken down into component “sub-dimensions.”
A set of indicators is identified for each sub-dimension. Most of these are qualitative, but a number of indicators are quantitative, aimed at measuring the “intensity” of policy interventions. The results of the assessment are expressed as numerical indices (scores) on a scale of 0 to 5 – or, in the case of this 2018 ASEAN assessment, on a scale of 1 to 6 – and they are calculated at both sub-dimension and dimension level. To calculate these results, indicator scores are weighted based on perceived importance and relevance.
Integration of the 2016-2025 SAP SMED
The SAP SMED 2016-2025 specifies five strategic goals, 12 desired outcomes and 62 action lines for SME development in ASEAN. The 2018 assessment framework was developed in reference to the priorities identified in the SAP SMED in order to provide policymakers with an additional tool to monitor its implementation.
Figure A.2 and Figure A.3 illustrate the correspondence between the 2018 assessment and the 2016-2025 SAP SMED.
Integration of pre-existing data from other sources
Some thematic blocks take the form of existing OECD, World Bank, and ERIA data (Table A.1). The data for some of these indicators were converted into 4-level (0.33.66) or 5-level (0.25.5.75) scales. The intervals for each level were determined based on two considerations: i) where the indicator had already been used in other OECD SME Policy Index assessments, this scaling system was duplicated (mainly taking the 5-level scale form); ii) where the indicator was used for the first time, a 4-level scale was applied. In the latter case, the interval is determined by the standard deviation of the indicator across ASEAN.
Wherever possible the 2018 assessment strove to supplement the framework’s indicators with additional quantitative information such as the budget allocated, the number of participants in the activity and so on. Due to limited data or comparability, these indicators were often unscored, but were used to inform the scoring of other indicators as well as the assessment write-up.
On account of significant regional disparities, some sub-dimensions and/or indicators were regarded as being inapplicable for a number of ASEAN countries. In these cases, the country was not scored for the sub-dimension and/or indicator in question, and weights were adjusted proportionally. In particular:
-
Brunei Darussalam and Singapore are not scored on sub-sub-dimension 3.2.2 (on the availability microfinance instruments) or sub-dimension 5.3 (on measures to tackle the informal economy), given their small territory and high income.
-
Singapore is also not scored on the indicator pertaining to the presence of a credit guarantee scheme, given the country’s high level of financial sector development and the fact that it provides alternative risk-sharing instruments.
-
Malaysia is not scored on sub-dimension 5.3 (measures to tackle the informal economy), given the relatively marginal level of informality in the country.
The methodology has undergone significant alterations since the 2014 assessment, and therefore only limited comparisons with the previous assessment are possible. These changes have been indicated elsewhere in the report, and include the addition of new indicators, the incorporation of different dimensions and sub-dimensions, the application of new weights, a new graduated approach to scoring and the use of the median to calculate regional scores, rather than the mean that was used previously.