Annex A. Reader’s guide
Most of the data presented in this Outlook, including the composite indicators, are the results of the 2014, 2017 and 2021 Regulatory Indicator Surveys. This Reader’s guide aims to help readers understand the scope of the data collected through these surveys and some of the limitations related to the use of indicators. Please note that this edition of the Outlook also features results of new survey questions that were designed in conjunction with the Measuring Regulatory Performance (MRP) Steering Group on ex post evaluation, reflecting the developed normative thinking from the recently published Best Practice Principles (OECD, 2020[1]). The Secretariat updated the ex post evaluation composite indicator prior to the launch of the survey in 2020. In order to maintain an accurate time series, a limited number of answers from 2014 and 2017 relating to new questions needed to be completed that formed part of the composite indicator for ex post evaluation. Questions relating to reviewing the legality of regulations were also revised.
New survey questions have also been added in the areas of regulatory policy and risk as well as coherence across all levels of government. These questions have not been used to develop composite indicators and have a different scope than the other questions in the Regulatory Indicators Survey.
The Regulatory Indicators Surveys gathered information at three points in time: as of 31 December 2014, 31 December 2017, and 1 January 2021. Data for 2014 are from 34 OECD member countries and the European Union whilst data for 2017 are from 36 OECD members and two accession countries (at the time of data collection) as well as the European Union. The 2021 survey collects data from the 38 OECD member countries and the European Union. The surveys focus on countries’ regulatory policy practices as described in the 2012 OECD Recommendation of the Council on Regulatory Policy and Governance (OECD, 2012[2]).
The surveys investigate in detail three principles of the 2012 Recommendation: stakeholder engagement, regulatory impact assessment (RIA) and ex post evaluation. For each of these areas, the surveys have collected information on formal requirements and have gathered evidence on their implementation. The methodology of the survey and the composite indicators are described in detail in Chapter 2.
While RIA, ex post evaluation and stakeholder engagement are all very important elements of regulatory policy, they do not constitute the whole better regulation framework. The 2021 Regulatory Indicators Survey also includes a range of questions relating to reviewing the legality of regulations, and coherence across all levels of government (Chapter 2), the institutional setup of regulatory policy and oversight (Chapter 3), international regulatory co-operation in line with Principle 12 of the 2012 Recommendation (Chapter 4), as well as risk-based regulation (Chapter 6). Information might be collected in the future on the implementation of other principles in the Recommendation.
The survey focuses on the processes of developing regulations (both primary and subordinate) that are carried out by the executive branch of the national government and that apply to all policy areas. However, questions regarding ex post evaluation cover all national regulations regardless of whether they were initiated by parliament or the executive. Based on available information, most national regulations are covered by survey answers, with some variation across countries. Most countries in the sample have parliamentary systems. The majority of their national primary laws therefore largely originate from initiatives of the executive. This is not the case, however, for the United States where no primary laws are initiated by the executive, or, to a lesser extent, for Colombia, Costa Rica, Czech Republic, Korea, Mexico, and Portugal where the share of primary laws initiated by the executive is low compared to other OECD member countries.
Survey results are used throughout the Outlook in multiple ways. First, results of individual questions are displayed to show trends in the number of countries picking up particular practices. Second, qualitative information and examples provided through the survey are used to enrich the analysis. Third, composite indicators for RIA, stakeholder engagement and ex post evaluation were constructed to provide an overview of country practices.
Each composite indicator is composed of four equally weighted categories: 1) Systematic adoption which records formal requirements and how often these requirements are conducted in practice; 2) Methodology which gathers information on the methods used in each area, e.g. the type of impacts assessed or how frequently different forms of consultation are used; 3) Oversight and quality control records the role of oversight bodies and publically available evaluations; and 4) Transparency which records information from the questions that relate to the principles of open government, e.g. whether government decisions are made publically available.
In interpreting the survey results, it is important to bear in mind the methodological limitations of composite indicators, particularly those that, as in the current survey, are based on categorical variables.
Composite indicators are useful in their ability to integrate large amounts of information into an easily understood format (Freudenberg, 2003[3]). However, by their very nature, cross-country comparable indicators cannot be context specific and cannot fully capture the complex realities of the quality, use and impact of regulatory policy. While the current survey, compared to previous editions, puts a stronger focus on evidence and examples to support country responses, it does not constitute an in-depth assessment of the quality of country practices. For example, while countries needed to provide examples of assessments of some specific elements required in RIA to validate their answers, the OECD Secretariat did not evaluate the quality of these assessments nor discussed with stakeholders the actual impact of the RIAs on the quality of regulations.
In-depth country reviews are therefore required to complement the indicators. Reviews provide readers with a more detailed analysis of the content, strengths and shortcomings of countries’ regulatory policies, as well as detailed and context-specific recommendations for improvement. OECD member countries have a wide range of governance structures, administrative cultures and institutional and constitutional settings that are important to take into consideration to fully assess regulatory practices and policies. While these are taken into account in OECD member country peer reviews, it is not possible to reflect all these country specific factors in a cross-country comparison of regulatory practices.
It is also important to bear in mind that the indicators should not be interpreted as a measurement of the quality of regulation itself. While the implementation of the measures assessed by the indicators aim to deliver regulations that meet public policy objectives and will have a positive impact on the economy and society, the indicators themselves do not assess the achievement of these objectives.
The results of composite indicators are always sensitive to methodological choices, unless country answers are homogeneous across all practices. It is therefore not advisable to make statements about the relative performance of countries with similar scores. Instead composite indicators should be seen as a means of initiating discussion and stimulating public interest (OECD/European Union/EC-JRC, 2008[4]). To ensure full transparency, the methodology for constructing the composite indicators and underlying data as well as the results of the sensitivity analysis to different methodological choices, including the weighting system, has been made available publicly on the OECD website.
References
[3] Freudenberg, M. (2003), “Composite Indicators of Country Performance: A Critical Assessment”, OECD Science, Technology and Industry Working Papers, No. 2003/16, OECD Publishing, Paris, https://dx.doi.org/10.1787/405566708255.
[1] OECD (2020), Reviewing the Stock of Regulation, OECD Best Practice Principles for Regulatory Policy, OECD Publishing, Paris, https://doi.org/10.1787/1a8f33bc-en.
[2] OECD (2012), Recommendation of the Council on Regulatory Policy and Governance, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264209022-en.
[4] OECD/European Union/EC-JRC (2008), Handbook on Constructing Composite Indicators: Methodology and User Guide, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264043466-en.