Reader’s guide
Most of the data presented in this Outlook, including the composite indicators, are the results of the 2014 and 2017 Regulatory Indicators Surveys. This Reader’s guide aims to help readers understand the scope of the data collected through these surveys and some of the limitations related to the use of indicators. Please note that this edition of the Outlook also features results of new survey questions on the institutional setup of regulatory policy and oversight. These questions serve a ‘mapping exercise’ to help illustrate the breadth and diversity of regulatory oversight across all jurisdictions covered. These questions have not been used to develop composite indicators and have a different scope than the other questions in the Regulatory Indicators Survey. Details on these survey questions are described in Chapter 3.
The Regulatory Indicators Surveys gathered information at two points in time: as of 31 December 2014 and 31 December 2017. Data for 2014 are from 34 OECD member countries and the European Union. Data from the 2017 survey are from the 38 OECD member and accession countries (at the time of data collection) and the European Union.1 The surveys focus on countries’ regulatory policy practices as described in the 2012 OECD Recommendation of the Council on Regulatory Policy and Governance (OECD, 2012[1]).
The surveys investigate in detail three principles of the 2012 Recommendation: stakeholder engagement, regulatory impact assessment (RIA) and ex post evaluation. For each of these areas, the surveys have collected information on formal requirements and have gathered evidence on their implementation. The methodology of the survey and the composite indicators are described in detail in Annex A.
While RIA, ex post evaluation and stakeholder engagement are all very important elements of regulatory policy, they do not constitute the whole better regulation framework. Beyond new survey questions on the institutional setup of regulatory policy and oversight, the 2017 Regulatory Indicators Survey also includes a range of questions relating to international regulatory co-operation in line with Principle 12 of the 2012 Recommendation, which are presented in Chapter 4. Information might be collected in the future on the implementation of other principles in the Recommendation.
Scope of the Regulatory Indicators survey data and its use in the Outlook
The survey focuses on the processes of developing regulations (both primary and subordinate) that are carried out by the executive branch of the national government and that apply to all policy areas. However, questions regarding ex post evaluation cover all national regulations regardless of whether they were initiated by parliament or the executive. Based on available information, most national regulations are covered by survey answers, with some variation across countries. Most countries in the sample have parliamentary systems. The majority of their national primary laws therefore largely originate from initiatives of the executive. This is not the case, however, for the United States where no primary laws are initiated by the executive, or, to a lesser extent, for Mexico and Korea where the share of primary laws initiated by the executive is low compared to other OECD member countries (34% over the period 2014-16 in Mexico and 13% in Korea over the same period).
Survey results are used throughout the Outlook in multiple ways. First, results of individual questions are displayed to show trends in the number of countries picking up particular practices. Second, qualitative information and examples provided through the survey are used to enrich the analysis. Third, composite indicators for RIA, stakeholder engagement and ex post evaluation were constructed to provide an overview of country practices.
Each composite indicator is composed of four equally weighted categories: 1) Systematic adoption which records formal requirements and how often these requirements are conducted in practice; 2) Methodology which gathers information on the methods used in each area, e.g. the type of impacts assessed or how frequently different forms of consultation are used; 3) Oversight and quality control records the role of oversight bodies and publically available evaluations; and 4) Transparency which records information from the questions that relate to the principles of open government, e.g. whether government decisions are made publically available.
Limitations of the Regulatory Indicators survey and composite indicators
In interpreting the survey results, it is important to bear in mind the methodological limitations of composite indicators, particularly those that, as in the current survey, are based on categorical variables.
Composite indicators are useful in their ability to integrate large amounts of information into an easily understood format (Freudenberg, 2003[2]). However, by their very nature, cross-country comparable indicators cannot be context specific and cannot fully capture the complex realities of the quality, use and impact of regulatory policy. While the current survey, compared to previous editions, puts a stronger focus on evidence and examples to support country responses, it does not constitute an in-depth assessment of the quality of country practices. For example, while countries needed to provide examples of assessments of some specific elements required in RIA to validate their answers, the OECD Secretariat did not evaluate the quality of these assessments nor discussed with stakeholders the actual impact of the RIAs on the quality of regulations.
In-depth country reviews are therefore required to complement the indicators. Reviews provide readers with a more detailed analysis of the content, strengths and shortcomings of countries’ regulatory policies, as well as detailed and context-specific recommendations for improvement. OECD member countries have a wide range of governance structures, administrative cultures and institutional and constitutional settings that are important to take into consideration to fully assess regulatory practices and policies. While these are taken into account in OECD member country peer reviews, it is not possible to reflect all these country specific factors in a cross-country comparison of regulatory practices.
It is also important to bear in mind that the indicators should not be interpreted as a measurement of the quality of regulation itself. While the implementation of the measures assessed by the indicators aim to deliver regulations that meet public policy objectives and will have a positive impact on the economy and society, the indicators themselves do not assess the achievement of these objectives.
The results of composite indicators are always sensitive to methodological choices, unless country answers are homogeneous across all practices. It is therefore not advisable to make statements about the relative performance of countries with similar scores. Instead composite indicators should be seen as a means of initiating discussion and stimulating public interest (OECD/EU/JRC, 2008[3]). To ensure full transparency, the methodology for constructing the composite indicators and underlying data as well as the results of the sensitivity analysis to different methodological choices, including the weighting system, has been made available publicly on the OECD website.
References
[2] Freudenberg, M. (2003), “Composite Indicators of Country Performance: A Critical Assessment”, OECD Science, Technology and Industry Working Papers, No. 2003/16, OECD Publishing, Paris, https://doi.org/10.1787/405566708255.
[1] OECD (2012), Recommendation of the Council on Regulatory Policy and Governance, OECD Publishing, Paris, https://doi.org/10.1787/9789264209022-en.
[3] OECD/EU/JRC (2008), Handbook on Constructing Composite Indicators: Methodology and User Guide, OECD Publishing, Paris, https://doi.org/10.1787/9789264043466-en.
Note
← 1. On 3 May 2018, the OECD Council invited Lithuania to become a member. At the time of preparation, the deposit of Lithuania’s instrument of accession to the OECD Convention was pending and therefore Lithuania does not appear in the list of OECD members and is not included in the calculation of the OECD average.