2. Assessment methodology and process

Introduction

The publication series, Competitiveness in South East Europe: A Policy Outlook (Competitiveness Outlook), provides a comprehensive assessment of competitiveness reforms in the six Western Balkan (WB6) economies: Albania, Bosnia and Herzegovina, Kosovo, Montenegro, North Macedonia, and Serbia. The publication uses international and OECD good practice to design sustainable economic reform agendas that enable policy makers to achieve greater competitiveness for their economies.

As the third edition of the Competitiveness Outlook series, the Competitiveness Outlook 2021 provides an important benchmarking tool that monitors the development, implementation and performance of policies affecting a broad range of policy areas. Its methodology has been only marginally refined in comparison to the 2018 edition, thus making it possible to track progress on policy design and implementation over time. This integral feature supports the domestic policy cycle by equipping policy makers with a tool to measure progress, ensure stronger co-ordination and consistency between policies and identify policy priorities. Likewise, the Competitiveness Outlook’s collaborative assessment process is itself a vital exercise that fosters government capacity by integrating the perspectives of government and non-government stakeholders in order to identify the primary economic challenges in an economy and develop tailored policy reforms to address them. This collaborative approach seeks to provide policy makers with a robust source of evidence-based guidance on how to achieve greater economic competitiveness and prosperity.

For the first time in the Competitiveness Outlook series, economy-specific profiles for each of the six WB6 economies offer an in-depth analysis of the key policy areas for strengthening competitiveness post-COVID-19. The inclusion of economy-specific profiles enables this edition of the Competitiveness Outlook to provide individualised structural reform recommendations tailored to the specific challenges of each of the Western Balkan economies.

This assessment cycle of the Competitiveness Outlook took place during the COVID-19 pandemic that began in early 2020. The COVID-19 related developments in France and the Western Balkans had a significant impact on the project activities. Multiple lockdowns in Europe and the Western Balkan economies over the course of 2020 created new challenges associated with remote working and other atypical working conditions. The extraordinary challenges presented by the pandemic necessitated major modifications to the planned project activities. Accordingly, this assessment process used a flexible and innovative approach to successfully conduct data collection and analysis, as well as to engage in the extensive policy dialogue that this assessment requires (Box 2.2).

Scope

Although the term “competitiveness” is frequently used by academics, policy makers and the general public, it lacks a common definition and can have different meanings in different contexts, such as a firm, industry, regional, national or supranational context. Academics have attempted to find general definitions that recognise the multi-dimensional character of competitiveness and that can be more practical when applied at the macro level.

The Competitiveness Outlook uses the following three definitions of competitiveness, which all include the concept of sustainability and go beyond an exclusive focus on cost competitiveness and GDP:

• “The ability of companies, industries, regions, nations or supranational regions to generate, while being and remaining exposed to international competition, relatively high factor income and factor employment levels on a sustainable basis” (Hatzichronoglou, 1996[1]).

• “A measure of an economy’s ability to provide its population with high and rising standards of living and high rates of employment on a sustainable basis” (EC, 2012[2]).

• “The ability of a country (region, location) to deliver the beyond-GDP goals for its citizens, today and tomorrow” (Aiginger, Bärenthaler-Sieber and Vogel, 2013[3]).

These holistic definitions imply that to increase an economy’s competitiveness, policy reforms should not be pursued in isolation. Instead, they should seek a comprehensive approach with co-ordinated actions across a variety of policy areas, enabling them to build on one other. In order to achieve this holistic policy-making process, the Competitiveness Outlook 2021 used a collaborative assessment approach encompassing 16 policy dimensions. These dimensions are grouped under four key pillars that are crucial for strengthening competitiveness (Table 2.1).

Table 2.1. Policy dimensions assessed in the Competitiveness Outlook 2021
Pillar	Policy dimension
I. Business environment	1. Investment policy and promotion
	2. Trade policy
	3. Access to finance
	4. Tax policy
	5. Competition policy
	6. State-owned enterprises
II. Skills and capacity	7. Education policy
	8. Employment policy
	9. Science, technology and innovation
	10. Digital society
III. Economic structure	11. Transport policy
	12. Energy policy
	13. Environment policy
	14. Agriculture policy
	15. Tourism policy
IV. Governance	16. Anti-corruption policy

Methodology

The Competitiveness Outlook’s methodology is designed to provide an evidence-based assessment of progress in the design and implementation of policies related to overall economic competitiveness. The assessment is based on a set of indicators that have been tailored to each of the 16 policy dimensions and evaluated using a highly participatory process that involves more than 700 local stakeholders.

The overall assessment approach

Each policy dimension is divided into several sub-dimensions that highlight the key elements of that policy area. The sub-dimensions in turn are composed of qualitative and quantitative indicators (Figure 2.1), which capture the design, implementation and performance of policies, processes and institutions. To reflect the latest trends in OECD economic policy research and good practice developed since the previous Competitiveness Outlook (2018) (OECD, 2018[4]), new qualitative and quantitative indicators have been included across the 16 assessment frameworks. Moreover, the assessment has been designed to inform economies on their progress towards achieving the UN Sustainable Development Goals.

Figure 2.1. The Competitiveness Outlook 2021 overall assessment approach

The indicators in the Competitiveness Outlook’s 2018 edition were taken as a basis for this publication and further refined in order to increase the focus on critical areas and to integrate additional OECD tools and instruments (Box 2.1). The set of indicators used for each of the 16 policy dimensions can be found in the “assessment framework” included at the beginning of each chapter.

Box 2.1. Refinements to the Competitiveness Outlook 2021

The Competitiveness Outlook 2021 has introduced new features that build on the assessment framework used during the 2018 edition to strengthen the analysis and increase its salience and impact. These include:

• refined assessment materials to increase the focus on certain critical areas identified by updated OECD methodologies and best practice

• collaborative, cloud-based assessment materials to encourage stronger participation across governments

• more extensive collaboration with OECD directorates and bodies, including an increased application of OECD instruments to Western Balkan economies and more detailed questionnaires to capture more nuanced qualitative information

• six economy-specific profiles containing:

  • an overview of the economy’s key economic features and structural challenges

  • an overview of the economy’s progress towards achieving the UN Sustainable Development Goals

  • a comprehensive, economy-level analysis and targeted policy advice for each of the 16 policy dimensions

Qualitative indicators

Qualitative indicators assess whether competitiveness-enhancing policy settings, strategies, processes or institutions exist and, if so, the extent to which they have been adopted, implemented, monitored and updated. Each qualitative indicator is assigned a numerical score that reflects the level of policy development and implementation in order to facilitate the comparison of performance among the Western Balkan economies (Figure 2.2).

Figure 2.2. Qualitative indicators: what does the scoring mean?

Table 2.2 illustrates how the general structure of the scoring scale used to measure qualitative information has been translated into level descriptors, using the environment policy dimension’s climate change mitigation and adaptation indicator as an example.

Table 2.2. Climate change mitigation and adaptation scoring levels
Level	Level descriptor
Level 5	Impact assessments on climate change mitigation and adaptation policy, legal and institutional outcomes are regularly conducted with clear accountability mechanisms for government bodies in place.
Level 4	Advanced implementation of approved climate change adaptation and mitigation policy framework, with the policy framework aligned with related policy areas including waste management and sectoral policies such as trade, innovation, industry and agriculture. The implementation is regularly monitored and adjusted if needed.
Level 3	Evidence of policy implementation as measured by programme outputs and inputs of the approved climate change adaptation and mitigation policy. Government bodies’ human and financial resources are adequate for executing relevant responsibilities.
Level 2	A policy framework addressing climate change adaptation and mitigation is in place and has clear strategic vision linked with national economic and social objectives, clear and measureable objectives, defined actions and measures.
Level 1	A policy framework addressing climate change mitigation and adaptation is under development – government activity in framework drafting, conducting stakeholder consultations, and/or progressing through the adoption process.
Level 0	A policy framework addressing climate change adaptation and mitigation does not exist or is obsolete.

Policy dimension and sub-dimension average scores are generated by calculating a simple average across the relevant qualitative indicator scores. Indicators are not weighted because the relative importance of each indicator may be different to different stakeholders. Average scores should therefore be interpreted with caution and taken only as a rough estimate of overall policy development.

Unlike the other 15 policy dimensions assessed in this edition, where indicators are allocated a score from one to five, the assessment of the four qualitative policy areas and one quantitative policy area in the Competition policy dimension is based on yes/no (coded as 1/0) answers to 71 questions in a questionnaire (listed in the Annex to the Competition policy chapter). Where a response to a question is yes (coded as 1), this is referred to as an adopted criterion. Each of the five policy areas has a different number of possible criteria that can be stated as having been adopted. For more information, please see the Competition policy chapter.

Quantitative indicators

The quantitative indicators are input and output factors pertinent to the assessment of policies, policy making, institutional conditions and policy results – e.g. public or private spending in the policy field in question; the share of actors engaging in a certain activity; or the volume of a certain output resulting from a policy or economic activity. They complement qualitative indicators by supplying quantifiable information on the performance of policy settings, processes and institutions.

The Assessment framework part in each of the 16 policy dimension chapters provides an overview of the qualitative and quantitative indicators used to assess this policy dimension.

The assessment process

The Competitiveness Outlook 2021’s assessment process consisted of four main phases, conducted between October 2019 and June 2021: 1) the design phase; 2) the evaluation phase; 3) the consolidation phase; and 4) the review and publication phase:

1. 1. Design phase (October 2019 – March 2020)

In this phase, the OECD developed the qualitative and quantitative questionnaires to assess each of the 16 policy dimensions. This process constituted extensive research to ensure that the latest international and OECD good practice were incorporated into the assessment frameworks for each policy dimension. For each qualitative indicator assessed, the OECD compiles a table, which lists the necessary elements that need to be met to attain a certain score level. Respondents have to complete a set of yes/no, multiple-choice and open-ended questions. The answers to these questions and the listed criteria for each score level enable the respondents to self-evaluate the score for this indicator. For every answer, the OECD asks respondents to provide supporting evidence (e.g. adopted strategies, legislation, monitoring reports, etc.), which is subsequently verified and the score level either confirmed or revised. The quantitative questionnaires are tailored to each economy depending on the extent to which statistical data are already available to the OECD. Only quantitative indicators for which data are not available publicly are included in the quantitative questionnaire. The qualitative and quantitative questionnaires had to be digitalised and complemented with additional guidance documents following the impact of the COVID-19 pandemic (Box 2.2).

1. 2. Evaluation phase (April – November 2020)

This phase consisted of the government self-assessment. It was led by a designated Government Co-ordinator (see Acknowledgements for a list of names), and co-ordinated with the help of policy dimension and statistical office contact points in each economy. Since the qualitative and quantitative questionnaires for each policy dimension always cover several sub-dimensions (Figure 2.1), their completion called for an inclusive approach which led to extensive collaboration among different ministries, public agencies and statistical offices in each WB6 economy, often underpinned by dedicated working groups. Completing the OECD questionnaires thus enabled WB6 policy makers of different institutions working in the same policy field to reflect together on the most pressing challenges, priorities and most suitable way forward for their policy field. This phase also enabled WB6 policy makers to reflect on the roles of their respective services in the policy dimension in question, forge closer ties across different institutions, and to improve future inter-ministerial collaboration for holistic policy making beyond this assessment. Remaining information and data gaps were filled through virtual fact-finding meetings with relevant government stakeholders as well as local non-government stakeholders.

1. 3. Consolidation phase (December 2020 – March 2021)

The results of the government self-assessments were consolidated into draft economy profiles and draft regional policy chapters whose key findings and initial policy recommendations were presented at seven stakeholder meetings (one for each WB6 economy and one regional meeting). While these meetings were held in the six Western Balkan capitals in previous assessment cycles, COVID-19 restrictions meant that the roundtables had to be held virtually. The meetings brought together Western Balkan government officials as well as international donor representatives and ambassadors from a number of interested EU and OECD countries. The consolidated assessment and first policy recommendations for each policy area were discussed with the meeting participants and formed the basis for completing the draft economy profiles and regional chapters of the Competitiveness Outlook 2021 publication.

1. 4. Review and publication phase (January – June 2021)

The 6 draft economy profiles and 16 regional chapters were shared with Western Balkan government officials and experts across the OECD, the European Commission, and regional expert organisations for comprehensive review. Their helpful feedback and comments helped to maximise the accuracy of the publication’s analysis as well as its salience for local policy makers. Subsequently, the OECD prepared the report for publication. The publication was released at the Berlin Process Summit on 5 July 2021.

Box 2.2. The impact of COVID-19 on the project

The COVID-19 pandemic triggered lockdowns around the world, requiring a novel approach to this assessment cycle of the Competitiveness Outlook. In light of these extraordinary developments, the OECD team adopted the following measures:

• All assessment materials (qualitative and quantitative questionnaires) were transformed into digital formats designed to be accessible and easy to use for all who contributed to the assessment. This approach made it possible for multiple persons to remotely access and work on the assessment materials simultaneously, which was an essential feature during the multiple lockdowns and confinements implemented across the Western Balkans.

• Since the assessment materials could not be explained to WB6 policy makers during dedicated assessment launch meetings, the OECD team prepared extensive explanatory documents for the completion of each policy dimension’s assessment materials, including dimension-specific tutorials explaining the assessment framework and the specific elements that merit further clarification, as well as two guidance notes with step-by-step explanations for the completion of the qualitative and quantitative questionnaires.

• In lieu of the planned assessment launch events in each WB6 economy, the OECD team organised individual dimension-specific virtual launch meetings, requiring 77 virtual kick-off meetings to be organised with government dimension co-ordinators, statistical office representatives and their teams.

• In order to retain the vital policy dialogue that underpins each phase of the assessment cycle, the preliminary key findings of each economy profile and the regional policy chapter were presented virtually during 7 virtual stakeholder meetings attended by a total of about 750 stakeholders. This enabled policy makers in the Western Balkans to further contribute to the OECD’s findings and incorporate policy developments that occurred after the initial data collection phase was finalised.

Despite these measures, the challenges created by the COVID-19 pandemic caused some delays in the input collection process. A number of dimension co-ordinators of the Competitiveness Outlook 2021 became ill with COVID-19 and needed to be temporarily replaced, necessitating multiple additional kick-off meetings and calls with individuals newly included in the assessment process. Moreover, mandatory lockdowns and teleworking in the Western Balkans negatively affected the co-ordination across various administrative units, which also slowed down the government self-assessment data-collection process.

The participatory nature of the Competitiveness Outlook 2021 assessment process

The Competitiveness Outlook’s methodology is complemented by a participatory assessment process which has been designed to foster peer learning, to create consensus on reform priorities and to facilitate stakeholder co-ordination. The participatory basis of the publication is reflected in the high number and diversity of stakeholders included in the assessment and the Competitiveness Outlook meetings and roundtables (Table 2.3).

Table 2.3. Competitiveness Outlook assessment: stakeholders and meetings
Involved stakeholders:	8 Western Balkan Government Co-ordinators 128 Western Balkan Policy Dimension Contact Points 95 Western Balkan Statistical Office Contact Points More than 550 Western Balkan Government Officials 6 local expert consultant agencies Representatives from the private sector, academia, civil society and international donor organisations
Meetings and roundtables:	77 assessment kick-off meetings 27 fact-finding meetings 6 economy-specific stakeholder meetings 1 regional stakeholder meeting

The involvement of these stakeholders strengthened the analysis in terms of data collection. In turn, the stakeholders engaged in gainful exchanges among themselves and with the OECD on concrete policy issues. This participatory approach is a particular strength of the Competitiveness Outlook, enabling the assessment process to:

• catalyse positive organisational change through government self-assessment and roundtables which ask stakeholders to judge the success of current policies and resource allocations, while identifying possible directions for improvement

• strengthen inter-ministerial consultation by asking Western Balkan government officials across various ministries and agencies to exchange and co-ordinate with one another as they assess the policy dimensions, as well as by bringing them together during economy-specific stakeholder meetings to reflect on the roles of their respective services in the policy dimension in question

• encourage Western Balkan statistics offices and government bodies to produce new or more frequent statistics through the process’s demand for key indicators for assessing competitiveness

• support regional integration and a regional perspective on policy issues by virtue of the regional virtual stakeholder meeting.

The strengths and limitations of the Competitiveness Outlook 2021

The Competitiveness Outlook 2021 possesses a number of strengths which make it a uniquely valuable tool for Western Balkan policy makers, citizens, researchers, as well as investors and international donor organisations. Table 2.4 lists the Competitiveness Outlook’s main strengths, as well as its limitations.

Table 2.4. Strengths and limitations of the Competitiveness Outlook 2021
Strengths	Limitations
Six economy profiles provide accurate and salient analysis and recommendations tailored to the specific context and needs of each economy	While the Competitiveness Outlook 2021 focuses on areas crucial for strengthening competitiveness in the Western Balkans, it does not cover all areas of competitiveness exhaustively
Independent and rigorous assessment enables performance to be benchmarked against peer economies and OECD good practice	Statistical coverage of many issues is limited in the region and the economy-specific context of the quantitative data cannot always be acknowledged
The government self-evaluation acts as a change management tool and creates a process that enhances the quality of pro-competitiveness policy development	As the same set of indicators is applied to all economies in the region, certain economy-specific characteristics may not be fully reflected in the scoring
Good practice examples and policy recommendations offer guidance to policy makers in designing or redefining a sustainable economic reform agenda to foster competitiveness	The benchmarking potential of certain policy areas is limited as some qualitative indicator score levels were refined for the 2021 edition of the Competitiveness Outlook
The analysis draws on both original data collected by the OECD and existing data collected by other organisations
The participatory assessment process enables stakeholder dialogue on policy, joint learning, and agreement on identified strengths and shortcomings to help build consensus for future reform
Scoring levels by policy dimension helps public officials communicate more effectively on policy progress and areas where future reform is necessary
Economy-specific contexts and other wide-ranging factors that affect competitiveness and policy development underpin the analysis and supplement the scores

References

[3] Aiginger, K., S. Bärenthaler-Sieber and J. Vogel (2013), Competitiveness under New Perspectives, European Commission, Brussels, http://www.oecd.org/economy/Competitiveness-under-New-Perspectives.pdf.

[2] EC (2012), Operational Guidance for Assessing Impacts on Sectoral Competitiveness Within the Commission Impact Assessment Toolkit, a ’Competitiveness Proofing’ Toolkit for use in Impact Assessment, European Commission, Brussels, https://ec.europa.eu/transparency/regdoc/rep/2/2012/EN/2-2012-91-EN-F1-1.PDF.

[1] Hatzichronoglou, T. (1996), “Globalisation and Competitiveness: Relevant Indicators”, OECD Science, Technology and Industry Working Papers, No. 1996/5, OECD Publishing, Paris, https://dx.doi.org/10.1787/885511061376.

[4] OECD (2018), Competitiveness in South East Europe: A Policy Outlook 2018, Competitiveness and Private Sector Development, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264298576-en.