5. Enable competent education authorities to strengthen system evaluation and improve co-ordination
Education stakeholders in Bosnia and Herzegovina have taken steps to establish some of the basic building blocks needed to monitor and evaluate education policy and guide system improvements. However, major gaps in system evaluation frameworks remain, namely the availability of comparable data on learning outcomes, which is lacking in most jurisdictions. This chapter presents possible pathways that competent education authorities and other BiH actors could take to strengthen collaboration and co-ordination in the education sector, as well as increase the use of evidence for planning and policy development.
Education system evaluation is central to improving educational performance. System evaluation provides governments and other stakeholders with information to formulate effective policies, and reinforces accountability for meeting high-level education goals. Bosnia and Herzegovina (BiH) has taken steps to establish some fundamental components of system evaluation but there are major gaps in the country’s system evaluation framework that prevent the government and other stakeholders from effectively monitoring – and through this, improving – system performance.
In Bosnia and Herzegovina (BiH), the governance of education is highly complex and decision-making power in the area of school education has been delegated to twelve competent education authorities (CEA) at the entity, canton and district level (see Chapter 1). This arrangement provides the Republika Srpska (RS), the 10 cantons of the Federation of BiH (FBiH) and the Brčko District with more autonomy to direct education policy. However, it also creates particular challenges for monitoring and improving quality. While competent education authorities in the country have defined some goals for their education systems, the majority covered by this review lack adequate resources and quality data to translate these goals into concrete implementation plans and measurable objectives.
This chapter recommends a set of measures that could help competent education authorities in BiH to build a more coherent direction for system improvement and strengthen system evaluation through greater collaboration and co-ordination. In particular, it recommends ways to produce richer, more comparable data to support a technical dialogue around the performance of different education systems in BiH. The chapter also recommends initiatives to intensify co-operation and improve peer learning so that good practices and tools can be scaled across the country. Implementing these reforms could enable BiH’s education authorities to focus more effectively on improving education outcomes for students, especially in light of the negative consequences caused by the COVID-19 pandemic.
Education system evaluation is essential to improve educational performance. A robust framework for system evaluation involves defining goals, establishing tools to provide reliable data on the system’s inputs, outputs and outcomes, and conducting regular reviews of system performance. Through this framework, governments provide important accountability information to the public and education authorities, and periodically review whether policies are meeting their ascribed goals and how education needs might be evolving. Education system evaluation is thus a critical lever to ensure that education systems deliver high-quality instruction, and that public resources are well spent. BiH has taken steps to produce richer data on its education systems and to set strategic directions for improvement. However, there are major gaps in the country’s system evaluation frameworks (Table 5.1).
Competent education authorities have taken steps to establish goals for system improvement
Most OECD countries have established system-wide education goals, to provide a coherent direction for education reform and an anchor against which performance can be assessed. In BiH, some administrative units have set specific goals for the education sector, while others only have capacity to carry out administrative functions. There have been past efforts to set common directions for education policy at the country level, recognising that this could help BiH to fulfil its international commitments, align education policies to facilitate greater student mobility (among other benefits), and establish baseline standards for learning. However, these efforts have been hampered by an absence of implementation planning and measurable objectives that could help to translate “big picture” goals into concrete actions. At the same time, the lack of co-operation and the existence of political differences create additional hurdles to defining education goals directed at system improvement and better student outcomes.
All competent education authorities have set strategic directions for education improvement, but many are impeded by resource constraints
During meetings with the review team, all competent education authorities reported that setting longer-term directions for education improvement helped to guide their work – as a way to organise resources, make policy more coherent, and structure dialogue with different partners. At the same time, the extent to which different competent education authorities can formulate these strategies and articulate implementation plans varies widely. At the time of this review, Republika Srpska had a dedicated five-year education strategy in place (2016-21), which outlined goals and included a five-year Action Plan with clear measures and activities. The RS government is also currently developing a new strategy that will guide the entity’s education sector through 2022-30. Sarajevo Canton also has a standalone education strategy. However, in other parts of FBiH, most cantons have defined education goals as part of broader cantonal development plans. This is the case in the West Herzegovina and Central Bosnia cantons, though the former is currently drafting a dedicated Education and Science Strategy. Brčko District has also defined some education goals as part of its district-level development strategy.
Country-level education laws, strategies and standards have been defined in the past, but their implementation has been hampered by a lack of political will and concrete guidelines
BiH’s competent education authorities have made efforts to improve the coherence of education policies and to harmonise them with practices found in European Union (EU) countries. Specifically, there have been country-level framework laws, strategies and proposed standards for teachers and student learning that aim to strengthen the performance of BiH education systems. For example, in 2003, BiH’s Education Ministers adopted a new (at that time) country-wide Framework Law on Primary and Secondary Education, to facilitate student mobility across BiH, promote greater school autonomy, and increase parent and teacher involvement in the organisation of schooling. In 2007, the country also established the 2008–2015 Strategic Directions for the Development of Education in BiH, to provide guidance on policies that competent education authorities could include in their development plans, using EU education models as a reference. This strategic directions document has since expired.
While BiH has developed a range of high-level education documents in the last two decades, the only strategic documents and platforms that currently exist at the state-level relate to higher education, VET, pre-school education, and other topics, such as entrepreneurial education and lifelong learning (MoCA, n.d.[2]). Most of these documents were developed as part of commitments made to international bodies, such as the Council of Europe and the EU. The Framework Law, for instance, was developed with support from the Council of Europe and fulfilled part of BiH’s commitments in the area of education (OSCE, 2003[3]). Notably, there are no current strategic documents or platforms at the state-level in BiH related to primary and secondary schooling.
Implementation of state-level policy documents within each administrative unit has been a challenge. Positively, the Ministry of Civil Affairs reports to the Council of Ministers of BiH once a year on the extent to which competent education authorities have implemented the principles contained in these documents, based on input from each jurisdiction. However, this reporting indicates that implementation is slow and uneven across different parts of the country. Actors commonly cite a lack of political will, as well as the absence of concrete operational guidelines and measurable progress indicators as reasons for inconsistent implementation (USAID, 2017[4]). In general, there have been no steps to define a framework of indicators to comprehensively measure education performance at the BiH-level, and the completeness of reporting information provided by different competent education authorities varies substantially.
Mechanisms to promote more country-level co-ordination, communication and alignment around education policy have been created, but sustaining momentum is a challenge
BiH has country-level bodies to help co-ordinate, communicate and align education policies implemented by the competent education authorities. In 2008, the Conference of Ministers of Education in BiH was established, with the goal of overseeing “the fundamental reform of the existing parallel education systems of BiH as a matter of high priority.” In 2009, the Agency for Pre-Primary, Primary and Secondary Education (APOSO) was established to develop a Common Core Curriculum Based on Learning Outcomes, evaluate learning, and serve other evaluation roles, as a successor to the former Agency for Standards and Assessment for FBiH and RS. The Ministry of Civil Affairs of BiH also plays a co-ordination role, helping to track country-level initiatives and lead engagement with international partners in the field of education. For example, the Ministry of Civil Affairs performs general policy co-ordination tasks, including consolidating the development plans of the entities and cantons, which provides it with a perspective on how each authority’s education policy is linked to its overarching development context and plans.
While mechanisms for country-level policy co-ordination exist, sustaining momentum is a challenge. Most co-ordination mechanisms were initially set up to fulfil commitments made to international bodies, and find momentum and support difficult to maintain once this moment has passed – which is not to understate their impressive achievements. The review team heard, for instance, that meetings of the Conference of Education Ministers take place infrequently, and are sometimes attended by working-level officials who may lack expertise in the subjects being discussed. While country-level bodies like the Ministry of Civil Affairs continue to co-ordinate education governance across BiH, these bodies are overloaded, and struggle to engage with some authorities at the entity, canton and district level.
Tools to conduct system evaluation are incomplete
Most OECD countries use a range of tools to monitor education system performance. Administrative data on students, teachers and schools is typically held in a comprehensive information system, and can be easily extracted for analysis. Most OECD countries now also compile trend data on learning outcomes collected through regular national and international standardised assessments. In BiH, the Agency for Statistics of Bosnia and Herzegovina (BHAS) compiles education statistics across the country for reporting internationally. However, data for important indicators cannot be aggregated at the country level, and information is missing in critical areas, notably data on learning outcomes.
The BHAS compiles country-level education statistics for reporting internationally
The BHAS collects data on the education sector once a year through a network of canton- and entity-level reporting units to produce its education statistics. These units collect information from schools through a (mainly paper-based) questionnaire, compile a report, and then send this information to the BHAS to be aggregated into country-level statistics. A legal framework has been established, which regulates the collection and systematisation of data at the country and entity levels, and most cantons have harmonised their laws accordingly (World Bank, 2019[5]). However, the BHAS still faces the challenge of processing incomplete statistical forms and notes concerns over the accuracy of some of its key education indicators. For instance, data on net participation rates at different levels of education must be compiled through referencing data collected from schools against the country’s most recent census, which was last conducted in 2013, meaning that baseline population figures are not available for children of pre-primary and primary age. Despite these challenges, the BHAS continues to improve the quality of BiH’s education data. For instance, the agency is currently implementing a project with UNESCO to migrate its method of data collection from pre-school, primary and secondary schools to a web-based form – aiming to make data collection more efficient for international reporting and address some concerns around data quality.
Each competent education authority compiles its own administrative data but many cannot provide information for country-level reporting purposes
In most OECD countries, administrative education data are collected according to national and international standardised definitions, enabling data to be collected once, used across the country’s education sector and reported internationally. In BiH, each competent education authority applies its own data management system, and there is no country-level agreement on statistical concepts, definitions, or on information management in the field of education. As a result, the BHAS can not make use of the administrative data produced by education authorities to compile its education statistics and must collect its own data directly from schools, reducing the accuracy of certain key indicators.
While some competent education authorities have a relatively comprehensive education management information systems (EMIS), others do not have any structured data collection and processing systems. Republika Srpska, for instance, can access disaggregated data on school financing, human resources, the working week and learning outcomes in real time through its EMIS (known as EDUIS), while West Herzegovina Canton can access records of subject teachers per student/class, curriculum implementation, test grades and student absences on a daily basis through its system. In smaller jurisdictions, like Brčko District, such systems often do not exist, preventing authorities from conducting analysis on system performance trends over time. Few competent education authorities compile itemised data on teachers and students, which hinders meaningful analysis.
In the early 2000s, a World Bank project sought to establish a country-wide EMIS in BiH – with the objective of providing more information on teaching and learning to primary school teachers, promoting a more efficient and equitable use of public resources, promoting more co-operation and co-ordination among the country’s three constituent groups, and to test the viability of a “per-student” budgeting model. The project was initially piloted in RS, Tuzla Canton and Central Bosnia Canton, with the aim of gradually rolling out to all other competent education authorities. However, in most cantons, the EMIS was either never fully implemented or fell out of use (World Bank, 2012[6]).
Data on learning outcomes is limited
Most competent education authorities in BiH compile data on student learning in the form of grade point average and other teacher-graded marks. However, these results are not standardised and cannot be used to reliably assess learning across schools. Over the past twenty years, many OECD countries have expanded the use of regular standardised assessments to provide comparative measures of student achievement. In BiH, too, RS has started to implement an annual external assessment of Serbian language and Mathematics skills at Grade 5, which is mandatory for all schools under the Law on Primary Education of RS; however, the details of this assessment and the use of its results are unclear. In addition, two cantons (Sarajevo and Tuzla) conduct external standardised assessments but these are designed primarily for certification purposes rather than system monitoring (see Chapter 2). In Sarajevo Canton, examination results are coded and uploaded to the EMIS and school-level data is published periodically on school noticeboards; however, actors outside of the ministry and school must make a specific request to view the data. Importantly, these assessment instruments were developed independently by their respective competent education authority, and thus can not produce comparable data.
Implementing a country-level standardised assessment implies significant costs. Participation in international learning assessments can therefore be an effective way to produce data on learning outcomes until such a tool is made available. Positively, BiH participated in two international learning assessments in recent years – PISA (in 2018) and TIMSS Grade 4 (in 2019) – after more than a decade without reliable data on learning outcomes. Participation is not regular, however, which means performance cannot be measured meaningfully over time. Notably, BiH will not participate in PISA 2022 and is unlikely to participate in TIMSS 2023, making it the only Western Balkan economy that will not have a means to monitor learning outcomes in relation to EU and international education goals. BiH is also the only Western Balkan economy that is unlikely to participate in the next cycle of the OECD Teaching and Learning International Survey (TALIS), further limiting the country’s efforts to better understand factors that influence student outcomes – e.g. teachers’ working conditions and school learning environments.
Education authorities do not regularly report on system performance
Publishing regular reports on system performance allows many OECD countries to interpret system-level data and provide policy messages for accountability and improvement efforts. BiH does not regularly publish reports on the performance of its education systems or evaluations of major policies. Positively, some education authorities and development partners in the country have organised conferences and produced research and analysis on the sector and specific education issues, such as inclusive education. APOSO also has a mandate to conduct system evaluations at the state-level and it plays an important role in advancing technical dialogue around education in BiH. However, the agency is under-resourced and some competent education authorities struggle to fully understand the agency’s role and may not engage with its initiatives.
There is limited demand for evidence and analysis on education system performance among domestic education stakeholders
Aside from a lack of resources, one of the reasons for limited system evaluation in BiH is a lack of demand from key sectoral stakeholders. Competent education authorities report limited use of evidence to inform planning and policy development. Many do not have a full picture of the evidence available and may not have capacity and time to interpret it. Aside from competent education authorities, BiH’s academic community has not expressed a strong interest in carrying out education system evaluations. Secondary analysis of BiH’s results from PISA, TIMSS is limited to the work of APOSO, and research on other aspects of the education system, which could be used to inform policy, is rarely conducted. As a result, the wider stakeholder community, including parents and community leaders, do not have evidence or opportunities to engage with the competent education authorities or Pedagogical Institutes on system performance.
APOSO is a state-level agency that sets standards and evaluates education quality, in co-operation with competent education authorities
APOSO has a mandate to set standards, evaluate education quality, and help to co-ordinate participation in international learning assessments. To date, the agency has carried out secondary analysis on the results of PISA and TIMSS, translated these results into Bosnian, Croatian and Serbian (as well as English), and made these results freely available online. APOSO is a strong advocate for advancing evidence-informed, technical discussion on the state of education in BiH, and it produces analysis and technical tools (for instance, the Common Core Curriculum) that can underpin this discussion. The agency continues to invite debate on the results from international learning assessments among education stakeholders in BiH. In 2020, for instance, APOSO posted key findings from PISA and TIMSS on its news page and invited “education authorities to take the results … seriously, as an incentive to speed up the reform processes (APOSO, 2020[7]).” It also organised stakeholder conferences and workshops to discuss their results – for instance, with teachers, school principals and pedagogical specialists.
BiH has traditionally underinvested in its system evaluation function
The decentralised nature of education policymaking in BiH has left the country’s system evaluation function under-resourced. Neither competent education authorities nor country-level co-ordination bodies have the resources or capacity to produce periodic system evaluations. BiH also lacks much of the comparable data needed to produce meaningful analysis on the performance of its education systems. This context presents a serious challenge for BiH, as education authorities struggle to build decisions around the governance of education on a more impartial, evidence-informed footing.
BiH works closely with international partners in the sphere of education
Most international development partners have identified education as a priority area for engagement with BiH. Over the period 2014-20, for instance, 9% of the financial assistance BiH received from the EU’s Instrument for Pre-Accession Assistance (IPA) II programme went to education, employment and social policies (EC, 2018[8]). International development partners based in Sarajevo convene regularly, and have attempted to co-ordinate their activities with BiH to support more strategic engagement with the country. At the same time, development support can be piecemeal, due to the absence of a long-term vision or a strong co-ordination mechanism at the state-level. Producing better-quality information on system performance could enable BiH to make more strategic use of available development assistance, and allow international development partners to review the impact of their support measures over time.
In some OECD countries, decentralisation has improved the quality of education by enabling education authorities to strengthen partnerships with local stakeholders and better identify and address their specific needs. At the same time, decentralised education systems can be more vulnerable to challenges related to effectiveness, equity and accountability, especially when co-ordination and co-operation mechanisms are missing (EASNIE, 2017[9]). In BiH, the governance of education policy is decentralised, and the lack of strong collaboration among various education authorities presents significant challenges for accountability and system improvement. While competent education authorities in BiH have the power to develop and implement their own education policies, many lack the resources to identify measurable goals for their education system and measure progress systematically. In many cases, competent education authorities also need scale to make their goals, data and reporting meaningful.
Most OECD members with decentralised education systems have country-level bodies and initiatives to help facilitate co-ordination and co-operation. In some instances, they also use these bodies to share resources more equitably across jurisdictions. Such co-ordination efforts must be designed carefully, to provide the support needed while simultaneously preserving the independence of local authorities. BiH has established mechanisms to co-ordinate education policy and there are positive examples of recurring collaboration between competent education authorities. The primary challenge is to expand meaningful collaboration at the country level and progressively strengthen system evaluation to support accountability and guide improvement efforts. To do this, BiH should revitalise the Conference of Education Ministers with a mandate to establish a set of common goals for school education, as well as clear action plans and reporting procedures. To support reporting against these goals and policy more broadly, BiH authorities should co-operate to produce richer education statistics and more robust data on learning outcomes. Finally, BiH should strengthen transparency and trust across its education system by building demand for system evaluation among researchers and creating new platforms for broader stakeholder engagement.
Many OECD countries with decentralised education systems establish education goals and other governance initiatives at the country-level, to stimulate improvement, facilitate peer learning and reduce territorial disparities. This approach enables governments to set standards that are coherent at the country level as well as internationally, to define goals that are more ambitious and outcome-focused, and to pool resources and know-how across the country’s different education systems. At present, there are no state-level strategic goals for school education in BiH. The governance of school education in BiH is decentralised and collaboration across jurisdictions is often limited. This creates challenges for system evaluation and can prevent actors from agreeing on a set of common goals. While competent education authorities in the country have defined goals for their education systems, the majority lack sufficient resources and quality data to translate these goals into concrete implementation plans and measurable objectives. Establishing school education goals at the country-level would also enable BiH to make better use of donor support and to move closer to EU standards.
Positively, BiH’s competent education authorities have already collaborated to develop country-level framework laws, standards and strategies for parts of the education sector. In addition, the Conference of Education Ministers provides a platform for country-level education policy co-operation and dialogue. These initiatives have helped to set standards for education in BiH and improved the country-level coherence of policies set by the competent education authorities. However, the implementation of country-level documents at canton and entity level has proved a challenge, and the Conference has lost momentum and lacks a clear programme of work. The Conference should be revitalised to chart a common vision and goals for raising the quality of education in BiH in the wake of COVID-19. To ensure that this exercise leads to real change, the Conference should establish a vision that is evidence-informed, focused on results and widely accepted by stakeholders. It should devise an indicator framework and action plans that would translate broad goals into concrete activities and enable authorities to measure progress over time. Finally, the Conference should ensure that the results of its work and system performance more generally are reported to the public on a regular basis, to strengthen trust, transparency and engagement in .
Recommendation 5.1.1. Establish a common, widely-approved vision and goals for pre-tertiary education in BiH
The decentralised governance of education and strained collaboration among jurisidctions has created barriers to reform within some competent education authorities and barriers to co-operation at the country level. In a number of OECD countries, establishing a country-level vision for system improvement has helped to depoliticise the education debate and focus attention on improving outcomes. In Australia, for instance, the 1989 Hobart Declaration played a critical role in strengthening intra-state co-operation in the area of education and establishing a demand for monitoring education outcomes in Australia (Santiago et al., 2011[10]). While the BiH context is different, Australia’s experience shows an example of independent education sub-systems, each with their own powerful stakeholder groups, reaching consensus around a set of goals that are seen to have common relevance – and finding value in having achieved this result. The move to establish common goals for the school sector was underpinned by a view that this was essential to secure Australia’s future productivity and international competitiveness, and the Declaration makes a clear reference to this objective (Box 5.1).
Defining country-level approaches to education policy can have other benefits, such as helping to improve the coherence of education policy across different jurisdictions. For instance, it can be used to address local discrepencies in college and career readiness and to ensure that certification of attainment and achievement is standardised and recognised across the country. In BiH, country-level initiatives could help education authorities to review the organisation of schooling in their jurisdictions and shift the focus of stakeholders towards helping students learn throughout school and transition successfully into further studies or work. To establish an implementable, country-level vision for system improvement, BiH should establish a discrete set of education goals, aligned with its international commitments and labour market needs. It should define clear responsibilities in this process, to ensure that education goals are formulated through a reflection on BiH’s current needs and based on a broad consultation to secure stakeholder support.
Develop a discrete set of education goals that are clear, focused on results and informed by evidence
System-level goals should give coherent direction across different levels of government and to a multiplicity of education actors. For this to happen, goals should be clear, feasible and above all, generally relevant across all parts of the education system. These features also enable system-level goals to be picked up and embedded into key reference frameworks, such as school evaluation standards (see Chapter 4), which can strengthen education policy coherence and help all actors to work towards their achievement.
In a growing number of OECD countries, high-level education goals have been established that set out the overarching results a country would like to achieve, as well as the strategies or objectives to achieve them. This approach can provide clarity of focus, reinforce accountability and improve resource efficiency – by establishing not only a clear outcome that all stakeholders know they must work towards, but also by establishing the approach and processes that stakeholders should use to get there. Many high-performing systems define only a limited number of goals, to support this clarity of focus. In Australia, for instance, the Hobart (1989) and Adelaide (1998) Declarations defined just ten and then eighteen common goals, establishing a basic approach to education that should guide all states.
Alongside a growing focus on outcomes, OECD countries are increasingly including equity objectives in their high-level, country-wide goals. This move has enabled many countries to reinforce efforts to deliver on their Sustainable Development Goal 4 (SDG4) commitments and sends a clear message that education goals are designed to serve all citizens. BiH may also decide to include equity objectives in its high-level goals – echoing the country’s Constitution, which sets out that every child should be able to access education on equal terms.
The Melbourne Declaration on Educational Goals for Young Australians, released in December 2008, and agreed to by all education ministers through the Ministerial Council on Education, Early Childhood Development and Youth Affairs (MCEECDYA), articulates future directions and aspirations for Australian schooling. It sets young Australians at the centre of the agenda for educational goals and provides a framework for developing curriculum and assessment. The Melbourne Declaration has two overarching goals for schooling in Australia:
All young Australians become successful learners, confident and creative individuals, and active and informed citizens.
The national goals for schooling are supported by the MCEECDYA Four-Year Plan 2009–2012 (MCEETYA, 2009), which was endorsed by all Australian education ministers in March 2009. The plan is closely aligned with the Council of Australian Governments agreements. It outlines the key strategies and initiatives Australian governments will undertake in the following eight inter-related areas in order to support the achievement of the educational goals outlined in the Melbourne Declaration:
improving educational outcomes for Indigenous youth and disadvantaged young Australians, especially those from low socio-economic backgrounds
Source: (Santiago et al., 2011[10]), OECD Reviews of Evaluation and Assessment in Education: Australia, https://doi.org/10.1787/9789264116672-en.
BiH’s competent education authorities should formulate a discrete set of common goals for schooling in the 21st century that focus on the outcomes they would like to achieve. These goals should focus on the desired economic and social outcomes of schooling, and should enable BiH to align more closely with the EU and advance international commitments like SDG4. There are a number of areas in which many of BiH’s competent education authorities are facing common challenges and needs, and these include:
Increasing participation in early childhood education. Currently, BiH presents one of the lowest rates of enrolment in early childhood education in the EU. While rates vary among jurisdictions, the country level was around 18% in 2018, compared to 95.3% on average in the EU (European Commission, 2019[11]). Numerous studies have found that the first five years of a child’s life are crucial to their development. PISA results, for instance, show that students who attended early childhood education and care (ECEC) typically score higher in reading at age 15 (OECD, 2018[12]), and that investment in quality early childhood education yields important economic and social returns – often more than at other levels of education (OECD, 2020[13]). This goal could be a cost-effective way for BIH to improve student outcomes in the long-term.
Improving outcomes in core learning areas. Results from PISA 2018 suggested that 41% of students in BiH did not achieve the minimum skills (Level 2) in all three PISA domains. This is significantly higher than the average percentage of students who did not achieve minimum skills in both the OECD (13.4%) and in the Western Balkans (38.7%). PISA aims to assess whether students have acquired complex, higher-order thinking skills and can apply these skills to unseen problems, because it assumes that as incomes grow and the century advances, jobs are likely to constantly evolve and become more technology-intensive (OECD, 2011[14]). BiH may aim to address this issue as part of its common goals.
Raising digital literacy and ICT skills. Strong digital literacy and ICT skills will be critical to thrive in 21st century work and life. Data compiled by Eurostat through the EU’s Digital Competence Framework suggests that only 24% of individuals in BiH had basic or above basic digital skills in 2019, compared to 56% in the EU-27 (EC, 2019[15]). Emphasing these competences in education systems across BiH would help position the labour force to compete in today’s digital world.
Improving education outcomes for Roma and students with disabilities. BiH is home to around 25 000-50 000 Roma, and some estimates place around 6.5 percent of the country’s children between two and nine with some form of disability (UNICEF, 2020[16]). According to official estimates, only around half of Roma children of primary school age are enrolled in school (UNICEF, 2020[17]), very few participate in pre-primary education (an estimated 3% of those aged 3-5, compared to 33% in Albania), and no progress has been made to improve the completion rate for compulsory education or to increase continuation in schooling afterwards (Robayo-Abril and Millán, 2019[18]). In addition, it has been noted that children with disabilities do not receive the same quality of education as their more abled peers (UNICEF, 2020[16]). BiH may wish to target this issue to improve inclusion of education and work towards achieving SDG4.
Addressing performance gaps between rural and urban areas. Results from PISA 2018 show that BiH presents considerable rural-urban gaps in reading performance. An in-depth analysis found that students in urban schools outperformed those in rural schools by about 50 score points – and this gap remains statistically significant once the data has been controlled for students’ socio-economic status (OECD, 2020[19]).
Supporting school to work transitions. BiH has one of the highest youth unemployment rates in the Western Balkans, at 33.8% in 2019, compared to 27.0% in Albania and 25.3% in Montenegro for instance, and compared to 12.5% in the OECD (ILO, 2019[20]). Youth unemployment is an issue in many countries, as the transition from school to work becomes increasingly challenging for young people. This problem reflects supply-side issue, as participation in basic education reaches near-universal levels, as well as demand-side issues, as employers seek higher-order skills and competences and jobs become increasingly changeable. To address the problem, many countries are making a deliberate effort to support school to work transitions.
Establish distinct roles and responsibilities for elaborating a common vision and goals
While school education policy will continue to be set by RS entity, cantons of FBiH or Brčko District authorities, BiH should establish a technical Task Force to support the Conference of Education Ministers in elaborating a common vision and goals for school education at the state level. The Task Force should be comprised of technical representatives of cantonal and entity-level competent education authorities, as well as other key stakeholder groups, such as the private sector, APOSO, the BHAS, the Ministry of Civil Affairs, the donor community, academia and, potentially, an international expert. The body would be responsible for ensuring that common goals are established and implemented through evidence and consultation. Their tasks would include developing background analysis, organising stakeholder consultations, overseeing the elaboration of an indicator framework and devising a structure for regular analytical report on the state of education.
The Conference of Education Ministers would be given the role of steering the formulation of a common vision and goals for education. The Conference already has a complementary mandate (to drive “the fundamental reform of the existing parallel education systems of BiH as a matter of high priority” (OHR, 2000[21])), giving it the authority to help ensure that education goals are concluded with political support. The Conference, at Minister level, should commit to meet at least four times over the course of one year to provide feedback on and approve a common vision and goals for school education. Once this has been achieved, the Conference, at Minister level, should commit to meet at least once a year to discuss performance against common goals, to engage in mutual learning, and to discuss how support from the international community could be best leveraged. The meeting could take place at end-Q3 to start-Q4 each year, to coincide with planning for the entities’ and cantons’ annual budgets. Updates linked to a biennial/trienniel analytical report on the state of education (Recommendation 5.1.3) should be released just prior, to inform this dialogue.
Ensure that education goals are established through broad-based consultation
The obstacles to education reform and improvement in BiH are complex, and there are few straightforward solutions. Across the OECD, for instance, policymakers often use background characteristics as a lens to review education outcomes – recognising that education systems can underprovide for minority groups. In BiH, however, ethnic background continues to have a out-sized influence on the organisation of schooling.This lens has a strong influence on public debate around education and can be an important obstacle to reform. Evidence suggesting that certain social groups underperform others may therefore not be sufficient to propel targeted initiatives. The limited collaboration among competent competent education authorities in BiH could present roadblocks to defining common goals, reporting on progress, and engaging in state-level projects, even if robust evidence is available to inform debate.
To address this challenge, education goals should be established through broad-based consultation. Different forms of consultation will be needed at different stages of the process of establishing common goals and targets. In Canada, for instance, the Council of Education Ministers, Canada consults extensively with different stakeholder groups to elaborate its education priorities and its five-year strategic plans (OECD, 2015[22]).Overall, the elaboration of common goals for school education in BiH should involve input from business associations, important stakeholder groups, civil society organisations, and other actors. Consultations should be constructive, and organised around a set of technical issue areas, with interlocutors clearly briefed on the purpose of both the consultation and the overarching goal. APOSO should be involved in organising and mediating these consultations.
Recommendation 5.1.2. Formulate action plans and a country-level indicator framework to monitor progress against the common education goals
BiH’s country-level education strategies and laws have failed to translate into concrete change, partly due to a lack of measurable objectives and concrete implementation plans (World Bank, 2019[5]). A growing number of OECD countries now use planning and reporting tools to frame how they will advance and monitor progress against their education goals. These tools include action plans, that help education authorities break down how they are going to achieve a certain goal, and indicator frameworks, that enable education authorities to monitor progress incrementally. Countries use these tools to provide a clear sense of direction to education actors on the goals they are working towards, and the nature of their role. Clarity of focus is particularly important in the education sector, given the multiplicity of actors involved, and the fact that achieving change often necessitates incremental efforts over many years.
In BiH, efforts to implement country-level reform agendas and standards will necessarily involve many different education actors, across a complex governance landscape. In this context, establishing a clear direction on the actions that must be taken, by who and by when, becomes key to achieving progress. Another important step would be to translate common education goals into a country-level indicator framework. This framework would provide different education actors with a tool to objectively monitor and report on system progress. In BiH, the tool would provide a common reference point to align different education actors, rallying all to the objective of achieving results, and helping to structure dialogue when planned activities may need to change course.
Enable competent education authorities to develop their own action plans linked to achieving country-level goals for the school education sector
Change hinges on aligning policy activities behind strategic goals for system improvement, and for this, action plans are key. There are a number of actions that BiH will need to take at the country level to achieve common school education goals. Competent education authorities may wish to develop a country-level action plan to achieve these goals, or to integrate actions linked to achieving country-level goals into their own action plans. To ensure the coherence of action plans and the coverage of all competent education authorities, the Task Force may decide to develop a template action plan that can be customised by each authority but aligns with state-level goals and plans. There are various areas where BiH’s competent education authorities may wish to carry out collaborative activities. These include:
Initiatives to improve existing school funding models. Despite spending close to 5% of GDP on education, schools in BiH remain under-resourced. PISA 2018 results, for instance, suggest that 67% of BiH students attend schools whose principal reported that the school’s capacity to provide instruction is hindered by a lack of educational material, which includes ICT equipment. This compares to 49.6% in Albania, 47.3% in Romania and 28.4% in the OECD (OECD, 2020[23]). The majority of funding goes to teachers’ salaries, which are still perceived as low. The World Bank has proposed initiatives to strengthen school funding mechanisms in BiH, which could be covered in competent education authorities’ action plans, including:
Initiatives to address issues in rural schools. In many OECD member and partner countries, the quality of rural schooling has been improved by efforts to address multi-grade classrooms, size and efficiency issues (e.g. through school network rationalisation), limited access to early childhood education, digital connectivity gaps, and gaps in teacher quality, among other initiatives (OECD, 2017[24]).
In developing action plans, competent education authorities should incorporate features that have supported implementation planning across the OECD. In addition to the features in Table 5.2, these include:
Aligning actions with clear and specific goals. Desired outcomes should be clearly stated and included in action plans so that actors know what they are working towards (the outcome)..Desired outcomes should be clearly stated and included in action plans. For instance, an expected result outlined in the EU’s Instrument for Pre-Accession Assistance (2014-20) is to see a reduced number of early school leavers and school drop-outs. To align with this aim, BiH could establish an action plan goal to “establish an early warning system in each competent education authority.”
Ensuring actions are clear and specific. Similar to goals, actions and sub-actions should be operationally clear and specific. For example, to establish an early warning system in each competent education authority, two relevant actions could be to identify funding and to identify a research partner to develop a methodology for an early warning system.
Including an indication of timing and points of contact. BiH could consider developing mid-term outcomes or milestones to monitor progress continuously. For example, a mid-term outcome for establishing early warning systems across BiH could be that a methodology has been approved for early warning systems and that this methodology has been tested.
Reviewing progress indicators and assigning clear targets. Clear targets will help the action plan’s architects to track progress against the plan’s goals. Alongside output and outcome indicators, the plan’s architects could also identify progress indicators. Progress indicators provide implementors with a picture of the different steps in a process, and can thus help implementors keep sight of the bigger picture and major milestones that should be achieved along the way.
Identifying and planning for resource needs. For the action plan to be financially viable, the issues addressed must be sufficiently important and produce desirable results at reasonable and forseeable levels of expediture (Bryson, 2018[25]). This requires a constructive discussion with funding partners, both domestic and external (i.e. development partners). To make this discussion constructive, the agency leading formulation of the action plan (here, the Conference’s Task Force) should develop a realistic budget that prioritises actions and measures results. Decisions should align with the government’s (or governments’) broader development agenda and adequate resources should be allocated with more predictability, based on strategic plans.
Once completed, these action plans could be compiled by the Task Force for monitoring and review purposes – potentially in an abridged form.
Define a country-level indicator framework, linked to common goals
Defining a country-level indicator framework could help BiH to measure and communicate progress towards achieving its country-level education goals. A robust indicator framework will effectively translate policy goals into measurable targets, and will provide regular and objective feedback on progress. Alongside timebound targets, a robust framework will stipulate the data sources to be used and the frequency of reporting around each indicator. These steps strengthen co-ordination between different system actors, by providing clarity around the data points that they should pay attention to. A robust indicator framework will set clear expectations and reinforce transparency. In BiH, the Conference’s Task Force should elaborate a set of outcome indicators that link to the country’s common education goals. In selecting indicators, the Task Force should take care to choose indicators that can be underpinned by good-quality, regularly-released data. The Task Force may also choose to select indicators that have been prioritised in BiH’s reporting to the EU and other international partners.
Each indicator should be associated with timebound targets. BiH’s competent education authorities may decide to establish single targets at the country level, or to establish differentiated targets at the level of each competent education authority. Through the latter option, each competent education authority would have flexibility to prioritise the common education goals that are most important for them (for instance, by setting more ambitious targets for some indicators, less ambitious targets for others). However, this approach would still require baseline targets to be set at the country-level, to ensure that actions taken by competent education authorities still help BiH to progress in delivering upon its international commitments and aligning with the EU. For instance, competent education authorities may agree to set a baseline target for the rate of youth not in employment, education or training (NEET) upon completion of a five-year action plan. In 2019, this rate was 21.9% for BiH as a whole (ILO, 2020[26]). Competent education authorities may agree to set a target of 15% or 16% at the end of a medium- to long-term planning period, which would bring BiH more in line with the youth NEET rate in the newest EU member states, such as Bulgaria (13.7% in 2019) and Romania (14.7% in 2019) (ILO, 2020[26]).
Recommendation 5.1.3. Strengthen reporting on education performance and policy
In many OECD countries, regular reporting on system performance has helped build trust and a shared understanding of the system’s structure and dynamics. Regular reporting could help BiH’s competent education authorities to strengthen public confidence and engagment in reform, by establishing an evidence-informed dialogue on the sector’s main challenges and needs. In Portugal, for instance, an annual analytical report was introduced as part of a major education reform process, while Switzerland has established a quadrenniel education report as part of efforts to strengthen monitoring and reporting across the Confederation (Wong et al., 2017[27]). BiH does not regularly report on system performance. While agencies like the BHAS and APOSO provide good information on the sector’s fundamental characteristics and performance at the country-level, this information is still sparse, scattered across different platforms and there is limited information on major education policy initaitives and activities that is publicly available.
Launch a periodic State of Education in BiH report
BiH should establish a regular analytical report on the state of education to strengthen reporting on system performance, connected to priorities outlined under the country’s common goals for schooling. This report should be released on a biennial or triennial basis, to allow adequate time to carry out quality assurance of the data and conduct meaningful analysis. The report could be funded by the Conference of Education Ministers through an international development partner. The partner would ideally commit to fund multiple cycles of the report or until the Conference of Education Ministers can reach a cost sharing agreement among the country’s education systems. The report could be drafted by researchers within BiH, and/or by a group of researchers commissioned internationally, based on data provided by the competent education authorities, BHAS and APOSO.
The Conference’s Task Force would be well placed to devise the report’s structure and items to be included. The report could open with an introductory chapter on the educational context, setting out demographic trends and economic factors, among other data, as is common to most state of education reports. The report should include data disaggregated for each competent education authority, and the report’s analysis of overarching trends at the country level should include references to trends in specific authorities, where this enriches the analysis. Disaggregated data and analysis is a common feature of reporting in other countries with decentralised education systems, and helps to ensure that the exercise is useful for policy and supports peer learning. Germany’s periodic report on education, for instance, provides indicator-based information that is disaggregated for each of its federal states (Länder) and compared internationally. In addition, the country provides an online platform (https://www.bildungsbericht.de/) where a complete set of data tables can be accessed.
To complement more generalised reporting on the state of education, each iteration of the report could include a thematic chapter, on a topic selected by the Task Force. Some OECD countries have used general themes – the Swiss Education Report presents thematic chapters on effectiveness, efficiency and equity, for instance, while Norway’s Education Mirror presents thematic chapters on learning outcomes, the learning environment, upper-secondary education completion rates, school resources and school facts. In BiH, the Task Force may choose to link thematic chapters to overarching themes of the country’s common goals for schooling.
Alongside quantitative data, the report could also include qualitative insights. The Task Force should, for instance, consider including information on instructional and managerial practices and workloads in schools, provided by the Pedagogical Institutes (a need cited by the World Bank, in (World Bank, 2019[5]), To promote mutual learning between competent education authorities, the Task Force should consider featuring snapshots of successful interventions undertaken by these authorities, and any insights obtained through recent policy evaluations conducted by competent education authorities. This approach has been taken in Norway, which includes qualitative information on schools and on national initiatives to promote better local monitoring of quality in its Education Mirror (Nusche et al., 2011[28]).
Establish a web platform dedicated to school education in BiH
While basic information about the country’s education sector is available on the Ministry of Civil Affairs’ website, BiH does not have a single portal that provides information on education policies and performance. At the country level, the Ministry of Civil Affairs, APOSO and the BHAS present important information on education policies and system performance. These resources include sector-specific data and reports that interpret the findings of PISA 2018 and TIMSS 2019 for BiH. However, information on the education systems of BiH is limited, and scattered across different platforms. As a result, it is difficult to obtain a clear picture on how education funding is being used, the outputs that this funding is producing, and – most importantly – to feed reliable, objective information on system performance into public debates on education.
Concerted efforts to strengthen transparency are particularly important in countries where trust in government is low. In BiH, surveys suggest that around 64% of the population finds their education systems to be corrupt or extremely corrupt (Transparency International, 2018[29]). To strengthen transparency around education system performance, the Conference’s Task Force should consider establishing a web portal that provides data and analysis on education in BiH, and presents recent policy initiatives. This could take the form of a dedicated site or be linked to the website of the Ministry of Civil Affairs, through a page on the Conference of Education Ministers. The first option would be more costly, and it would entail more efforts to maintain and update. However, this option could provide the platform’s architects with more flexibility to develop innovative functionalities and maintain the independence of content posted.
Data are integral to system evaluation. Over the past two decades, technological advancements have enabled policymakers to compile increasingly granular and timely data, and provided new tools to analyse this data. As a result, policymakers are able to more accurately measure the direct effects of education outputs (for instance, the employment and earning benefits of pursuing higher education), as well as to measure the contribution made by specific steps in a process to reach an end result. This is important because it provides the data needed for more effective, formative policy. An ability to capture data on outcomes and processes helps to shift policy focus away from inputs (e.g. expenditure on education, the number of teachers) and outputs (e.g. enrolment and completion rates), and towards outcomes (i.e. progress towards the results that a system would like to achieve) and processes (i.e. measuring the extent to which different steps are helping or hindering the achievement of a specific result). The availability of increasingly granular data also enables policymakers to track differentiated impacts on specific demographic groups, and this supports policies to reduce inequities. However, as data are used to inform policy more often, it also becomes more important to ensure that this data is accurate, complete and timely.
Competent education authorities in BiH cannot access high-quality data support system evaluation and benchmarking purposes. While the BHAS tries to produce country-level education statistics for reporting internationally, it finds quality assurance a challenge. The BHAS informed the review team that access to reliable, comparable and timely administrative data would enable the agency to produce more accurate, timely and granular education statistics, but it cannot make use of administrative data produced by competent education authorities due to methodological differences. In parallel, many competent education authorities would like to review the performance of students, teachers and schools in their own jurisdiction, but not all have the resources needed to establish the rich data management systems that could support this analysis.
Further co-operation at the country level would help BiH to strengthen its education data. Through stronger country-level data governance, the BHAS would be able to exploit data produced by competent education authorities, and competent education authorities would be able to access better-quality data to produce more meaningful jurisdiction-specific analysis. Establishing common standards would also facilitate the country-level reporting recommended in Policy issue 5.1. At the same time, efforts to build data collection capacity at the competent education authority level would tackle “gaps” in data collection and could increase demand for education data among decision-makers. Finally, subscribing to future cycles of major international learning assessments would provide BiH with data on learning outcomes, helping the country to track performance against one of the most important measures of the effectiveness of a school system – i.e. student learning. Through these surveys’ questionnaires, policymakers can also identify the variables associated with performance that can help inform policy responses.
Recommendation 5.2.1. Progressively improve country-level data governance
Country-level efforts to improve data governance have helped many OECD countries to strengthen the quality, accessibility and use of education data. The BHAS continues to improve the quality of country-level data governance in BiH and works closely with partners like Eurostat and UNESCO-UIS to build statistical capacity. However, administrative data compiled by competent education authorities is another valuable resource that should produce rich insights for system evaluation. To unlock this resource, the BHAS and competent education authorities should work together to strengthen the country-level governance of education data. In OECD countries with decentralised education systems, such as the United States, this has been achieved through establishing common data standards, creating assurances around the privacy and security of individual data, and addressing gaps in data collection, among other initiatives.
Institute an inter-agency council or board for education data governance in BiH
Data on the education system in BiH is currently collected through two parallel processes – once by the BHAS and once by the respective competent education authority. BiH does not currently have a country-level agreement on statistical concepts, definitions or information management in the field of education, and this precludes the BHAS from being able to use data compiled by education authorities. The BHAS and competent education authorities should consider establishing an inter-agency council or board to develop a set of rigorous principles and guidelines around the collection, storage and reporting of education data. This body should be chaired by the BHAS and include representatives of the competent education authorities as well as other public users of education statistics. The body could build from an existing working group that the BHAS has established to implement its project with UNESCO. This group comprises the BHAS, representatives of entity-level statistical agencies and public users of education statistics, including the MoCA, all competent education authorities, APOSO and the Agency for Development of Higher Education and Quality Assurance of BiH. The group has already carried out extensive consultations with all of BiH’s entities, cantons and cities, and thus has a good picture of the landscape for reporting education data across BiH and familiarity with the stakeholders involved.
In other countries, these Boards are responsible for a range of tasks that support data quality and integrity. In the United States, for instance, the EDFacts Governing Board is responsible for identifying common standards, developing operating policies, and implementing processes for managing data. They have helped to resolve data ownership issues, support information sharing, proactively manage data vulnerabilities, and improve the quality of data collection, reporting and use (Edfacts, 2020[30]). The Board meets once a month, and data issues are raised by members who then lead a workgroup to analyse the issue with education stakeholders and other Board members. Background information on the topic is shared in advance through a common drive to support the analysis and facilitate transparent decision making.
Establish common data standards for BiH’s competent education authorities
One of the tasks of this council or board should be to develop a set of common data definitions and protocols for BiH’s competent education authorities. This is common in countries with decentralised education systems, where local authorities often apply their own standards, techniques, and develop their own solutions to resolve emerging issues. By implementing common data standards, country-level policymakers can be confident that data from different administrative units have the same meaning and can be relied upon to inform country-level decision-making. Establishing standards is particularly important in countries where double shift and multi-grade schools exist, as in BiH. In these cases, authorities will need to make an additional effort to ensure that schools use common standards in their reporting.
In BiH, the council or board should develop a formal data dictionary and sharing protocol for use by competent education authorities and their schools. In particular, the body should ensure that it establishes standards that would facilitate reporting against the country-level indicator framework outlined in Policy issue 5.1. As one example, it may wish to establish common definitions and specifications to record student participation and attendance, where considerable variations in coverage and comparability currently exist. Australia, for instance, has established a national standard for reporting on student attendance, which is identified as a key performance indicator in its Measurement Framework for Schooling in Australia. This standard applies to students in years 1-10 for all government, Catholic and independent schools in Australia, and establishes the following criteria: attendance rate calculation formula; actual days in attendance (numerator); number of possible school days (denominator); level of disaggregation; data collection period; school types; student enrolment types; movement during collection period; part-day absences; ungraded students, and treatment of incidents/absences. These standards will not be legally binding, but CEAs should be encouraged to adopt them, once developed and validated by a variety of stakeholders and international experts.
Review rules around the privacy and security of individual-level education data in BiH
Another task of this council or board should be to review and refine BiH-level rules around the privacy and security of individual student and teacher data. Over time, a growing share of competent education authorities should be able to compile individual-level data in order to conduct meaningful analysis and fulfil state-level and international reporting requirements (Recommendation 5.2.3). However, the compilation of individual-level data brings risks and justifiable concerns – for instance, on who will be able to access the data, and how the data could be used, both in the present and in the future. Privacy and security is particularly important in the management of student data, since these individuals are typically not old enough to provide informed consent. In BiH, ethnic sensitivities could increase concerns around the collection of individual-level data.
Decentralised education authorities are often more inclined to report individual data to country-level authorities (and schools to share this data with their respective education authorities), if they can ensure that this data is anonymised, and that the privacy of students and teachers is protected. In the majority of federal education systems across the OECD (for instance, the United States), most personal student information stays local, and each level of governance compiles data in a different way and has different access rights. Typically, rules governing the storage, use and exchange of individual-level data are developed and established at the country level, where they can benefit from more expert input and can access additional tools for enforcement. This has been the case in the United States and in Canada (Box 5.2).
In the United States, the Federal Government has established a law to protect the privacy of student education records, known as the Family Educational Rights and Privacy Act (FERPA). Under FERPA, schools that receive federal funds can disclose directory information and de-identified data to a discrete list of authorities, but they must do this in-line with a set of guidelines designed to safeguard the personal privacy of students. Under these rules, “directory information” is information that is not considered harmful or an invasion of privacy if disclosed. De-identified data is information where the student’s personally identifiable information has been removed and there is a reasonable likelihood that the student is no longer personally identifiable.
To supplement FERPA, the US federal government has established roadmaps and guidelines to help state education authorities strengthen their data protection procedures, as part of its Data Quality Campaign (DQC). One such example is the DQC’s Roadmap to Safeguarding Student Data. This roadmap establishes principles for safeguarding the privacy of student data at the state level. These principles focus on the procedures and personnel practices that are in place and include:
Policies and procedures. Levels of data sensitivity are clearly defined, and data are categorized by these levels, with appropriate differences in levels of protection depending on how sensitive the data are. Processes and practices ensure that encryption or other protection is in place during movement or transmission of sensitive or confidential data and that these protections are routinely reviewed and kept up-to-date
Personnel. Staff annually review the student data privacy policy and provide written assurances that they will meet their data privacy responsibilities as a prerequisite to accessing data; orientations for new staff regarding data responsibilities begin soon after employment; access to student personally identifiable information is based on staff roles and responsibilities; background checks are performed on new employees that can access student information.
In other OECD countries, privacy rules have been established to govern many different areas of public life – and are thus not specific to the education sector, but do apply to it. In Canada, for instance, the Government has recently passed the Digital Charter Implementation Act, (2020), which has clarified and expanded rules around personal privacy protection and strengthened country-level enforcement mechanisms. One development, for instance, has been to clarify how personal information can be used once it has been de-identified, recognising that it is increasingly easy to re-identify individuals by linking sets of data.
Source: (Data Quality Campaign, 2016[31]), Roadmap to Safeguarding Student Data: Key Focus Areas for State Education Agencies, http://dataqualitycampaign.org/wp-content/uploads/2016/03/DQC-roadmap-safeguarding-data-June24.pdf (accessed on 15 November 2021); (Government of Canada, 2020[32]), Fact Sheet: Digital Charter Implementation Act, 2020, https://www.ic.gc.ca/eic/site/062.nsf/eng/00119.html (accessed on 1 October 2021).
In BiH, reviewing the privacy laws that govern personal data could help competent education authorities to progressively establish unique identifiers for education data that are linked to civil identification numbers (Recommendation 5.2.3). Providing the right framework for protecting personal data - aligned with EU, BiH and local level regulations - would make it easier for competent education authorities to establish unique identifiers that can be linked to richer personal data, providing more insights for system evaluation.
Renew attempts to establish a (limited) EMIS at the country level
Over time, competent education authorities should renew efforts to establish an information system that compiles and stores education data at the country level. A country-level information system would enable BiH to compile comparable and timely information on its education systems, linked to the indicator framework outlined in Policy issue 5.1. Competent education authorities could be offered an interface that provides access to additional, optional modules – and this could enable authorities without an EMIS to start compiling robust data for reporting and planning, without having to build this system from scratch.
At the same time, any initiatives here should be handled carefully – to ensure that any concerns around data management and disclosure are addressed. This is evidenced by the fact that a comprehensive World Bank project to develop an EMIS for BiH, with a budget of USD 2 million, never led to country-wide implementation due to a lack of political interest (World Bank, 2005[33]). Aspects that could be considered for a country-level EMIS in BiH include:
Rules and protocols. Competent education authorities may need to conclude a memorandum of understanding (MoU) to establish a country-level EMIS. This instrument may be essential to ensure that the system is acceptable to all parties – and therefore that they report to it and it is useful for them. The MoU should articulate system features and the responsibilities of different parties, and should touch upon all issues that may be initially contentious, such as access rights and server location. In addition, the MoU should stipulate a calendar for data collection. This calendar should outline the various steps in the process and the roles of different players. The latter, for instance, could include: the dissemination of survey questionnaires; the completion of survey questionnaires at school level; the start of data capturing; the completion of the data capturing; and release of the data (UNESCO, 2020[34]).
Funding. The authorities driving this project would need to identify funding for the project – both an initial lump sum to establish the system, as well as a multi-year funding stream to finance basic running costs. This funding could be requested from an international partner, and/or come from voluntary contributions provided by participating CEAs.
Compulsory module coverage. The system architects should establish a set of compulsory modules linked to BiH’s country-level goals’ monitoring framework (see Policy issue 5.1) and international reporting requirements. In addition, system architects could develop a set of optional modules that would provide more support for system planning. These modules could be used on a voluntary basis by competent education authorities, for their own policy formulation and planning.
Optional module coverage. As mentioned above, the authorities driving this project could gradually provide competent education authorities with the option of a dedicated web interfaces, which would provide them with exclusive access to certain sets of data. This option may be particularly interesting for smaller competent education authorities that do not have an EMIS in place already, such as Brčko District. These competent education authorities may wish to upload their data directly into this system, rather than developing their own separate system from scratch. This is one area in which a country-level system could bring significant value.
Additional functionalities. Providing additional functionalities that make education data more accessible would enable this resource to strengthen trust and transparency in the system, and could create new pressures for competent education authorities to participate. It would also make the system more useful, and thus could strengthen these authorities’ interest in contributing to it.
Recommendation 5.2.2. Commit to participate in future cycles of international assessments
BiH does not implement a country-wide external assessment of learning outcomes, nor does this look feasible in the near term, though there is scope to move in this direction (see Chapter 2). Given this absence of reliable country data, BiH’s ability to participate in international learning assessments is more important. In comparison with other countries in Europe and other emerging economies, BiH stands out for the lack of sustained participation in PISA or other major surveys, such as TIMSS and PIRLS.
Producing data on learning outcomes is a feature of evaluation frameworks for education in most OECD countries because it provides information on the final results that an education system is trying to achieve (OECD, 2009[35]). Large-scale assessment results are a commonly-used reference for measuring learning outcomes. BiH’s lack of data on learning outcomes inhibits the country from setting concrete policy goals, but it also weakens policy planning at the competent education authorities level. Few competent education authorities have produced data on learning outcomes to inform the implementation of major policy changes. In RS, for instance, the ministry conducted an analysis of learning outcomes to inform whether it should reform the entity’s primary school curriculum. The analysis confirmed that the existing curriculum supported the acquisition of declarative rather than functional knowledge, and the curriculum was revised accordingly (USAID, 2016[36]).
Establish a formal commitment to participate in at least two rounds of PISA and TIMSS
Participation in international assessments can produce data on learning outcomes until countries have built domestic capacity to implement their own more regular, customised assessments. However, BiH’s participation in international assessments is not regular, and BiH recently missed its deadline to participate in PISA 2022 due to a political deadlock. To guarantee periodic data on on learning outcomes, BiH should establish a formal commitment to participate in at least two rounds of PISA (in 2024 and 2027) and TIMSS Grade 4 (in 2023 and 2027). By participating in TIMSS at Grade 4, BiH would produce learning outcomes data linked to the start of the second cycle of primary school – allowing for more course correction. Should the entities/cantons decide that they would like more entity-specific outcomes data, they could request to over-sample in these assessments. Securing participation in future international assessment cycles could enable BiH to include a measure of learning outcomes in its country-level indicator framework – for instance, an indicator on the share of students below Level 2 in PISA.
Progressively explore options for piloting a standardised assessment in FBiH or RS
In a number of OECD countries with decentralised education systems, participation in international assessments has strengthened demand for data on learning outcomes and provided comparative, objective information on student achievement that can then be disaggregated to provide insights for delegated education authorities. This has been the case in Australia, for instance, where participation in PISA has facilitated standardised reporting on learning outcomes at the country level since 2001. Since then, Australia has published a national report presenting PISA results disaggregated by school sector and by state and territory (Santiago et al., 2011[10]). In BiH, too, participation in PISA and other international studies would provide comparative data that can be disaggregated for each competent education authority, and presented with analysis in a country-level report (see Recommendation 5.1.3). Over time, many education authorities in the OECD have seen the value of having comparative data on their learning environments and outcomes, and have started to develop their own large-scale assessments. Australia, for instance, adopted a standardised, country-wide assessment in 2008, that provides nationally comparable student achievement data linked to its long-term goals. A similar trend can be observed in BiH’s neighbouring countries. Albania, North Macedonia and Serbia have participated in at least three of the last four cycles of PISA, for instance, and are now developing or have recently developed a country-level standardised assessment that can provide more regular and country-specific data on learning outcomes. As BiH secures its participation in international assessments, entity bodies may begin to explore options for developing their own standardised assessment, starting with a pilot. In the case of the FBiH, this could be for optional participation by the cantons.
Recommendation 5.2.3. Build competent education authorities’ capacity to compile high-quality data
Efforts to strengthen co-operation at the country level should be complemented by targeted support to strengthen data collection capacity in specific competent education authorities. These steps will be necessary to address “gaps” in data collection that would hamper country-level and international reporting. Targeted support could also raise demand for education data among decision-makers within these authorities. Alongside targeted technical support and outreach from the BHAS, BiH’s governance body for education data (Recommendation 5.2.1) should help competent education authorities to establish a data expert position in each of their jurisdictions and progressively build capacity to implement the use of identification numbers to compile student and teacher data.
Identify the most critical gaps for reporting on country-level goals at the competent education authority level and devise an action plan to address them
While the BHAS is continuing to strengthen education data reporting within BiH’s administrative unit, BiH may need a more deliberate strategy to support reporting at the country level (Recommendation 5.1.3). In order to improve the quality and availability of country-level and competent education authority-level education data, the BHAS and other partners should map the data collection and management capacities of each competent education authority, and identify critical gaps where they exist. Implementing this exercise while the Conference’s Task Force is elaborating a measurement framework for their common goals could help to ascertain the information that could be collected, reliably, from all units, and the extent of capacity building needs in each authority. Once a measurement framework has been established, the BHAS should draw up a plan for capaciy building activities, prioritising the competent education authorities that have the greatest needs and the largest school systems.
Enable each competent education authority to create at least one data analyst position
To strengthen their collection and use of education data, BiH’s competent education authorities will require at least one expert that can maintain and improve the country’s information system, flag potential data errors, and has quantitative analysis skills for processing data and creating thematic reports. This is a requirement to manage EMIS in most OECD and peer countries. In Georgia, for example, the EMIS employs five statisticians solely for responding to data and research requests, in addition to department leadership, administrative support and software developers who manage the system (Li et al., 2019[37]). Where necessary, BiH’s governance body for education data (Recommendation 5.2.1) should help competent education authorities to unlock funding for this position. The BHAS could also progressively offer technical assistance and training opportunities to these staff members, to develop their technical skills and remain up-to-date with changes in the EMIS, user needs and changing technologies (Abdul‐Hamid, 2014[38]), potentially in partnership with Eurostat, UNESCO-UIS and other development partners.
Support the adoption and use of individual identification numbers by competent education authorities
Over the past few years most of BiH’s regional peers have integrated unique identification numbers into their EMIS. Using unique identifiers has helped these countries enhance the analytical functions of education data and it has provided insights to support progress against national education goals. Through linking unique identifiers to civil identification numbers, these countries have been able to additionally link education data to information on an individual’s background characteristics. This functionality enables countries to monitor the education outcomes of vulnerable demographic groups, and thus more easily report progress against the equity aspects of SDG4 and – in the case of EU neighbourhood countries – against the EU’s inclusive education targets.
In BiH, the implementation of unique identifiers for education data is hampered by particularities of the country’s unique identification system. Under BiH law, the responsibility for issuing identification numbers has been delegated to the entities, via their Ministries of Interiors. In the FBiH, this responsibility has subsequently been delegated to the cantons. Currently, RS uses a special identification number (ID) to store student-level data in its EMIS, but this ID is not linked to a civil identification number. According to data collected by the review team, no other competent education authority currently uses unique identifiers.
The Ministry of Civil Affairs and competent education authorities should work to identify the challenges that competent education authorities face in implementing unique identifiers and linking these to a civil identification number. This analysis could be carried out in consultation with the EU as a technical assistance project through BiH’s Pre-Accession Assistance Programme. This topic should be a development focus, because learning losses from COVID-19 have impacted certain demographic groups more than and differently to others. However, education authorities are not able to obtain data on access to and participation in education by gender, ethnic origin, disability and level of education, according to a rapid assessment undertaken by UNICEF (UNICEF, 2020[39]). In undertaking this study, the experience of particular jurisdictions could be explored. West Herzegovina Canton, for instance, is currently developing a new EMIS, in partnership with the University of Mostar and all cantons that cover the Croatian language. This system will purportedly implement unique identifiers, and these identifiers should also be linked to an administrative ID. These efforts should not involve the sharing of individual data, and should continue to ensure that personal privacy laws, at all levels, are respected.
BiH’s recent participation in two international learning assessments has generated comparative, quality data on learning outcomes, and sparked international and country-wide debate on the system’s performance. BiH’s competent education authorities have also established links with a handful of “hub” universities – such as the University of Mostar, the University of Banja Luka and the University of Sarajevo – that provide support for different research and development projects. At the same time, there is surprisingly little domestic demand for data and analysis on system performance, not only among unit-level education authorities, but also among local researchers and the wider public. Though outcome data is now available through TIMSS and PISA, minimal secondary analysis has been conducted and political tensions jeapordise future participation in these studies.
BiH’s decentralised system could be leveraged to support policy experimentation and mutual learning. BiH also has a sizeable diaspora and development partners that could be mobilised to produce outward-looking analysis and debate on how its education systems are performing. Investment in system-level data and outcomes monitoring, as outlined in Policy issue 5.2 would support these efforts. To build a stronger culture of education research in BiH, the Conference should consider creating an international scholarship programme for research in education, with a dedicated pool of funding. To improve accountability and local scrutiny, the Conference should establish citizen assemblies, that would meet on a periodic basis to provide input on implementing important reforms.
Recommendation 5.3.1. Create an international scholarship programme for research in education
Objective, policy-relevant and methodologically-sound research and analysis on the education system is an important source of governance information for policymakers and other sectoral stakeholders. This resource is particularly important for the governance of education, where the costs of negative effects from major reforms are high, but may take a while to manifest clearly. In BiH, many competent education authorities do not make active use of research and analysis on BiH’s education system to guide policymaking, and this information is not widely available. There have been no concerted efforts at the country level to co-ordinate, consolidate and extensively commission research in education, and most competent education authorities do not have the resources to do this independently.
This issue is not limited to BiH – it has been encountered in other decentralised education systems like Switzerland. In Switzerland, for instance, research findings were used for some time to formulate policy, but in a way that was not systematic (OECD, 2006[40]). Rather, the use of education research to inform policy depended strongly on each canton’s general environment – e.g. the existence and quality of local universities and other research bodies. Recognising this, Switzerland has made a concerted effort to support the more systematic production and use of education research since the 1960s. One of the main instruments it chose was to create country-level institutions like Swiss Co-ordination Centre for Research in Education, which plays an important role in co-ordinating, compiling and disseminating education research (OECD, 2006[40]). BiH may choose to initially create a scholarship programme for research in education, leveraging expertise and funding from international partners.
Elaborate a proposal for an international scholarship programme for research in education
To ensure that research findings from the scholarship programme are used systematically to inform policy, the Conference’s Task Force should “own” the programme and elaborate its main features. A first step would be to elaborate a proposal for the scholarship programme that outlines a general picture of its design, its benefits for advancing education policy in BiH, and an indicative estimate of its funding needs. This proposal should be elaborated following a consultation with some of BiH’s major research hubs, such as the Universities of Sarajevo, Banja Luka and Mostar, as well as the country’s Pedagogical Institutes.
Approach development partners that can provide sustained funding
A second step would be to identify a source of sustained funding. The Conference’s Task Force may wish to initially approach development partners that have established relationships with BiH’s main universities and other research hubs, such as USAID, which partnered with the Universities of Sarajevo, Mostar, Banja Luka, Zenica and Tuzla, for instance, through its Enhancing and Advancing Basic Learning and Education in Bosnia and Herzegovina project (USAID, 2016[41]). Through these links, the programme’s architects could ensure that the research projects it sponsors are coherent with the topics already being researched in BiH’s universities and other research hubs, and that findings from these research projects link back into these bodies’ own research activities. In approaching potential donors, it would be important to ascertain the level of funding that might be available, since this would likely determine the scheme’s design features.
Alongside development partner funding, BiH’s Pedagogical Institutes should be encouraged to contribute a nominal amount to the programme’s funding. This contribution would signal the development of research on the education system as a core mission of the Pedagogical Institutes, and provide the Pedagogical Institutes with a stake in setting research priorities and selecting projects. The latter would be enriched by information that Pedagogical Institutes have acquired from interactions with schools and from observing classroom practices. Findings from the scholarship programme should also be helpful for Pedagogical Institutes. During interviews with the Pedagogical Institute of Republika Srpska, for instance, the review team heard that the Institute would like to strengthen capacity for formative assessment in RS schools. To accomplish this task, the Institute may benefit from having more evidence and analysis on effective formative teaching and assessment approaches for students based on characteristics like age and socio-economic status. This information would help the Institute develop a more nuanced and targeted work programme, and thus continue to advance the relevance, effectiveness and efficiency of its work.
Elaborate the scholarship programme’s design
While the source and level of funding for the scholarship programme will likely influence its design features, the Task Force (spearheaded by APOSO) should lead its design, to ensure that it remains relevant for BiH. In elaborating the programme’s design, the Task Force may wish to consider the following features:
Limited yet competitive eligibility criteria. Limiting eligibility criteria to just three to four conditions can help expand the pool of potential candidates. However, eligibility criteria should help to filter the most competitive candidates. For instance, candidates may need to demonstrate outstanding academic performance during undergraduate and postgraduate studies, including a proficiency in research methods and a final thesis.
A competitive selection process. The selection process should be competitive and have transparency mechanisms built in. For instance, candidates could be selected through two stages – the first involving an assessment by representatives of the Conference’s Task Force, the second involving an assessment by an international selection panel. Any special weights ascribed for selection should also be made public – for instance, the programme may wish to assign special weights to research proposals addressing cross-entity education issues or comprised of cross-entity team members.
Clearly-defined benefits and expectations. The benefits and expectations of research grants should be clearly-defined, to ensure accountability. For instance, the programme may stipulate requirements to participate in specific events, to fulfil all requirements of the host university, to notify the programme’s administrators of any changes to their situation, and to fulfil all necessary reporting as requested by the programme’s administrators.
Provide APOSO with a mandate to administer the scholarship programme
APOSO would be well-placed to administer the programme. As an independent, state-level expert body, it would have an informed perspective on the most critical policy questions around education reform in BiH, and it could select research projects on that basis. The Conference’s Task Force, which should also comprise a representative of APOSO, should identify a set of pressing research topics for BiH. To identify topics, the Task Force could also consult with BiH’s major universities, such as the Universities of Sarajevo, Banja Luka and Mostar. These universities have a history of constructive collaboration with BiH’s education authorities, and also provide initial teacher education. In addition, Pedagogical Institutes in BiH should help identify research projects for the programme, providing information to address the critical quality development challenges they observe in their interactions with schools. Potential research areas could include modalities for school funding, student assessment, and effective teacher policies (for an example of research on teacher policy linked to PISA, please see (OECD, 2018[42]).
Ensure that research findings are reported to the public and fed into policy planning
Research findings from the programme should be published by the Task Force, potentially via a web portal dedicated to education in BiH (Recommendation 5.1.3). The Task Force could preface these reports with remarks on how the findings will be used to support policy planning in BiH and to provide more accountability information to the public. In addition, APOSO could organise an annual conference for education authorities and HEIs (as well as other research bodies) at the start of each year (to follow the release of entity-level budgets) to present key findings from research carried out through the programme, as well as any other important recently-released studies. These conferences could be general or thematic, but they should provide education authorities with an easy opportunity to access recent research on the education sector that is relevant to BiH, enable them to ask questions, and invite them to offer their own feedback on the findings and to highlight critical challenges that they currently face. The information gathered should help competent education authorities to develop their annual action plans, identify areas for common projects, and support the Task Force’s annual planning.
Recommendation 5.3.2. Establish citizens’ assemblies to provide input for planning the implementation of important reforms
Education remains a politically sensitive topic in BiH, which can create roadblocks for reform. At the same time, most BiH citizens do not have the opportunity to engage in decision-making around education policy nor do they have a comprehensive picture on how the education system is performing. Parents, in particular, lack access to information that wil help them reflect on the progress their children made in school, the educational choices available and how to engage more broadly with the education system. Trust in government is also very low. According to the 2019 edition of the The Gallup World Poll survey, only 23% of people in BiH report confidence in their government. This is the lowest level in the Western Balkans and has deteriorated at the fastest rate since 2007 (OECD, 2020[43]).
In a number of OECD countries, policymakers have used citizens’ assemblies to deliberate on, and get buy-in for, important issues and initiatives. These processes aim to address complex policy problems, particularly those that will have an important impact on future generations, through an inclusive discussion (OECD, 2020[44]). Citizens’ assemblies have served as an important tool for deliberative democracy in Ireland, where the government is establishing a Citizens’ Assembly on the Future of Education. Ireland faces pressure to reorganise its school network, to ensure equity-based provision of education, and to make important decisions on curriculum and assessment reform, particularly in the wake of COVID-19. Through the Citizens’ Assembly on the Future of Education, Ireland aims to achieve “a shared understanding of the value of education,” and to address “how education can prepare people of all ages to meet new societal, environmental, technological and economic challenges” (Education Matters, 2020[45]). Importantly, the initiative aims to ensure that “the voices of young people and those being educated are central” (Education Matters, 2020[45]). In BiH, citizen assemblies should strengthen trust in the education system, show that this system is accountable to the public it serves and provide the public with a role in shaping its development.
Structure a format for assembly meetings
Citizen assemblies typically work well for three types of problem: i) value-driven dilemmas; ii) complex problems that require trade-offs; and iii) long-term issues that go beyond electoral cycles (OECD, 2020[44]). Since citizen assemblies take time and resources to organise, they should only be used for public policy questions where the costs of action and/or inaction are high – in BiH, for instance, a topic for deliberation could be around the allocation of school funding, or the assessment and certification of students (see Chapter 2). To ensure that citizen assemblies serve to diminish rather than amplify the role of politics in debates around education and can take a broader perspective on education challenges, these assemblies should be organised at the country level, but they should ensure fair representation of citizens from different jurisdictions (see below).
In terms of their duration, most citizens’ assemblies across the OECD do not meet for less than four full days in-person, unless a shorter time frame can be justified (OECD, 2020[44]). This standard duration is designed to give participants time to review evidence, hear the views of their co-assembly members, and reflect on this information in-between meetings. In BiH, however, a shorter duration may be necessary and virtual meetings should be considered to circumvent barriers to attendance. The Task Force should decide on the occurrence and exact format of citizen assembly meetings. The Task Force should also ensure that ample imformation is provided to participants in order to facilitate constructive dialogue.
Determine the composition of assemblies
Citizens’ assemblies aims to reflect the broad characteristics of the country’s electorate – usually, the electorate’s gender, ethnicity, social class, and the area in which they reside. Candidates are selected through random selection and stratified sampling, and this method attempts to include those that may have been traditionally excluded from public decision-making (OECD, 2020[44]). Where policymakers have capacity, stratification can also be made on attitudinal criteria. In certain instances, some OECD countries over-sample particular demographics during the random sampling stage, in order to achieve representativeness (OECD, 2020[44]). In BiH, organisers should ensure that stratification takes into account a fair representation of citizens from different administrative units, to ensure that the direction provided by citizen assemblies reflects local contexts and concerns. In addition, organisers may wish to consider additional background criteria during the sampling process – such as including the representatives of groups that advocate for those with learning difficulties and/or other disabilities. Ensuring that the education system caters for students with learning difficulties and/or other disabilities will be a critical challenge across most of BiH over the years ahead, particularly as these students appear to exhibit particularly stark learning losses following the COVID-19 pandemic and related school closures (UNICEF, 2020[39]).
Establish how assembly proceedings will be used to inform policymaking
In order to be productive, citizens’ assemblies should influence public decisions. Assembly organisers should ensure the utmost transparency during this process. For instance, the main conclusions of the assembly’s deliberation should be made public, and the commissioning authority (here, the Conference of Education Ministers) should publicly commit to respond to or act on participants’ recommendations in a timely manner. The assembly’s conclusions and the Conference’s responses could be published via a web portal dedicated to education in BiH (Recommendation 5.1.3). Ideally, the Task Force should publish subsequent updates on whether and how it has implemented the assembly’s recommendations through a regular public progress report (OECD, 2020[44]).
References
[38] Abdul‐Hamid (2014), What Matters Most for Education Management Information Systems: A Framework Paper, World Bank Group, http://hdl.handle.net/10986/21586 (accessed on 18 December 2021).
[7] APOSO (2020), TIMSS Report and Recommendations for Bosnia and Herzegovina, https://aposo.gov.ba/en/timss-report-and-recommendations-for-bosnia-and-herzegovina/ (accessed on 10 October 21).
[1] BiH (2021), Country Background Report for the OECD Review of Evaluation and Assessment in Education: Bosnia and Herzegovina, Unpublished.
[25] Bryson, J. (2018), Strategic Planning for Public and Nonprofit Organizations: A Guide to Strengthening and Sustaining Organizational Achievement, John Wiley & Sons.
[31] Data Quality Campaign (2016), Roadmap to Safeguarding Student Data: Key Focus Areas for State Education Agencies, http://dataqualitycampaign.org/wp-content/uploads/2016/03/DQC-roadmap-safeguarding-data-June24.pdf (accessed on 15 November 2021).
[9] EASNIE (2017), Decentralisation in Education Systems, https://www.european-agency.org/sites/default/files/Decentralisation%20in%20Education%20Systems_0.pdf (accessed on 18 December 2021).
[15] EC (2019), “Individuals who have basic or above basic overall digital skills (%)”, Eurostat website, https://appsso.eurostat.ec.europa.eu/nui/show.do?dataset=isoc_sk_dskl_i&lang=en (accessed on 29 September 2021).
[8] EC (2018), Annex to the Commission Implementing Decision amending Commission Decision C(2014) 9495 of 15.12.2014 adopting the Indicative Strategy Paper for Bosnia and Herzegovina for the period 2014-2020, https://ec.europa.eu/neighbourhood-enlargement/system/files/2018-12/20180817-revised-indicative-strategy-paper-2014-2020-for-bosnia-and-herzegovina.pdf (accessed on 18 December 2021).
[30] Edfacts (2020), Introduction to the U.S. Department of Education Data Governance, U.S. Department of Education, https://edfacts.communities.ed.gov/api/ApplicationMedia/GetDownload/86100 (accessed on 18 December 2021).
[45] Education Matters (2020), Citizens Assembly for Education Could Be a Game-Changer, https://educationmatters.ie/citizens-assembly-on-education/ (accessed on 18 December 2021).
[11] European Commission (2019), Analytical Report Accompanying the Document: Communication from the Commission to the European Parliament and the Council Commission Opinion on Bosnia and Herzegovina’s application for membership of the European Union, https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/20190529-bosnia-and-herzegovina-analytical-report.pdf (accessed on 18 December 2021).
[32] Government of Canada (2020), Fact Sheet: Digital Charter Implementation Act, 2020, https://www.ic.gc.ca/eic/site/062.nsf/eng/00119.html (accessed on 1 October 2021).
[26] ILO (2020), ILOSTAT, https://ilostat.ilo.org/ (accessed on 18 December 2021).
[20] ILO (2019), “Unemployment, youth total (% of total labor force ages 15-24)”, ILOSTAT database, https://ilostat.ilo.org/ (accessed on 20 November 2020).
[37] Li, R. et al. (2019), OECD Reviews of Evaluation and Assessment in Education: Georgia, OECD Reviews of Evaluation and Assessment in Education, OECD Publishing, Paris, https://doi.org/10.1787/94dc370e-en.
[2] MoCA (n.d.), Education strategic documents, Ministry of Civil Affairs of Bosnia and Herzegovina, http://www.mcp.gov.ba/Content/Read/obrazovanje-dokumenti (accessed on 18 December 2021).
[28] Nusche, D. et al. (2011), OECD Reviews of Evaluation and Assessment in Education: Norway 2011, OECD Reviews of Evaluation and Assessment in Education, OECD Publishing, Paris, https://doi.org/10.1787/9789264117006-en.
[13] OECD (2020), Early Childhood Education: Equity, Quality and Transitions: Report for the G20 Education Working Group, https://www.oecd.org/education/school/early-childhood-education-equity-quality-transitions-G20.pdf (accessed on 18 December 2021).
[19] OECD (2020), Education in the Western Balkans: Findings from PISA, PISA, OECD Publishing, Paris, https://doi.org/10.1787/764847ff-en.
[43] OECD (2020), Government at a Glance: Western Balkans, OECD Publishing, Paris, https://doi.org/10.1787/a8c72f1b-en.
[44] OECD (2020), Innovative Citizen Participation and New Democratic Institutions: Catching the Deliberative Wave, OECD Publishing, Paris, https://doi.org/10.1787/339306da-en.
[23] OECD (2020), PISA 2018 Results (Volume V): Effective Policies, Successful Schools, PISA, OECD Publishing, Paris, https://doi.org/10.1787/ca768d40-en.
[12] OECD (2018), Early Learning Matters: Brochure for the International Early Learning and Child Well-being Study, https://www.oecd.org/education/school/Early-Learning-Matters-Project-Brochure.pdf (accessed on 18 December 2021).
[42] OECD (2018), Effective Teacher Policies: Insights from PISA, PISA, OECD Publishing, Paris, https://doi.org/10.1787/9789264301603-en.
[24] OECD (2017), “Country roads: Education and rural life”, Trends Shaping Education Spotlights, No. 9, OECD Publishing, Paris, https://doi.org/10.1787/ea43a39d-en.
[22] OECD (2015), Education Policy Outlook: Canada, OECD Publishing, Paris, https://www.oecd.org/education/EDUCATION%20POLICY%20OUTLOOK%20CANADA.pdf (accessed on 3 February 2022).
[14] OECD (2011), Lessons from PISA for the United States, Strong Performers and Successful Reformers in Education, OECD Publishing, Paris, https://doi.org/10.1787/9789264096660-en.
[35] OECD (2009), Measuring Government Activity, OECD Publishing, https://doi.org/10.1787/9789264060784-en.
[40] OECD (2006), OECD/CERI Review: Educational Research and Development in Switzerland - Country Background Report, CORECHED, https://www.oecd.org/education/ceri/38011656.pdf.
[21] OHR (2000), Meeting of the Conference of the Ministers of Education of Bosnia and Herzegovina – Declaration, http://www.ohr.int/ohr_archive/meeting-of-the-conference-of-the-ministers-of-education-of-bosnia-and-herzegovina-declaration-2/ (accessed on 18 December 2021).
[3] OSCE (2003), International organizations welcome adoption of law on education in Bosnia and Herzegovina, https://www.osce.org/bih/55404 (accessed on 18 December 2021).
[18] Robayo-Abril, M. and N. Millán (2019), Breaking the Cycle of Roma Exclusion in the Western Balkans, The World Bank Group, https://documents1.worldbank.org/curated/en/642861552321695392/pdf/Breaking-the-Cycle-of-Roma-Exclusion-in-the-Western-Balkans.pdf (accessed on 18 December 2021).
[10] Santiago, P. et al. (2011), OECD Reviews of Evaluation and Assessment in Education: Australia 2011, OECD Reviews of Evaluation and Assessment in Education, OECD Publishing, Paris, https://doi.org/10.1787/9789264116672-en.
[29] Transparency International (2018), National Results: Bosnia and Herzegovina (GBC 2013), https://www.transparency.org/gcb2013/country?country=bosnia_and_herzegovina (accessed on 23 March 2020).
[34] UNESCO (2020), Efficiency and Effectiveness in Choosing and Using an EMIS: Guidelines for Data Management and Functionality in Education Management Information Systems (EMIS), UNESCO Publishing, Paris, http://tcg.uis.unesco.org/wp-content/uploads/sites/5/2020/09/EMIS-Buyers-Guide-EN-fin-WEB.pdf (accessed on 15 November 2021).
[39] UNICEF (2020), Rapid Situation and Needs Assessment - Education in Bosnia and Herzegovina: Phase II, UNICEF Bosnia and Herzegovina, Sarajevo, https://bosniaherzegovina.un.org/sites/default/files/2020-10/UNICEF_UNESCO%20RNA%202%20Phase%202_final.pdf (accessed on 18 December 2021).
[16] UNICEF (2020), UNICEF Bosnia and Herzegovina Country Office: Every Child Has an Equitable Chance in Life - Children with Disabilities, https://www.unicef.org/bih/en/children-disabilities (accessed on 18 December 2021).
[17] UNICEF (2020), UNICEF Bosnia and Herzegovina Country Office: Every Child Has an Equitable Chance in Life - Roma Children, https://www.unicef.org/bih/en/roma-children (accessed on 18 December 2021).
[4] USAID (2017), Brief Overview of Main Challenges in Primary and Secondary Education in BiH, USAID/Bosnia and Herzegovina, Sarajevo, http://www.measurebih.com/uimages/Overview20of20Main20Challenges20in20Primary20and20Secondary20Education20in20BiH.pdf (accessed on 18 December 2021).
[41] USAID (2016), Enhancing and Advancing Basic Learning and Education in Bosnia and Herzegovina: First Quarterly Progress Report, United States Agency for International Development, https://pdf.usaid.gov/pdf_docs/PA00TQK2.pdf (accessed on 18 December 2021).
[36] USAID (2016), Monitoring and Evaluation Support Activity (MEASURE-BiH): Brief Assessment of Basic Education in Bosnia and Herzegovina, http://www.measurebih.com/uimages/Basic%20Education%20Assessment.pdf (accessed on 18 December 2021).
[27] Wong, K. et al. (2017), Federalism and Education: Cross-National Lessons on Governance, Standards, and Accountability for the 21st Century, Forum of Federations, Ontario, http://www.forumfed.org/wp-content/uploads/2014/01/OC_22_Forumfed.pdf (accessed on 18 December 2021).
[5] World Bank (2019), Bosnia and Herzegovina: Review of Efficiency of Services in Pre-University Education - Phase I Stocktaking, https://documents1.worldbank.org/curated/en/719981571233699712/pdf/Bosnia-and-Herzegovina-Review-of-Efficiency-of-Services-in-Pre-University-Education-Phase-I-Stocktaking.pdf (accessed on 18 December 2021).
[6] World Bank (2012), Bosnia and Herzegovina: Challenges and Directions for Reform - A Public Expenditure and Institutional Review, World Bank, Washington D.C., https://documents1.worldbank.org/curated/en/143251468005404225/pdf/662530ESW0BA0P0C0disclosed080200120.pdf (accessed on 18 December 2021).
[33] World Bank (2005), Implementation Completion Report on a Credit in the Amount of SDR 8.0 Million (USD 10.6 Million Equivalent) to Bosnia And Herzegovina for an Education Development Project, https://documents1.worldbank.org/curated/en/454241468013228497/pdf/320330rev.pdf (accessed on 18 December 2021).