Chapter 2. Improving the value of school-based assessments and central examinations for teaching and learning

This chapter looks at how the assessment system of Serbia measures and shapes student learning. Classroom assessments in Serbia are often summative and have high stakes for students. Developing the assessment literacy of teachers and ensuring a better balance between school-based formative and summative assessment can help shift attention from grades towards student learning. There is also a need to review the design of a new final examination (Matura) at the end of upper secondary education, especially the new system for admission into higher education. Finally, Serbia should strengthen the technical quality of the central examination at the end of basic education (Grade 8). These are essential to improving the quality of Serbia’s exam system, creating a fairer basis for student selection and encouraging broader learning across the curriculum.

    

Introduction

The primary purpose of student assessment is to determine what students know and are capable of doing to help them advance their learning and make informed decisions to further their education. In Serbia, efforts are being made to reform school-based assessment practices and centralised examinations to better serve this primary purpose. For example, learning standards and new curricular plans and programmes are modernising teaching and learning expectations based on a competency-based approach. In addition, there are plans for a new central Matura examination at the end-of-upper-secondary education, a prerequisite to obtaining a secondary school completion certificate. Students’ results in the Matura exam will also be the main criterion for selection in a new system of admissions into higher education.

These policy efforts are promising but challenges remain. Work is needed to improve the design and implementation of these reforms. For example, there is a need to ensure a better balance between school-based formative and summative assessment; whereas classroom assessments for learning are poorly understood, valued and practised, assessments of learning are very frequently practised and have high stakes for students. Shifting the attention of teachers and students from grades to learning will require a clear governmental mandate that redefines expectations of how classroom assessment ought to be practised. There is also a need for further reflection on the design and implementation of a new Matura at the end-of-upper-secondary education. While the current concept for Matura is generally well developed and establishes a firm foundation for the reform, there are several underdeveloped areas which require additional consideration. These include a lack of clarity in the new admission system into higher education institutions, the need to conduct a pilot study to review and complete Matura’s examination model and the development of a realistic timeframe and organisational model for implementation. Another key issue identified by this review is the need to strengthen the technical quality and implementation of the central examination at the end of basic education (Grade 8).

Key features of an effective student assessment system

Student assessment refers to the processes and instruments used to evaluate student learning. These include assessment by teachers as part of school-based, classroom activities, such as daily observations and periodic quizzes, and through standardised examinations and assessments designed and graded outside schools.

Overall objectives and policy framework

At the centre of an effective policy framework for student assessment is the expectation that assessment supports student learning (OECD, 2013[1]). This expectation requires clear and widely understood national learning objectives. Assessment regulations must orient teachers, schools and assessment developers on how to use assessment to support learning goals.

To these ends, effective assessment policy frameworks encourage a balanced use of summative and formative assessments, as well as a variety of assessment types (e.g. teacher observations, written classroom tests and standardised instruments). These measures help to monitor a range of student competencies and provide students with an appropriate balance of support, feedback and recognition to encourage them to improve their learning. Finally, effective assessment frameworks also include assurance mechanisms to regulate the quality of assessment instruments, in particular central, standardised assessments.

The curriculum and learning standards communicate what students are expected to know and be able to do

Common expected learning outcomes against which students are assessed are important to determine their level of learning and how improvements can be made (OECD, 2013[1]). Expectations for student learning can be documented and explained in several ways. Many countries define them as part of national learning standards. Others integrate them into their national curriculum frameworks (OECD, 2013[1]).

While most reference standards are organised according to student grade level, some countries are beginning to organise them according to competency levels (e.g. beginner and advanced), each of which can span several grades (New Zealand Ministry of Education, 2007[2]). This configuration allows for more individualised student instruction but requires more training for teachers to properly understand and use the standards when assessing students.

Types and purposes of assessment

Assessments can generally be categorised into classroom assessments, national examinations and national assessments. Assessment has traditionally held a summative purpose, aiming to explain and document learning that has occurred. Many countries are now also emphasising the importance of formative assessment, which aims to understand learning as it occurs in order to inform and improve subsequent instruction and learning (see Box 2.1) (OECD, 2013[1]). Formative assessment is now recognised to be a key part of the teaching and learning process and has been shown to have one of the most significant positive impacts on student achievement among all educational policy interventions (Black and Wiliam, 1998[3]).

Box 2.1. Purposes of assessment

• Summative assessment – assessment of learning summarises learning that has taken place in order to record, mark or certify achievements.

• Formative assessment – assessment for learning identifies aspects of learning as they are still developing in order to shape instruction and improve subsequent learning. Formative assessment frequently takes place in the absence of marking. For example, a teacher might ask students questions at the end of the lesson to collect information on how far students have understood the content and use the information to plan future teaching.

Source: OECD (2013[1]), Synergies for Better Learning: An International Perspective on Evaluation and Assessment, https://dx.doi.org/10.1787/9789264190658-en.

Figure 2.1. Student assessment and learning

Classroom assessment

Among the different types of assessment, classroom assessment has the greatest impact on student learning (Absolum et al., 2009[4]). It supports learning by: regularly monitoring learning and progress; providing teachers with information to understand student learning needs and guide instruction; and helping students understand the next steps in their learning through the feedback their teachers provide.

Classroom assessments are administered by teachers in classrooms and can have both summative and formative purposes. They can be delivered in various formats, including closed multiple-choice questions, semi-constructed short-answer questions and open-ended responses such as essays or projects. Different assessment formats are needed for assessing different skills and subjects. In general, however, assessing complex competencies and higher-order skills requires the use of more open-ended assessment tasks.

In recent decades, as most OECD countries have adopted more competency-based curricula, there has been a growing interest in performance-based assessments such as experiments or projects. These types of assessments require students to mobilise a wider range of skills and knowledge, and demonstrate more complex competencies such as critical thinking and problem solving (OECD, 2013[1]). Encouraging and developing effective, reliable, performance-based assessment can be challenging. OECD countries that have tried to promote this kind of assessment have found that teachers have required far more support than initially envisaged.

Effective classroom assessment requires the development of teachers’ assessment literacy

Assessment is now seen as an essential pedagogical skill. In order to use classroom assessment effectively, teachers need to understand how national learning expectations can be assessed – as well as the students’ trajectory in reaching them – through a variety of assessments. Teachers need to know what makes for a quality assessment – validity, reliability, fairness – and how to judge if an assessment meets these standards (see Box 2.2). Feedback is important for students’ future achievement and teachers need to be skilled in providing constructive and precise feedback.

Box 2.2. Key assessment terms

• Validity – focuses on how appropriate an assessment is in relation to its objectives. A valid assessment measures what students are expected to know and learn as set out in the national curriculum.

• Reliability – focuses on how consistent the assessment is measuring student learning. A reliable assessment produces similar results despite the context in which it is conducted, across different classrooms or schools for example. Reliable assessments provide comparable results.

Source: OECD (2013[1]), Synergies for Better Learning: An International Perspective on Evaluation and Assessment, https://dx.doi.org/10.1787/9789264190658-en.

Many OECD countries are investing increasingly in the development of teachers’ assessment literacy, starting with initial teacher education. In the past, teachers’ initial preparation in assessment has been primarily theoretical; countries are now trying to make it more practical, emphasising opportunities for hands-on learning, where teachers can develop and use different assessments for example. Countries encourage initial teacher education providers to make this shift by incorporating standards on assessment in programme accreditation requirements and in the expectations of new teachers listed in national teacher standards.

It is essential that teachers’ initial preparation on assessment is strengthened through ongoing, in-school development. Changing the culture of assessment in schools – especially introducing more formative approaches and performance-based assessments, and using summative assessments more effectively – requires significant and sustained support for teachers. Continuous professional development, such as training on assessment and more collaborative opportunities in which teachers can share effective assessment approaches, provides vital encouragement. Pedagogical school leaders also play an essential role in establishing a collaborative culture of professional enquiry and learning on the subject of assessment.

Finally, countries need to invest significantly in practical resources to ensure that learning expectations defined in national documents become a central assessment reference for teachers and students in the classroom. These resources include rubrics that set out assessment criteria, assessment examples aligned to national standards and marked examples of student work. Increasingly, countries make these resources available online through interactive platforms that enable teachers to engage in developing standards, which facilitates a greater feeling of ownership of the resources and makes it more likely that they will be used.

National examinations

National examinations are standardised assessments developed at the national or state level with formal consequences for students. The vast majority of OECD countries (31) now have exit examinations at the end of upper secondary education to certify student achievement and/or for selection into tertiary education, reflecting rising expectations in terms of student attainment and the importance of transparent systems for determining access to limited further education opportunities (see Figure 2.2). National examinations are becoming less common at other transition points as countries seek to remove barriers to progression and reduce early tracking. Among those OECD countries (approximately half) which continue to use national examinations to inform programme and/or school choice for entrants to upper secondary education, few rely solely or even primarily on the results of examinations to determine a student’s next steps.

While classroom assessment is the most important assessment for learning, evidence shows that the pace of learning slows down without external benchmarks such as examinations. National examinations signal student achievement and in many countries carry high stakes for students’ future education and career options, which can help to motivate students to apply themselves (Bishop, 1999[5]). They are also more reliable than classroom assessment and less susceptible to bias and other subjective pressures, making them a more objective and arguably fairer basis for taking decisions when opportunities are limited, such as access to university or high-demand schools.

However, there are limitations related to using examinations. For instance, they can only provide a limited snapshot of student learning based on performance in one-off, time-pressured exercises. To address this concern, most OECD countries complement examination data with classroom assessment information, teachers’ views, student personal statements, interviews and extracurricular activities to determine educational pathways into upper secondary and tertiary education.

Another concern is that the high stakes of examinations can distort teaching and learning. If examinations are not aligned with the curriculum, teachers might feel compelled to dedicate excessive classroom time to examination preparation instead of following the curriculum. Similarly, students can spend significant time outside the classroom preparing for examinations through private tutoring. To avoid this situation, items on examinations must be a valid assessment of the curriculum’s learning expectations and encourage high-quality learning across a range of competencies.

Most OECD countries are taking measures to address the negative impact that examination pressure can have on student well-being, attitudes and approaches to learning. For example, Korea has introduced a test-free semester system in lower secondary education with activities such as career development and physical education to develop students’ life skills and reduce stress (OECD, 2016[6]).

Figure 2.2. National examinations and assessments in public school in OECD countries

Note: Number of subjects covered in the assessment framework (subjects may be tested on a rotation basis).

Data for the national examinations and assessments in Lithuania are drawn from authors’ considerations based on OECD (2017[7]), Education in Lithuania, Reviews of National Policies for Education, OECD Publishing, Paris. https://doi.org/10.1787/9789264281486-en.

Source: OECD (2015[8]), Education at a Glance 2015: OECD Indicators, https://doi.org/10.1787/eag-2015-en.

National assessments

National assessments provide reliable information on student learning with no consequences for student progression. Across the OECD, the vast majority of countries (30) have national assessments to provide reliable data on student learning outcomes, comparative across different groups of students and over time (see Classroom assessment). The main purpose of a national assessment is system monitoring and, for this reason, they provide essential information for system evaluation (see Chapter 5).

Countries might also use national assessments for more explicit improvement purposes, such as to ensure that students are meeting national achievement standards and identify learning gaps needing further support. In these cases, providing detailed feedback to teachers and schools on common problems and effective responses is critical.

Many OECD countries also use national assessments for school accountability purposes, though there is considerable variation in how much weight is given to the data. This is because student learning is influenced by a wide range of factors beyond a school or teacher’s influence – such as their prior learning, motivation, ability and family background (OECD, 2013[1]).

National assessment agencies

Developing high-quality national examinations and assessments requires a range of assessment expertise in fields such as psychometrics and statistics. Many OECD countries have created government agencies for examinations and assessments where this expertise is concentrated. Creating a separate organisation with stable funding and adequate resources also helps to ensure independence and integrity, which is especially important for high-stakes national examinations.

Student assessment in Serbia

Serbia is in the process of introducing major changes to student assessment both at the classroom level and nationally. Initiated in 2018, the roll-out of the new competency-based curriculum and subject- and grade-specific learning standards will help teachers better understand student learning levels and use that information in designing assessment and lesson plans. The introduction of a new centralised examination at the end of upper secondary education to certify completion of schooling and inform selection into tertiary education will also help improve the reliability and fairness of the exam at this important transition point. However, many gaps are yet to be addressed to ensure the success of these reforms. In particular, teachers’ assessment capacity remains relatively weak, notably for formative assessment, and many elements of the new end-of-upper-secondary examination reform are yet to be determined.

Table 2.1. Student assessment in Serbia: Current practices and expected reforms
Type of student assessment	Guidelines documents	Standards	Body responsible	Process	Use
School-based summative assessment	Teaching and learning plans and programmes Rulebook on student assessment (for basic and upper secondary) Competency-based assessment in vocational education	Standards of student achievement Interdisciplinary competencies in LFES	Teachers	Grade 1: no summative assessment Grades 2-8: at least 8 numerical marks for each subject each year, and at least 4 marks for subjects taught once a week Grades 9-12: at least 6 numerical marks in each subject each year and at least 4 marks for subjects taught once a week	Grades 6-8: selection into upper secondary education Grades 9-12: selection into tertiary education
School-based formative assessment		Standards of student achievement Interdisciplinary competencies in LFES	Teachers	Grade 1: descriptive (qualitative) marks Initial (diagnostic) test	Initial test used for teaching planning
Central examination: end of basic education (Grade 8)	Rulebook on the final exam programme in basic education	Standards of student achievement	IEQE (Exam Centre) and MoESTD	Compulsory exam in three domains: Serbian language (or mother tongue), mathematics, and natural and social sciences (five subjects combined)	Requisite for basic education completion certificate Exam scores considered for selection into upper secondary
School-level examination: end of upper secondary education	Rulebook on the content and method of the Matura exam in gymnasium VET schools: internal school acts based on the law of secondary school		School	General test in Serbian language (or mother tongue) and either mathematics or foreign language, plus an individual essay in any subject	Exam is requisite for secondary education completion certificate
Note: IEQE: Institute for Education Quality and Evaluation; LFES: Law on the Foundations of Educational System; MoESTD: Ministry of Education, Science and Technological Development; VET: Vocational education and training.

Overall objectives and policy framework

The curriculum in most grades is not aligned with the learning standards

The Institute for Education Quality and Evaluation (IEQE) introduced learning standards for the end of lower secondary education (Grade 8) in 2010, followed by end of primary (Grade 4) in 2011 and end of upper secondary (Grade 12) in 2013. Primary and lower secondary are considered to be part of the same cycle in Serbia and commonly referred to as “primary education”. These standards were Serbia’s first attempt at introducing a competency-based approach to teaching and learning. Their design compares positively to standards in OECD countries such as New Zealand which have end-of-learning-cycle standards. Standards are designed for most subjects and include three performance levels for each competency: basic, intermediate and advanced. The end-of-lower-secondary standards are the reference document for the end-of-basic-education national examination at Grade 8.

However, a decade after the introduction of these learning standards, teaching practices in Serbia remain predominantly knowledge-based. While learning standards based on a new competency-based approach were fully introduced by 2013, the curriculum remained mostly and narrowly knowledge-based until 2017 when Serbia started rolling out the new competency-based curriculum aligned with learning standards. Thus, teachers tend to place little emphasis on how knowledge is applied or on higher-order cognitive processes. Moreover, there are no guidelines describing students’ learning progression across a cycle, which is important to help teachers identify where individual students are in their learning and determine appropriate next steps. Without such guidance, teachers in Serbia are not able to use the learning standards in their classroom practices.

The curriculum framework is highly prescriptive, leaving little space for teachers to adapt teaching to the needs of their students

The curriculum in Serbia is relatively prescriptive compared to practices in OECD countries (OECD, 2013[9]). The curriculum framework in Serbia includes teaching plans that define the list of subjects (compulsory and elective) and pedagogical activities for each grade (e.g. regular, project-based, remedial, optional and outdoors lessons), as well as the yearly and weekly number of lessons per grade, subject and pedagogical activity. This leaves little space for teachers to adapt their teaching practices to the specific learning needs of students. Teachers met by the review team said that they felt the curriculum was too dense, leaving them limited time to review notions or competencies not achieved by some of their students. This curriculum overload is exacerbated by the relatively limited instruction time in Serbia compared to European countries (a 100 hours fewer in primary and 30 hours in lower secondary) (European Commission/EACEA/Eurydice, 2018[10]).

Serbia is introducing a new competency-based curriculum at all education levels

The ongoing curriculum reform addresses some of the issues stated above. The Institute for Improvement of Education (IIE) has developed a new competency-based curriculum and the roll-out started in 2018 with Grades 1, 5 and 9 (starting grade of each cycle). The curriculum includes learning outcomes for each grade which should help teachers better understand how their students may reach the end of cycle standards of learning. The reform also introduces transversal competencies such as problem solving and digital skills (see Box 2.3). The new curriculum includes didactic and methodological recommendations about student assessment. It distinguishes between formative and summative assessment, and underscores the desirability for teachers to provide continuous feedback to students on their progress, based on an initial diagnostic evaluation of the student’s level.

The IIE is developing a training programme for teachers on the new curriculum with support from the European Union and the United Nations Children’s Fund (UNICEF). The programme includes a three-day seminar to familiarise teachers with the new materials and approach to learning. The IIE also developed an e-learning platform with materials on the curriculum such as examples of lesson plans, activities and assessments. The IIE hopes to reach approximately 40 000 education professionals in primary and secondary education through this training by the end of 2019 (European Commission, 2016[11]). However, the IIE does not have the human capacity to provide more continuous support, such as mentoring teachers or monitoring implementation of the curriculum in schools. While some school advisors in the Regional School Authorities (RSAs) try to support teachers in implementing the new curriculum, they do not currently have the resources or a clear mandate to provide the needed assistance.

Box 2.3. Interdisciplinary (transversal) competencies in Serbia

According to Article 12 of the Law on the Foundation of Education System (LFES), interdisciplinary (transversal) competencies are a combination of knowledge, skills and attitudes relevant to different real-life contexts that require their functional application. These are also an integral part of the Rulebook on Competence Standards in General Subjects for the End of Secondary Education (2013). Interdisciplinary competencies are based on key competencies for lifelong learning, developed through the teaching of all subjects, applicable in different situations and contexts in solving various problems and tasks. They are required by all students for personal achievement and development, for inclusion in social trends and employment, and form the basis of lifelong learning.

Interdisciplinary competencies for the end of compulsory basic education are: 1) competency for learning; 2) responsible participation in a democratic society; 3) aesthetic competency; 4) communication; 5) responsible attitude towards the environment; 6) responsible attitude towards health; 7) entrepreneurship and orientation towards entrepreneurship; 8) working with data and information; 9) problem solving; 10) co-operation; 11) digital competency.

Interdisciplinary competencies for the end of secondary education are: 1) competencies for lifelong learning; 2) communication; 3) working with data and information; 4) digital competency; 5) problem solving; 6) co-operation; 7) responsible participation in a democratic society; 8) responsible attitude towards health; 9) responsible attitude towards the environment; 10) aesthetic competency; 11) entrepreneurship and entrepreneurial competency.

Sources: MoESTD (2019[12]), Zakon o Osnovama Sistema Obrazovanja i Vaspitanja [Law on the Foundations of the Education System], Ministry of Education, Science and Technological Development, Belgrade; MoESTD (2013[13]), O Opštim Standardima Postignuća za Kraj Opšteg Srednjeg Obrazovanja i Srednjeg Stručnog Obrazovanja u Delu Opšteobrazovnih Predmeta [Rulebook on Competence Standards in General Subjects for the End of Secondary Education], Ministry of Education, Science and Technological Development, Belgrade.

Classroom assessment

Teachers must assign at least 8 numerical marks per year in Grades 2-8 and at least 6 in upper secondary education

Students start receiving numerical marks as early as Grade 2 in Serbia. In basic education (Grades 2 to 8), teachers of subjects taught more than once per week must assign every student at least 4 numerical marks per semester for each subject. These marks are considered when computing a student’s final mark at the end of each semester and year. In upper secondary education (Grades 9 to 12), teachers of subjects taught more than once per week must assign at least 3 numerical marks in each subject (see Table 2.1). The number of prescribed marks that teachers must give is relatively high compared to OECD countries (OECD, 2013[1]).

Students are marked using a five-point scale with five points being the highest mark. This practice, common among former socialist countries, is becoming rare among OECD countries that tend to have longer marking scales to allow for better differentiation of marks and a more precise description of performance levels (source). Students who receive a grade point average (GPA) of one out of five have to repeat the grade but this rarely happens as teachers rarely give a mark below three. In 2017, only 1% of Grade 8 students had a GPA of 2 while virtually no students had a GPA of 1. About two-fifths of Grade 8 students had a GPA of 5 (“excellent”), 35% achieved “very good” and 21% achieved “good” (IEQE, 2017[14]).

Only descriptive feedback is used in Grade 1

In the first grade of primary, numerical marks have not been allowed since 2004. Students receive qualitative (descriptive) feedback only. However, teachers are given little support on how to implement this practice. Until 2018, there were no learning outcomes and performance levels for Grade 1 teachers could refer to in assessing the progress of their students. Moreover, until 2018, the school report card left no space for teachers to provide detailed feedback to students, which could make it seem less important. The report card has been changed in 2018 to allow teachers some space for descriptive feedback but this space is limited for each subject and does not allow for a detailed description of students’ reached learning levels. Changes were also made to simplify information given to parents on the progress of their children. Still, it has been reported to the OECD review team that some teachers translate descriptors into marks under the pressure of parents, for example by using descriptive symbols that suggest a numerical equivalency.

National regulations define the criteria of classroom assessment and the type of assessments

National policies provide a framework that is relatively prescriptive in terms of student classroom practices. Indeed, the minimum number of assessments as well as the type of assessments and criteria to be used are defined nationally. For example, national policies state that the types of assessment a teacher can use include, among other things, oral and written tests, presentations and reports, group projects, peer or self-assessment and student portfolio.

A mark for student participation is included in the GPA

The rulebooks on classroom assessment mandate that the GPA of students from Grade 6 to 12 should take into account the marks of both students’ academic performance and their class participation and engagement. This practice encourages using the GPA as a means to punish misbehaviour or students who do not engage in the classroom. Because of this, most OECD countries separate marking of student assessment from marking of classroom behaviour and participation (OECD, 2012[15]).

Classroom marks contribute to selection into upper secondary

Students’ GPA in Grades 6, 7 and 8 is considered alongside scores in the central examination at the end of primary for selection into upper secondary school. For students applying to general or vocational secondary schools (3 or 4 years), the “entrance score” consists of the weighted sum of the average school marks for Grades 6, 7 and 8 (60%) and the score in the end-of-primary examination (40%). At least 50 points is needed to enter a gymnasium or a 4-year vocational secondary school. Students who do not reach this threshold can enrol in a three-year vocational school.

Improving teachers’ assessment literacy is a national priority for professional development in Serbia

Teachers’ assessment literacy is relatively weak in Serbia. In 2017, external school evaluations’ results showed that the use of assessment to inform learning (formative assessment) and adapting teaching to students need is weak or very weak in almost half of basic education schools and two-thirds of upper secondary schools (Figure 2.3). Teachers are aware that they need training on assessment. One-third of teachers surveyed by the IIE in 2017 reported needing professional development on student assessment (IIE, 2017[16]).

In response, the Ministry of Education, Science and Technological Development (hereafter referred to as the ministry) identified student assessment as a priority for teachers’ professional development in Serbia for 2017-20; thus, all Serbian teachers are required to follow some training in this area over the three years. Student assessment is also addressed during the three-day training seminar being implemented to familiarise teachers with the new curriculum and approaches to learning. However, the take-up rate for professional development remains low, mostly because of financial constraints and the format of training provided (see Chapter 3). One-off seminars held outside of schools comprise the main mode of professional development in Serbia; they are not adequate to help teachers develop more effective assessment practices.

Figure 2.3. Results of external school evaluation for the school quality area “Teaching and Learning”, 2017

Note: Ranked in descendant order of share of schools that scored weak by this indicator.

Source: IEQE (2017[17]), Izveštaј o spoљašњem vrednovaњu kvaliteta rada obrazovno-vaspitnih ustanova u školskoј 2016/2017 [Report on External Evaluation of the Quality of Work of Educational Institutions in the 2016/2017 School Year], Institute for Education Quality and Evaluation.

Teachers are required to conduct a diagnostic test at the start of each school year

All teachers in Serbia must prepare and conduct a diagnostic test (called “initial test” in Serbia) to assess their students’ acquired levels of achievement in particular subjects and topics at the start of the school year. The results are supposed to help teachers develop lesson plans adapted to the learning needs of their students. In the schools that the OECD visited, teachers and school managers were clearly aware of this requirement and reported that they were conducting this initial test yearly.

However, there are questions about the quality of this practice. Despite being a requirement, teachers receive very little guidance on how to design diagnostic tests and use the results to inform learning. There are, for instance, no national guidelines or manuals on how to develop an effective diagnostic test. Since 2010 the Exam Centre, an agency within the IEQE, has occasionally developed templates (e.g. examples of tests and marking schemes in mathematics and language) for some Grades (5, 7, 9 and 12). This was also done for lower Grades (1 to 4) from 2007 to 2012. However, as this is not part of the Exam Centre’s regular programme of work and because there is no dedicated budget, these templates have not been developed systematically for all grades. The Exam Centre intends to conduct a system-level analysis of the results in initial tests at some point in the future.

National examinations

Students in Serbia take two high-stake examinations, one at the end of compulsory education (Grade 8) and one at the end of upper secondary education. The examination at the end of secondary education is currently the focus of a major reform.

A central examination certifies completion of compulsory education in Grade 8

All students completing Grade 8 of compulsory education take a compulsory and centrally designed final examination called the end-of-basic-education exam. In addition to assessing student achievement, this exam has two main purposes. It is a requisite to obtain the certificate of completion of compulsory education. The results are also used to inform the allocation of students into upper secondary schools, together with a student’s secondary school application “wish list”. Introduced for the first time in 2011, this exam marked a significant achievement, both in terms of establishing more reliable data on student learning outcomes in Serbia and of providing a more objective, fair and trusted basis for selection into secondary school.

The exam includes a test in mathematics, in Serbian (or a recognised minority) language and a “combined” test of social and natural sciences that includes five subjects (biology, chemistry, geography, history and physics). The combined test was introduced in 2014 in order to motivate students to apply themselves in more subjects during the final years of compulsory education. The exam can be adapted to better accommodate the needs of students with disabilities, for example by printing it in Braille. Students from national minorities educated in their mother tongue can opt to take the exam in their language of instruction (a range of eight languages). In the final exam of June 2018, 6.2% of the students took the test in a language other than Serbian. Offering the option to test in other languages is a legal requirement, even though some of the languages are taken by very few students (e.g. only eight students in 2017 took the test in Bulgarian). In 2017, a total of 63 111 students took the exam in Serbian at the June session (IEQE, 2017[14]).

Table 2.2. Examinations in Serbia
	End-of-basic-education examination (small Matura)	School-based examination at the end of upper secondary (until 2020 for gymnasium and four-year VET schools)	State Matura examination (planned for 2020)
Components	Mathematics; Serbian (or a recognised minority) language; combined test (biology, geography, chemistry, history and physics)	Final exam - Gymnasium Written tests: Serbian or mother tongue language, mathematics or foreign language Project assignment Final exam - Vocational (four-year VET schools): Written test (Serbian or mother tongue language and an elected subject) Practical assignment (including oral exam) Final exam - Vocational (three-year VET schools): Practical assignment (including oral exam)	General Matura Two compulsory tests (Serbian language and literature or a recognised minority language and mathematics, for all students with over two years of mathematics in upper secondary) At least 1 elective test from a list of 13 electives subjects (biology, chemistry, history, foreign language, etc.) Vocational (VET) Matura (four-year VET schools): Two compulsory general tests (Serbian language and literature or a recognised minority language and mathematics, for all students with over two years of mathematics in upper secondary)* One compulsory vocational test: Professional test Art Matura Two compulsory tests (Serbian language and literature or a recognised minority language and mathematics, for all students with over two years of mathematics in upper secondary) Artistic test
Eligibility	Compulsory for students in Grade 8	Compulsory for students in Grade 12 (including general education and 4-year VET secondary schools) and in Grade 11 (3-year VET secondary schools)	Compulsory for students in Grade 12 (including 4-year VET secondary schools)
Item development	IEQE Exam Centre	Schools	IEQE Exam Centre and IIE Centre for Vocational Education and Adult Education for the professional test in VET Matura
Question format	Multiple-choice and open-ended questions	Multiple-choice and open-ended questions	Not yet determined
Marking	Maximum score on final exam is 40 (13 for mathematics and 13 Serbian language test, and 14 for combined test) Final examination score is added to the GPA for Grades 6, 7 and 8	Not standardised. Each school determines its marking system	Not yet determined
Marking	Teachers in the school	Teachers in the school	Not yet determined
Purpose	Certification and selection into upper secondary education	Certification of completion of upper secondary education	Certification of completion of upper secondary education and selection into tertiary education
Reporting	Accessible online on the ministry’s official website with student ID and password	The final results are not publicly available	Not yet determined
* In addition to compulsory subjects, students may also take the Matura in elective subjects.
Source: MoESTD (2014[18]), Pravilnik o organizaciji i sprovođenju ispita [Rulebook on the Organisation and Conduct of Examinations], Ministry of Education, Science and Technological Development, Belgrade.

The IEQE Exam Centre is responsible for test design

The Exam Centre is responsible for the design of the test and sets the assessment framework (specification grid), which defines the competencies to be measured and the type of questions to be included in the tests (World Bank, 2012[19]). The assessment framework is well-aligned with the end of cycle learning standards and levels of student achievement. Subject-specific working groups, which mainly include teachers from primary and secondary schools, university teachers and staff from the IEQE and IIE are responsible for writing the test items under the guidance and supervision of the exam centre’s professional staff.

The end-of-basic-education exam includes multiple-choice and open-ended items of varying levels of difficulty

Each of the three tests includes 20 items distributed over three levels of difficulty: basic, intermediate and advanced. Each test has approximately the same balance between the different levels of difficulty. Most items are of the basic levels (around 9-10 items in each of the three tests). Moreover, the tests include a mix of multiple-choice questions and close-ended questions. In the Serbian language test, the majority of items are multiple-choice while in the mathematics test, open-ended items prevail. The “combined” test includes multiple choice, matching and open-ended items with short answers.

The end-of-basic-education exam is marked in schools but a rigorous supervision system is in place

Students take the final exam in their own schools in June, and exceptionally in August if they scored one or lower in at least one subject or were not able to take the exam in June due to ill health or another valid reason. The IEQE Exam Centre provides schools with guidance on how to prepare their students for the exam. For example, the Exam Centre organises information seminars for schools to explain the examination procedure. Additionally, a mock examination designed by the Exam Centre is carried out in April to familiarise students with the exam and to test out the procedures of administering it.

The tests are marked by teachers from the school organised in scoring commissions. The ministry appoints external supervisors (teachers from other schools) to monitor the conduct of exam classrooms and the work of test scoring commissions. Supervisors submit an electronic report to the Republic Commission, composed of representatives from national institutions such as the ministry and the IEQE, and to the RSA on the day of the exam. A randomly selected sample of tests is also reviewed at the district level (the district commission). Nationally, the Exam Centre selects a sample of schools and assesses the extent to which their school commissions followed prescribed procedures in administering and scoring the exam. The results of this analysis are presented to the ministry by request in the “Report on the results of the quality control of test scoring”, which has not been made public.

Results of the end-of-basic-education exam are used to place students in upper secondary programmes

Results from the end-of-basic-education exam are combined with the student’s GPA from lower secondary education (Grades 6, 7 and 8) to constitute the final score. The GPA accounts for 60% of this score while the mark from the exam itself accounts for 40%. The final score is then used along with students’ wishes to define their placement in upper secondary schools and programmes. Students applying to some specialised art schools might be required to take a school-specific entrance test in addition. However, these cases are relatively rare.

Final results at the level of individual schools are publicly available on the ministry’s website. Aggregated results at the municipal, district and national levels are published in the IIE yearly reports. The results of individual students are not publicly available but can be accessed by pupils themselves by entering their personal six-figure code in the government’s dedicated website or through their school information boards for enrolment in upper secondary education.

A school-based examination is used to certify completion of upper secondary education

At the moment, all students completing upper secondary education in academic, art or vocational schools take a school-based Matura exam which certifies completion of secondary education. For gymnasium, this school-based exam comprises two parts: a written exam and a graduation paper. The written exam includes a test and essay in Serbian language (or mother tongue) and literature, and a test in either mathematics or a foreign language, depending on the stream chosen by the student (social or natural sciences). The student can write the graduation paper on any subject they choose (see Table 2.2).

The exam is designed, administered and marked by schools with no oversight from the Exam Centre or the ministry. The lack of standardisation has resulted in great differences across schools in terms of content, implementation procedures and assessment criteria (Matura Working Group, 2017[20]). While students have to pass the school-based exam to qualify for tertiary education, the school-based exam does not serve the purpose of selection into tertiary education, which is determined by tertiary institutions.

Tertiary education institutions administer their own entrance examinations

Students who wish to enter tertiary education must apply to each faculty and/or institution separately and take the entrance examinations for the desired programmes. This system raises concerns of transparency, quality and equity. To increase chances of enrolment, students typically take admission exams at two or more tertiary institutions, which tends to be costly in particular for students living in remote or rural areas who need to travel to the university to take the test. Additionally, private tutoring has become a widespread and commonly accepted practice in preparation of admission exams in Serbia which adds to the cost burden from parents and students. These tutoring programmes are commonly offered by the tertiary institution itself which creates issues about the transparency and fairness of the entrance exam system (OECD, 2012[21]).

For these reasons, a major reform is being planned to replace the current system with a new, central and standardised final exam, to improve fairness and transparency.

A new central Matura exam is planned to certify completion of upper secondary school and determine selection into tertiary education

For the first time, Serbia is introducing a national examination at the end of upper secondary education called the State Matura that will serve both to certify completion of upper secondary education and to select and orient students into tertiary education programmes (see Table 2.2). This is consistent with trends in neighbouring countries and the OECD, where a majority of countries have centralised exams at the end of upper secondary education that serve – either or (increasingly) both – a certification and selection role (OECD, 2017[22]). This major reform, currently developed with financial and technical support from the European Union, is scheduled to be rolled out in 2020. This is Serbia’s second attempt at introducing a Matura exam. Earlier plans of a roll-out in the 2014/15 school year stalled due to a lack of consensus between stakeholders and funding.

In 2018, the Ministry of Education developed a concept note which spells out clearly the reasons for introducing the new Matura exam and the principles underpinning the proposed policies (see Box 2.4). The reform has two main goals. It is intended to strengthen the reliability of the final grades of upper secondary education by introducing a common, national examination that is standardised in design and delivery. This improved robustness will increase trust in the results and value in the certification they confer. Additionally, the reform seeks to improve the fairness and integrity of the process for selection into tertiary institutions, by creating a new centralised admission system that will process student preferences in relation to their results on the national examination and high school GPA as well as programme entry criteria. Importantly, universities will no longer be able to set their own admissions tests (except in a few specialised areas not covered by the Matura). The concept note also defines the lists of subjects to be tested in both general and vocational education programmes.

However, there are several areas in which plans for Matura remain underdeveloped and require additional consideration. The ministry has still to decide on the design and process of selection into tertiary education institution. Details of the examination model such as the items to be used and the marking scale are still to be determined. Finally, at the time of the review, plans for how the Matura will be administered were still being developed. With so many elements still to be decided, the timeline for implementation – which aims at full roll-out by 2021 – appears unrealistic.

Box 2.4. Serbia’s new State Matura

Plans for a new central examination at the end of secondary education (Matura) have been in the making for several years. The European Union provided funding and technical support to the project in its early steps through an Instrument for Pre-Accession Assistance (IPA) project. In 2016, the ministry formed a working group that developed the concept of the new final exam at the end of secondary education. The working group included members of the Ministry of Education, Science and Technological Development, the IEQE, the National Education Council (NEC) and other governmental agencies, as well as representatives of higher education institutions and secondary school communities. The proposal of the working group was adopted by the ministry in 2017.

Under the new system, all students who complete general, vocational or artistic upper secondary education will take a compulsory, centrally designed, final exam. This exam will have a dual function: it will be a prerequisite to obtaining a secondary school completion certificate and scores will be incorporated in the tertiary education admissions system. The cut-off mark for passing the examination has yet to be defined.

There will be three different exams: the General Matura; the Vocational (VET) Matura; and the Art Matura. To obtain the certificate for their upper secondary studies, gymnasium students must take the General Matura, students in four-year professional schools the VET Matura, and students in art schools the Art Matura. Nevertheless, students in vocational and art schools are also allowed to also take the General Matura if they want to apply to university programmes that require a General Matura for admission.

The Matura exam will include a minimum of three tests: Serbian language or mother tongue (compulsory for all students), mathematics (compulsory for students with more than two years of mathematics in upper secondary) and at least one test in an elective subject chosen from a list of subjects. Students taking the General Matura will choose electives from general subjects such as biology, chemistry, geography, history, physics, eight languages of national minorities, and Serbian as a non-mother tongue. In addition to Serbian and mathematics, those taking the VET Matura have to sit a third compulsory exam that assesses professional knowledge and practical work in a VET subject. Similarly, the Art Matura will assess students’ in art subjects. Students can take as many elective subject tests as they want, according to their plans for enrolment in higher education. Tests will be designed to measure achievement of learning standards at the end of secondary education. While learning standards exist for most subjects, they are yet to be developed for some elective subjects such as information science, music and psychology.

Results in the Matura exam and marks in upper secondary school classroom assessments (GPA) will be used alongside students’ choices to determine admission into tertiary education. Out of a maximum of 100 points, students’ GPA in Grades 10 to 12 will weigh 40 points and Matura results 60 points. Contrary to current practices, universities will no longer be allowed to set additional admission tests on subjects already covered by Matura. However, universities will be able to choose which Matura elective subjects to consider for admission into specific study programmes and the weight given to each of these subjects in the student’s final application score. The universities will also be allowed to set additional specialised admission tests to cover specific subject areas not covered in the list of subjects included in the Matura, such as art, music, architecture, etc. Higher education institutions (HEIs) must publish information on their study programmes enrolment criteria two years in advance, to allow students and their schools enough time to select the exams they will take to prepare for them.

Sources: Matura Working Group (2017[20]), Opšta, Stručna i Umetnička Matura i Završni Ispit u Srednjem Stručnom Obrazovanju [General, Professional and Art Matura and Final Examination in Secondary Education], Ministry of Education, Science and Technological Development, Belgrade; European Commission (2015[23]), Instrument for Pre-accession Assistance (IPA II) 2014-2020: Serbia European Integration Facility, https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/pdf/serbia/ipa/2015/pf_01_european_integration_facility.pdf (accessed on 17 October 2019).

The IEQE Exam Centre is responsible for designing and administering both the end-of-basic-education exam and the new Matura exam

The IEQE Exam Centre’s responsibilities have increased significantly over the past couple of years. From being responsible for only one examination – the end-of-basic-education exam – the Exam Centre is now responsible for two examinations and several large-scale student assessments. The centre is the main agency responsible for designing and administering the future Matura examination and the new national assessment of learning outcomes that will be introduced in 2020 (see Chapter 5). The Exam Centre has also recently acquired responsibility for running the international student assessments that Serbia participates in – the Programme for International Student Assessment (PISA), Progress in International Reading Literacy Study (PIRLS) and Trends in International Mathematics and Science Study (TIMSS) – previously managed by the University of Belgrade (PISA) and the Independent Institute for Educational Research (TIMSS).

The financial, human and material capacities of the centre are, at their current level, insufficient to carry out all these tasks. With only 15 technical staff, the IEQE Exam Centre does not have the capacity to carry out all its planned activities, most of which have overlapping timelines. Moreover, the centre is missing some key technical profiles for designing the assessments. For example, it does not have psychometricians or statisticians on staff and relies on staff from other IEQE departments who have these profiles. The Exam Centre lacks the funds needed to carry these activities. Both attempts at introducing a national assessment and a Matura exam in the past have failed due to lack of sustainable funding. The centre does not have the adequate resources and capacity to administer a modern national examination such as the Matura. It has, for example, limited capacity for printing, packing and storing large examination papers in a safe manner.

Policy issues

As Serbia seeks to change the culture of education to become more learner-centred, it will need to ensure that student assessment practices, both in schools and nationally, reinforce this message. To do so, teachers will require much more support on how to assess student learning in relation to the new competency-based curriculum. The extensive use of simple summative tests and marks must give way to more meaningful assessments and more constructive feedback that provides each individual student with an understanding of where they are in their learning and how they can advance.

At the national level, small modifications to the end-of-basic-education exam – which is a strength of the Serbian assessment system – would further promote this culture change. This review proposes adjustments to the exam’s design and administration that would help focus attention on competencies valued in the curriculum and also give direction in building greater public trust in the results. This will be important for the successful implementation of the new Matura exam at the end of secondary education. The latter is one of the most ambitious reforms on the education agenda in Serbia. If it succeeds, the reform will improve the integrity and equity of student selection into tertiary education and, through its rigour and reliability, create a positive backwash on learning during upper secondary schooling. This review provides recommendations on how this can be achieved.

Policy issue 2.1. Ensuring a better balance between formative and summative purposes in school-based assessment

In Serbia, there is a marked imbalance between school-based assessment for learning (formative assessment) and assessment of learning (summative assessment). On the one hand, summative assessments are frequently practised because it is compulsory for teachers to assign a minimum number of numerical marks to each student every year. School-based summative assessments also have high stakes for students, with a student’s cumulative GPA significantly influencing their options at the upper secondary and tertiary levels. By contrast, formative assessment is underdeveloped, largely because summative assessment weighs so heavily, but also because the formative purposes of assessment are poorly understood, valued and practised. The only formative assessment practice mandated by law is the initial diagnostic test that teachers must administer at the beginning of the school year. Even in this case, it is unclear whether teachers are using the results from the initial test to adapt their instructional practices and focus on individual student needs. This imbalance has negative consequences for student learning. It generates strong pressure for students and parents to focus on getting good marks rather than on authentic learning. Some teachers and schools respond to this pressure by inflating student grades.

The ministry, the IEQE and the IIE need to play a more active role in encouraging teachers and students – and parents and society at large – to focus more on learning and less on summative marks. The ministry should revise the assessment framework to redefine expectations about how classroom assessment ought to be practised, notably by extending the marking scheme and linking marks to performance levels. The IEQE and the IIE need to work together to provide teachers with adequate support to develop their assessment literacy, in particular in formative assessment.

Recommendation 2.1.1. Revise the student assessment framework to encourage a shift in focus from marks to learning

Serbia needs to revise its national framework for student assessment to ensure that teachers are given the space to use assessment in a more formative manner. Current policies for classroom assessment are relatively prescriptive and leave little space for teachers to implement formative assessment. The high frequency of summative numerical marks and the limited marking scale constrain teachers’ capacity to use assessment to inform teaching and learning continuously. While the ministry has introduced a new competency-based curriculum and defined learning standards and levels of achievement for each cycle, teachers have received little guidance on how to use this new curriculum framework to assess their students’ learning achievements. All teachers in the system need to understand what the new curriculum and standards mean in concrete terms in order to use it in their everyday assessment practices.

Define clearly the core principles of student assessment in Serbia

Introducing a new curriculum provides an opportunity to give renewed impetus to long-standing efforts to change the focus of student assessment in Serbia. The new curriculum resources already provide valuable orientation towards a more balanced approach – notably in terms of highlighting the central importance of formative feedback. This review also suggests corresponding amendments to rulebooks and legislation. However, transforming classroom assessment culture will require more direct communication efforts to help teachers and society understand the rationale for change and what is at stake. OECD countries use a variety of ways to communicate the fundamental purpose and principles of assessment, such as position papers and national guidelines. In Canada, the Principles for Fair Student Assessment for Education in Canada provides a good example of how a clear normative reference document can serve as a guide for both teachers and for those responsible for developing policies, handbooks and tests (see Box 2.5).

Box 2.5. The Principles for Fair Student Assessment for Education in Canada

The Principles for Fair Student Assessment for Education in Canada were developed by a working group and guided by a joint advisory committee representing the School Board Association and provincial and territorial ministries and departments of education. They came as a response to observed inadequate assessment practices in Canadian classrooms and aim to build consensus on what constitutes and guides a fair assessment of students. The principles are designed to guide the design and implementation of assessment in Canadian schools and ensure the fairness of practices. The text acts both as a set of parameters and a handbook for assessment. The principles are organised in two parts: the first part lists principles for classroom-based assessments in elementary and secondary schools; the second part focuses on standardised assessments developed externally (i.e. by departments of education, local school jurisdictions and others).

The list below summarises the principle and the following seven guidelines for “Developing and choosing methods of assessment” by teachers:

Assessment methods should be appropriate for and compatible with the purpose and context of the assessment

1. 1. Assessment methods should be developed or chosen so that inferences drawn about the knowledge, skills, attitudes and behaviours possessed by each student are valid and not open to misinterpretation.

2. 2. Assessment methods should be clearly related to the goals and objectives of instruction, and be compatible with the instructional approaches used.

3. 3. When developing or choosing assessment methods, consideration should be given to the consequences of the decisions to be made in light of the obtained information.

4. 4. More than one assessment method should be used to ensure comprehensive and consistent indications of student performance.

5. 5. Assessment methods should be suited to the backgrounds and prior experiences of students.

6. 6. Content and language that would generally be viewed as sensitive, sexist or offensive should be avoided.

7. 7. Assessment instruments translated into a second language or transferred from another context or location should be accompanied by evidence that inferences

Source: Rogers, W. (1993[24]), “Principles for fair student assessment practices for education in Canada”, https://doi.org/10.1177/082957358500900111.

How such principles are developed is as important as the final document. The training programme on the new curriculum provides a good forum to collect feedback from teachers on core concerns and misunderstandings that need to be addressed. Interviews with teachers, principals and parents conducted as part of this review suggest the value of including in the document:

• A statement about what an effective system of student classroom assessment looks like – including, as key messages, that student learning must be the central goal of the assessment practices, and that balance between formative and summative assessment is necessary to promote better student learning.

• A statement on the importance of formative assessment as a pedagogical approach, a description of different types of formative assessments (e.g. initial diagnostic test, frequent interactive checks of student understanding to identify learning needs and adapt teaching strategies, etc.) and an explanation of how results from formative assessments can be used to identify aspects of learning as it is developing in order to deepen and shape subsequent learning. This would provide a key reference for the formative assessment guidelines that this review recommends Serbia develop.

• A description of summative assessment as a pedagogical approach, of different summative assessments, and examples of how numerical marks can be linked to standards to ensure transparency in the evaluation criteria and provide substantive feedback to students.

• A statement that assessment is a core pedagogical competency that all teachers need to master and seek continuously to develop. It should also be stated that a teacher’s professional assessment judgement should be respected and that teachers should have the professional discretion to adapt central guidelines to their classroom context. This statement should also be echoed in the teacher standards which should define the assessment literacy expected of teachers at different career stages (see Chapter 3).

Extend the marking scale to allow for a more refined description of students’ abilities

To make marking more meaningful from an educational perspective, the ministry should consider replacing the traditional 1-5 scale used for reporting classroom outcomes with a longer common scale. Marking schemes vary across countries but most feature a greater number of categories than the five currently used in Serbia (European Commission, n.d.[25]). Examples include letter grades A-F with + and – qualifications and/or countries that use numerical grades ranging from 1-10 allowing for further discrimination, such as the example of Ontario (see Box 2.6). Having more available marks gives teachers more flexibility over how they report student results and relieves some of the pressure they might currently feel with so few marks from which to choose.

Box 2.6. Reporting scales in Ontario, Canada

In Ontario, Canada, a 6-point letter grade scale is used to report student achievement against provincial curriculum expectations in each subject or course in Grades 1 to 6 (see example below), and 6-point numeric scales are used for Grades 7 to 8 and Grades 9 to 12. Each point on the achievement scale is accompanied by a descriptor and aligns with a provincial standard level, which is the reporting scale used for province-wide student assessments. This information is included in student report cards to help parents and students understand students’ results.

Letter Grade	Achievement of the Provincial Curriculum Expectations
A- to A+	The student has demonstrated the required knowledge and skills with a high degree of effectiveness. Achievement surpasses the provincial standard (Level 4)
B- to B+	The student has demonstrated the required knowledge and skills with considerable effectiveness. Achievement meets the provincial standard (Level 3)
C- to C+	The student has demonstrated the required knowledge and skills with some effectiveness. Achievement approaches the provincial standards (Level 2)
D- to D+	The student has demonstrated the required knowledge and skills with limited effectiveness. Achievement falls much below the provincial standards (Level 1)
R	The students has not demonstrated the required knowledge and skills. Extensive remediation is required.
I	Insufficient evidence to assign a letter grade

A four-point rating scale is also used to report on students’ learning skills and work habits: E-excellent; G-good; S-satisfactory; and N-needs improvement.

Sources: The Star (2017[26]), “Report card, curriculum changes on the way in Ontario”, https://thestar.com/news/queenspark/2017/09/06/report-card-curriculum-changes-on-the-way-in-ontario.html (accessed on 23 May 2019).; London Region MISA PNC (2011[27]), Comment Framework: Progress Reports and Report Cards, http://misalondon.ca/PDF/a&e/Comment_Framework_Feb_2011.pdf (accessed on 20 June 2019); Ontario Ministry of Education (2010[28]), Growing Success: Assessment, Evaluation and Reporting in Ontario Schools, http://edu.gov.on.ca/eng/policyfunding/growSuccess.pdf (accessed on 23 May 2019).

Link marks to performance levels and require teachers to provide descriptive feedback to students

Improving the learning value of classroom assessment requires making a series of changes to the way teachers are providing feedback to students. Currently, teachers assign numerical marks without being required to provide the rationale for their decisions or to explain to students how they can improve their future performance. For example, the student report card leaves little space for teachers to include written feedback to students. Given the misalignment between the curriculum in basic education and the learning standards, teachers are not in the habit of using the learning standards to design their assessment and provide feedback to students on their achievement level. The ministry should require teachers to provide descriptive feedback to students. It should also require this feedback be based on the performance levels and learning outcomes defined by the new curriculum. To encourage this practice, the ministry should consider:

• Linking marks with achievement levels: Linking the marking scale with the performance levels in an explicit manner will help teachers and students better appropriate the learning standards. It will also help teachers provide a fairer assessment of student learning. For example, OECD countries such as Australia, Finland, France, Ireland and Israel use a combination of numerical marks and qualitative achievement levels in primary education (OECD, 2013[1]).

• Giving teachers examples of what performance at each level means: The IIE and IEQE Exam Centre should work together to collect examples of students’ work at each performance level and make these available on the teacher e-learning platform. Examples need to be accompanied by commentary explaining how the student demonstrates a given level of achievement. These examples are important in helping teachers to understand different levels of performance and provide more reliable judgements and feedback on student learning achievement. For example, Ireland has made available on the curriculum website (curriculumonline.ie) examples of student work illustrating the three levels of achievements (at expectation, ahead of expectation or yet to meet expectation) for each learning outcome included in the curriculum. These examples can be easily accessed by teachers by clicking on a given learning outcome (NCCA, n.d.[29]).

• Requiring teachers to record descriptive feedback for at least some assessments: Teachers are required in Serbia to record the marks of students’ assessments in their classroom records. They should be required for at least some marks to also include descriptive feedback and justification for the mark vis-à-vis performance levels. Such practice will help school advisors monitor the effective use of performance levels and provide teachers with feedback on how to improve. Teachers should, however, not be required to record such descriptive feedback for all marks as to avoid administrative burden.

Limit the frequency of summative numerical marks to create space for more formative dialogue

The ministry needs to revise the requirements related to summative assessment and marking to give teachers both more flexibility and more space to engage in formative practices. Specifically, the ministry should consider:

• Ending numerical marking in the first cycle of basic education (Grades 1 to 4): Instead, assessment judgements would take the form of descriptive feedback in relation to the learning standards. Using descriptive feedback will help focus the discussion between teachers, students and parents on whether students have mastered competencies and their learning progression rather than on grades. This reform will also help gradually change attitudes towards learning and instil in students at an early stage the value of learning for its own sake. Students will then carry with them this value into later stages of education. Ending numerical marks in early grades is a trend in most OECD countries (OECD, 2013[1]). Neighbouring Albania has also ended the use of numerical marks in the first three grades of primary, replacing them by qualitative descriptors aligned with the learning standards. The ministry will need to provide teachers and school principals with the tools to communicate this change to parents as past experience with similar initiatives in Serbia show they may be resistant to changes to the marking system.

• Reducing the number of compulsory summative marks that teachers have to assign in each subject and each semester in the second cycle of basic education (Grades 5 to 8): This will give teachers greater opportunity to engage in more formative practices and also more complex, longer-term assessment assignments such as individual or team projects. The latter are important under the new curriculum, which places a stronger emphasis on student agency and the application of knowledge. Serbia should also consider introducing an upper limit to the number of summative assessments a teacher may assign. Teachers in some of the schools visited by the review team reported assigning double the number of assessments required by law.

• Limit the number of grades where GPA contributes to the final scores of the certifications and admission systems: Currently, all marks from lower secondary education (Grades 6 to 8) are included in the end of lower secondary aggregate GPA which contributes 60% of the final entrance score to upper secondary education. Similarly, all marks from Grades 9 to 12 will contribute 40% of the final score for upper secondary certification (school marks and Matura exam). This creates a backwash effect with teachers, students and parents focusing primarily on marks rather than learning. Instead, Serbia might consider only including the 2 final grades of lower secondary (Grades 7 and 8) in the end-of-basic-education examination final mark. Similarly, the concept note for the new Matura might be revised to include only the GPAs of Grades 11 and 12. Only including the final years of a given cycle is also a more accurate reflection of students’ current development.

Recommendation 2.1.2. Strengthen the support provided to schools in conducting formative assessment

In a system such as the Serbian education system where summative assessment is so ingrained, a clear direction from the central government is needed to make sure that schools are effectively using assessment results to inform teaching and learning practices. With the new conceptual definitions and regulations described above, the practical support and guidance that schools and teachers receive from the centre about formative assessment need to be strengthened. The ministry will also need to give higher priority to developing training for teachers on how to use information from assessment to adapt their practices.

Strengthen the support provided by the IEQE Exam Centre in using diagnostic assessment (the initial test)

The initial test mandated at the beginning of the academic year is a positive feature of Serbia’s assessment framework. Teachers can use this diagnostic assessment to adapt their teaching to students’ needs, identify where students might need to go back over material from the previous year and develop a baseline for evaluating individual student progress throughout the year. However, teachers in Serbia need more support to make the most of this tool and truly embed this practice in their teaching. At the moment, while the Exam Centre has developed some templates for certain grades and subjects, most teachers are developing their own assessment tools without having clear guidance on how to design a test that effectively identifies students’ knowledge and skill with respect to national expectations. The Exam Centre should scale up and ensure the sustainability of this initiative by making it part of its regular programme of work that reaches all schools in the system and a larger number of grades and subjects. This new regular programme should include the following activities:

• Developing a standardised initial test for key transition grades in basic education: The IEQE Exam Centre should develop nationally standardised initial tests for key transition grades. These tests need to be aligned with the learning standards and mapped against the performance level to help teachers understand what students learnt in the previous cycle and their performance level at the start of the cycle. Having such a tool developed by the Exam Centre will help ensure the reliability of the initial test in key transition grades and provide teachers with an example of what they might develop themselves for other grades. The Exam Centre should consider a priority developing such a national initial test in Grade 1 (right after the mandatory preparatory year), to assess students’ readiness to learn, and Grade 5 (start of lower secondary education) for mathematics and Serbian (or language of instruction) to inform teaching and learning in the 2 cycles of basic education. Teachers should also receive training on how to mark these initial tests and use the results. As is the case in France, schools should be instructed to share the results of this initial test and discuss them with parents to make sure that parents feel engaged and have calibrated expectations for their child’s grades (see Box 2.7).

• Creating a pool of initial test items mapped against the learning standards: The IEQE’s website currently allows users to access old copies of initial test questions. However, the roll-out of the new curriculum presents an opportunity to broaden the pool of available test items and also make examples of student answers and feedback templates available for all grades and subjects. In particular, the Exam Centre should map student initial test answers against the learning standards’ performance levels. This could help teachers identify the level at which their students are performing at the start of the school year. The Exam Centre can first prioritise some core subjects, such as mathematics and Serbian, then gradually develop a comprehensive item bank. These materials should be added to the online e-learning platform currently being developed by the IIE and UNICEF and that this review suggests transforming into a one-stop-shop “school improvement hub” (see Chapter 4).

• Giving teachers space in the curriculum to adapt teaching based on the initial test results: Teachers should be given some flexibility to adapt the curriculum to the needs of their students based on the results of the initial test. The ministry might need to review the curriculum and pacing charts to make sure that teachers can go slower or faster on some competencies based on how their students perform in the initial tests.

Box 2.7. Diagnostic assessments in French primary schools

In France students who enter elementary school (cours préparatoire) are evaluated as part of a national diagnostic evaluation in French language and mathematics. The French language assessment focuses on basic literacy skills and knowledge, and evaluates a student’s ability to communicate orally, their phonological awareness and their knowledge of the alphabet. In mathematics, the assessment focuses on counting and reading numbers up to ten. The evaluation is a written assessment, with each student receiving a booklet in which they respond to the questions. Teachers also receive a booklet that provides detailed guidance on how to administer the assessment to the whole class. Student booklets are collected at the end and evaluated by a student’s classroom teacher.

The diagnostic assessment provides teachers with information so that they can adapt their teaching practices to students’ needs. It also provides school inspectors with information to understand the needs of the schools within their district, enabling them to provide relevant support to the teaching staff. The results are also shared with parents, and together parents and the student’s classroom teacher discuss how to best support the student’s learning and development needs. Results are also anonymised at the school level and shared with the relevant district to provide direction for future professional development training for teachers.

Source: Ministère de l’Éducation nationale et de la Jeunesse (2018[30]), Évaluations diagnostiques en CP [Diagnostic Evaluation in CP], http://eduscol.education.fr/cid119562/evaluation-diagnostique-en-cp.htm (accessed on 17 June 2019).http://eduscol.education.fr/cid119562/evaluation-diagnostique-en-cp.html.

Provide guidelines and tools to encourage teachers’ use of formative assessment

While the initial test is a good way to embed more formative student-centred approaches, teachers also need to be provided with a range of other resources if they are to enhance the learning value of assessment. This means guidance on how to integrate formative assessment approaches within their regular classroom activities – how to provide feedback in a way that is sensitive, constructive and motivational, but also how to engage students in setting and monitoring their own goals and in providing feedback on their peers (see Table 2.3 for an example of quality feedback from a tertiary education study). This also means introducing assessment requirements that encourage teachers to engage consistently in these approaches, such as the requirement that teachers work with students to develop a portfolio.

Student portfolios are selected collections of a student’s work that demonstrate evidence of a student’s progress in relation to learning goals. Using student portfolios helps students to see where they are in their learning by engaging them in their own assessment. The collection of student work contained in a portfolio can also act as a basis for more meaningful teacher-student-parent conversations about student progress and can encourage parents to be more involved in their children’s education (Qvortrup and Keiding, 2015[31]). In Norway for example, teachers are expected to keep the documentation of their formative assessment of students so they can meet with pupils and their parents for a discussion of a student’s progress every term (OECD, 2013[1]). In Serbia, portfolios could help teachers in shifting the discussion with students and parents away from marks and more on student learning.

Towards this end, the IIE should make sure that the following tools are available on the online teacher education platform:

• Guidelines and examples on how to provide formative feedback to students: The IIE should develop examples of good feedback that teachers can use to provide students with clear direction on how to improve. The IIE can learn from the experience of the National Council for Curriculum and Assessment (NCCA) in Ireland which has developed materials to help teachers and schools expand their assessment toolkit. For example, multimedia and materials such as samples of students’ work with teacher commentary and classroom video footage are available in its Assessment for Learning website, that also includes checklists and reflection tools for teachers and other school staff to develop their assessment competencies (OECD, 2013[1]).

• Guidelines and examples to encourage the use of student portfolios: Student portfolios seem to be rarely used by teachers in Serbia. For example, none of the teachers interviewed by the review team use student portfolios in their classroom practices. The IIE and the IEQE Exam Centre need the resources to develop supporting tools that explain the purpose and use of portfolios and encourage teachers to make use of this assessment practice in their classrooms. Providing regular support and guidance to teachers on how to use student portfolios should also be part of the mandate of the new body of assessment coaches that this review recommends creating (see Recommendation 2.1.3).

Table 2.3. Examples of types of feedback
Feedback type	Example
Identifying errors	Underline or circle words, “?”
Explaining misunderstandings	This data is out of date… Don’t forget… recent data shows…
Demonstrating correct practice	Inserting corrections, new sentence
Engaging students in thinking	Why?, Is this logical?, Does this follow? Is there an alternative interpretation?
Suggesting further study	“See…for information”, “Try reading… to develop your thinking further".
Justifying marks	“I could not award a higher mark because of xxx”. “This analysis made a strong contribution to your grade”
Suggesting approaches to future work	“In future assignments I recommend…” “Try to develop your…”
Aligning progress from previous attainment	“I can see how you have developed this…” “You have made progress here”
Source: Adapted from Orsmond, Paul; Merry, Stephen (2011[32]), “Feedback alignment: Effective and ineffective links between tutors’ and students’ understanding of coursework feedback”, https://doi.org/10.1080/02602930903201651.

Provide teachers with further training on differentiating teaching to adapt to students’ learning levels

In addition to better understanding students’ levels of learning through formative assessment, teachers need support to develop their capacity to use the information from assessments to adapt their lesson plans to the learning needs of students. This is an area where many teachers struggle in Serbia. In an IIE survey of teachers’ professional development needs, more than half of surveyed teachers (57.3%) reported needing training in adapting teaching to students’ needs (IIE, 2017[16]). Effective use of the initial test and other formative tools will hinge on teachers knowing what to do with the results. The IIE can, for example, work with the IEQE Exam Centre to identify the most common mistakes in the initial tests and provide teachers with examples of pedagogical strategies to deal with these common problems. The IIE also needs to ensure that pedagogues (pedagogical support staff in the schools) and psychologists are sufficiently trained in how to help teachers adapt their teaching practices or organise good remediation classes. In a growing number of OECD countries, school support staff are trained to work with teachers to design comprehensive approaches to support learning. For example, in Finland, the school support staff work with teachers at the beginning of the year to set a learning plan and meet regularly throughout the year to ensure that students’ learning is supported through adequate interventions (Borgonovi, Ferrara and Maghnouj, 2018[33]).

Recommendation 2.1.3. Develop teachers’ assessment literacy

Serbia recognises the urgency of improving teachers’ assessment literacy and has made it a priority area of professional development for 2017-20. External school evaluations show that teachers struggle with assessing students learning in more than half of Serbia’s schools (IEQE, 2017[17]). Strengthening teachers’ assessment literacy is particularly urgent for “subject teachers” teaching in Grade 5 and above. These, for the most part, graduated in a field other than education and entered teaching with little or no training on pedagogical practices – including student assessment (see Chapter 3). Once in schools, training on assessment is relatively limited. In 2019, only 11 of the 398 programmes included in the IIE training catalogue were on assessment literacy. Their quality and practical value should be improved to make them both more relevant and attractive for teachers. While research shows that the most effective forms of professional development is embedded in teachers’ regular work and are enquiry based, the majority of training provided in Serbia are seminar-based and last at most a day. The IIE and the ministry need to encourage more in-school training on assessment and develop peer learning by encouraging schools to share their experiences.

Moreover, take-up of professional development is particularly low in Serbia due to limited financial support and because there is limited data on the relevance and quality of training (see Chapter 3). If the ministry and the IIE want to improve take-up of training on assessment, financial support and incentives need to be provided to teachers. They also need to further develop the online teacher education platform set up as part of the curriculum roll-out process in order to make sure that teachers have access to tools and examples to guide their assessment practices.

Make sure that all in-service teachers have a minimum level of assessment competency

Teachers with important weaknesses regarding their assessment literacy should be provided with access to training to help them reach a minimum level of competency. Such training should be free and mandatory. To do so, school principals need to be given authority to require that teachers undertake training if regular appraisal shows that their knowledge and skills are lacking in this area (see Chapter 3). For this process to work, school principals need to be provided with guidelines and tools to appraise a teacher’s assessment literacy. They also need to receive training on how to encourage a positive assessment culture in their school and develop teacher practice in this area (such as by developing a school assessment policy, making time for teachers to co-develop assessments and moderate marking, and providing guidance on how to engage parents and manage their expectations). Modules on a school leader’s role in assessment should be included as part of their initial and in-service training, and the external inspection process should be used to reinforce good practice through the judgement and feedback given to schools (see Chapter 4).

Further develop in-school professional development and peer learning on assessment

The IIE should do more to promote in-school professional development on student assessment. At the moment, most government-sponsored training takes place in the form of short, one-off workshops (see Chapter 3). For example, the IIE is offering a training module on student assessment to about 18 000 teachers as part of the new curriculum reform roll-out. This training takes the form of single three-day seminars held outside of schools. It is not sufficient to change practice. Promoting ways to embed training on assessment in schools is not only a much more effective approach to developing teachers’ practices, as research on other countries has demonstrated (OECD, 2013[1]). It will also be important for engaging those teachers most in need of support, recognising that participation in external seminars tends to be self-selective, benefitting least those who are most resistant to change.

To develop in-school training and support on student assessment, the IIE and the ministry should consider the following strategies:

• Provide grants to schools for in-school training on assessment: Currently, most schools have to fundraise or find other means to fund training activities which creates a risk of inequality between schools in rich and poor areas. As discussed in Chapter 3, the ministry should provide schools with grants to access relevant training and also organise more in-school professional development activities, prioritising disadvantaged schools or schools with a large number of at-risk students. In order to ensure that some funds are used for training on assessment, part of the professional development grants can be earmarked for this area. For example, a small grant can be allocated to hiring coaches on student assessment to come to schools and help teachers understand how to provide constructive feedback, for example, or build a portfolio. Such a model was used by Norway in its Knowledge Promotion programme between 2006 to 2010 as part of Norway’s national effort to improve the quality of classroom assessment (Nusche et al., 2011[34]).

• Train teachers to become assessment champions: The IIE should consider using a “train the trainer” model and prepare at least one teacher per school to serve as a school point of contact for questions related to student assessment. In the short term, they should prioritise schools that performed very low in the teaching and learning area at the last external school evaluation and gradually expand the model to all schools. This assessment champion would be tasked with organising in-school training and working with classroom or subject groups on such activities as joint assessment design and moderation. This role needs to be recognised in the revised career structure recommended by this review (see Chapter 3).

• Encourage the use of assessment moderation as a form of professional development: Moderation is a set of measures that seek to improve the consistency of marking, for example through teachers reviewing or cross-marking each other’s assessment within a school. Moderation can also contribute to building a shared understanding of marking criteria or standards (Timperley et al., 2007[35]). To encourage schools to set up moderation processes, the IIE can make sure that it is part of the mandate of the assessment expert to ensure that moderation activities are effectively taking place. The external school evaluation should review the quality of this moderation during the school visit and provide schools with recommendations if the quality is not satisfactory.

• Make peer-learning on student assessment a key component of the SHARE project: SHARE is a peer-learning initiative that pairs high-performing schools with those that performed poorly in the external school evaluation (see Chapters 3 and 4). As part of the SHARE project this review recommends expanding, the external school evaluation should be used to identify schools where student assessment practices are particularly weak and match them with schools that demonstrated good teaching and learning practices. Teachers from the weaker school can work with colleagues from the higher performing school to design better assessment tools. Teachers can also work together across the two schools to set joint moderation committees and exchange practices on marking and good written feedback.

Encourage teachers to share examples of good assessments through an online e-learning platform

The online teacher education platform developed by the IIE in collaboration with UNICEF includes some modules on student assessment as well as some materials (e.g. exercises, templates) linked to this training module. As discussed in Chapter 3, the IIE should develop this platform as a national repository of teaching and learning resources which all teachers are encouraged to draw upon. In the area of student assessment, the platform should include instruments developed centrally by the IEQE Exam Centre and the IIE, such as initial test templates and marking grids aligned with the learning levels in the curriculum. It should also give teachers the opportunity to upload examples of student assessment and create a forum where teachers can exchange ideas and experiences. The IIE should ensure the quality of the material uploaded to the platform by hiring and training moderators (see Chapter 3).

Improve initial teacher education in assessment

This review recommends that the ministry plays a more active role in monitoring the quality of initial teacher education programmes by setting programme-specific accreditation criteria and developing guidelines for the design of initial teacher education programmes (see Chapter 3). These criteria and guidelines should cover teacher preparation on student assessment and specify the expected outcomes in assessment literacy at the end of initial teacher education, aligned with the teach standards for novice teachers this review recommends that Serbia develops (see Chapter 3). In developing these criteria, the ministry could look at the example of New South Wales, Australia, and its assessment literacy competencies for novice teachers (see Box 2.8). These standards were defined following a review in 2013 of teachers’ learning gaps in assessment at the end of initial teacher education, and accredit and keep providers accountable for the quality of their assessment training. Moreover, candidates’ assessment literacy should be tested in the certification and selection examination this review recommends introducing at the end of initial teacher education (see Chapter 3).

Box 2.8. BOSTES’ key elements of assessment knowledge, skills and understanding for beginning teachers

In 2013, the Board of Studies, Teaching and Educational Standards (BOSTES) (now the New South Wales Education Standards Authority) conducted a study to determine how the state’s initial teacher education programmes were covering student assessment and reviewed the research literature identifying gaps in teachers’ student assessment competencies in Australia. The board then established 24 key elements of assessment knowledge, skills and understanding or competencies that beginning teachers should develop in their initial teacher education programmes. These elements provided a framework for assessment content that initial education programme providers are now expected to cover in their programmes.

Below is a selection from the 24 key elements of knowledge, skills and understanding that the board determined new teachers needed to develop in their initial teacher education programmes on classroom assessment. These key elements expand on the Australian professional standards for graduate teachers, which set out what new teachers should know and be able to do in relation to assessment. They emphasise, in particular, that new teachers need to be able to conduct assessments appropriate to the state’s school curriculum.

• Beginning teachers need to understand how teaching, learning, assessment, feedback and reporting can be aligned and integrated in practice.

• Beginning teachers need to know the purposes of summative and formative assessment and how the two can be brought together. They need to know how to incorporate both purposes for assessment into teaching and learning programmes.

• Beginning teachers should have a working knowledge of the vocabulary of assessment. They should understand and be able to apply concepts of validity and reliability to the development of their own assessment activities and tasks and to broader measures such as examinations and standardised testing programmes.

• Beginning teachers should understand the importance of developing criteria for judging different levels of performance in response to assessment activities or tasks.

• Beginning teachers need to be able to formulate questions to help them analyse student performance for feedback to students and, just as importantly, to feed forward into their teaching.

• Beginning teachers should know about ways that the reliability of their judgements can be improved, for example through moderation.

Source: BOSTES (2016[36]), Board of Studies Teaching and Educational Standards New South Wales, www.boardofstudies.nsw.edu.au (accessed on 24 June 2019).

Policy issue 2.2. Planning for the successful implementation of a new final examination (Matura) at the interface of upper secondary and tertiary education

The plan to introduce a new final examination (Matura) at the interface of secondary and tertiary education is a vital reform that promises to enhance fairness, transparency and efficiency in how decisions are taken at this critical juncture in a student’s education and life (see Box 2.4). Introducing the national Matura will improve the reliability of students’ results that determine upper secondary school certification and selection into tertiary education. The associated introduction of a new centralised tertiary admissions system will allocate places based on student choice and merit in a way that is rational and impartial. Drawing on results from the curriculum-based Matura instead of tests administered by tertiary institutions will improve transparency, limit the need for tutoring and help ensure students have an equal chance of access to places and scholarships, regardless of their socio-economic background or where they live. For all these reasons and more, this review supports the planned reforms and underscores the importance of preparing carefully for their implementation. With less than two years to go until the first cohort of students is intended to take the Matura, many aspects of the design are yet to be defined. For example, the ministry and the Exam Centre are still to decide how the centralised admission system will work and how the exam will be administered and marked. These are critical elements, and the roll-out of the Matura can only begin once they have been determined and piloted. While it is important to move forward quickly, the introduction of a Matura has already been delayed once and it is disconcerting that there are discussions of rethinking some key decisions, such as ending additional tertiary admissions tests. Serbia needs to set a more reasonable date for introducing the Matura if its implementation is to be a success and its results trusted and accepted by all key stakeholders.

Recommendation 2.2.1. Develop the concept of the new system of student admissions into tertiary education

The lack of a clear explanation of how students will be allocated to tertiary programmes is a major gap in the current Matura concept. Thus far, the design of the new certification and selection function has focused primarily on the Matura exams but it does not provide enough clarity about the new system for student admissions into tertiary education which is a crucial component of the reform. It is understood that the admissions system will consider students’ wishes, their achievement scores (i.e. Matura exam scores and school GPA) and the list of available places in study programmes. Tertiary institutions will also be asked to set and publish the requirements for entry to different programmes and the weight given to different elements of a student’s results. Serbia should define how these variables will be taken into account to allocate students to university programmes, as this will have implications both for the Matura exam (e.g. the list of subjects, the marking scale, etc.) and how the exam is administered, marked and reported upon. The ministry also needs to decide whether to introduce a common admission system (CAS), as recommended by this review, or continue with the current proposal to let the admissions process be managed by each tertiary institution.

Develop a common admission system

The most recent plan communicated to the review team leaves the decision of admission to each individual university. Each institution would establish an internal commission to rank applicants based on Matura and GPA results, then decide who to admit. Such a model would not help address two major issues with the current system of student admission into universities: inefficiency and lack of transparency. The concept note for the Matura rightly includes transparency as a key principle of the new system. However, the current proposal of granting universities the authority to manage admissions would not give the ministry any means to ensure the fairness and integrity of the selection process. It also does not address the need for greater efficiency in how students are allocated to solve the major mismatch between demand and supply of university programmes in Serbia. A decentralised admissions system operated by individual tertiary institutions with no co-ordination, such as the one currently discussed in Serbia, does not allow for finding the best match between students’ wishes and performance and the availability of seats in a university programme.

The ministry should instead consider introducing and implementing a common admission system (CAS). CAS is a special type of centralised university admission system that automatically allocates students to available places in study programmes according to rule-driven mathematical algorithms (Bethell and Zabulionis, 2014[37]). A well-designed CAS guarantees the transparency of the admission process as well as the efficiency in terms of avoiding under/over-subscription and maximising the chances of all places being filled (see Box 2.9 on the Irish CAS). Serbia’s CAS should take into account the principles that have already been established as guiding tenants of the new system: applicants’ wishes of study programmes, university programmes’ entry requirements and applicants’ achievement scores. This implies the following with respect to the design of the CAS:

• Fairness of the selection process: All applicants should be treated fairly and placement should be based on merit. To ensure this principle is followed, the Matura, an un-biased external assessment, should be the main measure used to rank students. The ministry should also correct for perceived disadvantages by providing some applicants with additional points (i.e. bonuses). For example, the ministry may consider giving special educational needs (SEN) students bonuses. In Tajikistan, bonuses are added for candidates with disabilities (Bethell and Zabulionis, 2014[37]).

• Transparency of process and criteria: The CAS’s rules and processes need to be clear for all stakeholders including applicants, universities and upper secondary schools. To do this, the CAS should provide applicants with information that allows them to see why they were awarded a particular “wish” and denied “higher wishes”. The ministry should include brochures detailing the selection process in simple and accessible terms on the CAS online platform. For example, the Central Application Office in Ireland has on its website a visual diagram detailing the key stages of the selection process (see Box 2.9). Current plans of requiring universities to disclose programmes’ eligibility criteria at least two years in advance is a very positive aspect of the Matura concept, as it will allow students and schools to prepare accordingly.

• A unique offer is provided to students based on their wishes and abilities: The CAS should provide candidates with a unique offer. This means that the algorithm behind the CAS should result in a finite and unique solution to the problem of matching student wishes with available places. To do so, the CAS should have a comprehensive list of all available university programmes and their eligibility criteria (e.g. how subjects are weighted in the Matura). The number of ranked wishes that a student can make should also reflect this principle. While the number of wishes allowed varies considerably across OECD countries with a CAS (from two in Canada to no maximum number of applications in France, Italy and New Zealand), these design choices are aimed at finding the optimal solution that matches students with an offer (OECD, 2018[38]). In Serbia, a large number of student preferences (e.g. at least ten or more) should be allowed in order to maximise students’ opportunities for enrolment and the chance of all available places being filled.

• Timeliness: The CAS should provide all students with an initial offer in two weeks. This would enable Serbia to organise a second and third round of placements, once students have accepted or refused the initial offer. Delays and unpredictability may reduce trust in the system.

Box 2.9. The Central Application Office in Ireland

The Central Application Office (CAO) was established in 2014, with the purpose of assisting students with their applications to bachelor’s programmes in Irish higher education institutions (HEIs). Students apply for almost all full-time bachelor’s programmes through the central office. The office is responsible for processing applications and recording acceptances, while HEIs retain the function of making final decisions on admissions. The central office provides a handbook that lists all the study programmes on offer (including the minimum entry requirements for each one) and gives information to students on how to apply.

The admission process includes five stages:

• Stage 1: Students first register on the central office system and provide details of qualifications.

• Stage 2: Students are then invited to enter and rank ten bachelor’s programmes of their choice and ten short-cycle tertiary programmes.

• Stage 3: The central office system takes into account students’ choices and their assessment results including the Leaving Certificate Examination and rank them based on merit for each programme they applied to and are eligible for.

• Stage 4: The HEIs instruct the central office about how many offers to make based on available seats. Students receive one offer and must accept, defer or decline it.

• Stage 5: If an applicant does not receive an offer in the first round, they may receive an offer in subsequent rounds. There are three rounds of offers.

Sources: CAO (2019[39]), The CAO Handbook 2019, http://www2.cao.ie/handbook/handbook2019/hb.pdf (accessed on 24 June 2019); CAO (CAO, n.d.[40]), The CAO: A Guide for Parents and Guardians, http://www2.cao.ie/downloads/documents/CAOparentsguide.pdf (accessed on 24 June 2019).

Use the CAS to allocate scholarships based on merit and resources

The CAS should be used to centralise the distribution of state scholarships based on criteria of merit (e.g. scores in the Matura) and socio-economic background. Candidates for the tertiary programme should be asked to provide two separate lists of wishes, one with scholarship and one without. Such a model has been used in many countries with a CAS such as Georgia and Tajikistan as it allows for greater flexibility and fairness in the distribution of state scholarships.

Recommendation 2.2.2. Review and complete the Matura’s examination model

While the concept note for the new Matura clearly defines some aspects of the examination model, such as the list of subjects that students can take and their weights, other important aspects are still being discussed. The Matura concept needs to be completed to include clear instructions on how the Matura tests will be scored and how the results will be reported to the public. The type of items to be used also should be agreed upon. Moreover, the ministry and the IEQE should consider reviewing current plans for making mathematics compulsory for only those who took more than two years of mathematics in upper secondary schools. Rather, the Matura should require minimum numeracy skills for a student’s certification of completion of upper secondary, with the responsibility placed on the ministry and schools to ensure that all students have adequate time in the curriculum to study toward this goal.

Make mathematics compulsory for all students and assess it using a dual-level exam

In the current design of the Matura, mathematics is not a compulsory subject for students who took less than two years of mathematics in upper secondary education. This is the case for students enrolled in art schools (Music and Ballet) and those philological gymnasium– currently around 1 150 students per year – who are not expected to be assessed on their numeracy skills. However, Serbia is currently reviewing the curriculum of gymnasium and mathematics will become a mandatory subject. The planned changes are in line with trends in OECD countries where mathematical literacy, alongside reading and writing, are considered to be some of the core competencies that students should acquire at school. They are not only essential for life and work, but they also provide the foundations for other domains such as the humanities and sciences. For this reason, most OECD countries make it compulsory for students to study mathematics until the end of upper secondary education and many OECD countries assess mathematics externally as a compulsory subject in national examinations (OECD, 2015[8]). The ministry and the IEQE should review the current design of the Matura with a view to moving in this direction. Specifically, they should consider:

• Introducing a dual-level exam for mathematics: It is recommended that students’ numeracy competencies be assessed using a dual-level exam (e.g. different tests covering different ranges of mathematics ability). This will help ensure that all students at the end of upper secondary have attained basic functional numeracy competencies while allowing students with the knowledge and competencies in more advanced mathematics to be assessed on these. Serbia can thus have one test covering the minimum numeracy competencies that all students should have achieved by the end of their schooling while a second test would assess more advanced levels of mathematics. The IEQE can learn from the experience of many OECD countries with similar models such as Ireland, the Netherlands and Norway (see Box 2.10).

• Giving students the choice to choose their test: This is important to give all students a fair chance to demonstrate their numeracy aptitudes and enter the faculty of their liking. For example, students with less than two years of mathematics in upper secondary should still be given the option of taking the more advanced test should they want to. Similarly, students not wishing to enter a university science programme or those with weaker numeracy skills from any upper secondary track should be allowed to take the test of basic mathematics competency.

Box 2.10. Setting mathematics examinations at different levels in Ireland, the Netherlands and Norway

In Ireland, the Leaving Certificate Examinations, the final examinations taken at the end of the secondary school system, are available at two levels – ordinary and higher-level in a variety of subjects including English language, natural sciences, humanities and the arts. In addition, the examinations for Irish language and mathematics are also available at the foundation level. Students can take a combination of higher-level and ordinary-level examinations.

To certify school completion, students must pass examinations at any level in five subjects. Students who meet this criterion can also access post-secondary non-tertiary courses that usually last one year and in many cases provide access to higher education institutions.

In the Netherlands, the voorbereidend wetenschappelijk onderwijs (VWO), meaning “preparatory scientific education” in Dutch, is an upper secondary programme available in the country designed for students who want to continue their studies at university. Mathematics is a compulsory subject for the four subject clusters in the programme. Students take the programme’s mathematics test at three different levels called A, B or C, depending on their track (economics and society, culture and society, science and technology).

The Vitnemål fra den Videregående Skole (Certificate of Upper Secondary Education) in Norway certifies completion of upper secondary general programmes. Mathematics is a compulsory subject for certification and is considered at a different level of complexity depending on the general education track. For example, students in social science studies take “Mathematics S” courses while natural science and mathematical students take “Mathematics R” with a stronger focus on pure mathematics and a small amount of probability. The Mathematic R test is taken at the end of the course and is divided into two parts. The first part of the paper consists of two exercises which must be answered and handed in after two hours of the examination. Then, in the second part, students must complete five exercises and hand them in after the five hours have elapsed (counting from the beginning of the exam).

Sources: Department of Education and Skills (2018[41]), The Education System, Ireland, www.education.ie/en/The-Education-System/ (accessed on 1 October 2019); Ofqual (2012[42]), International Comparison in Senior Secondary Assessment: Full Report, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/372211/2012-06-12-international-comparisons-in-senior-secondary-assessment.pdf (accessed on 17 October 2019).

Use a combination of multiple-choice and constructed-response items

It is not clear what combination of item types will be used in the Matura. Similar to current practices at the end-of-basic-education examination, Serbia should use a combination of multiple-choice items and open-ended items. The item type should be suitable for assessing the breadth and depth of the curriculum. Open-ended questions or constructed responses items that call for extended written responses (e.g. essays) are best suited for assessing higher-order competencies (Ku, 2009[43]). Choice of item types should also be informed by capacities to mark and process results in a reliable manner. Multiple-choice items tend to be the most reliable as they leave little space to the marker’s interpretation.

Define Matura scoring, scaling and reporting procedures

The Matura concept is unclear about the procedures for scoring, scaling and reporting student results. These need to be precisely defined and agreed upon as part of the exam concept. The IEQE will need to define the following:

• The scoring scale: The score scale should be defined so the CAS can easily rank students based on their performance. This will require a somewhat long and quasi-continuous scale to allow for sufficient discrimination of students’ performance.

• A threshold for certification: The IEQE should define the minimum score needed to pass the Matura and receive the certificate of completion of upper secondary studies. This threshold should ensure that students who pass the Matura have attained the “basic level” as defined in end-of-upper-secondary learning standards. It also should be tested to make sure that it is accessible to most students.

• Map scores against criterion-referenced achievement levels: Score points need to be mapped against the achievement levels defined in end-of-upper-secondary learning standards. Both the specific score point and the achievement level need to be reported on a student’s Matura result bulletin. This will allow the user of these bulletins (e.g. students, parents, employers and upper secondary schools) to understand the student’s performance at a more granular level.

Recommendation 2.2.3. Set up sustainable administrative and IT systems to implement the Matura

Designing and implementing the administrative processes needed for both the Matura and CAS are yet to be discussed and agreed upon. For example, the information technology (IT) system used for both the Matura and the CAS has still not been determined. These processes need to be sustainable over time to avoid losing the public’s trust in the quality of the Matura. To ensure this sustainability, the IEQE needs to identify the right actors to carry out the administrative tasks needed to implement the Matura and ensure that they are sufficiently trained. They also need to secure central government funding for the Matura beyond 2021.

Assign responsibilities and secure capacity for the Matura’s key administrative tasks

The ministry will need to identify the agencies and actors that will be responsible for key implementation tasks. The IEQE Exam Centre should be the lead agency responsible for the overall Matura process, including administration of the CAS. The implementation of administrative tasks might be distributed as follows (see Table 2.4):

• The Exam Centre: In addition to being responsible for the overall quality of the Matura, the Exam Centre should be in charge of defining the examination standards that will guide development and marking. It would be responsible for checking the quality of items by testing them and for auditing the marking by checking the quality of sample copies. The Exam Centre should have a secure space for test production (i.e. printing, packing and storing) and develop a process for the distribution of tests to test-taking centres. The centre will need additional human and material resources to carry out these tasks (see Recommendation 2.2.4).

• Regional exam centres: In contrast to the end-of-basic-education exam, which is currently administered and marked by in-school commissions, the Matura’s administration and marking will need to carried out by independent commissions outside of the school that report directly to the Exam Centre. Serbia can learn from the experience of neighbouring Albania which set up five regional exam centres under the authority of the national exam centre responsible for administering the State Matura locally and marking it (see Box 2.11). These regional exam centres are run by permanent staff of the exam centre but include mostly teachers trained in marking the exam.

• Certified teachers: The Exam Centre will need to train and certify teachers to participate in the subject working groups in charge of test design as well as those working in the regional exam centres to mark Matura tests. The Exam Centre should define a code of professional ethics describing the standards of integrity, professionalism and confidentiality that certified teachers should follow. These roles should also be recognised in the teacher career structure (see Chapter 3).

Table 2.4. Suggested responsibilities for key administrative tasks for the new Matura
	The IEQE Exam Centre	Teachers’ working groups	Regional exam centres	Candidate’s school
Overall responsibility for Matura and quality control	●
Items design	●	●
Test production
Registration of candidates				●
Test administration			●
Test marking and moderation	●		●
Dissemination of results	●
Addressing students’ appeals against results	●
Exporting data into CAS	●

Box 2.11. The administration of the State Matura in Albania

The creation of a centralised system for the design, administration and evaluation of examinations was an important change brought by the State Matura reform from 2006 in Albania. Before that, exams were drafted by the Ministry and administered and corrected by the schools’ academic staff.

Albania has now set up five regional exam centres in order to locally administer and mark the State Matura. The national exam centre, ESC (Educational Service Centre), oversees the examination but is not responsible for training and certifying teachers who will mark the exam when questions cannot be evaluated using technology. Evaluation of tests is carried out in six assessment centres with appropriate and safe environments. The centres are selected by the ESC and approved by an order of the Minister of Education and Sports.

Source: OECD (forthcoming[44]), OECD Reviews of Evaluation and Assessment in Education: Albania, OECD Reviews of Evaluation and Assessment in Education, OECD Publishing, Paris.

Develop an integrated IT system for the Matura

The information system currently used for the central exam at the end of basic education should serve as the basis to develop the new IT system for the Matura. Serbia already has a basic CAS system in place for placing students in upper secondary schools. This system needs to be reviewed and further developed to accommodate the specificities of the Matura process, which include more subjects and electives than the end-of-basic-education exam. The IT platform will need to be capable of processing the Matura exam data and the CAS. A list of modules that such an IT platform needs to include is provided in Table 2.5.

Table 2.5. Suggested list of modules to include in an IT system
	List of modules to be developed
Matura examination processes	registration of candidates allocation of candidates to examination centres/rooms/seats production of administrative protocols (e.g. printing requirements, packing lists, candidate attendance lists, production of control barcodes, etc.) processing of candidate responses (objective items) and entry of candidate scores (examiner marked) processing of candidate scores (scaling etc.) issue (publication) of results for individual applicants statistical analysis and reporting
CAS	registration of all available places on all HEI courses with their characteristics (e.g. state-paid and candidate-paid places) and entry requirements (including any “weighting” of scores by subject) registration of all examination candidates and applicants with their characteristics and, most importantly, their HEI wish lists algorithm for matching applicant wishes and scores with HEI places issue (publication) of places awarded for individual applicants possibility to run a second round of CAS to fill unfilled places

The ministry should also consider building the IT system in such a way that it would allow Serbia to move towards computer-based assessment (CBA) in the near future without having to rebuild a new IT system. While a fully digital Matura is not feasible in the short run, this should be considered in the medium term as it would allow for increased security, faster marking and allow the use of more refined testing instruments such as adaptive testing.

Ensure the sustainability of the new Matura over the long term

Once the new Matura model is introduced, it becomes an ongoing commitment that will require predictable and adequate funding to cover the human, technical and physical resources required for its effective implementation. At the moment, only the concept-design and early implementation phases are funded (up to 2020). The Serbian government needs to guarantee recurrent funding for the Matura for at least the coming ten years. This is key to building trust among universities and school actors that will need to adapt to a new exam and tertiary admission system. This funding should be adequate to cover all core costs.

A thorough review is required of the Exam Centre’s staffing and technical and physical so that it is able to meet its responsibilities for the new Matura. These are barely adequate for its current roles. A range of additional capacities is likely to be required to meet the demands of the Matura system. This will certainly involve increasing the number of staff, as well as strengthening specialised profiles in statistical and psychometric analysis. Judging from the experience of countries that have introduced similar systems in recent years, significant investment is also likely to be needed in terms of accommodation and equipment. A good approximate reference for development costs comes from World Bank projects to establish national exam centres. For example, the cost of setting up an exam centre in Tajikistan responsible for developing and administering a new national examination similar to the Matura was USD 6.1 million (World Bank, 2013[45]).

Recommendation 2.2.4. Set a realistic timeframe for implementation and build public understanding of and support for the new system

While the ministry aims to have the first Matura exam in June 2021, many gaps in the design of the Matura need to be addressed before this begins. Most importantly, how results of the Matura will be used to allocate students to tertiary programme needs to be agreed upon and the IT system administering the Matura and the CAS need to be secured. This will make the 2021 deadline difficult to meet without risks of jeopardising the quality of the assessment and its administration. It is therefore important that the ministry revise the timeline to leave sufficient time for addressing the gaps in the design discussed in this policy issue. The revised timeline should also leave sufficient time for proper piloting and revisions and for establishing an effective communication campaign.

Delay the implementation of the Matura by two years to leave sufficient time for an effective roll-out

Given the amount of work that is still ahead, the target date for introducing the Matura and CAS systems by 2021 appears overly ambitious and should be reconsidered. The ministry and the IEQE are yet to finalise the concept notes for the Matura and tertiary admission system, define the administrative procedures and run pilots (see Figure 2.4). All these steps require time to be successfully executed and are not feasible within a year and a half timeframe. International experience shows that the design and implementation of a high-stakes examination at the end of upper secondary education takes at least five years. In Slovenia, the development and implementation of a new Matura exam took approximately six years (1989-95). In Tajikistan, the development and implementation of a unified examination system and CAS took approximately eight years (2006-14) from the formation of the initial concept to the first full roll-out of the examination (World Bank, 2015[46]; 2016[47]).

Serbia should change the law in order to delay the final introduction of the Matura by at least two years so that the new target date for the full-scale implementation of Matura is set for 2023. This revised timeline will give the ministry and IEQE time to complete the Matura and CAS designs and test them by end of 2020. In parallel, an information campaign targeting students in first and second years of upper secondary education needs to be organised to explain the new model. The following two years should be dedicated to setting up administrative and IT systems and revising the Matura and CAS in light of the pilot’s results.

Figure 2.4. Steps needed to implement a new examination and suggested timeline

Conduct two robust pilot studies before full-scale implementation

Pilot studies are a central feature of education assessment reforms as the quality of instruments and administrative processes can only be fully checked when tested in conditions as close as possible to the full roll-out plan (OECD, 2013[1]; van Teijlingen and Hundley, 2002[48]). Current plans for the Matura include a two-stage pilot. The Matura exam would be piloted in about 40% of schools to test the exam instruments and the administration process. Based on the results, a revised version of the Matura would then be piloted in all upper secondary schools with the participation of all students. This plan of a two-stage pilot is sound and should be implemented: indeed, a two-stage pilot allows for testing both the Matura exam itself (twice) as well as the administration of the exam and the tertiary admission system in conditions similar to those of the real test. Delaying the implementation of the Matura as recommended above is necessary to run such pilots and revise the tools and processes accordingly.

In the hypothetical undesirable scenario in which the launch of the Matura is not delayed until 2023 and thus there is no time for a two-stage pilot study, the key principle to keep in mind is that the Matura exam should be piloted at least once and revised based on the results before full-scale implementation begins.

Develop an information campaign and engage stakeholders in the design and implementation of the Matura

The Matura concept note was developed in a participatory manner through a working group that included national and provincial level governmental agencies, representatives from higher education institutions, teachers, councils in the field of education and school communities. This principle of collaboration needs to be continued as the reform enters the delicate phase of clarifying the test design and deciding on the tertiary admission system. For example, university professors involved formerly in the design of university entrance tests should be involved in the working group tasked with writing the Matura tests. This would help to build trust and buy-in, which also ensuring continuity and relevance for university programmes. Representatives from upper secondary schools and universities should also be involved in developing the new CAS model recommended by this review. Universities can also engage in the process of monitoring and evaluating the implementation of the Matura exam.

In addition, consulting key stakeholders in the design of the Matura, the ministry and the IEQE need to set up a communication strategy to make sure that students, parents, schools and universities understand and accept the reform. There is at the moment high levels of uncertainty and doubt among stakeholders about the Matura and planned changes to tertiary admissions. For example, while some of the schools visited by the review team were already preparing their students for the new Matura based on the information received from the ministry, others were doubting that the Matura would enter into effect and some were unsure whether mathematics would be a required subject for their students. To build support and prepare key stakeholders, the ministry should consider the following actions:

• Set up a communications team or unit in charge of translating these reforms into clear messages to be disseminated through materials for students, parents and school staff, and through communication media such as television, social media, Internet and printed media (e.g. newspapers, magazines). The communication team within the Ministry of Education in France, for example, has created clear information materials on the reform of France’s academic qualification exam, the baccalaureate (baccalauréat) (see Box 2.12).

• Create a website dedicated to the Matura reform that provides information on the Matura and allows the public to ask questions and request information. In 2017, England in the United Kingdom dedicated a website to guide students, parents and school staff through the first reformed General Certificate of School Education (GCSE) (see Box 2.12).

Box 2.12. Examples of communication strategies on exam reforms in England and France

France is undergoing a reform of its academic qualification exam, which students are required to take in order to graduate from high school and access tertiary education. The new baccalauréat, as it is called, will have its first session in 2021 and therefore the government has already started communicating on the terms of the reform. The Ministry of Education in France released a set of informative documents – which can be accessed online – containing key information on the new exam explained through infographics and clear messages in order to facilitate the understanding of the reform’s key components and timeline. A good example is the press kit available for download on the government’s website. One of the infographics available in the press kit translates into clear steps a student’s path from the beginning of high school to the new baccalauréat exam.

In England, a reform has been implemented to modify the country’s General Certificate of Secondary Education (GCSE). The first reformed General Certificate of Secondary Education – an academic qualification taken in a number of subjects by pupils in secondary education – was introduced in 2017 in English language, English literature and mathematics. Some of the main features of the new exams are: a new marking scale from 1 to 9, 9 being the top grade; more demanding content developed by the government and exam boards; and courses designed over two years of study, with students taking all their exams in one period at the end of their course.

As one of the biggest reforms in England, the government has massively invested in publicity and a communication campaign to inform on its main points and engage stakeholders. In 2017 for example, England’s Office of Qualifications and Examinations Regulation - the exams regulator - stated it was essential that major changes were communicated to a wide audience as independent research carried on their behalf showed that public understanding of the new marking system and other reforms related to the exam had increased since the first campaign. The Office and the Department for Education in England have set the communication campaign around original films, printed materials and social media advertising. They have developed online material, including a website and a page in the Office’s blog to inform the public on the progress of the reform and address their questions. The pages available on line inform the public on how the exam will look like, with updated information about the reform, including its main features and following steps.

Sources: Ministère de l’Éducation nationale et de la Jeunesse (2018[49]), Communication en conseil des ministres : la réforme du baccalauréat [Communication at the Council of Ministers: the reform of the Baccalauréat], www.education.gouv.fr/cid126564/communication-en-conseil-des-ministres-la-reforme-du-baccalaureat.html (accessed on 31 May 2019); UK Government Department for Education (2018[50]), Get the Facts: GCSE Reform, www.gov.uk/government/publications/get-the-facts-gcse-and-a-level-reform/get-the-facts-gcse-reform (accessed on 31 May 2019); The Independent (2017[51]), “Government to spend half a million pounds explaining confusing GCSE exam reforms”, www.independent.co.uk/news/education/education-news/gcse-exam-results-government-watchdog-ofqual-500000-half-million-explain-confusing-education-a7861586.html (accessed on 3 June 2019).

Policy issue 2.3. Strengthening the technical quality of the central examination at the end of basic education

The central examination that students take at the end of basic education (Grade 8) is a policy instrument that, since its creation in 2011, has effectively served its dual purpose: it certifies students’ completion of basic education and provides the scores which are used in their automatic placement into general secondary and VET schools – taking into account their “wish list” of desired schools. However, almost a decade after its introduction, the exam would benefit from some refinement in terms of both its design and administration. Regarding design, the exam can be reviewed to assess a wider range of competencies in line with the recent curriculum reform. The exam’s administration in schools should also be strengthened to improve the reliability of results and the public’s trust in them. Building confidence in the quality and integrity of the “small” Matura will be important for gaining support for the crucial reforms of examinations at the end of upper secondary school.

Recommendation 2.3.1. Develop the exam to measure a wider range of competencies and levels of achievement

While the design of the Grade 8 exam is largely fit for purpose, it could be improved to assess a wider range of competencies that are included in end-of-basic-education learning standards. There is also scope to extend the number of items and the marking scale to enable more refined discrimination of achievement levels. Despite the Exam Centre’s efforts to include more diverse item types that assess more complex and higher-order outcomes, the end-of-basic-education exam remains largely a knowledge-based instrument which tests mostly lower and intermediate learning outcomes. It focuses primarily on the reproduction of facts and the use of routine cognitive procedures, and the more complex problem-solving tasks that are included tend to be limited to familiar contexts which reduces their level of difficulty. The exam also makes little attempt to assess the transversal competencies that are emphasised in the learning standards. While the Exam Centre will need to allocate its resources with the Matura as a priority, some limited adjustments to the Grade 8 exam could help to develop item-writing and other skills that will be important for the quality of the exam in Grade 12.

Increase the number of questions in exam tests to allow for more space to measure advanced competencies

With only 20 items per test, there is little space in the exam for testing competencies across the ability range, especially advanced ones. Each test includes seven items of “basic level” competencies, nine of “medium level” and only four questions of “advanced level” competencies. The exam is thus not discriminating enough for higher levels of ability. This is very apparent when looking at students results. For example, about 40% of the students who took the Serbian language exam in 2017 scored 15 or more points out of a maximum of 20 points (see Figure 2.5) (IEQE, 2017[14]). The Exam Centre should consider increasing the number of items for the advanced level of competencies and increase the overall number of items above the current arbitrary number of 20.

The recommended increase in items can be achieved without lengthening the test-taking time and without forcing students to rush to respond to an excessive number of questions. The time spent per question in Serbia’s end-of-basic-education exam is longer than common assessment practices internationally. Students have 2 hours (120 minutes) to respond to each 20-question test; this means that that students have, on average, 6 minutes per question, which is more time than in most other countries with similar exams, such as Singapore. For example, in Singapore’s Primary School Leaving Examination (PSLE), the first part of the mathematics test includes a total of 30 multiple-choice and short-response questions. Students are allowed 1 hour to complete the first part of the test which means an allowance of about 1.3 minutes per question. Similarly, the science test in the examination includes 40 questions (28 multiple-choice and 12 open-ended) and 1 hour and 45 minutes to answer, thus students have on average 2.6 minutes per question to answer this test (Singapore Examinations and Assessment Board, 2019[52]).

Figure 2.5. Distribution of students’ scores by test subjects, 2016-17

Average score by percentile

Source: IEQE (2017[14]), Izveštaj o Rezultatima Završnog Ispita na Kraju Osnovnog Obrazovanja i Vaspitanja u Školskoj 2016/2017 [Results of Primary School Final Examination 2016-2017].

Revise the scoring system using a longer score scale and allow for partial credits

Introducing more tasks, including complex tasks, will require adjustments in the scoring system. Currently, the results in all exam tests are graded on a 0-20 scale with a general rule that each task carries 1 point. While partial credits are allowed (half a point), they are rarely used which limits the tests’ capacity to assess partial achievement. The Exam Centre should revise the scoring system to allow and support the use of a wider range of partial points, at least for items using complex multiple-choice in constructed-response formats. For example, mathematics tasks requiring multiple steps or constructed tasks involving the use of 2 or more skills could be marked using a range of points for full and partial credit (e.g. 0, 1 point, 2 points). This more complex scoring rule would improve the technical quality of the exam because it would better measure students’ ability to solve specific problems (distinguishing between students who can solve part or parts of a problem and those who cannot solve the problem in any part). Additionally, the 20-points scale could be lengthened to allow for more refined discrimination of students’ result.

Improve item-writing capacity among associate teachers and within the IEQE

The ministry needs to invest in improving the capacity of test writers if more complex items are to be introduced as recommended above. At present, Serbian teachers who are contracted by the Exam Centre as associates to write items on specific subjects are generally better at writing simple questions that require students to memorise content information than at writing items that require higher-order thinking. This is due to the limited training provided to item-writers and overall lack of familiarity with competency-based curriculum and assessment.

To address this issue, the Exam Centre should provide teachers participating in the item-writing commissions with training on how to assess higher-order competencies and write constructed test items. The Exam Centre can, for example, invite experts from other European countries with experience in assessing higher-order competencies such as Austria and Denmark (OECD, 2013[1]). This exposure to competency-based assessment will not only improve the quality of test items but also teachers’ classroom assessment practices.

Replace the combined test with subject-specific assessment instruments

Introducing the combined test in 2014 was a positive step because it enabled the assessment of a slightly larger breadth of the curriculum. Teachers also informed the review team that the test also led to students paying attention to a larger set of subjects during the final years of basic education. Analysis performed for this review suggests that the test is a good measure of a student’s overall cognitive ability, as shown by the strong correlation between a student’s results in the combined test and results in the mathematics and Serbian language tests (see Table 2.6). However, the combined test is a relatively weak measure of subject-specific competencies. With only four items per subject, the combined test is not a valid measure of the level of student achievement in any of its five component sub-domains (biology, chemistry, geography, history and physics). The test merely assesses basic knowledge of each subject.

The Exam Centre should consider replacing the combined test with more valid assessments of student competencies in different subject areas. For example, the combined test could be replaced by two separate tests: a test of natural sciences (biology, chemistry and physics) and a combined test of history and geography. Alternatively, Serbia may consider introducing a natural sciences test and a foreign language test, as both science, technology, engineering and mathematics (STEM) and foreign language are priority areas in Serbia’s education strategy.

Table 2.6. Final examination for school year 2017/18: Descriptive statistics and correlation coefficients between tests included in the exam
	Language (Serbian)	Mathematics	Combined test
Number of test-takers (students)	65 129	65 129	65 129
Mean score	11.99 (60%)	10.04 (50.2%)	12.60 (63%)
Standard deviation	4.17	4.31	4.17
Correlation (r) with language (Serbian)	..	0.730	0.708
Correlation (r) with mathematics	0.730	..	0.729
Correlation (r) with combined test	0.708	0.729	..
Notes: .. : Not available.
The table does not show data for students who passed the final exam in their mother tongue.
Correlation coefficients for language (Serbian) and mathematics are based on Grade 8 school marks in the corresponding subject. The correlation for the combined test is based on the average Grade 8 score across 5 subjects: biology, chemistry, geography, history and physics.
Source: IEQE (2018[53]), OECD Review of Evaluation and Assessment: Country Background Report for Serbia, Institute for Education Quality and Evaluation.

Create a new school-based project aimed at assessing interdisciplinary competencies

The end-of-basic-education exam does not at present measure the transversal competencies included in the curriculum. Learning from the experience of many OECD and neighbouring countries, the Exam Centre should strongly consider introducing a mandatory project-based assessment to test students’ transversal competencies such as communication or collaborative problem solving. Project-based assignments are long-term, in-depth projects that students complete within their school by applying skills they learnt throughout the grades prior to the examination in a practical manner (Kaldi, Filippatou and Govaris, 2011[54]; Blumenfeld et al., 1991[55]).

The Exam Centre should consider the following in designing the project-based assignment:

• Ensure comparability of the test: While students and schools should be given some flexibility designing the project assignment, the Exam Centre needs to provide clear guidelines to ensure comparability of results. These guidelines should define the competencies that the student will be assessed against (see Box 2.13 for an example from the United Kingdom) and define a list of topics that students and schools can choose from. The Exam Centre could also review a randomly selected sample of project assignments to ensure quality.

• Make it part of the student’s final score: To make the project more relevant for all actors, marks in this project should be made part of the final graduation score used for student selection into upper secondary schools. For example, the project-based assignment could account for 10 points out of the total 100 points of the final graduation score. This could be achieved by reducing the weighting of the student GPA to 50 points (instead of 60) while leaving the weighting of the aggregate written examination score at 40 points.

Box 2.13. Project assignments in England, Northern Ireland and Wales

In England, Northern Ireland and Wales (United Kingdom), students completing their A Levels at the end of upper secondary can also produce an optional “extended project”. The extended project provides students with the opportunity to develop and demonstrate their project management skills and extended writing.

• Subjects: the extended project can be completed in one or more of the student’s study areas and/or areas of interest related to a student’s main study programme, in agreement with their examination centre (often their school). Examples of acceptable titles for extended projects are available online.

• Outcome: a design, performance, report, dissertation or artefact.

• Assessment: the extended project is internally assessed by a candidate’s examination centre. Candidates must produce a written log verified by a supervisor, a written report, supplementary evidence and a presentation.

Students are assessed against four objectives. Each objective has contributed a specific weight to the student’s overall mark:

1. 1. Manage – identify, design, plan and complete the individual project or task within a group project, applying organisation skills and strategies to meet stated objectives. Contributes 15%-25% to the final mark.

2. 2. Use resources – obtain and select information from a range of sources, analyse data, apply relevantly and demonstrate understanding of any appropriate linkages, connections and complexities of their topic. Contributes 15%-25% to the final mark.

3. 3. Develop and realise – select and use a range of skills, including new technologies, to solve problems, to take decisions critically, creatively and flexibly, and to achieve planned outcomes. Contributes 35%-45% to the final mark.

4. 4. Review – evaluate outcomes including own learning and performance. Select and use a range of communication skills and media to convey and present outcomes and conclusions. Contributes 15%-25% to the final mark.

Marking grids are provided to demonstrate student performance at three levels for each assessment outcome and how marks may be allocated.

• Learning hours: 120 hours in total. Approximately 50 hours of taught time and 70 hours preparing for assessment.

• Grades: A*-E.

Source: UCAS (n.d.[56]) Extended Project Qualification (EPQ), https://qips.ucas.com/qip/extended-project-qualification-epq (accessed on 14 January 2019).

Recommendation 2.3.2. Build public confidence in the examination system

The procedures that the IEQE Exam Centre has put in place to guarantee the correct implementation of the exam seem generally well-designed and adequate. External supervisors monitor the test administration in schools and checks are performed at the local and national levels to ensure there were no irregularities in test administration and scoring. However, the Exam Centre regularly finds evidence of malpractice in some schools. Some schools are not implementing the test administration procedures as they should be and some school staff deliberately cheat by giving their students the right answers. While this problem is not widespread according to analysis by the Exam Centre, it leads many parents and students to mistrust the end-of-basic-education exam results. The ministry and the Exam Centre should address these concerns by strengthening the administration procedures and providing more information to the public on how the exam is run. These measures to improve public trust in the exam will help likewise to build confidence in the government’s capacity to administer the new Matura.

Continue to develop and use control measures to improve security and accountability of implementation

While instances of irregularity are rare, there are still measures that the Exam Centre and the ministry can take to strengthen the administration of the exam and prevent cheating. The following measures should be considered:

• Appoint exam supervisors from other municipalities: At the moment, exam supervisors come from other schools in the same municipality which means that they might have pre-existing personal or professional relations with the staff and students of the schools they are supervising. To avoid any appearance of bias, the ministry should instead bring in teachers from other municipalities to serve as supervisors.

• Introduce greater penalties for malpractice: Organised cheating in a national high-stakes examination is considered in most countries a serious offence with severe consequences. In England for example, for the 2018 summer exam series (GCSE, AS and A level examinations), 620 penalties were issued to 475 members of school or college staff (e.g. teachers and invigilators). The sanctions ranged from written warnings, requirement for training or mentoring, to suspensions from involvement in exams and even teaching depending on the category and impact of the malpractice (Ofqual, 2018[57]). Conversely in Serbia, instances of cheating by school staff are rarely punished. For example, it was reported to the review team that a school principal who provided the answer key to students taking the test received only a small fine. The ministry needs to make sure that school principals or teachers caught cheating are fined and barred from public service to deter others, signalling the seriousness of this offence.

• Increase the scale of checks made on the marking process by the district commission. Specifically, the ministry should increase the number of schools from which tests are checked and make sure these are done as soon as possible after the marking process.

• Introduce targeted reviews for schools where past irregularities were observed: Schools flagged by the Exam Centre’s annual analysis as having abnormal results (e.g. schools with uniformly high results in the exam) should be the target of stronger surveillance during the following exam year. For example, students from these schools can be sent to take the test in other schools.

Communicate the strengths and problems in implementing the exam

The end-of-basic-education exam has a reputation for not being very fair and transparent. In fact, many parents and students believe that cheating is widespread despite very little evidence that this is the case. The ministry and Exam Centre need to make more information available to educate the public and raise awareness of the improvement, ensuring a fair and transparent administration of the exam as well as the real extent of irregularities and how they are handled.

• Make the Exam Centre’s annual summary analysis on the exam’ quality control public: Such a measure would help increase public accountability of schools. It will also help show to the public that organised cheating is, in fact, rare and thus help improve trust in the exam.

• Provide more public information in advance of each session: the ministry should make available on its website information targeted at the general public on the exam’s procedures and the measures taken to ensure integrity. Schools should also be provided with guidance and resources to use for communicating with students and parents. In addition, there is the need for a more proactive approach to briefing the media so that coverage of the exam is constructive and keeps actors accountable. On its webpage, the Ministry of Education in France made available a list of questions and answers about the baccalauréat 2019 almost a year before it took place. Among the data available, the public – in particular, students – can find information on the progress of exams, the correction process, the communication of results, etc. (Ministère de l'Éducation nationale et de la Jeunesse, 2018[58]).

Table of recommendations

Policy issue	Recommendations	Actions
2.1. Ensuring a better balance between formative and summative purposes in school-based assessment	2.1.1. Revise the student assessment framework to encourage a shift in focus from marks to learning	Define clearly the core principles of student assessment in Serbia
		Extend the marking scale to allow for a more refined description of students’ abilities
		Link marks to performance levels and require teachers to provide descriptive feedback to students
		Limit the frequency of summative numerical marks to create space for more formative dialogue
	2.1.2. Strengthen the support provided to schools in conducting formative assessment	Strengthen the support provided by the IEQE Exam Centre in using diagnostic assessment (the initial test)
		Provide guidelines and tools to encourage teachers’ use of formative assessment
		Provide teachers with further training on differentiating teaching to adapt to students’ learning levels
	2.1.3. Develop teacher’s assessment literacy	Make sure that all in-service teachers have a minimum level of assessment competency
		Further develop in-school professional development and peer-learning on assessment
		Encourage teachers to share examples of good assessments through an online e-learning platform
		Improve initial teacher education in assessment
2.2. Planning for the successful implementation of a new final examination (Matura) at the interface of upper secondary and tertiary education	2.2.1. Develop the concept of the new system of student admissions into tertiary education	Develop a common admission system
		Use the CAS to allocate scholarships based on merit and resources
	2.2.2. Review and complete the Matura’s examination model	Make mathematics compulsory for all students and assess it using a dual-level exam
		Use a combination of multiple-choice and constructed-response items
		Define Matura scoring, scaling and reporting procedures
	2.2.3. Set up sustainable administrative and IT systems to implement the Matura	Assign responsibilities and secure capacity for the Matura’s key administrative tasks
		Develop an integrated IT system for the Matura
		Ensure the sustainability of the new Matura over the long term
	2.2.4. Set a realistic timeframe for implementation and build public understanding of and support for the new system	Delay the implementation of the Matura by two years to leave sufficient time for an effective roll-out
		Conduct two robust pilot studies before full-scale implementation
		Develop an information campaign and engage stakeholders in the design and implementation of the Matura
2.3. Strengthening the technical quality of the central examination at the end of basic education	2.3.1. Develop the exam to measure a wider range of competencies and levels of achievement	Increase the number of questions in exam tests to allow for more space to measure advanced competencies
		Revise the scoring system using a longer score scale and allow for partial credits
		Improve item-writing capacity among associate teachers and within the IEQE
		Replace the combined test with subject-specific assessment instruments
		Create a new school-based project aimed at assessing interdisciplinary competencies
	2.3.2. Build public confidence in the examination system	Continue to develop and use control measures to improve security and accountability of implementation
		Communicate the strengths and problems in implementing the exam

References

[4] Absolum, M. et al. (2009), Directions for Assessment in New Zealand: Developing Students’ Assessment Capabilities, Ministry of Education, Wellington, http://assessment.tki.org.nz/Research-and-readings (accessed on 20 June 2019).

[37] Bethell, G. and A. Zabulionis (2014), “University Admissions Systems: Evolution Versus Intelligent Design”, Education Today, Vol. 64/2, https://www.researchgate.net/publication/284154532_University_Admissions_Systems_Evolution_versus_Intelligent_Design (accessed on 20 June 2019).

[5] Bishop, J. (1999), Are National Exit Examinations Important for Educational Efficiency?, Cornell University ILR School, http://digitalcommons.ilr.cornell.edu/articles (accessed on 17 October 2019).

[3] Black, P. and D. Wiliam (1998), “Assessment and classroom learning”, Assessment in Education: Principles, Policy & Practice, Vol. 5/1, pp. 7-74, https://doi.org/10.1080/0969595980050102.

[55] Blumenfeld, P. et al. (1991), “Motivating project-based learning: Sustaining the doing, supporting the learning”, Educational Psychologist, Vol. 26/3-4, pp. 369-398, https://doi.org/10.1080/00461520.1991.9653139.

[33] Borgonovi, F., A. Ferrara and S. Maghnouj (2018), “The gender gap in educational outcomes in Norway”, OECD Education Working Papers, No. 183, OECD Publishing, Paris, https://dx.doi.org/10.1787/f8ef1489-en.

[36] BOSTES (2016), Board of Studies Teaching and Educational Standards New South Wales, https://www.boardofstudies.nsw.edu.au/ (accessed on 24 June 2019).

[39] CAO (2019), The CAO Handbook 2019, Central Applications Office, Galway, Ireland, http://www2.cao.ie/handbook/handbook2019/hb.pdf (accessed on 24 June 2019).

[40] CAO (n.d.), The CAO: A Guide for Parents and Guardians, Central Applications Office, Galway, Ireland, http://www2.cao.ie/downloads/documents/CAOparentsguide.pdf (accessed on 24 June 2019).

[41] Department of Education and Skills (2018), Irish Education System, https://www.education.ie/en/The-Education-System/ (accessed on 1 October 2019).

[11] European Commission (2016), Sector Reform Contract: Education Reform in Serbia – Strengthening Links with Employment and Social Inclusion, https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/ipa2016-039806.09-serbia-sector_reform_contract_education_reform_in_serbia.pdf (accessed on 17 October 2019).

[23] European Commission (2015), Instrument for Pre-acession Assistance (IPA II) 2014-2020: Serbia European Integration Facility, European Union, Brussels, https://ec.europa.eu/neighbourhood-enlargement/sites/near/files/pdf/serbia/ipa/2015/pf_01_european_integration_facility.pdf (accessed on 17 October 2019).

[25] European Commission (n.d.), Serbia: Assessment in Primary Education, https://eacea.ec.europa.eu/national-policies/eurydice/content/assessment-primary-education-40_en?cookies=disabled (accessed on 23 September 2019).

[10] European Commission/EACEA/Eurydice (2018), Recommended Annual Instruction Time in Full-time Compulsory Education in Europe – 2017/18, https://doi.org/10.2797/616811.

[53] IEQE (2018), OECD Review of Evaluation and Assessment: Country Background Report for Serbia, Institute for Education Quality and Evaluation, Belgrade.

[14] IEQE (2017), Izveštaj o Rezultatima Završnog Ispita na Kraju Osnovnog Obrazovanja i Vaspitanja u Školskoj 2016/2017 [Results of Primary School Final Examination 2016-2017], Institute for Education Quality and Evaluation, Belgrade.

[17] IEQE (2017), Izveštaј o spoљašњem vrednovaњu kvaliteta rada obrazovno-vaspitnih ustanova u školskoј 2016/2017 [Report on External Evaluation of the Quality of Work of Educational Institutions in the 2016/2017 School Year], Institute for Education Quality and Evaluation, Belgrade.

[16] IIE (2017), Izveštaj: Ispitivanje Potreba za Stručnim Usavršavnjem [Report: Examining the Needs for Professional Improvement], Institute for Improvement of Education, Belgrade.

[54] Kaldi, S., D. Filippatou and C. Govaris (2011), “Project-based learning in primary schools: Effects on pupils’ learning and attitudes”, Education 3-13, Vol. 39/1, pp. 35-47, https://doi.org/10.1080/03004270903179538.

[43] Ku, K. (2009), “Assessing Students’ Critical Thinking Performance: Urging for Measurements Using Multi-Response Format”, Thinking Skills and Creativity, Vol. 4/1, pp. 70-76, https://www.sciencedirect.com/science/article/pii/S1871187109000054 (accessed on 17 October 2019).

[27] London Region MISA PNC (2011), Comment Framework: Progress Reports and Report Cards, London Region MISA PNC, http://misalondon.ca/PDF/a&e/Comment_Framework_Feb_2011.pdf (accessed on 20 June 2019).

[20] Matura Working Group (2017), Opšta, Stručna i Umetnička Matura i Završni Ispit u Srednjem Stručnom Obrazovanju [General, Professional and Art Matura and Final Examination in Secondary Education], Ministry of Education, Science and Technological Development, Belgrade.

[58] Ministère de l’Éducation nationale et de la Jeunesse (2018), Baccalauréat 2019, https://www.education.gouv.fr/cid60987/bac-2016-questions-reponses.html#D%C3%A9roulement_des%20%C3%A9preuves (accessed on 17 June 2019).

[49] Ministère de l’Éducation nationale et de la Jeunesse (2018), Communication en conseil des ministres : la réforme du baccalauréat [Communication at the Council of Ministers: the reform of the Baccalaureat], https://www.education.gouv.fr/cid126564/communication-en-conseil-des-ministres-la-reforme-du-baccalaureat.html (accessed on 31 May 2019).

[30] Ministère de l’Éducation nationale et de la Jeunesse (2018), Évaluations diagnostiques en CP [Diagnostic Evaluation in CP], http://eduscol.education.fr/cid119562/evaluation-diagnostique-en-cp.htm (accessed on 17 June 2019).

[12] MoESTD (2019), Zakon o Osnovama Sistema Obrazovanja i Vaspitanja [Law on the Foundations of the Education System], Ministry of Education, Science and Technological Development, Belgrade.

[18] MoESTD (2014), Pravilnik o Organizaciji i Sprovođenju Ispita [Rulebook on the Organization and Conduct of Examinations], Ministry of Education, Science and Technological Development, Belgrade, http://www.tehniar.com/?page_id=189 (accessed on 8 September 2019).

[13] MoESTD (2013), O Opštim Standardima Postignuća za Kraj Opšteg Srednjeg Obrazovanja i Srednjeg Stručnog Obrazovanja u Delu Opšteobrazovnih Predmeta [Rulebook on Competence Standards in General Subjects for the End of Secondary Education], Ministry of Education, Science and Technological Development, Belgrade.

[29] NCCA (n.d.), Curriculum Online, https://www.curriculumonline.ie (accessed on 5 June 2019).

[2] New Zealand Ministry of Education (2007), The New Zealand Curriculum, http://nzcurriculum.tki.org.nz/content/download/1108/11989/file/The-New-Zealand-Curriculum.pdf.

[34] Nusche, D. et al. (2011), OECD Reviews of Evaluation and Assessment in Education: Norway 2011, OECD Reviews of Evaluation and Assessment in Education, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264117006-en.

[38] OECD (2018), “How do admission systems affect enrolment in public tertiary education?”, Education Indicators in Focus, No. 63, OECD Publishing, Paris, https://dx.doi.org/10.1787/41bf120b-en.

[22] OECD (2017), Education at a Glance 2017: OECD Indicators, OECD Publishing, Paris, https://dx.doi.org/10.1787/eag-2017-en.

[7] OECD (2017), Education in Lithuania, Reviews of National Policies for Education, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264281486-en.

[6] OECD (2016), Education Policy Outlook: Korea, OECD, Paris, http://www.oecd.org/education/policyoutlook.htm (accessed on 10 January 2019).

[8] OECD (2015), Education at a Glance 2015: OECD Indicators, OECD Publishing, Paris, https://dx.doi.org/10.1787/eag-2015-en.

[9] OECD (2013), PISA 2012 Results: What Makes Schools Successful (Volume IV): Resources, Policies and Practices, PISA, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264201156-en.

[1] OECD (2013), Synergies for Better Learning: An International Perspective on Evaluation and Assessment, OECD Reviews of Evaluation and Assessment in Education, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264190658-en.

[15] OECD (2012), Grade Expectations: How Marks and Education Policies Shape Students’ Ambitions, PISA, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264187528-en.

[21] OECD (2012), Strengthening Integrity and Fighting Corruption in Education: Serbia, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264179646-en.

[44] OECD (forthcoming), OECD Reviews of Evaluation and Assessment in Education: Albania, OECD Reviews of Evaluation and Assessment in Education, OECD Publishing, Paris.

[57] Ofqual (2018), Malpractice for GCSE, AS and A level: Summer 2018 Exam Series, Ofqual, Coventry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/765268/Malpractice_for_GCSE_AS_and_A_level_summer_2018_exam_series.pdf (accessed on 10 June 2019).

[42] Ofqual (2012), International Comparisons in Senior Secondary Assessment: Full Report, Ofqual, Coventry, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/372211/2012-06-12-international-comparisons-in-senior-secondary-assessment.pdf (accessed on 10 June 2019).

[28] Ontario Ministry of Education (2010), Growing Success: Assessment, Evaluation and Reporting in Ontario Schools, Ontario Ministry of Education, Toronto, http://edu.gov.on.ca/eng/policyfunding/growSuccess.pdf (accessed on 23 May 2019).

[32] Orsmond, P. and S. Merry (2011), “Feedback alignment: Effective and ineffective links between tutors’ and students’ understanding of coursework feedback”, Assessment & Evaluation in Higher Education, Vol. 36/2, pp. 125-136, https://doi.org/10.1080/02602930903201651.

[31] Qvortrup, A. and T. Keiding (2015), “Portfolio assessment: Production and reduction of complexity”, Assessment & Evaluation in Higher Education, Vol. 40/3, pp. 407-419, https://doi.org/10.1080/02602938.2014.918087.

[24] Rogers, W. (1993), “Principles for fair student assessment practices for education in Canada”, Canadian Journal of School Psychology, Vol. 9/1, pp. 110-127, https://doi.org/10.1177/082957358500900111.

[52] Singapore Examinations and Assessment Board (2019), PSLE, https://www.seab.gov.sg/home/examinations/psle (accessed on 4 March 2019).

[51] The Independent (2017), “Government to spend half a million pounds explaining confusing GCSE exam reforms”, https://www.independent.co.uk/news/education/education-news/gcse-exam-results-government-watchdog-ofqual-500000-half-million-explain-confusing-education-a7861586.html (accessed on 3 June 2019).

[26] The Star (2017), Report card, curriculum changes on the way in Ontario, Toronto Star, https://thestar.com/news/queenspark/2017/09/06/report-card-curriculum-changes-on-the-way-in-ontario.html (accessed on 23 May 2019).

[35] Timperley, H. et al. (2007), Teacher Professional Learning and Development Best Evidence Synthesis Iteration [BES], Ministry of Education of New Zealand, Wellington, http://www.oecd.org/education/school/48727127.pdf (accessed on 17 October 2019).

[56] UCAS (n.d.), Extended Project Qualification (EPQ), https://qips.ucas.com/qip/extended-project-qualification-epq (accessed on 14 January 2019).

[50] UK Government Department for Education (2018), Get the Facts: GCSE Reform, https://www.gov.uk/government/publications/get-the-facts-gcse-and-a-level-reform/get-the-facts-gcse-reform (accessed on 31 May 2019).

[48] van Teijlingen, E. and V. Hundley (2002), “The Importance of Pilot Studies”, Social Research Update 35, https://doi.org/10.7748/ns2002.06.16.40.33.c3214.

[47] World Bank (2016), Improving People’s Lives in Tajikistan, http://projects-beta.worldbank.org/en/results/2016/04/13/improving-peoples-lives-in-tajikistan (accessed on 31 May 2019).

[46] World Bank (2015), The World Bank Group - Tajikistan Partnership Program Snapshot.

[45] World Bank (2013), National Testing Center Inaugurated in Dushanbe, http://www.worldbank.org/en/news/press-release/2013/12/20/national-testing-center-inaugurated-in-dushanbe (accessed on 4 March 2019).

[19] World Bank (2012), SABER student assessment country report : Serbia 2012 (English), World Bank, Washington DC, http://documents.worldbank.org/curated/en/905731468102269374/pdf/799420WP0SABER0Box0379797B00PUBLIC0.pdf (accessed on 10 June 2019).