Chapter 4. Undue recognition of learning achievement in primary and secondary education in Ukraine1
The chapter examines undue recognition of learning achievement of students in Ukraine: intentional over-marking or under-marking by teachers for personal gain, such as money, gifts or services to the teacher or the school.
Serious weaknesses in the assessment of learning outcomes, combined with a culture of acceptance of gifts, provide teachers with opportunities to mismark - and parents with the expectation that marks are negotiable. All sides involved have reasons to engage in the integrity violation: parents in the conviction that better marks can secure admission to good higher education, schools because of their dependence on parental contributions and teachers because of their unsatisfactory income.
The chapter recommends ways to improve classroom assessment, including wider and earlier use of low-stake, external assessments; raising awareness about the limited importance of school marks for admission to higher education institutions (HEI); and the adoption of marking moderation. The report notes that incentives for malpractice might be diminished if teacher salaries were raised, but only after evaluation of actual teacher income and working conditions to determine what changes to compensation would be fair and effective.
Regulatory and policy background
At the time of this integrity review, the learning achievement of school students was assessed in three ways:
-
By classroom teachers on a continuous basis in every year, at the end of each semester, and at the end of the school year, as stipulated in the 1999 Law on Secondary Education.
-
At the end of primary (Year 4) lower secondary (Year 9) and upper secondary (Year 11) education through a state final exam (державна підсумкова атестація) set by the Ministry of Education and Science (MoES) and administered by the schools, which after Years 9 and 11 provides students with a School Leaving Certificate (SLC) (Annex 4.A1, ref. 1).
-
Through an external standardised HEI entry test at the end of Year 11 (External Independent Testing – EIT or зовнішнє незалежне оцінювання), administered by the Ukrainian Centre for Education Quality Assessment (CEQA) and its regional centres on behalf of the MoES. Since 2014, the EIT has also replaced the exit exam (state final exam) at the end of schooling. Thus, since 2014 the EIT results count for HEI admission and as the final marks for the School Leaving Certificate for all Ukrainian school leavers.
Additionally, central and regional inspections can administer external diagnostic tests to monitor quality in the education system, which do not have an impact on the school career of students. Table 4.1 presents a summary of all forms of assessment of learning achievement in schools in Ukraine.
In classroom assessment, teachers mark performance in each subject on a 12-point scale established by the MoES: marks 1-3 are unsatisfactory and marks 10-12 very good/outstanding. The transition of students between all levels of school education is based on the results of biennial and annual assessments administered by the teachers.
The grade averages achieved in the last few school years, and particularly in the final year, are reflected in each student’s School Leaving Certificate (SLC), which is a requirement for progression to post-secondary, non-tertiary or tertiary education. However, for students aspiring to HEI, the marks achieved in the External Independent Testing (EIT) introduced in 2008 have become much more important than high school certificate marks. Ukraine’s Law on Higher Education of 2014 limits the weight that can be given to the SLC marks to a maximum of 10% of the composite score that determines which students are given places at HEIs applied for, and which of the accepted students will get state funding. Also, starting with the 2015/16 academic year, all school leavers must take the EIT in 3 subjects (Ukrainian language and literature, foreign language and either mathematics or history). These examination results count as the final marks for the School Certificate for all Ukrainian school leavers, and, for those apply to an HEI, as part of their entrance examination scores.
A. Description of integrity risk and violation
Every country’s school system aims to equip students with the knowledge and skills they need for adult life, work and further learning. More than years of schooling, it is learning - or the acquisition of competencies - that improves individual productivity and earnings, with gains for aggregate income in the economy.
Every country trusts its schools to ensure that students master its curriculum and achieve the required standards. Teaching students is not enough; schools also need to prove to stakeholders - national and local government, parents and students themselves – that students have learnt and have achieved desired outcomes. A reliable and effective measurement (assessment) of student progress is paramount for monitoring individual and school system success. It can also be used for improving education quality. The results can guide the design and implementation of programmes to improve teaching and learning in classrooms. They can also be used – and increasingly are used – for holding school leaders, teachers and education policy-makers accountable to stakeholders.
A precondition for trust in the value and reliability of assessment results is that student outcomes are assessed fairly and honestly, without bias or favour. Stakeholders expect to see good student learning rewarded with good marks, poor student learning reflected in lower marks. Biased assessment undermines faith and trust in education and its outcomes, and is therefore especially harmful.
Undue recognition of learning achievement occurs where an assessor knowingly gives a student higher marks than their work merits, or where the recognition due to a deserving student is knowingly withheld. Teachers in any country may over-mark or under-mark individual students for a range of reasons. Some of them might be technical. For instance, if the assessment system that teachers are required to operate is flawed and produces invalid or unreliable results, unfair marking will be unintentional and random, and teachers will not be to blame. Other reasons might be due to personal reasons and incompetence. If individual students are unfairly marked because of the teacher’s personal likes, dislikes or prejudices, combined with ignorance of how to set appropriate standards or assess performance against them, this indicates a lack of professionalism and personal integrity and reflects badly on the teacher, the school and the education system concerned; but the teacher is not necessarily acting corruptly.
Corrupt actions imply that trusted authority is misused for personal benefit (OECD, 2015a). Therefore, a risk of over-marking or under-marking as an integrity violation is present only where the marker mismarks knowingly, and in the hope or expectation of personal gain.
Disparities in student marking
Recent studies conducted by the Ukrainian Centre for Education Quality Assessment (CEQA) show that undue recognition in the form of over-marking is wide-spread, and that there are considerable disparities in the results of classroom-based assessment of students across Ukraine. The studies compare EIT scores, which are external and criterion-referenced, to SLC marks, which are school-based and norm-referenced. Specifically, the Centre has examined the relationship between high levels of achievement in SLC marks that result in gold and silver medals for students, and EIT results.
Based on students’ academic performance, schools decide to award medals at a joint meeting of the Teachers’ Council and the School Council; the decision is approved by the school principal and agreed with the local education administration. Students are eligible to receive a gold or silver medal if they obtain an SLC average score between 10 and 12 (on a 12-point scale) in their final year. Gold medal winners must have at least 10 in all subjects, while silver medal winners may have marks of 9 in one or two subjects. Until 2014 and the new Law on Higher Education that made SLC marks more important than medals, high school medals facilitated access to higher education.
The analysis shows that the percentage of school leavers awarded gold and silver medals has grown every year in recent years as the importance of SLC average for HEI entrance increased. In 2009, when HEI entry depended exclusively on the results of the EIT, and SLC scores could not be taken into account, 4.5% of school leavers were medal-winners. In 2010, when the system changed and SLC scores were taken into account alongside EIT results, the medal-winner percentage jumped to 6.2%. By 2014, the national percentage of medal-winners had reached 8%, though it varied between regions, from 3.5% in Kyiv to 14% in Ternopil. (CEQA, 2014). These figures and other evidence presented below, invite the conclusion that between 2009 and 2014 SLC marks were progressively inflated as schools marked favoured students more and more generously to improve their chances of HEI entry.
The CEQA research provides more evidence along these lines. Table 4.2 shows the EIT scores of medal-winning students across Ukraine in 2014. These highest-performing students have a 10-12 SLC average score (on a 12-point scale) which corresponds to approximately 175-200 EIT average (on a 200-point scale). The average national score for all 193 117 EIT test-takers in 2014 was 149.5 (CEQA, 2014). The data shows that more than 50% of all medal-winning students scored less than 175 points and almost 3% scored lower than the national average, which suggests that some schools heavily over-mark their students.
Where does over-marking most frequently occur? Important discrepancies between the expected medal-level performance and the actual EIT score are recorded in 16 regions of Ukraine scoring below the national average of medal-winners (Table 4.3). With only one exception (Lviv), the largest deviation from the national average is observed in the rural areas of all oblasts.
The CEQA regional analysis by type of schools also shows that lyceum and specialised school medal-winners score closer to national average than medal-winners from general secondary schools where negative deviations from national average are the largest. These results suggest that over-marking is most widespread in general secondary schools.
Table 4.4 offers another comparison of teacher assessment and external assessment based on assessment results in Ukrainian Language and Literature in the Khmelnytskyi region. In each rayon, city and town - 26 in all - the researchers identified how many of the students who were given the highest marks (10-12) by their teachers achieved a corresponding score in the EIT, between 175 and 200. Similarly, they calculated how many of the students given the lowest marks (1-3) by their teachers achieved correspondingly low scores (100-123) in the EIT. They then calculated the ratio of students with high/low EIT scores to high/low SLC scores to reach the percentage “consistently assessed” by their teachers.
School-based assessment leads to many more high scores than external, independent assessment. In 25 of the 26 rayons or cities presented in the table, the number of students with high SLC results far outnumbered the number with high EIT results: on average, high SLC results were twice as common as high EIT results. Conversely, school-based assessment leads to far fewer poor results than does external assessment. Comparison of EIT and SLC results showed classroom-based over-assessment in every oblast/rayon. On average, schools awarded only 22% of students who obtained poor EIT results similarly low SLC results. More than three out of four students who obtain passing scores subsequently received unsatisfactory EIT results.
How much of the undue recognition is intentional and thus an integrity violation?
Is over-marking or under-marking the result of teachers mismarking knowingly and in the hope or expectation of personal gain? Teachers and school principals with whom the review team met were not willing to admit to taking bribes or to acknowledge that corrupt undue recognition happened at their school. However, they rarely, if ever, denied that it happened elsewhere in Ukraine. At some meetings with the review team teachers admitted to being offered bribes by parents.
In interviews carried out during the site visits, the team was able to collect anecdotal evidence from NGOs and parental groups about common forms of undue recognition for personal benefit in Ukrainian schools. They offered these examples:
-
A teacher gives a student higher marks than their work merits because the student’s family have given or promised money, gifts or services to the teacher and/or the school.
-
Because the student’s family has given or promised money, gifts or services, the teacher allows the student to submit work that is not all his/her own: the work has been ‘improved’ by cheating, plagiarism or direct help from the teacher.
-
A teacher gives a student unduly low marks and informs parents that the student’s work is of a lower standard or their chances in the EIT poorer than is actually the case, in order to induce the parents to offer money, gifts or services, in exchange for which the teacher will give the student special help. This may involve giving the student extra help and attention in school (during the hour per week each subject teacher is required by national rules to devote to helping struggling students catch up, or at the end of the school day): or it may involve private tuition out of school (see Chapter 5). Whichever form of help is given, the teacher shows the parents that their child is ‘improving’ by awarding higher school marks after receiving the reward.
-
A teacher gives unduly low marks to children of parents who have not, or not yet, paid them (in gifts or by engaging them as private tutors), by giving those children less help and support in class. This reduces the likelihood that those children will realise their potential in end-of-term or end-year tests. If a parent then decides to pay, the teacher gives the child more help and marks their work more generously.
-
One parent representative reported that teachers at her child’s school fiercely resisted an offer by parents to install CCTV in classrooms to help resolve complaints of unfairly low marking of students whose families were unable or unwilling to make financial contributions.
At the stakeholder seminar in Kyiv in March 2016, which gathered a representative selection of stakeholders from the education system to respond to preliminary results from this review, there was consensus that the undue recognition of student achievement was grounded in the desire of teachers to obtain material benefit.
B. Factors that create opportunities for the violation
Weaknesses in the system for assessing learning achievement
Classroom-based assessment in Ukraine falls short of international good practice and its deficiencies create opportunities for widespread irregularities in the assessment of the learning achievement of students. The analysis in this section reviews assessment practices in Ukraine against the background of international practice and identifies four deficiencies that put valid, reliable, and unbiased marking especially at risk by creating opportunities for integrity violations.
Summative assessment of students, or assessment of learning, aims to summarise what has been learnt so far, in order to record, mark or certify achievements. Its purpose is to monitor educational outcomes, often for purposes of external accountability, and usually results in a test score. Because they result in a test score, students and parents care a lot about the outcomes: summative tests tend to be “high stakes”. Formative assessment or assessment for learning – which includes diagnostic testing - is a range of formal and informal assessment procedures conducted by teachers during learning in order to modify teaching and learning activities to improve student attainment. It usually results in qualitative feedback focusing on what is good about a piece of work and why it is good; also what is not so good and how the work could be improved. The results of formative assessments are “low stakes” or “no stakes”.
Summative assessments can be conducted internally or externally. Internal summative assessment, or school-based assessment, is designed and marked by the students’ own teachers as part of regular classroom instruction. External, or standardised, summative assessment is designed and marked outside schools to ensure that the questions, test conditions, scoring and marking are consistent and comparable among students. External summative assessments (if properly designed) are more likely to be valid – a highly valid assessment ensures that all relevant aspects of student performance are covered by the assessment – and reliable – a highly reliable assessment ensures that the assessment is accurate and will produce consistent results, whoever is the assessor and whatever the assessment occasion.
Source: Elaborated by the authors based on OECD (2013b), Synergies for Better Learning: An International Perspective on Evaluation and Assessment, https://doi.org/10.1787/9789264190658-en.
The first weakness of classroom assessment in Ukraine is that it relies exclusively on internal summative assessments by classroom teachers throughout primary and secondary schooling (see Box 4.1 for definitions). Assessment items (tasks) are set by classroom teachers, administered by teachers, and marked by classroom teachers based on guidelines provided by the Ministry of Education and Science (MoES). While its subjects and format are set by the MoES on the basis of the national curriculum, the content of tasks is determined by each school separately, and the marking is done by the class teachers in their respective subjects, without a reference to nationally-normed criteria or competency framework. Classroom assessments in Ukraine take place frequently, so there are many occasions on which undue recognition of learning achievement could occur. In secondary schools, tests to measure and compare student performance typically take place not only at the end of every term, but several times during each term.
In the majority of OECD member countries, school students take at least one external test or examination, either during their school career or as a school-leaving test. The first opportunity that students in Ukraine have to learn about their performance through an external, standardised and nationally normed test is at the conclusion of their schooling when they take the EIT.
External and independent assessments are useful at the time of school leaving or when a high-stakes decision, such as HEI entry, must be taken; however, their principal benefit is to earlier stages of schooling. The information that external examinations provide helps to ensure the integrity, reliability and comparability of school-based assessment. In countries where students have both internal and external summative assessments during their school careers, it is possible for students and families to compare the marks awarded to each student in their internal and external assessments, to identify significant differences in how they performed in the two assessments, and to seek explanations for any differences found. Teachers benefit from the assessments, since they are provided reliable benchmarks on student progress within the curriculum, and a framework that they can use to develop and calibrate their classroom-based assessment activities.
The great majority of OECD countries have national summative assessments with no stakes for students, and these assessments are intended either to provide formative and diagnostic information to teachers, or to monitor the overall performance of the education system. Out of 36 OECD education systems studied, 29 had such assessments at primary level and 27 had such assessments at lower secondary level (OECD, 2013b).
In the United Kingdom (England), pupils aged 10-11 in their last year of primary school are assessed by their teachers, and also take national standardised tests in English and mathematics. Both assessments measure the mastery of the national curriculum and national curriculum level reached using the same criteria and the same marking scale.
Teachers look at and learn from any differences between their own assessments of each pupil and how that pupil performed in their test. Did the teacher’s assessment over-rate or under-rate the pupil? Did the teacher fail to prepare the pupil for some topic or problem that came up in the test? Or, did that pupil just have a bad day, or a personal problem, and perform atypically in the test? Because the test is external and standardised and national, teachers and schools can also compare their pupils’ test results with those of pupils in other primary schools locally, regionally and nationally, and look at and learn from any differences.
The second weakness and source of opportunity for assessment malpractice is that national guidance on classroom-based assessment is too general to signal clearly to teachers which marks students should be given in every set of circumstances. The MoES guidance, which teachers use when assessing, is not comprehensive or specific enough to ensure that teachers will interpret it in the same way, and that different teachers will award the same mark to any given piece of work. The guidance gives a broad description of each point on the 12-point marking scale, and broad criteria for performance at each point on that scale, but is too short and general to work effectively as a marking handbook. The MoES also developed indicative requirements for the assessment of students in primary and lower secondary education. It is more specific than the guidance but does not cover all levels of education and is not linked to educational programmes and standards. Overall, subject teachers need much more detail if they are to understand clearly which mark is appropriate for the work they must assess. They need to know what learning objectives a pupil in their class should have achieved by every point at which they are to be assessed. They need to be told how to judge whether and to what extent those learning objectives have been achieved. They also need to be equipped with clear information and examples, so that they can relate their students’ achievements precisely to the points on the scale and distinguish clearly and fairly between each point.
Teachers’ professional judgment is not able to fill in the gaps in assessment guidance. As the OECD study Synergies for Better Learning noted,
“While most countries regulate the use of a particular marking scale [. . .], especially in secondary education, this does not mean that the meaning of a particular mark is necessarily equivalent across schools. Even if schools use the same marking scale, they may have different marking criteria. It is difficult to ensure that the marks awarded in one school align with similar marks in another school. In addition, the same teacher will not necessarily be consistent in the application of criteria across students and over time. Such inequity in marking becomes problematic when […] marks are used for high-stakes decisions” (OECD, 2013b).
It is particularly problematic where, as in Ukraine, marking may be influenced by the potential for personal benefit.
The third weakness and source of opportunity is that teachers are insufficiently trained in assessment to compensate for the lack of specificity in the guidance. Initial teacher training does not provide them with the solid grounding in assessment principles and supervised practice that is needed if summative assessments are to be valid and reliable. The review team interviewed teachers who had recently completed a refresher training course undertaken every fifth year. They reported that the refresher course module on assessment had enabled them to understand the national guidance, and that before coming on the course they had not understood it. Refresher training seemed to have succeeded where initial teacher training failed. However, it was unfortunate that these teachers had been assessing students without understanding what the national assessment guidance required them to do, for at least their first five years in the profession. The review team was not able to judge whether the teachers’ confidence in their new knowledge survived their first experiences of student assessment after the course.
The fourth weakness is that there is limited cross-checking or validation of teacher-based assessments within the education system, for example, by requiring senior colleagues within the school, or external moderators, to check or counter-sign all “high stakes” internal summative assessments. There is not a well-established arrangement through which external moderators test, replicate, and validate internal summative assessments, as is implemented in some other schooling systems (see Box 4.3).
In Ukraine, teachers – subject to the school principal’s review - are the sole and final arbiters of their students’ marks. School principals and local education inspectors may make classroom visits to observe teachers. However, these visits are not focused on validating marking practices, and provide weak checks on mismarking. Leaving assessments in the hands of a single teacher exposes students to the risk of mismarking that arise both from error and corrupt practice. Experience and seniority alone – without external validation - provide no assurance of reliability.
These deficiencies, taken together, make it likely that large numbers of students will be given marks higher or lower than their learning deserves. Some mismarking will be unintentional and unconscious: the teachers concerned wish to assess their pupils fairly, but cannot, because the national assessment system does not give them the necessary training, guidance, information, support and supervision. However, every one of these deficiencies opens the door to deliberate mismarking and presents corruptible teachers with opportunities to reap personal benefits by manipulating students’ marks. Moreover, the many separate assessments currently required during a school career leave a permanently open window of opportunity for corrupt bargains. A teacher who wishes to seek or receive donations and favours in return for better marks or extra help has the opportunity to do so at least every term.
A key way to increase the reliability of assessment and marking is to systematically implement moderation procedures that aim to ensure the quality and comparability of assessment judgement. This may involve teachers cross-marking each other’s assessments or discussing student performance in groups, or a competent external organisation systematically checking school-based marking. While in many settings moderation occurs informally within and between schools and may not be documented, some education systems have introduced systematic arrangements for moderation. This is particularly the case in education systems where centrally developed examinations with high stakes for students are corrected and marked locally by teachers.
-
France: Teachers examine their own students through continuous classroom assessment, while teachers from another school are responsible for marking written examinations leading to diplomas or certification.
-
Denmark: Centrally appointed external examiners correct examination papers and are assisted through national guidance materials such as performance criteria, exemplars, rubrics and keys. There is also moderation of marking by external examiners who attend oral examinations.
-
The Netherlands: Examinations are corrected by the students’ own teacher and moderated by a teacher from another school using a central scoring protocol. The school boards are responsible for the proper handling of the procedures. In case of disagreement, external moderation by a competent body is provided.
-
Queensland, Australia: The examination system is school-determined and based, but achievement standards and scoring are externally moderated. Moderation processes for the Senior Certificate (Year 12) involve subject-based panels of expert teachers providing advice to schools on the quality of their assessment programme and their judgements of quality of student performance based on sample portfolios. The system involves follow-up with schools where panels identify issues regarding assessment and standards. There is negotiation of the final results to be recorded on the Senior Certificate (Sebba and Maxwell, 2005 in Santiago et al., 2011). Similarly, procedures adopted by educational jurisdictions and particular schools for moderating internal summative teacher judgements (so-called A-E ratings) also facilitate common understanding of year level proficiency standards and foster the development of professional learning communities that can provide crucial support for improving opportunities for student learning and building teacher capacity.
-
New Zealand: An external moderation system is also in place to ensure the dependability of internal assessments in Years 11-13. The New Zealand Qualifications Authority directly checks the quality of internal assessment through a sampling approach. Schools are required to submit 10% of internally assessed student work for NZQA moderation to make sure the assessment is appropriately aligned with standards. The moderation process does not affect the marks assigned to assessment samples by teachers, but is intended to provide feedback to teachers and to inform future assessment policy development at the system level.
Source: adapted from OECD (2013b), Synergies for Better Learning: An International Perspective on Evaluation and Assessment, https://doi.org/10.1787/9789264190658-en; Santiago, P. et al. (2011), OECD Reviews of Evaluation and Assessment in Education: Australia 2011, https://doi.org/10.1787/9789264116672-en; Nusche, D. et al. (2012), OECD Reviews of Evaluation and Assessment in Education: New Zealand, https://doi.org/10.1787/9789264116917-en.
A tradition of gift-giving and a culture of acceptance
The citizens of Ukraine have a low opinion of the probity of staff working in the public sector. Transparency International’s Corruption Perception Barometer 2013 (Transparency International, 2013) found that 82% of respondents in Ukraine thought that public officials and civil servants in general were corrupt/extremely corrupt; 69% considered that education systems were corrupt/extremely corrupt; and 37% said they had actually paid a bribe to 1 of 8 services in the previous 12 months, among which education. The 2015 Corruption Perception Survey (Razumkov Centre, 2015) found that 20.9% of the population considered school education to be “absolutely corrupted”; 31.8% considered it to have “widespread corruption”; and 29.3% thought it had “certain cases of corruption”. Just 10.1% thought it had no corruption, while 7.9% found it difficult to answer the question.
Parents with means, who believe that payments to teachers - like payments elsewhere in the public sector - will improve their children’s marks and prospects, are often willing to offer substantial amounts of money to teachers and school leaders. This follows a well-established pattern of paying a bribe for a public service which might otherwise be difficult to obtain. The vast majority of parents in public schools make regular payments to their children’s class teachers and school administrators through so-called “class” and “school” funds, discussed in detail in Chapter 2. In a recent survey of parents, 91% said that such funds existed in their children’s schools; 68% said that their schools had both class and school funds; 47% of respondents recognised contributing to the class and 41% to school funds each month. While the primary purpose of these funds is to meet the costs of school operation, 45% of the parents surveyed said that class funds were also spent on buying gifts for, or making additional payments to, the class teacher. A smaller proportion of respondents suggested that payments were made to teachers from school funds (ERA, 2014).
This practice is supported by a well-established tradition of giving gifts to those who are in a position to benefit the giver, as a sign of recognition. This is a practice particularly well established in educational institutions. School teachers often receive presents or useful services from pupils and their parents. These gifts may be unsolicited or may be a one-off exchange between a parent and a teacher.
Gift-giving is particularly likely to be associated with undue recognition of learning achievement if it happens at the end of the term, the school year or the student’s last year at school. These are all times when students have their performance assessed and receive marks from their teachers. For teachers or others who wish to manipulate marking for personal gain, the class or school fund provides a perfect means of collecting substantial sums or services from a parent without arousing suspicion. Similarly, substantial gifts of money or services can be portrayed as a simple and natural mark of the parent’s gratitude. A non-governmental organisation worker told the review team that he, like all the parents in his child’s class, had been asked for a “gift” of UAH 3 000 on his child’s graduation. Though this was a large amount of money, he had not found the request surprising or unreasonable, given the importance of graduation and the tradition of gift-giving in Ukrainian society.
C. Factors that create incentives for the violation
Parental information and motivation
Most Ukrainian parents want their children to go to HEIs after leaving school (IED, 2015). They are right in believing that their children’s academic performance is important to getting there. The marks achieved in school-based tests are important to the prospects of Ukrainian students of all ages. Low marks in the early years of education limit a student’s chances of entering or remaining in a specialised or elite secondary school that selects its pupils by ability.
Without high marks, students will not be eligible for prizes and awards, such as entering for scholarships and Olympiads; certificates from such competitions improve a student’s chances of entering a prestigious HEI. The SLC is a requirement for progression to vocational or higher education. Good SLC marks help a student’s chances of entering a good HEI with a state-funded place, though the most recent (2014) Law on Higher Education states that they may not count for more than 10% of the composite score. Poor school marks are a signal that students may not pass the EIT or not well enough to attend the HEI they prefer, or that they may not qualify for a state-funded place.
Parents are strongly motivated to have their children earn high marks, but they are not always well-informed about entry requirements for higher education, and thus many overestimate the importance of school marks. Asked what they believed to be of crucial importance when seeking admission, survey respondents gave the answers in Table 4.5.
The correct answer is “results of External Independent Testing (EIT)”, but only one in three parents (34.3%) gave this answer. An EIT pass is the only mandatory requirement for HEI entry, and has been since 2008. To sit the EIT a school leaver must have a School Leaving Certificate (SLC) to show that they have completed school education, but even the lowest average SLC marks do not prevent students from sitting the EIT and getting into the HEI of their choice if they do well. Good SLC grade averages may help a student get into a competitive HEI, other things being equal. However, they are not as relevant as they used to be before Ukraine’s 2014 Law on Higher Education limited the weight that can be given to SLC marks to a maximum of 10% of the composite score. It is therefore remarkable that 18.4% of parents, answering a survey in 2015, thought that the SLC grade point average (GPA) was more important than the results of EIT testing.
A factor that contributes to the misperceptions might be the regular changes in the EIT since its inception: in the way its results are used for admission to HEI; the relative weight of school graduation results in the composite score; and the selection of subjects that are available and/or mandatory for testing (Kovalchuk and Koroliuk, 2014). For instance, in 2008 and 2009, the EIT results were counted toward both school-leaving and higher education admission examinations; in 2010 they were counted only toward admission examination and students had to take a separate examination to graduate from high school; and until 2014 the relative weights of the SLC, the EIT, and the HEI admission examinations were changing before each admission campaign. To reflect these adjustments, the website of the Centre for Education Quality Assessment, the entity that manages the EIT, even features a section devoted to the “specifics” of the EIT in that year. Each year, the part of the website of the MoES that is devoted to the HEI admission campaign also provides an update in the way HEIs can use the EIT results.
Perceptions of inadequate compensation of teachers
Low salaries of teachers are a common explanation for the problems in the education system in Ukraine, including for those concerning malpractice. When the survey School and Reform 2015 (IED, 2015) asked “Which current problems of secondary education in Ukraine do you consider are most serious and require to be solved?” - allowing respondents to give more than one answer - 87.9% of school principals and 62.1% of teachers mentioned “Low salary, reduction in educationalist social status”. Asked what high priority steps should be taken to improve secondary education, 52.1% of parents said “Increase teachers’ remuneration”, as did 80.4% of teachers and 90.2% of school principals.
Teachers in Ukraine interviewed by the review team also consistently discussed remuneration and agreed that the statutory pay of education professionals, while adequate for a single person to live on, was inadequate to support a family. A group of trainee teachers, asked how they expected to survive on their salary once they became teachers, mentioned options such as working in a private school, giving private tuition, doing extracurricular work on an international project or going abroad to work, or having another part-time occupation, such as growing vegetables or running a small business.
The salary regulations allow one to calculate the statutory salary of teachers in Ukraine at different points of their career. Statutory salaries are scheduled salaries determined by official pay scales. The statutory salaries of educators in Ukraine are lower than the salaries of workers in other sectors of the economy with comparable levels of educational attainment. In the second half of 2016, the starting and mid-career2 salaries of teachers in primary and secondary schools in Ukraine were 30% and 40% of the average income of workers in finance, insurance, scientific and technical professions respectively. The statutory income of new teachers was 60% lower than average income in the civil service and that of mid-career teachers was only half of that average income (Table 4.6).
Viewed in comparison to teachers elsewhere in Eastern and Central Europe the statutory salaries of starting and mid-career teachers in Ukraine are low (see Figure 4.1). After adjusting for differences in purchasing power, the annual statutory salary of teachers in Ukraine in 2013 was USD 5 517 in the beginning of their career and USD 6 282 after 15 years of service (mid-career). The country with the next lowest level of teacher income is Hungary, where the annual starting salary was USD 10 647 and USD 13 061 after 15 years in the profession, while the OECD average starting salary was USD 31 013, and USD 42 825 for mid-career teachers.
Increases in teacher pay are a recurrent theme in policy discussions in Ukraine. Over the past decade the salaries of education professionals have been raised several times, and at the time of this integrity review further raises were under discussion. Increased teacher salaries do not offer, however, a clear path to improvements in integrity in education. There are two reasons for this.
First, there is no direct and simple link between low teacher salaries and the willingness of teachers to engage in – or refrain from - malpractice, such as intentional mismarking. Teachers and school leaders may act with integrity despite having a financial incentive to the contrary. Conversely, even well-compensated teachers who work in schools where leaders create performance pressures backed by a culture of fear, intimidation and retaliation may engage in “organised and systemic wrongdoing”, such as manipulation of test results (New York Times, 2015).
Second, there is no official information on how many teachers in Ukraine earn the low statutory salaries shown in Figure 4.1, and thus for how many teachers an increase in salary might change their incentives to engage in malpractice. Government policy (Annex 4.A1, refs. 2 and 3) permits teachers to earn supplementary monthly compensation for additional tasks such as checking homework, or managing a school museum. Table 4.7 lists a common combination of such supplementary tasks and additional payments. These calculations demonstrate that teacher compensation can be increased to levels well above the statutory salary shown in Figure 4.1 – and above the top of the teacher salary scale in some OECD member states, including Estonia, Hungary and the Slovak Republic. Not every teacher can benefit from supplementary payments, but most of the additional tasks for which these payments are provided are available at almost every school in the country.
Note: Countries are ranked in descending order of starting salaries for lower secondary teachers with typical qualifications.
1. Actual base salaries.
2. Salaries at top of scale and typical qualifications, instead of maximum qualifications.
3. Salaries at top of scale and minimum qualifications, instead of maximum qualifications.
4. Includes average bonuses for overtime hours.
5. The typical qualifications of starting teachers differs substantially from the typical qualifications of all the current teachers.
6. Data could not be verified as being based on full-time equivalents. Salary at top of the pay scale includes a selection of typical compensation payments for additional work and specific working conditions for teachers with a senior professional category who work double the minimum standard workload (stavka) of 18 hours.
Source: OECD (2015b), Education at a Glance 2015: OECD Indicators, https://doi.org/10.1787/eag-2015-en; Ukraine Decree of the Cabinet of Ministers of Ukraine No. 102 of 15 April 1993, with amendments; Decree of the Cabinet of Ministers of Ukraine No. 1298 of 30 August 2002; Decree of the Cabinet of Ministers of Ukraine No. 373 of 23 March 2011; Decree of the Ministry of Education and Science of Ukraine No. 557 of 26 September 2005, with amendment No. 991 of 17 August 2016.
Official statistics on public education spending records only aggregate expenditure on salaries from the national budget for education. There is no data on how much teachers actually earn in Ukrainian schools – how many of the teachers are compensated for what additional work, how many teachers work more than the standard number of hours and thus can earn more, and what additional bonuses they might be receiving from parents. A recent survey of teachers revealed large differences in their self-reported official monthly income, from as low as UAH 1 000 to as high as UAH 5 000 (IED, 2015).
Dependence on parental contributions
The public schooling system of Ukraine makes extensive use of financial support of households. Household contributions are obtained through school and class funds, collected on a voluntary basis and managed ad-hoc or through charitable organisations established and run by parents for the school (ERA, 2014). School administrations look regularly to students’ parents to cover unfunded school costs. The item for which parents paid the most in 2014 was the cost of procuring materials, equipment and inventory (62%) (SSSU, 2016). Some 87% of 357 parents surveyed in 2014 in selected oblasts in Ukraine stated that they paid for maintenance services, 54% claimed to have also contributed to the purchase of materials and technical equipment, 29% helped procure furniture and carpets, 40% bought hygiene and cleaning products, and 31% acquired textbooks and learning materials for the school libraries (ERA, 2014). The near-universality of these funds helps to explain why only 7.9% of parents surveyed in School and Reform 2015 (IED, 2015) absolutely agreed with the statement “secondary education in Ukraine gives everyone equal opportunities to study for free”. Some 91% of parents say that school funds and/or class funds exist in their children’s schools (ERA, 2014).
Schools, principals and teachers are dependent on the financial contributions, practical help and goodwill of their students’ parents. The support parents provide can play a valuable role in helping them achieve their educational mission. School leaders and teachers therefore believe it is important to keep parents happy, and to convince parents that their children are being well taught. Giving children high marks and glowing progress reports preserves parental goodwill and trust in the school. That being so, teachers and principals have strong incentives to give undue or undeserved recognition of learning achievement. They are also less likely to see undue recognition for financial gain as corrupt where they think they are acting “for the sake of the school”.
D. Policy options
Closing the opportunities for malpractice
Improve the assessment of student achievement through three types of reforms
Review and improve the national assessment framework and guidance. The first step towards a better assessment practice in the nation’s schools is to review and improve the national assessment framework and guidance. This should be led by the MoES, working in collaboration with the central School Inspectorate and representatives of classroom teachers. If teachers are to assess and mark students’ work with accuracy, reliability, and without bias, they need well-specified procedures and assessment criteria (OECD, 2013b). An improved national assessment framework should be developed for Ukraine, which should include the elements described below.
-
Accurate and reliable assessment requires learning objectives that describe what pupils should know and be able to do, in each subject, by the end of each school year. Ideally, these learning objectives should be closely aligned to the requirements of the national curriculum (as they are, for example, in England). They should also be SMART objectives: Specific, Measurable, Achievable, Relevant and Time-limited.
-
There should also be success criteria for each curriculum subject for each school year. Success criteria, which should be clear and comprehensive, enable teachers to judge whether, and to what extent, learning objectives have been met.
-
Assessment would benefit from a marking scale with enough points to enable teachers to distinguish between pupils whose performance is measurably, verifiably different but no more.
-
It is also important to provide examples of the performance against learning objectives to be expected at each scale point, so that teachers can judge how far learning objectives have been met, and can do so reliably and consistently. As an accompanying measure, it is important to give more time and attention to assessment and marking skills in teacher training, both in initial teacher training and in continuing professional development.
-
Provide pre-service and in-service training for teachers in formative/diagnostic assessment, as well as summative assessment, so that they can be continually assessing each pupil’s classroom performance in relation to their learning objectives and identifying areas in which pupils need extra help to catch up, between tests.
-
Schools should be organised, resourced and have time in their curriculum to provide extra individualised help where necessary. This will also help to reduce the perceived need for private tutoring (discussed in detail in Chapter 5).
-
Reducing opportunities for mismarking requires greater involvement of parents. The new system should include arrangements for class teachers to give parents regular feedback on their children’s progress and performance, and discuss with them what can be done to address any learning problems. Teachers should share with parents how their child’s test performance compares with that of classmates and with expected standards, and do this in writing, within a common format.
Make wider and earlier use of low-stakes, external and independent assessment to improve the consistency and integrity of marking
Second, the review team recommends the wider and earlier use of low-stakes, external and independent assessment. End-year tests at the end of primary school and the end of lower secondary school should be replaced by common national tests, set and marked by external test developers. External independent testing would identify where teachers are over-marking or under-marking certain pupils, in a way that further assessments by the same teacher cannot. It would identify students falling behind their peers while there is still time for the school system to take remedial action (avoiding the need to resort to private tutoring). It would also identify students whose potential has been underestimated. Promising directions with respect to assessment are under consideration: the MoES proposed in 2016 to introduce an EIT at the end of Year 9, after the completion of lower secondary education.
In addition, student achievement information from external independent testing can be used to inform other education policies. Comparable information can be used to: design and implement programmes to improve teaching, learning and teacher training programmes; identify struggling students so that they can get the support they need; and provide suitable help and training to low-performing teachers and schools. It can also be used to hold school leaders, teachers and education policy-makers accountable to students and their parents; to those who fund education; to those who take on school leavers (employers, HEIs, colleges); and to the general population (OECD, 2013b).
Improvements in the assessment system that follow these lines will also strengthen the capacity of schools to resist parental pressure to treat students more favourably. The successful implementation of these suggestions will lead to a more reliable and trusted assessment system.
Support the adoption of marking moderation to achieve more consistent and reliable marking
There are many varieties of moderation in use throughout the OECD, as Box 4.3 illustrates. The ministry, in consultation with education stakeholder throughout the country, should draw upon international practice to develop a model of moderation that is adapted to the needs and circumstances of Ukraine. In doing so, it should ensure that moderation is used as a means by which to improve the consistency and integrity of marking, and to support teacher professional development. Linking moderation to teacher careers and compensation should, however, be avoided.
Eliminating the incentives for malpractice
Raise awareness about the EIT and the SLC
Parents often misperceive how school assessment results matter for successful graduation and admission to higher education, Reducing misperceptions through better communication, well ahead of the graduation and HEI admission, can help to curb parental motivation to influence classroom assessment results through gifts and favours. Parental misperception can also be reduced by minimising the frequency of changes to the EIT and how it is used for admission. Where changes are unavoidable, they should be better communicated than at present.
Focus on teacher working conditions and on developing reliable information on earnings before adopting changes to compensation
Across-the-board salary increases for school teachers might reduce incentives for them to accept gifts and payments that compromise the integrity of learning assessment. However, without improvements to other working conditions and appropriate information about current teacher incomes, this could be a very costly and ineffective initiative. Before adopting any changes to teacher compensation levels, two prior actions are needed.
First, a careful and public review of teacher’s working conditions should be carried out. While the level of teacher compensation can be a factor influencing the behaviour of teachers, many other factors have an impact on the readiness of education professionals to engage in malpractice. These may include deficiencies in the system of teacher appraisal, unsatisfactory conditions of work and lack of employment security. Adding higher levels of compensation to a framework of teacher appraisal and advancement that helps to sustain malpractice in student assessment would be a costly and ineffective measure.
Second, authorities should develop accurate and reliable information about the actual incomes of teachers. The MoES - with the help of cities, rayons and municipalities - should establish the actual income of teachers in Ukrainian schools, including compensation payments and the extra bonuses they might be receiving with the help of parents. This will permit them to assess whether an increase in teacher pay could be funded through a fairer and more efficient distribution of salary resources rather than an undifferentiated salary increase. This will help to make a stronger case for adjustments in teacher pay, not necessarily by means of across-the-board salary increases, but through a more realistic, fair and efficient distribution of resources that are already earmarked for wages.
References
CEQA (2014), Comparison of Measurement of Outcome of Academic Achievement Obtained by Standardised Rating and Subjective Criteria-based Assessment of 2014 Graduates Awarded Gold and Silver Medals: Statistical and Analytical Report, Ukrainian Centre for Education Quality Assessment, Kyiv.
ERA (2014), School Funds: Peculiarities of Establishment and Expenditure, European Research Association, Kyiv.
Fasolia, O.I. (2016), Presentation by the Education Department Director of Khmelnitsky Region, given at a Meeting of Collegium of Oblast State Administration, 26 January 2016.
Hrynevych, L. (2009), “Ethical issues and examination systems in Ukraine”, in Transparency in Education in Eastern Europe, IIEP-UNESCO, Paris, pp. 54-70, http://unesdoc.unesco.org/images/0018/001864/186429E.pdf.
IED (2015), Середня освіта в Україні: думка вчителів та батьків [Secondary Education in Ukraine: Attitudes of Teachers and Parents], Institute for Education Development, Kyiv, http://iro.org.ua/ua/main/research/23.
Kovalchuk, S. and S. Koroliuk (2014), “The introduction of standardized external testing in Ukraine: Challenges and successes”, European Education, Vol. 44, No. 1, pp. 46-70.
New York Times (2015), “Atlanta educators convicted in school heating scandal”, www.nytimes.com/2015/04/02/us/verdict-reached-in-atlanta-school-testing-trial.html?_r=0.
Nusche, D. et al. (2012), OECD Reviews of Evaluation and Assessment in Education: New Zealand, OECD Publishing, Paris, https://doi.org/10.1787/9789264116917-en.
OECD (2015a), Consequences of Corruption at the Sector Level and Implications for Economic Growth and Development, OECD Publishing, Paris, https://doi.org/10.1787/9789264230781-en.
OECD (2015b), Education at a Glance 2015: OECD Indicators, OECD Publishing, Paris, https://doi.org/10.1787/eag-2015-en.
OECD (2013a), PISA 2012 Results: What Makes Schools Successful (Volume IV): Resources, Policies and Practices, OECD Publishing, Paris, https://doi.org/10.1787/9789264201156-en.
OECD (2013b), Synergies for Better Learning: An International Perspective on Evaluation and Assessment, OECD Publishing, Paris, https://doi.org/10.1787/9789264190658-en.
Razumkov Centre (2015), Corruption Perception Survey 2015, Kyiv.
Santiago, P. et al. (2011), OECD Reviews of Evaluation and Assessment in Education: Australia 2011, OECD Publishing, Paris, https://doi.org/10.1787/9789264116672-en.
SSSU (2016), National Education Accounts of Ukraine, State Statistics Service of Ukraine, Kyiv.
Transparency International (2013), Global Corruption Barometer 2013, Transparency International.
-
Article 34, Law on General Secondary Education No. 28 (651-XIV) of 1999.
-
Decree of the Cabinet of Ministers of Ukraine No. 1298 of 30 August 2002.
-
Decree of the Cabinet of Ministers of Ukraine No. 373 of 23 March 2011.
References cited as sources of tables and figures:
Law of Ukraine on Education No. 34 (1060-XII) of 1991.
Law of Ukraine on General Secondary Education No. 28 (651-XIV) of 1999.
Tables 4.6 and 4.7, and Figure 4.1:
Decree of the Cabinet of Ministers of Ukraine No. 102 of 15 April 1993, with amendments.
Decree of the Cabinet of Ministers of Ukraine No. 1298 of 30 August 2002.
Decree of the Cabinet of Ministers of Ukraine No. 373 of 23 March 2011.
Decree of the Ministry of Education and Science of Ukraine No. 557 of 26 September 2005 with amendment No. 991 of 17 August 2016.
Notes
← 1. The statistical data for Israel are supplied by and under the responsibility of the relevant Israeli authorities. The use of such data by the OECD is without prejudice to the status of the Golan Heights, East Jerusalem and Israeli settlements in the West Bank under the terms of international law.
← 2. Salary progression in Ukraine depends on the professional category of teachers. The professional categories are “specialist”, “second category specialist (teacher)”, “first category specialist” and “highest category specialist”. Every five years, teachers must undergo a mandatory review, at which point they have the opportunity (but not the obligation) to apply for the next professional category (MoES Order No. 930 of 6 October 2010). The calculations in this chapter are based on the assumption that mid-career teachers in Ukraine (15 years of experience) will have typically progressed to the level of “first category specialist”.