copy the linklink copied! Chapter 10. Measuring global education goals: How PISA can help
This chapter describes how PISA helps countries monitor progress towards the internationally agreed targets of quality and equity in education, and how PISA contributes to improving the capacity of countries to develop relevant data.
In September 2015, the world’s leaders gathered in New York to set ambitious goals for the future of the global community. The 17 Sustainable Development Goals (SDGs) adopted by the 70th General Assembly of the United Nations in 2015, otherwise known as the Global Goals or the 2030 Agenda for Sustainable Development, are a universal call for action to end poverty, protect the planet and ensure that all people enjoy peace and prosperity. Developed through an inclusive intergovernmental process, the 2030 Agenda integrates the social, environmental and economic pillars of sustainability with peace and security objectives.
The fourth SDG (SDG 4), to be achieved by 2030, is to: “Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all”. SDG 4 is to be achieved by meeting ten targets, representing the most comprehensive and ambitious agenda for global education ever attempted.
-
The share of 15-year-old students, in grade 7 and above, who reached a minimum level of proficiency in reading (i.e. at least Level 2 on the PISA scale) ranged from close to 90 % in Beijing, Shanghai, Jiangsu and Zhejiang (China), Estonia, Macao (China) and Singapore, to less than 10 % in Cambodia, Senegal and Zambia, countries that participated in the PISA for Development assessment in 2017.
-
In mathematics, the share of 15-year-old students who attained minimum levels of proficiency (Level 2 and above on the PISA scale) varied even more – between 98 % in Beijing, Shanghai, Jiangsu and Zhejiang (China) and 2 % in Zambia.
-
Disparities in above-minimum proficiency related to socio-economic status were found in all countries and tend to be large. On average across OECD countries, there were only about 7 socio-economically disadvantaged students scoring above minimum levels in reading or mathematics for every 10 advantaged students scoring above these levels.
SDG 4 differs from the Millennium Development Goals (MDGs) on education, which preceded the SDGs and were in place between 2000 and 2015, in the following two ways:
-
Like all other SDGs, Goal 4 establishes a universal agenda, and does not differentiate between rich and poor countries. Every single country is challenged to achieve the SDGs.
-
Goal 4 puts the quality of education and learning outcomes front and centre. Access, participation and enrolment, which were the focus of the MDG agenda, are still important, and the world is still far from providing equitable access to high-quality education for all. But participation in education is not an end in itself; what matters for people and for development are the skills acquired through education. It is mainly the competencies and character qualities that are developed through schooling, rather than the qualifications and credentials gained, that contribute to people’s success and resilience in their professional and personal lives, support individual well-being, and strengthen the prosperity of societies.
In sum, Goal 4 requires all countries to monitor the actual learning outcomes of their young people. PISA, which provides measurement tools to this end, has started to improve, expand and enrich its assessment instruments to help countries in this exercise. This chapter describes how PISA is helping countries monitor progress towards the internationally agreed targets of quality and equity in education, and how PISA contributes to improving the capacity of countries to develop relevant data.
copy the linklink copied! Measuring countries’ progress towards meeting global education targets
By including PISA data in the United Nations’ global indicator framework (UNESCO Institute for Statistics, 2019[1]; United Nations Statistics Division, 2019[2]), the global community has recognised the role of PISA in monitoring progress towards the SDG for education over the next decade. PISA data are used for monitoring progress in the proportion of children and young people who, at the end of lower secondary education, have achieved at least minimum proficiency in reading and mathematics (SDG global indicator 4.1.1c). PISA-based indicators are also used to measure how close countries are to meeting other targets, particularly those related to equity and education for sustainable development.
In 2018, PISA assessed the reading, mathematics and science performance of 15-year-old students in 79 countries and economies. An additional seven countries collected comparable data about their students’ foundational skills in 2017, as part of the PISA for Development initiative.1 That project enhanced the PISA paper-based tests to provide more nuanced measures of the reading, mathematics and science competences of 15-year-olds who scored at or below proficiency Level 2. These enhanced pen-and-paper tests will be offered to all countries that wish to continue to assess their students in pen-and-paper mode starting with PISA 2021. A pilot assessment of the reading and mathematics skills of 15-year-olds who, for whatever reason, do not attend school, was also conducted in 2018 in five countries (Panama, and four countries that took part in PISA for Development: Guatemala, Honduras, Paraguay and Senegal). Results of that assessment will be released in the first quarter of 2020.
SDG Target 4.1
The global indicator for the first SDG 4 target is a measure of the “Proportion of children and young people [at different stages of their education career] achieving at least a minimum proficiency level in (i) reading and (ii) mathematics, by sex”. PISA provides both a way of defining what “minimum proficiency level” means, through its described scale of proficiency, and a way of measuring this proportion, in an internationally comparable manner, amongst students who are close to the end of lower secondary education (or have recently completed lower secondary education). The UNESCO-led Technical Co-operation Group (TCG) on the Indicators for SDG 4 has officially recognised PISA as a source of data for this global indicator (UNESCO Institute for Statistics, 2019[1]).
Changes in technology and society will continue to influence the demand for skills and the contexts in which adults and young people will use their competence in literacy and numeracy. Nevertheless, PISA Level 2 proficiency, which is used in PISA reports (including this one) to identify low-achieving students in reading and mathematics, can represent the “minimum proficiency level” referred to in Target 4.1.2 This definition of minimum proficiency was accepted by the TCG.
Table I.10.1 shows, for each country and economy, the proportion of 15-year-old students who attained proficiency Level 2 in reading and mathematics in 2018 and, where available, the change in this proportion since 2009 (for reading) and 2012 (for mathematics). In countries with marked changes in enrolment rates over these periods, it is important to account for these differences when measuring progress towards greater inclusion and quality in education. For this reason, an alternative measure of progress towards this target is also included for countries where the coverage rate of PISA samples was below 75 % (meaning that 25 % or more of 15-year-olds were either out of school, in school but enrolled below 6th grade, or excluded from PISA) in 2018. This alternative measure neutralises the impact of changes in enrolment rates (or, more precisely, in the coverage rate of the PISA sample with respect to the 15-year-old population) by computing the proportion of students who scored above the minimum proficiency level not only amongst students represented by PISA samples, but amongst the entire population of 15-year-olds.
The share of 15-year-old students, in grade 7 and above, who reached a minimum level of proficiency in reading (i.e. at least Level 2 on the PISA scale) ranged from close to 90 % (in Beijing, Shanghai, Jiangsu and Zhejiang [China], Estonia, Macao [China] and Singapore) to less than 10 % in Cambodia, Senegal and Zambia, countries that participated in the PISA for Development assessment in 2017 (Table I.10.1). In mathematics, the share of 15-year-old students who attained minimum levels of proficiency (Level 2 and above on the PISA scale) varied even more – between 98 % in Beijing, Shanghai, Jiangsu and Zhejiang (China) and 2 % in Zambia. On average across OECD countries, 77 % of 15-year-olds attained the minimum level of proficiency in reading, and 76 % attained that level in mathematics. These numbers show that, in 2018, all countries still have some way to go towards reaching the global goals for quality education.
Table I.10.1 also shows those countries that have made significant progress over the past decade towards the objective of ensuring that all children reach minimum levels of proficiency in reading and mathematics by the end of lower secondary education. The share of students who scored above minimum levels in reading grew by more than 10 percentage points in the Republic of Moldova, the Republic of North Macedonia, Peru and Qatar, in particular; in mathematics, similarly large increases in the share of students performing above minimum proficiency in reading were observed in Albania, Malaysia, Montenegro, Peru and Qatar.
For countries where the share of students represented by PISA (Coverage Index 3) corresponds to less than 75 % of all 15-year-olds (often as a result of early dropout, late or discontinuous enrolment, and grade-retention in primary school), an alternative measure of progress towards the target is also presented in Table I.10.1. Instead of comparing shares of students over time, this alternative measure relates the number of students who performed above the minimum level of proficiency to the total population of 15-year-olds in the country. Assessed in this way, progress can result either from increases in the share of students who performed above the target or, if this share remains stable, from increases in the proportion of 15-year-olds who were in school in grade 7 or above.
This measure combines aspects related to the “quantity” of schooling (i.e. the share of 15-year-olds who are enrolled in school, in grade 7 and above) with measures of the “quality” of education outcomes (i.e. the share of students who scored above the minimum level of proficiency). In doing so, the measure encourages countries that, in 2018, still had comparatively low educational attainment amongst a significant share of young people, to work not only to improve the quality of teaching and learning in school, but also to make their secondary education systems more inclusive. By this measure, Turkey should also be counted amongst the countries that made rapid progress towards Target 4.1 over the past decade.
The children who are expected to meet the target of minimum proficiency in core subjects by 2030 have already been born. For most countries, the numbers presented in Table I.10.1 represent more than a baseline against which future progress can be measured. They represent an urgent call to action to ensure that, as these children progress through the various stages of education – from pre-primary, to primary to secondary education – there are social and education policies in place to support families, communities and schools in their efforts to help all children realise their potential.
SDG Target 4.5
Target 4.5 is dedicated to equity: “By 2030, eliminate gender disparities in education and ensure equal access to all levels of education and vocational training for the vulnerable, including persons with disabilities, indigenous peoples and children in vulnerable situations”. This target is cross-cutting by nature and encompasses all types of inequality across all education outcomes.
PISA is helping countries monitor progress in reducing disparities, particularly with respect to the attainment of minimum levels of proficiency (SDG Target 4.1). The TCG on the Indicators for SDG 4 identifies “parity indices” as the main measure to be used in monitoring inequalities (see Annex A3). Amongst the many dimensions of inequality and vulnerability identified for Indicator 4.5.1, PISA can help monitor gender disparities and inequalities related to family resources, through statistics based on the PISA index of economic, social and cultural status.3
Equity in education is analysed in detail in PISA 2018 Results (Volume II): Where All Students Can Succeed (OECD, 2019[4]), which contains a wide set of indicators on within-country inequalities in learning outcomes, and on the fairness and inclusiveness of education systems. Table I.10.2 shows, for each country/economy, only a single indicator of gender and socio-economic inequalities in minimum proficiency. This indicator, called the parity index, compares the share of 15-year-old students who reached at least Level 2 performance across two groups of students that differ in some background characteristics. The parity index varies between 0 and 2. It is equal to 1 if the share of 15-year-old students scoring above minimum levels is the same for both groups (no disparity).4 For example: if the share of girls scoring above Level 2 is 40 %, and the share of boys is 50 %, the gender parity index is 0.8 (40 %/50 %). Conversely, if the share of girls is 50 % and the share of boys is 40 %, the gender parity index is 1.2 (2 – 40 %/50 %). Values close to 1 indicate either a small percentage-point difference between the two shares or, for a given percentage-point difference, a higher average share. In other words, the parity index is sensitive both to differences in performance and to overall levels of performance.
Table I.10.2 shows that gender disparities in minimum proficiency are often in favour of girls in reading (as indicated by values of the parity index above 1) and of boys in mathematics. In both subjects, these disparities tend to be limited, as indicated by parity indices between 0.85 and 1.15.
In contrast, socio-economic disparities are more systematic across subjects and only a few countries/economies had limited disparities in above-minimum proficiency related to socio-economic status. These include Beijing, Shanghai, Jiangsu and Zhejiang (China), Estonia, Hong Kong (China) and Macao (China). Across OECD countries, the average parity index for socio-economic differences in performance above minimum levels (i.e. at Level 2 and above) was 0.72 in reading and 0.68 in mathematics. This means that, on average across OECD countries, there were only about seven socio-economically disadvantaged students who scored above the minimum proficiency level in reading or mathematics for every 10 advantaged students who scored above that level.5 Disparities were even wider in several low- and middle-income countries, including Cambodia, the Dominican Republic, Guatemala, Panama, Peru, the Philippines and Zambia, where the socio-economic parity index was lower than 0.30 in both reading and mathematics.
Other thematic targets and means of implementation
PISA also provides useful data for monitoring some thematic indicators that are relevant to Target 4.7 ( “ensure all learners acquire knowledge and skills needed to promote sustainable development […]”), particularly through its assessments of science (15-year-olds’ ability to engage with science-related issues, and with the ideas of science, as reflective citizens) and global competence (their ability to understand and appreciate the perspectives and world views of others). PISA indicators of students’ global competence are discussed in PISA 2018 Results (Volume VI): Are Students Ready to Thrive in Global Societies? (OECD, forthcoming[5]).
In addition, data on the context in which students learn enable countries to monitor two of the three “means of implementation” for SDG 4. In particular, PISA data can be used to monitor the quality of education facilities (Target 4.a: “facilities that are child, disability and gender sensitive and provide safe, non-violent, inclusive and effective learning environments for all”); and the supply of qualified teachers (Target 4.c: “teachers that are in sufficient number and adequately trained, qualified, motivated and supported”).6 PISA indicators related to resources, including teachers, are discussed in PISA 2018 Results (Volume V): Effective Policies, Successful Schools (OECD, forthcoming[6]).
copy the linklink copied! How PISA and the OECD are helping countries build national systems for monitoring learning goals
Through participation in PISA, countries can also enhance their capacity to develop relevant data to monitor national and international learning targets at different levels of education. While most countries that have participated in PISA already have adequate systems in place, that is not true for many low- and middle-income countries. To this end, the OECD PISA for Development initiative not only aimed to expand the coverage of the international assessment to include more middle- and low-income countries, it also offered these countries assistance in building their national assessment and data-collection systems. These capacity-building components of the PISA programme are now offered to all new countries joining PISA for its 2021 or 2024 cycle.
Countries that took part in PISA for Development prepared for their participation through a process that began with an analysis of their capacity to implement PISA and make use of PISA data, and included planning to strengthen that capacity. Countries were supported by the OECD and its contractors at each stage of the assessment cycle. This process helped countries overcome two potential barriers to participation in PISA: a lack of capacity to implement the assessment and a lack of experience in using PISA data and results. To overcome the latter obstacle, the OECD and its contractors offered training and assistance in data analysis, the interpretation of PISA results, report writing and communication.
During the analysis phase of the project, analysts confirmed that the test instruments measured what they purported to measure, and that the population statistics derived from tests and questionnaires could be compared internationally and used to monitor global learning goals. National analysis teams therefore could use the assessment results for a report that included relevant comparisons to inform decisions concerning national policies. Each country’s report highlighted main messages from the results as well as policy options to pursue to improve learning outcomes.7
Other OECD data, such as those derived from the Survey of Adult Skills (a product of the OECD Programme for the International Assessment of Adult Competencies [PIAAC]) and the OECD Teaching and Learning International Survey (TALIS), provide a solid evidence base for monitoring education goals more widely. PIAAC, in particular, is the principal source of data for measuring progress towards SDG Target 4.6 – adult literacy and numeracy. OECD data complement and inspire national data systems and promote peer learning, as countries can compare their experiences in implementing policies through their own analyses, or through reviews and reports co-ordinated by the OECD.
References
[7] European Council (2009), Council conclusions of 12 May 2009 on a strategic framework for European cooperation in education and training ( “ET 2020 “), Office for Official Publications of the European Communities, pp. C 119/2-10, https://doi.org/10.3000/17252423.C_2009.119.eng.
[4] OECD (2019), PISA 2018 Results (Volume II): Where All Students Can Succeed, PISA, OECD Publishing, Paris, https://doi.org/10.1787/b5fd1b8f-en.
[3] OECD (2018), PISA for Development Technical Report, OECD Publishing, Paris, http://www.oecd.org/pisa/pisa-for-development/pisafordevelopment2018technicalreport/ (accessed on 3 October 2019).
[6] OECD (forthcoming), PISA 2018 Results (Volume V): Effective Policies, Successful Schools, PISA, OECD Publishing, Paris.
[5] OECD (forthcoming), PISA 2018 Results (Volume VI): Are Students Ready to Thrive in Global Societies? PISA, OECD Publishing, Paris.
[1] UNESCO Institute for Statistics (2019), Quick Guide to Education Indicators for SDG 4, UNESCO Institute for Statistics, https://unesdoc.unesco.org/ark:/48223/pf0000265396 (accessed on 13 July 2019).
[2] United Nations Statistics Division (2019), Global indicator framework for the Sustainable Development Goals and targets of the 2030 Agenda for Sustainable Development, https://unstats.un.org/sdgs/indicators/indicators-list/ (accessed on 13 July 2019).
Notes
← 1. Evidence supporting the comparability of PISA for Development results with results from the PISA pen-and-pencil tests can be found in Chapter 12 of the PISA for Development Technical Report (OECD, 2018[3]).
← 2. Level 2 proficiency is already used as a normative benchmark in many countries. For example, the European Union’s strategic framework for co-operation in education and training (known as ET 2020), established in 2009, states: “By 2020, the share of low-achieving 15-year-olds in reading, mathematics and science should be less than 15 %” (as measured by the proportion of 15-year-old students performing below Level 2 in PISA) (European Council, 2009[7]).
← 3. In many countries, PISA samples are stratified according to geography and can also be used to monitor disparities related to location (e.g. region or city). These comparisons are not included here, as the location categories must be defined differently for each country.
← 4. If the share in the first group is smaller than that in the second group, the parity index is defined as the ratio of the share in the first group (e.g. girls) divided by the share in the second group (e.g. boys). If the share in the second group is smaller, the parity index is defined as two minus the inverse ratio.
← 5. It is possible to interpret socio-economic parity indices in this way because socio-economic groups (advantaged and disadvantaged) are defined to be of equal size, each comprising one quarter of a country’s/economy’s 15-year-old students.
← 6. Data from two other OECD programmes, the Teaching and Learning International Survey (TALIS) and the Indicators of Education Systems (INES) programme, can also be used to monitor progress towards Target 4.c.
← 7. PISA for Development capacity-needs analyses and national reports can be accessed through www.oecd.org/pisa/pisa-for-development/pisa-for-development-documentation.htm (accessed on 13 July 2019).
Metadata, Legal and Rights
https://doi.org/10.1787/5f07c754-en
© OECD 2019
The use of this work, whether digital or print, is governed by the Terms and Conditions to be found at http://www.oecd.org/termsandconditions.
