copy the linklink copied! Annex A2. The PISA target population, the PISA samples and the definition of schools
copy the linklink copied! Who is the PISA target population?
PISA 2018 assessed the cumulative outcomes of education and learning at a point at which most young people are still enrolled in formal education – when they are 15 years old.
Any international survey of education must guarantee the comparability of its target population across nations. One way to do this is to assess students at the same grade level. However, differences between countries in the nature and extent of pre-primary education and care, the age at entry into formal schooling, and the institutional structure of education systems do not allow for a definition of internationally comparable grade levels.
Other international assessments have defined their target population by the grade level that provides maximum coverage of a particular age cohort. However, this method is particularly sensitive to the distribution of students across age and grade levels; small changes in this distribution can lead to the selection of different target grades, even within the same country over different PISA cycles. There also may be differences across countries in whether students who are older or younger than the desired age cohort are represented in the modal grade, further rendering such grade-level-based samples difficult to compare.
To overcome these problems, PISA uses an age-based definition of its target population, one that is not tied to the institutional structures of national education systems. PISA assesses students who are aged between 15 years and 3 (complete) months and 16 years and 2 (complete) months1 at the beginning of the assessment period, plus or minus an allowed 1-month variation, and who are enrolled in an educational institution2 at grade 7 or higher.3 All students who met these criteria were eligible to sit the PISA assessment in 2018, regardless of the type of educational institution in which they were enrolled and whether they were enrolled in full-time or part-time education. This also allows PISA to evaluate students shortly before they are faced with major life choices, such as whether to continue with education or enter the workforce.
Hence, PISA makes statements about the knowledge and skills of a group of individuals who were born within a comparable reference period, but who may have undergone different educational experiences both in and outside of school. These students may be distributed over different ranges of grades (both in terms of the specific grade levels and the spread in grade levels) in different countries, or over different tracks or streams. It is important to consider these differences when comparing PISA results across countries. In addition, differences in performance observed when students are 15 may disappear later on if students’ experiences in education converge over time.
If a country’s mean scores in reading, mathematics or science are significantly higher than those of another country, it cannot automatically be inferred that schools or particular parts of the education system in the first country are more effective than those in the second. However, one can legitimately conclude that it is the cumulative impact of learning experiences in the first country, starting in early childhood and up to the age of 15, and including all experiences, whether they be at school, home or elsewhere, that have resulted in the better outcomes of the first country in the subjects that PISA assesses.4
The PISA target population does not include residents of a country who attend school in another country. It does, however, include foreign nationals who attend school in the country of assessment.
To accommodate countries that requested grade-based results for the purpose of national analyses, PISA 2018 provided a sampling option to supplement age-based sampling with grade-based sampling.
copy the linklink copied! How were students chosen?
The accuracy of the results from any survey depends on the quality of the information drawn from those surveyed as well as on the sampling procedures. Quality standards, procedures, instruments and verification mechanisms were developed for PISA that ensured that national samples yielded comparable data and that the results could be compared across countries with confidence. Experts from the PISA Consortium selected the samples for most participating countries/economies and monitored the sample-selection process closely in those countries that selected their own samples.
Most PISA samples were designed as two-stage stratified samples.5 The first stage sampled schools in which 15-year-old students may be enrolled. Schools were sampled systematically with probabilities proportional to the estimated size of their (eligible) 15-year-old population. At least 150 schools6 were selected in each country, although the requirements for national analyses often demanded a larger sample. Replacement schools for each sampled school were simultaneously identified, in case an originally sampled school chose not to participate in PISA 2018.
The second stage of the selection process sampled students within sampled schools. Once schools were selected, a list of each sampled school’s 15-year-old students was prepared. From this list, 42 students were then selected with equal probability (all 15-year-old students were selected if fewer than 42 were enrolled). The target number of students who were to be sampled in a school could deviate from 42 but could not fall below 20.
Data-quality standards in PISA required minimum participation rates for schools as well as for students. These standards were established to minimise the potential for bias resulting from non-response. Indeed, it was likely that any bias resulting from non-response would be negligible – i.e. typically smaller than the sampling error – in countries that met these standards.
At least 85 % of the schools initially selected to take part in the PISA assessment were required to agree to conduct the test. Where the initial response rate of schools was between 65 % and 85 %, however, an acceptable school-response rate could still be achieved through the use of replacement schools. Inherent in this procedure was a risk of introducing bias, if replacement schools differed from initially sampled schools along dimensions other than those considered for sampling. Participating countries and economies were therefore encouraged to persuade as many of the schools in the original sample as possible to participate.
Schools with a student participation rate of between 25 % and 50 % were not considered to be participating schools, but data (from both the cognitive assessment and questionnaire) from these schools were included in the database and contributed to the various estimates. Data from schools with a student participation rate of less than 25 % were excluded from the database.
In PISA 2018, five countries and economies – Hong Kong (China) (69 %), Latvia (82 %), New Zealand (83 %), the United Kingdom (73 %) and the United States (65 %) – did not meet the 85 % threshold, but met the 65 % threshold, amongst schools initially selected to take part in the PISA assessment. Upon replacement, Hong Kong (China) (79 %), the United Kingdom (87 %) and the United States (76 %) still failed to reach an acceptable participation rate.7 Amongst the schools initially selected before replacement, the Netherlands (61 %) did not meet the 65 % school response-rate threshold, but it reached a response rate of 87 % upon replacement. However, these were not considered to be major issues as, for each of these countries/economies, additional non-response analyses showed that there were limited differences between schools that did participate and the full set of schools originally drawn in the sample.8 Data from these jurisdictions were hence considered to be largely comparable with, and were therefore reported together with, data from other countries/economies.
PISA 2018 also required that at least 80 % of the students chosen within participating schools participated themselves. This threshold was calculated at the national level and did not have to be met in each participating school. Follow-up sessions were required in schools where too few students had participated in the original assessment sessions. Student-participation rates were calculated over all original schools; and also over all schools, whether original or replacement schools. Students who participated in either the original or in any follow-up assessment sessions were counted in these participation rates; those who attended only the questionnaire session were included in the international database and contributed to the statistics presented in this publication if they provided at least a description of their father’s or mother’s occupation.
This 80 % threshold was met in every country/economy except Portugal, where only 76 % of students who were sampled actually participated. The high level of non-responding students could lead to biased results, e.g. if students who did not respond were more likely to be low-performing students. This was indeed the case in Portugal, but a non-response analysis based on data from a national mathematics assessment in the country showed that the upward bias of Portugal’s overall results was likely small enough to preserve comparability over time and with other countries. Data from Portugal was therefore reported along with data from the countries/economies that met this 80 % student-participation threshold.
Table I.A2.6 shows the response rate for students and schools, before and after replacement.
-
Column 1 shows the weighted participation rate of schools before replacement; it is equivalent to Column 2 divided by Column 3 (multiplied by 100 to give a percentage).
-
Column 2 shows the number of responding schools before school replacement, weighted by student enrolment.
-
Column 3 shows the number of sampled schools before school replacement, weighted by student enrolment. This includes both responding and non-responding schools.
-
Column 4 shows the unweighted number of responding schools before school replacement.
-
Column 5 shows the unweighted number of sampled schools before school replacement, including both responding and non-responding schools.
-
Columns 6 to 10 repeat Columns 1 to 5 for schools after school replacement, i.e. after non-responding schools were replaced by the replacement schools identified during the initial sampling procedure.
-
Columns 11 to 15 repeat Columns 6 to 10 but for students in schools after school replacement. Note that the weighted and unweighted numbers of students sampled (Columns 13 and 15) include students who were assessed and those who should have been assessed but who were absent on the day of assessment. Furthermore, as mentioned above, any students in schools where the student response rate was less than 50 % were not considered to be attending participating schools, and were thus excluded from Columns 14 and 15 (and, similarly, from Columns 4, 5, 9 and 10).
copy the linklink copied! What proportion of 15-year-olds does PISA represent?
All countries and economies attempted to maximise the coverage of 15-year-olds enrolled in education in their national samples, including students enrolled in special-education institutions.
The sampling standards used in PISA only permitted countries and economies to exclude up to a total of 5 % of the relevant population (i.e. 15-year-old students enrolled in school at grade 7 or higher) either by excluding schools or excluding students within schools. All but 16 countries and economies – Sweden (11.09 %), Israel (10.21 %), Luxembourg (7.92 %), Norway (7.88 %), Canada (6.87 %), New Zealand (6.78 %), Switzerland (6.68 %), the Netherlands (6.24 %), Cyprus (5.99 %), Iceland (5.99 %), Kazakhstan (5.87 %), Australia (5.72 %), Denmark (5.70 %), Turkey (5.66 %), the United Kingdom (5.45 %) and Estonia (5.03 %) – achieved this standard, and in 28 countries and economies, the overall exclusion rate was less than 2 % (Table I.A2.1) When language exclusions9 were accounted for (i.e. removed from the overall exclusion rate), Estonia and Iceland no longer had exclusion rates greater than 5 %. More details can be found in the PISA 2018 Technical Report (OECD, forthcoming[1]).
Exclusions that should remain within the above limits include both:
-
at the school level:
-
schools that were geographically inaccessible or where the administration of the PISA assessment was not considered feasible
-
schools that provided teaching only for students in the categories defined under “within-school exclusions”, such as schools for the blind.
The percentage of 15-year-olds enrolled in such schools had to be less than 2.5 % of the nationally desired target population (0.5 % maximum for the former group and 2 % maximum for the latter group). The magnitude, nature and justification of school-level exclusions are documented in the PISA 2018 Technical Report (OECD, forthcoming[1]).
-
-
at the student level:
-
students with an intellectual disability, i.e. a mental or emotional disability resulting in the student being so cognitively delayed that he/she could not perform in the PISA testing environment
-
students with a functional disability, i.e. a moderate to severe permanent physical disability resulting in the student being unable to perform in the PISA testing environment
-
students with limited assessment-language proficiency. These students were unable to read or speak any of the languages of assessment in the country at a sufficient level and unable to overcome such a language barrier in the PISA testing environment, and were typically students who had received less than one year of instruction in the language of assessment
-
other exclusions, a category defined by the PISA national centres in individual participating countries and approved by the PISA international consortium
-
students taught in a language of instruction for the major domain for which no materials were available.
Students could not be excluded solely because of low proficiency or common disciplinary problems. The percentage of 15-year-olds excluded within schools had to be less than 2.5 % of the national desired target population.
-
Although exceeding the exclusion rate limit of 5 % (Table I.A2.1), data from the 16 countries and economies listed above were all deemed to be acceptable for the reasons listed below. In particular, all of these reasons were accepted by a data-adjudication panel to allow for the reliable comparison of PISA results across countries and economies and across time; thus the data from these countries were reported together with data from other countries/economies.
-
In Australia, Canada, Denmark, Luxembourg, New Zealand and Norway, exclusion rates remained close to those observed in previous cycles. In the United Kingdom, exclusion rates were also above 5 % but have decreased markedly across cycles.
-
In Cyprus, Iceland, Kazakhstan, the Netherlands and Switzerland, exclusions increased but remained close to the 5 % limit. The increase could be largely attributed to a marked increase in students who were excluded within schools due to intellectual or functional disabilities. Moreover, in the Netherlands, some 17 % of students were not excluded but assigned to UH (une heure) booklets, which were intended for students with special education needs. As these booklets did not cover the domain of financial literacy (PISA 2018 Results [Volume V]: Are Students Smart about Money?, OECD, forthcomingsee [2]), the effective exclusion rate for the Netherlands in financial literacy was over 20 %. This resulted in a strong upward bias in the country mean and other population statistics in that domain. Data from the Netherlands in financial literacy are not comparable with data from other education systems; but data from the Netherlands in the core PISA subjects were still deemed to be largely comparable.
-
The higher exclusion rate in Turkey was likely the result of a higher school-level exclusion rate due to a particular type of non-formal educational institution that was not listed (and hence not excluded) in 2015 but was listed and excluded in 2018.
-
The higher exclusion rate in Israel was the result of a higher school-level exclusion rate due to the lack of participation by a particular type of boys’ school. These schools were considered to be non-responding schools in cycles up to 2015 but were treated as school-level exclusions in 2018.
-
Sweden had the highest exclusion rate: 11.07 %. It is believed that this increase in the exclusion rate was due to a large and temporary increase in immigrant and refugee inflows, although because of Swedish data-collection laws, this could not be explicitly stated in student-tracking forms. Instead, students confronted with language barriers were classified as being excluded “for other reasons”, as were students with intellectual and functional disabilities. It is expected that the exclusion rate will decrease to previous levels in future cycles of PISA, as such inflows stabilise or shrink.10
Table I.A2.1 describes the target population of the countries participating in PISA 2018. Further information on the target population and the implementation of PISA sampling standards can be found in the PISA 2018 Technical Report (OECD, forthcoming[1]).
-
Column 1 shows the total number of 15-year-olds according to the most recent available information, which in most countries and economies means from 2017, the year before the assessment.
-
Column 2 shows the number of 15-year-olds enrolled in school in grade 7 or above, which is referred to as the “eligible population”.
-
Column 3 shows the national desired target population. Countries and economies were allowed to exclude up to 0.5 % of students a priori from the eligible population, essentially for practical reasons. The following a priori exclusions exceed this limit but were agreed with the PISA Consortium:
-
Canada excluded 1.17 % of its population: students living in the Yukon, Northwest Territories and Nunavut, and Aboriginal students living on reserves
-
Chile excluded 0.05 % of its population: students living on Easter Island, the Juan Fernandez Archipelago and Antarctica
-
Cyprus excluded 0.10 % of its population: students attending schools on the northern part of the island
-
the Philippines excluded 2.42 % of its population: students living in the Autonomous Region in Muslim Mindanao
-
Saudi Arabia excluded 7.59 % of its population: students living in the regions of Najran and Jizan
-
Ukraine excluded 0.37 % of its population: some students attending schools in the Donetsk and Luhansk regions
-
the United Arab Emirates excluded 0.04 % of its population: home-schooled students.
-
-
Column 4 shows the number of students enrolled in schools that were excluded from the national desired target population, either from the sampling frame or later in the field during data collection. In other words, these are school-level exclusions.
-
Column 5 shows the size of the national desired target population after subtracting the students enrolled in excluded schools. This column is obtained by subtracting Column 4 from Column 3.
-
Column 6 shows the percentage of students enrolled in excluded schools. This is obtained by dividing Column 4 by Column 3 and multiplying by 100.
-
Column 7 shows the number of students who participated in PISA 2018. Note that in some cases, this number does not account for 15-year-olds assessed as part of additional national options.
-
Column 8 shows the weighted number of participating students, i.e. the number of students in the nationally defined target population that the PISA sample represents.
-
Column 9 shows the total number of students excluded within schools. In each sampled school, all eligible students – namely, those 15 years of age, regardless of grade – were listed, and a reason for the exclusion was provided for each student who was to be excluded from the sample. These reasons are further described and classified into specific categories in Table I.A2.4.
-
Column 10 shows the weighted number of students excluded within schools, i.e. the overall number of students in the national defined target population represented by the number of students from the sample excluded within schools. This weighted number is also described and classified by exclusion categories in Table I.A2.4.
-
Column 11 shows the percentage of students excluded within schools. This is equivalent to the weighted number of excluded students (Column 10) divided by the weighted number of excluded and participating students (the sum of Columns 8 and 10), multiplied by 100.
-
Column 12 shows the overall exclusion rate, which represents the weighted percentage of the national desired target population excluded from PISA either through school-level exclusions or through the exclusion of students within schools. It is equivalent to the school-level exclusion rate (Column 6) plus the product of the within-school exclusion rate and 1 minus the school-level exclusion rate expressed as a decimal (Column 6 divided by 100).11
-
Column 13 shows an index of the extent to which the national desired target population was covered by the PISA sample. As mentioned above, 16 countries/economies fell below the coverage of 95 %. This is also known as Coverage Index 1.
-
Column 14 shows an index of the extent to which 15-year-olds enrolled in school were covered by the PISA sample. The index, also known as Coverage Index 2, measures the overall proportion of the national enrolled population that is covered by the non-excluded portion of the student sample, and takes into account both school- and student-level exclusions. Values close to 100 indicate that the PISA sample represents the entire (grade 7 and higher) education system as defined for PISA 2018. This is calculated in a similar manner to Column 13; however, the total enrolled population of 15-year-olds in grade 7 or above (Column 2) is used as a base instead of the national desired target population (Column 3).
-
Column 15 shows an index of the coverage of the 15-year-old population. The index is the weighted number of participating students (Column 8) divided by the total population of 15-year-old students (Column 1). This is also known as Coverage Index 3.
A high level of coverage contributes to the comparability of the assessment results. For example, even assuming that the excluded students would have systematically scored worse than those who participated, and that this relationship is moderately strong, an exclusion rate on the order of 5 % would likely lead to an overestimation of national mean scores of less than 5 score points on the PISA scale (where the standard deviation is 100 score points).12
copy the linklink copied! Definition of schools
In some countries, subunits within schools were sampled instead of schools, which may affect the estimate of the between-school variance. In Austria, the Czech Republic, Germany, Hungary, Japan, Romania and Slovenia, schools with more than one programme of study were split into the units delivering these programmes. In the Netherlands, locations were listed as sampling units. In the Flemish Community of Belgium, each campus (or implantation) of a multi-campus school was sampled independently, whereas the larger administrative unit of a multi-campus school was sampled as a whole in the French Community of Belgium.
In Argentina, Australia, Colombia and Croatia, each campus of a multi-campus school was sampled independently. Schools in the Basque Country of Spain that were divided into sections by language of instruction were split into these linguistic sections for sampling. International schools in Luxembourg were split into two sampling units: one for students who were instructed in a language for which testing material was available,13 and one for students who were instructed in a language for which no testing material was available (and who were hence excluded).
Some schools in the United Arab Emirates were sampled as a whole unit, while others were split by curriculum and sometimes by gender. Due to reorganisation, some schools in Sweden were split into two parts, each part with its own principal. Some schools in Portugal were organised into clusters where all units in a cluster shared the same teachers and principal; each of these clusters constituted a single sampling unit.
copy the linklink copied! The distribution of PISA students across grades
Students assessed in PISA 2018 were enrolled in various grade levels. The percentage of students at each grade level is presented, by country, in Table I.A2.8 and Table I.A2.9, and by gender within each country in Table I.A2.12 and Table I.A2.13.
copy the linklink copied! Tables available on line
https://doi.org/10.1787/888934028862
-
Table I.A2.3 PISA target populations and samples, by adjudicated regions
-
Table I.A2.5 Exclusions, by adjudicated regions
-
Table I.A2.7 Response rates, by adjudicated regions
-
Table I.A2.9 Percentage of students at each grade level, excluding students with missing grade information
-
Table I.A2.10 Percentage of students at each grade level, by adjudicated regions
-
Table I.A2.11 Percentage of students at each grade level, by adjudicated regions, excluding students with missing grade information
-
Table I.A2.12 Percentage of students at each grade level, by gender
-
Table I.A2.13 Percentage of students at each grade level, by gender, excluding students with missing grade information
-
Table I.A2.14 Percentage of students at each grade level, by gender and adjudicated regions
-
Table I.A2.15 Percentage of students at each grade level, by gender and adjudicated regions, excluding students with missing grade information
References
[3] OECD (2019), PISA 2018 Results (Volume II): Where All Students Can Succeed, PISA, OECD Publishing, Paris, https://doi.org/10.1787/b5fd1b8f-en.
[2] OECD (forthcoming), PISA 2018 Results (Volume IV): Are Students Smart about Money?, PISA, OECD Publishing, Paris.
[1] OECD (forthcoming), PISA 2018 Technical Report, OECD Publishing, Paris.
Notes
← 1. More precisely, PISA assessed students who were at least 15 years and 3 complete months old and who were at most 16 years and 3 complete months old (i.e. younger than 16 years, 2 months and roughly 30 days old), with a tolerance of one month on each side of this age window. If the PISA assessment was conducted in April 2018, as was the case in most countries, all students born in 2002 would have been eligible.
← 2. Educational institutions are generally referred to as schools in this publication, although some educational institutions (in particular, some types of vocational education establishments) may not be referred to as schools in certain countries.
← 3. As might be expected from this definition, the average age of students across OECD countries was 15 years and 9 months. The range in country means was 2 months and 13 days (0.20 year), from the minimum country mean of 15 years and 8 months to the maximum country mean of 15 years and 10 months (OECD, 2019[3]).
← 4. Such a comparison is complicated by first-generation immigrant students, who received part of their education in a country other than the one in which they were assessed. Mean scores in any country/economy should be interpreted in the context of student demographics within that country/economy.
← 5. Details for countries that applied different sampling designs are documented in the PISA 2018 Technical Report (OECD, forthcoming[1]).
← 6. Due to the small size of these education systems, all schools and all eligible students within these schools were included in the samples of Brunei Darussalam, Cyprus (see note 8), Iceland, Luxembourg, Macao (China), Malta, Montenegro and Qatar.
← 7. The threshold for an acceptable participation rate after replacement varies between 85 % and 100 %, depending on the participation rate before replacement.
← 8. In particular, in the case of the Netherlands and the United Kingdom, non-response bias analyses relied on direct measures of school performance external to PISA, typically from national assessments. More indirect correlates of school performance were analysed in Hong Kong (China) and the United States, due to the absence of national assessments. The non-response problem in Hong Kong (China) can be attributed to two causes: lack of initiative amongst schools and teachers to participate in PISA, and a large number of schools that were considered to be non-responding schools, as less than 50 % of sampled students in these schools sat the assessment.
← 9. These exclusions refer only to those students with limited proficiency in the language of instruction/assessment. Exclusions related to the unavailability of test material in the language of instruction are not considered in this analysis.
← 10. The preliminary attribution of school codes in the process of selecting, and then excluding, students and schools may have resulted in the double exclusion (at both the school and student levels) of some of the students with special education needs in Sweden. As a result, the overall exclusion rate in Sweden may have been overestimated by (at most) 0.5 of a percentage point. In this scenario, the overall exclusion rate would still be over 10 % and the highest amongst PISA-participating countries/economies.
← 11. The overall exclusion rate includes those students who were excluded at the school level (Column 6) and those students who were excluded within schools (Column 11); however, only students enrolled in non-excluded schools were affected by within-school exclusions, hence the presence of the term equivalent to 1 minus Column 6 (expressed as a decimal).
← 12. If the correlation between the propensity of exclusions and student performance were 0.3, then resulting mean scores would likely have been overestimated by 1 score point if the exclusion rate were 1 %; by 3 score points if the exclusion rate were 5 %; and by 6 score points if the exclusion rate were 10 %. If the correlation between the propensity of exclusions and student performance were 0.5, then resulting mean scores would likely have been overestimated by 1 score point if the exclusion rate were 1 %; by 5 score points if the exclusion rate were 5 %; and by 10 score points if the exclusion rate were 10 %. For this calculation, a model was used that assumed a bivariate normal distribution for performance and the propensity to participate.
← 13. Testing material was adapted to each country. Versions in the same language thus differed across countries, and students in Luxembourg who were not instructed in one of the three languages in which testing material was available (English, French and German) were unable to sit the PISA assessment, even if such material were available in their language of instruction in a different country.
Metadata, Legal and Rights
https://doi.org/10.1787/5f07c754-en
© OECD 2019
The use of this work, whether digital or print, is governed by the Terms and Conditions to be found at http://www.oecd.org/termsandconditions.
