Annex A. Data sources and methods
Data sources: overview
This publication reports the results of secondary analyses of data from several sources collected in surveys of students, teachers and principals. These data are drawn from PISA (Programme on International Student Assessment), TIMSS (Trends in International Mathematics and Science Study) and PIRLS (Progress in International Reading Literacy Study). PISA, TIMSS and PIRLS have been created to look at student achievements in maths and science (PISA and TIMSS) and text understanding (PISA and PIRLS). Background questionnaires provide relevant information about classroom or school practices which have been used to identify the extent to which they have changed over time. All these surveys are cross-sectional.
Coverage of the statistics
PISA is designed to assess learning outcomes of 15-year-old students and make comparisons over time. PISA focuses on the extent to which students can apply the knowledge and skills they have learnt and practised at school when confronted with situations and challenges for which that knowledge may be relevant.
PISA uses questionnaires to collect background information from students and data on various aspects of organisation and educational provision in schools from school principals.
The target population of PISA is 15-year-old students in grade 7 or higher who attend educational institutions, including those enrolled part-time and those in vocational training programmes. It is important to note that the sample is not designed to be representative of schools or classrooms and has not been reweighted. Results should be read as “the percentage of 15-year-old students who report …..”
TIMSS and PIRLS are designed to measure student achievement around the world and make comparisons over time. TIMSS has two target populations—all students enrolled at the 4th grade and all students enrolled at the 8th grade, although countries may choose to assess either or both student populations. Fourth and eighth grade represent four and eight years of schooling respectively, counting from the first year of ISCED Level 1, providing the mean age at the time of testing is at least 9.5 years/13.5 years.
The target population for PIRLS is all students enrolled at the 4th grade. All schools of all educational sub-systems that have students learning full-time in the target grade are part of the international target population, including schools that are not under the authority of the national Ministry of Education or its equivalent.
TIMSS and PIRLS are designed to pay particular attention to students’ curricular and instructional experiences and therefore sample intact classes of students. However, as with PISA, TIMSS and PIRLS are not designed to be representative of schools or classrooms and data have not been reweighted. Results should be read as “the percentage of 4th /8th grade students who report…..”
Country coverage
This publication incorporates information from 47 education systems or countries within the OECD, and 6 partner countries.
-
36 education systems within the OECD participated in PISA 2015, 34 in 2009 and 32 in 2006.
-
29 education systems within the OECD participated in TIMSS 2015, 38 in 2011 and 27 in 2007.
-
31 education systems within the OECD participated in PIRLS 2016, 29 in 2011 and 27 in 2006.
Sample sizes
Year coverage
This publication focuses on change across time and therefore requires data from the same questions asked in different years. There are many such questions in the datasets employed, but it should be noted that the years in which they were answered varies.
Where possible, analysis focuses on change between 2006 and 2016, although data from TIMSS presents change between 2007 and 2015, and PISA data between 2006 and 2015 or 2009 and 2015. The years included in the analyses are indicated in the chapters.
In some cases, data are also available for an additional year between the two end points. In this case, the data from all three data collection exercises are represented in figures but only the end points are discussed in the text.
Calculation of cross-country means and totals
Given the range of education systems covered in each chapter, cross-country means may not always incorporate the same countries or the same number of education systems. Where practical, the average cross-country statistics have been calculated using data for OECD countries (as in PISA, TIMSS, and PIRLS). In each indicator in TIMSS, PIRLS and PISA, the OECD average (unweighted) is computed taking into account the subset of OECD education systems with data available for all years concerned.
Calculation of effect sizes
Effect sizes are presented for all analyses in addition to tests of statistical significance. Tests of significance allow the reader to determine whether the difference between the two percentages reported could have happened by chance if the actual difference is zero and thus consider the quality of the instrument used for measurement. However, statistical significance is dependent on the sample size (the larger the sample and the more confident the reader can be that even small differences wouldn’t have happened by chance) and can, in principle, be improved simply by increasing the number of observations. Yet this does not tell the reader anything about how meaningful the observed effects are in real-world terms. For example, a change in classroom practice could be statistically significant but only amount to a few percentage points of relative change with no practical meaning.
The effect size provides important information about the size of the relationship between two statistics. The main difference between effect size and significance is that change is normalized by the standard deviation as opposed to standard error, which means that the result no longer depends on sample size. The precise form of calculation depends on the type of question asked, but is typically calculated as:
i.e. as the change between a treatment and control group (or any two subgroups of a sample; or – as in our case - two different years), divided by a “pooled” standard deviation:
Sometimes, the control group standard deviation or more complicated forms of pooled standard deviations are used instead of the one displayed. This book looks at effect sizes in two ways. One approach is to calculate country level effect sizes. Here, means and standard deviations refer to the individual country samples. The effect size calculation provides information about how much, in terms of their own standard deviation, a country has moved up (or down) over time. For country level effect sizes, and are estimated via σ =SE*√n (with n being the sample sizes), which provides a conservative (lower) estimate for the effect size (as n could potentially be overestimated by including invalid observations).
A second way of looking at effect size is required for questions that evaluate proportions, i.e. those that deal with categorical variables and ask, for example, “How often do you do this activity in class? Daily? At least weekly? At least monthly? Rarely or never?”. In this case, Cohen h is applied to carry out an arcsin-transformation, whereby h=2(arcsin √P1-arcsin √P2).
In accordance with common practices, effect sizes are assessed at three different levels. Effect sizes of less than 0.2 are considered negligible to very small, between 0.2 and 0.5 are come under small to modest, between 0.5 and 0.8, are large, and effect sizes above 0.8 are considered to be very large. While the usefulness of such cut-offs is debatable, this convention is followed by adding a colour coding in three different shades of blue when displaying effect sizes. The reader should interpret the colour coding with care as there is little practical difference between an effect size of 0.18 and 0.22, even if the colour coding is different.
Further resources
The publication uses the OECD StatLinks service. Below each table and Figure is a URL that leads to a corresponding Excel workbook containing the underlying data for that indicator. These URLs are stable and will remain unchanged over time. In addition, readers of the electronic version of this publication (the e-book) will be able to click directly on the links and the relevant workbook will open in a separate window. The tables in the Excel files contain additional information and computations that could not be presented in the paper version.