Chapter 5. Policies Governing School Systems and Low Student Performance

This chapter explores how some of the policies that govern school systems are associated with low student performance. Specifically, the chapter examines whether the incidence of underperformance in a school system is related to: the allocation of educational resources across schools in the system, the degree of school autonomy, the prevalence of private schools, and/or the grouping or selection of students into different tracks or programmes.

It is clear that students’ own behaviour and attitudes have an impact on their learning, as does the quality of resources, both human and material, that schools provide to their students. What might be less evident is the influence of policy, at the school-system level, on student performance. For example, only an analysis at the system level could show that when education systems are more socio-economically inclusive, the share of low performers in mathematics is smaller (Figure 5.1a) and the share of top performers is slightly larger (Figure 5.1b). A system-level perspective can also reveal relationships that are hidden or different from those found at the school and student levels. For instance, highly selective schools may benefit their own students through higher-quality resources, but they also tend to increase and reinforce social and academic segregation within a school system.

What the data tell us

Across PISA-participating countries and economies, higher-quality educational resources and physical infrastructure are associated with less low performance in mathematics. However, this relationship disappears when the quality of resources is above the OECD average.
In countries and economies where educational resources are distributed more equitably across schools, the incidence of low performance in mathematics is lower, even when comparing school systems that have educational resources of a similar quality.
When schools enjoy more autonomy over curricula and assessments, the share of low performers in mathematics across the education system is smaller; but this association is not observed when schools have more autonomy over resource allocation.

Some phenomena can only be, or are better, measured at the system level. This is particularly the case with measures of inequality, segregation and heterogeneity. New analyses in this chapter considers whether and how low – and high – performance in mathematics is associated with the quality of educational resources, the type of school governance, the level of school autonomy, and the degree of student grouping in PISA-participating school systems.

EDUCATIONAL RESOURCES AND LOW PERFORMANCE IN MATHEMATICS

Despite the conventional wisdom that higher investment leads to greater gains, there is no clear evidence that increasing public spending on education guarantees better student performance once a minimum level of expenditure is reached (Burtless, 1996; Hanushek, 1997; Nicoletti and Rabe, 2012; Woessmann, 2003). PISA results have shown that achieving excellence in education is not just about how much is spent, but how, when and where it is spent (OECD, 2013).

PISA 2012 asked school principals to report whether their schools’ capacity to provide instruction is hindered by a shortage or inadequacy of: physical infrastructure, such as school buildings, heating and cooling systems, and instructional space; educational resources, such as science laboratory equipment, instructional materials and computers; and/or qualified teachers in key areas. In addition, students who participated in PISA 2012 were asked to report the average number of students who attend their language-of-instruction class. Figure 5.2 shows that of these four factors, the quality of educational resources is most strongly associated with the incidence of low performance in mathematics at the country level, followed by the quality of physical infrastructure. In both cases, better quality means fewer low performers. Teacher shortage and class size are only weakly associated with low performance.

The association between teacher shortage and low performance is weak largely because school principals in several education systems with a relatively small number of underachievers, such as Germany, the Netherlands and Shanghai-China, reported that a lack of qualified teachers hinders instruction (Table 5.2). This weak relationship does not contradict the well-established fact that effective teaching is the most important in-school factor influencing strong academic performance (Chetty, Friedman and Rockoff, 2014; Rivkin et al., 2005). It may simply be that principals in different education systems may have different expectations and benchmarks to determine whether there is a lack of qualified teachers.

The weak association between low performance and class size largely reflects the fact that in certain Asian countries and economies, notably Hong Kong-China, Japan, Korea, Macao-China, Shanghai-China, Singapore, Chinese Taipei and Viet Nam, large classes co-exist with small shares of low performers. This is consistent with previous studies that focus on academic performance (Piketty and Valdenaire, 2006; Slavin, 1989), although some studies have also revealed that small classes may be particularly beneficial for at-risk students (Finn and Achilles, 1999; Krueger and Whitmore, 2001). The findings on the importance of different types of resources are in line with those reported by Woessmann (2003), who used data from the third Trends in International Mathematics and Science Study (TIMSS). According to reports by school principals, when there is a shortage or inadequacy of instructional materials, student performance suffers. However, this is not the case when classes are large and student-teacher ratios are high.

“When” resources are invested in a school system also matters. Improving the quality of a school’s physical infrastructure and educational resources can make a big difference for low-performing students when the initial quality of those resources is poor. Once principals in an education system report that the quality of their school resources is satisfactory, additional or better-quality resources appear to have little additional impact on the incidence of low performance (Figure 5.3). In other words, ensuring that every child has access to quality school buildings, teachers, books and other educational material can help to reduce the number of low performers. However, investing beyond a minimum level of quality has no appreciable impact on the incidence of low performance.

Investing resources in a school system is more beneficial for reducing the share of low performers than for increasing the share of top performers. Based on principals’ reports aggregated at the system level, the quality of schools’ physical infrastructure and educational resources and the degree of teacher shortage are better predictors of low performance in mathematics than of top performance (Figure 5.4 and Table 5.2). Class size has a different impact: larger language-of-instruction classes are associated with larger shares of both low performers and top performers in mathematics (Table 5.2).

“Where” resources are invested also has an impact on the incidence of underperformance (Card and Payne, 2002). Education systems can distribute resources proportionally, based on the number of students in schools; they can provide additional funding to disadvantaged schools to compensate for their larger share of at-risk students; or they may allocate funding that, intentionally or not, reinforces existing socio-economic inequalities. The latter most often occurs when school budgets rely on student fees, alumni donations and local taxes (Fernandez and Rogerson, 2003).

Equity in resource allocation, measured by the difference in the PISA index of quality of schools’ educational resources between socio-economically disadvantaged and advantaged schools, varies considerably across PISA-participating countries and economies. Croatia, Finland and Norway show the most equity in resource allocation, while Costa Rica, Mexico and Peru show the least equity (Table 5.2). As shown in Figure 5.5, in countries and economies where educational resources are distributed more equitably, the share of low performers in mathematics is considerably smaller, on average, even when comparing education systems with similar quality of educational resources. More important, equity in resource allocation is almost unrelated to the share of top performers in mathematics. This suggests that education systems can tackle inequalities in education while simultaneously promoting – and achieving – academic excellence.

SCHOOL AUTONOMY AND LOW PERFORMANCE

Evidence suggests that school autonomy is beneficial to student performance, which partly explains why education reforms since the early 1980s have focused on giving schools greater autonomy (Clark, 2009; Fuchs and Woessmann, 2004; OECD, 2013; Whitty, 1997). However, school autonomy is positively related to student performance in only certain situations. Using results from PISA 2000, 2003, 2006 and 2009, Hanushek, Link and Woessmann (2013) found that school autonomy is positively related to student performance only in developed and high-performing countries, presumably because in these countries school leaders and teachers are better prepared to reap the benefits of school autonomy. PISA 2012 data also show that only autonomy over curricula and assessments is clearly associated with low and high performance in mathematics (Figure 5.6). School autonomy over resource allocation is only weakly related to the share of low and high performers in mathematics across education systems.

The share of low performers in mathematics could be further reduced if education systems can increase school autonomy, particularly over curricula and assessments. To make the most of greater school autonomy, governments need to make sure that certain preconditions are met, including: having highly qualified teachers and strong school leaders to (re)design and implement rigorous internal evaluations and curricula, and having effective accountability systems to avoid opportunistic behaviour and identify low-performing schools (Hanushek, Link and Woessmann, 2013; OECD, 2013).

SCHOOL GOVERNANCE AND LOW-PERFORMING STUDENTS

Advocates of private schooling argue that private schools are more responsive to parents, more efficient, and increase competition, accountability and pedagogical diversity throughout the education system. Critics point to the detrimental effects of private schooling and the parental school choice that comes with it, including school segregation and a threat to social cohesion (Renzulli and Evans, 2005; Saporito, 2003; Schneider, Elacqua and Buckley, 2006; Willms, 1999). Other studies show that when enough middle-class families leave the public school system and the enrolment in private schools surpasses a certain “tipping” point, public schools can then enter a vicious circle of fewer students, less funding and deteriorating quality (Sonstelie, 1979). Equally, private schools can also suffer when the funding and quality of neighbouring public schools improve and the number of students enrolled in these schools increases as a result (Dinerstein and Smith, 2014; Husted and Kenny, 2002).

Chapter 4 shows that within education systems, differences in performance between students in public and private schools disappear if the schools have similar socio-economic profiles. But does the relative share of students enrolled in public, private-independent and private-dependent schools in an education system affect the incidence of low performance across the system as a whole? For instance, enrolling in private-independent schools may benefit individual students; but by increasing school segregation and reducing the support for public spending on education, it could weaken the overall performance of an education system. The impact of offering private schooling ultimately depends on why the school system opted to make that choice available, the levels of competition, autonomy and accountability (i.e. market mechanisms) already in place in the public school system, and how students and staff in public schools react to increased competition, if they do at all (Couch, Shughart and Williams, 1993; Ferraiolo et al., 2004; Waslander, Pater and van der Weide, 2010).

Data from PISA 2012 show that the percentage of low performers in mathematics decreases marginally as the percentage of students enrolled in private government-dependent schools1 rises, and remains virtually unchanged when the share of students in public schools increases (Figure 5.7). But for every additional percentage point of students enrolled in private-independent schools, the share of low performers in mathematics increases by 0.68 percentage point, on average across PISA-participating countries and economies (Figure 5.9). These results change when comparing the relationship between school governance and the share of top performers (Figure 5.8): the percentage of top performers increases as enrolment in government-dependent private schools increases; it decreases as enrolment in public schools increases; and it remains constant as the population of students enrolled in private-independent schools increases. These results suggest that, on average across PISA-participating countries and economies, the greater the number of students enrolled in privately operated, publicly funded schools in a given school system, the smaller the share of low performers and the larger the share of top performers in mathematics in that system. However, the analyses are correlational, certain countries and economies have a disproportionately large influence on the results, particularly Hong Kong-China, Macao-China and the Netherlands, and the coefficients that measure the association are small.

Why do education systems where more students are enrolled in private, government-dependent schools perform better overall, even if only marginally? One reason could be that having more of these schools results in a greater level of school autonomy across the entire school system, including public schools. When education systems grant similar levels of school autonomy over curricula and assessments to schools, the advantage of having a larger proportion of students enrolled in privately managed, publicly funded schools (and thus having a smaller proportion of underperformers) decreases by 50% (Figure 5.9). In other words, having more students enrolled in government-dependent private schools could be beneficial to the school system as a whole, partly because it introduces more school autonomy over curricula and assessments into the entire system (Figure 5.10) – which is associated with having fewer low performers in mathematics (Figure 5.6). Other potential benefits to school systems that are typically associated with having more students enrolled in private government-dependent schools, such as enhancing school competition or accountability, do not explain their negative association with low performance (Figure 5.9).

SELECTING AND GROUPING STUDENTS

School systems address diversity in students’ backgrounds, interests and performance in different ways (OECD, 2013). They can offer a single, comprehensive programme in which students of different abilities and aspirations are exposed to similar content, pedagogy and peers. Or they can group students of similar abilities, interests and motivation so that what is learned (content and difficulty) and how the content is taught (pedagogy and instruction) can be tailored to better meet students’ needs. This is known as stratification. However, grouping underperforming students together risks exacerbating their struggles with classwork and increasing inequalities in education (Epple, Newlon and Romano, 2002).

Ability groups, tracks or streams can be based on several factors: students’ age at selection; the flexibility of the grouping system (whether transfers between groups/tracks/streams are easy or difficult); the difficulty of course content; the programme orientation (e.g. academic or vocational); where the selection is applied (within classes, between classes, between grades, between schools); the intensity of the grouping (part/full day, some/all subjects); and selection criteria (students’ preference, past marks, placement exam scores, parent/teacher/school recommendations).

The analysis in this report focuses on three indices created by PISA: the index of vertical stratification, the index of ability grouping within schools and the index of between-school horizontal stratification.2

The effects of between-school horizontal stratification depend on the specific characteristics of the grouping. For example, selecting students at an early age strengthens the link between socio-economic background and student performance (OECD, 2013), which is why flexible systems are believed to be better. There is also evidence that placing students in different curricular tracks affects their academic performance, engagement and morale (Lucas, 1999; Trautwein et al. 2006), increases inequality in education opportunities (Maaz et al., 2008), and may be particularly detrimental for disadvantaged students (Epple, Newlon and Romano, 2002; Oakes, 2005; Pekkarinen, Uusitalo and Kerr, 2009).

At the system level, there is no association between the index of between-school horizontal stratification and the share of low and top performers in mathematics (Figures 5.11 and 5.12). This result is consistent with previous studies analysing the impact of the index of between-school horizontal stratification on countries’ and economies’ average PISA scores (Hanushek and Woessmann, 2006; OECD, 2013). Austria, Belgium and the Netherlands, for example, have high values on the index but small shares of underachieving students, while Argentina, Brazil and Chile have low values on the index but high percentages of low performers (Table 5.4). Apart from these specific cases, the association remains relatively weak – even after accounting for the share of disadvantaged students in the school system and for a country’s/economy’s average performance in mathematics (Figure 5.13).

There is little evidence that grade repetition is beneficial for academic and non-academic outcomes (Allen et al., 2010; Ikeda and García, 2014; Manacorda, 2012; Monseur and Lafontaine, 2012); however, many countries, including Belgium, Portugal and Spain, use the practice extensively (OECD, 2013). A significant proportion of the variation in grade repetition is observed at the system level (Goos et al., 2013). System-level analysis shows that more vertical stratification, including grade repetition, is related to a greater incidence of low performance in mathematics, but barely affects the share of top performers in a country/economy (Figures 5.11 and 5.12). Although the association weakens considerably when countries perform similarly in mathematics, it does not disappear entirely (Figure 5.13).

Ability grouping within the same school, the “softest” version of student stratification, appears to be becoming popular again (Garelick, 2013). A recent field experiment conducted by Duflo, Dupas and Kremer (2011) in Kenya observed significant academic gains from separating students, including low-performing students, by achievement into different school classes. These gains persisted one year after the programme ended. In a comparison of 27 strategies to improve student learning conducted for the Abdul Latif Jameel Poverty Action Lab (J-PAL),3 the Kenyan experiment was ranked fourth in a cost-benefit analysis, and first among eight pedagogical interventions (other interventions included adding computers, diagnostic feedback or remedial education). Similar beneficial effects of sorting students by achievement were observed by Borman and Hewes (2002), Collins and Gan (2013) and Zimmer (2003) in the United States. However, correlational evidence at the system level suggests that only a weak relationship exists between ability grouping within schools and the share of low/top performers in an education system. If there is an association, it is the opposite suggested by these studies: more ability grouping within schools is related to a greater number of low performers in mathematics, and fewer top performers (Figures 5.11, Figure 5.12 and 5.13).

References

Allen, C.S. et al. (2010), “Quality of research design moderates effects of grade retention on achievement: A meta-analytic, multi-level analysis”, Education Evaluation and Policy Analysis, Vol. 31/4, pp. 480-499.

Borman, G.D. and G.M. Hewes (2002), “The long-term effects and cost-effectiveness of ‘Success for All’”, Educational Evaluation and Policy Analysis, Vol. 24/4, pp. 243-266.

Burtless, G.T. (1996), Does Money Matter? The Effect of School Resources on Student Achievement and Adult Success, Brookings Institution Press, Washington, DC.

Card D. and A.A. Payne (2002), “School finance reform, the distribution of school spending, and the distribution of student test scores”, Journal of Public Economics, Vol. 83/1, pp. 49-82.

Chetty R., J.N. Friedman and J.E. Rockoff (2014), “Measuring the impacts of teachers II: Teacher value-added and student outcomes in adulthood”, American Economic Review, Vol. 104/9, pp. 2633-2679.

Clark, D. (2009), “The performance and competitive effects of school autonomy”, Journal of Political Economy, Vol. 117/4, pp. 745-83.

Collins, C.A. and L. Gan (2013), “Does sorting students improve scores? An analysis of class composition”,NBER Working Papers, No. 18848, National Bureau of Economic Research, Cambridge, MA.

Couch, J., W. Shughart and A. Williams (1993), “Private school enrolment and public school performance”, Public Choice, Vol. 76, pp. 301-312.

Dinerstein M. and T. Smith, (2014), Quantifying the Supply Response of Private Schools to Public Policies, Department of Economics, New York University, NY, http://econ.as.nyu.edu/docs/IO/37910/dinerstein_jmp.pdf.

Duflo, E., P. Dupas and M. Kremer (2011), “Peer effects, teacher incentives, and the impact of tracking: Evidence from a randomized evaluation in Kenya”, American Economic Review, Vol. 101/5, pp. 1739-1774.

Epple, D., E. Newlon and R. Romano (2002), “Ability tracking, school competition and the distribution of educational benefits”, Journal of Public Economics, Vol. 83/1, pp. 1-48.

Fernandez, R. and R. Rogerson (2003), “Equity and resources: An analysis of education finance systems”, Journal of Political Economy, Vol. 111/4, pp. 858-897.

Ferraiolo, K. et al. (2004), “Teachers’ attitudes and the success of school choice”, Policy Studies Journal, Vol. 32/2, pp. 209-224.

Finn, J.D. and C.M. Achilles (1999), “Tennessee’s class size study: Findings, implications, misconceptions”, Educational Evaluation and Policy Analysis, Vol. 21/2, pp. 97-109.

Fuchs, T. and L. Woessmann (2004), “What accounts for international differences in student performance? A re-examination using PISA data”, CESifo Working Papers, No. 1235, CESifo GmbH, Munich.

Garelick, B. (2013), “Let’s Go Back to Grouping Students by Ability”, The Atlantic, 26 March, Washington, DC.

Goos, M. et al. (2013), “How can cross-country differences in the practice of grade retention be explained? A closer look at national educational policy factors”, Comparative Education Review, Vol. 57/1, pp. 54-84.

Hanushek, E.A. (1997), “Assessing the effects of school resources on student performance: An update”, Educational Evaluation and Policy Analysis, Vol. 19/2, pp. 141-164.

Hanushek, E. and L. Woessmann (2006), “Does educational tracking affect performance and inequality? Differences-in-differences evidence across countries”, The Economic Journal, Vol. 116/510, pp. C63-C76.

Hanushek, E., S. Link and L. Woessmann (2013), “Does school autonomy make sense everywhere? Panel estimates from PISA”, Journal of Development Economics, Vol. 104, pp. 212-232.

Husted, T.A. and L.W. Kenny (2002), “The legacy of Serrano: The impact of mandated equal spending on private school enrolment”, Southern Economic Journal, Vol. 68/3, pp. 566-583.

Ikeda, M. and E. García (2014), “Grade repetition: A comparative study of academic and non-academic consequences”, OECD Journal: Economic Studies, Vol. 2013/1, pp. 269-315, https://doi.org/10.1787/eco_studies-2013-5k3w65mx3hnx.

Krueger, A.B. and D.M. Whitmore (2001), “The effect of attending a small class in the early grades on college‐test taking and middle school test results: Evidence from Project STAR”, The Economic Journal, Vol. 111/468, pp. 1-28.

Lucas, S.R. (1999), “Tracking inequality: Stratification and mobility in American high schools”, Sociology of Education Series, Teachers College Press, New York, NY.

Maaz, K. et al. (2008), “Educational transitions and differential learning environments: How explicit between‐school tracking contributes to social inequality in educational outcomes”, Child Development Perspectives, Vol. 2/2, pp. 99-106.

Manacorda, M. (2012), “The cost of grade retention”, Review of Economics and Statistics, Vol. 94/2, pp. 596-606.

Monseur, C. and D. Lafontaine (2012), “Structure des systèmes éducatifs et équité : Un éclairage international”, in M. Crahay (ed.), Pour une école juste et efficace, De Boeck, Brussels.

Nicoletti, C. and B. Rabe (2012), “The effect of school resources on test scores in England”, ISER Working Papers, No. 2012-13, Institute for Social and Economic Research, Essex.

Oakes, J. (2005), Keeping Track: Schools Structure Inequality, second edition, Yale University Press, New Haven, CT.

OECD (2013), PISA 2012 Results: What Makes Schools Successful (Volume IV): Resources, Policies and Practices, PISA, OECD Publishing, Paris, https://doi.org/10.1787/9789264201156-en.

Pekkarinen, T., R. Uusitalo and S. Kerr (2009), “School tracking and intergenerational income mobility: Evidence from the Finnish comprehensive school reform”, Journal of Public Economics, Vol. 93/7, pp. 965-973.

Piketty, T. and M. Valdenaire (2006), L’Impact de la taille des classes sur la réussite scolaire dans les écoles, collèges et lycées français : Estimations à partir du panel primaire 1997 et du panel secondaire 1995, ministère de l’Éducation nationale, de l’Enseignement supérieur et de la Recherche, Direction de l’évaluation et de la prospective, Paris.

Renzulli, L.A. and L. Evans (2005) “School choice, charter schools, and white flight”, Social Problems, Vol. 52/3, pp. 398-418.

Rivkin, S.G., E.A. Hanushek and J.F. Kain, (2005), “Teachers, schools, and academic achievement”, Econometrica, Vol. 73/2, pp. 417-458.

Saporito, S (2003), “Private choices, public consequences: Magnet school choice and segregation by race and poverty”, Social problems, Vol. 50/2, pp. 181-203.

Schneider M., G. Elacqua and J. Buckley (2006) “School choice in Chile: Is it class or the classroom?”, Journal of Policy Analysis and Management, Vol. 25/3, pp. 577-601.

Slavin, R.E. (1989), “Cooperative learning and student achievement: Six theoretical perspectives”, Advances in Motivation and Achievement, Vol. 6, pp. 161-177.

Sonstelie, J. (1979), “Public school quality and private school enrollments”, National Tax Journal, Vol. 32/2, pp. 343-353.

Trautwein U. et al. (2006), “Tracking, grading, and student motivation: Using group composition and status to predict self-concept and interest in ninth-grade mathematics”, Journal of Educational Psychology, Vol. 98/4, pp. 788-806.

Waslander, S., C. Pater and M. van der Weide (2010), “Markets in education: An analytical review of empirical research on market mechanisms in education”, OECD Education Working Papers, No. 52, OECD Publishing, Paris, https://doi.org/10.1787/5km4pskmkr27-en.

Whitty, G. (1997), “Creating quasi-markets in education: A review of recent research on parental choice and school autonomy in three countries”, Review of Research in Education, Vol. 22, pp. 3-47.

Willms, D.J. (1999), “Quality and inequality in children’s literacy”, in D.P. Keating and C. Hertzman (eds.),Developmental Health and the Wealth of Nations, Guilford Press, New York, NY.

Woessmann L. (2003), “Schooling resources, educational institutions and student performance: The international evidence”, Oxford Bulletin of Economics and Statistics, Vol. 65/2, pp. 117-170.

Zimmer, R. (2003), “A new twist in the educational tracking debate”, Economics of Education Review, Vol. 22/3, pp. 307-315.

Notes

← 1. In PISA, schools are categorised as public, private government-dependent and private government-independent. Public schools are managed directly or indirectly by a public education authority, government agency, or governing board appointed by government or elected by public franchise. Government-dependent private schools are schools that are directly or indirectly managed by a non-government organisation and receive 50% or more of their core funding (i.e. funding that supports the institution’s basic educational services) from government agencies. Government-independent private schools are schools that are managed directly or indirectly by a non-government organisation and receive less than 50% of their core funding from government agencies.

← 2. The index of vertical stratification is based on the degree of variation in 15-year-old students’ grade level across the education system, which reflects both the different starting ages for schooling and the prevalence of grade repetition. The index of ability grouping within schools is based on the extent to which ability grouping, with different content or difficulty, for all mathematics classes is used in the school, according to principals’ reports. The index of between-school horizontal stratification is based on five inter-related indicators: the number of education tracks, the prevalence of vocational or pre-vocational programmes, early selection, academic selectivity, and school transfer rates. All the indices have been standardised.

← 3. Established in 2003 at the Massachusetts Institute of Technology, the Abdul Latif Jameel Poverty Action Lab (J-PAL) is a global network of researchers who use randomised evaluations to answer critical policy questions in the fight against poverty.