Chapter 4. Data and monitoring to improve quality in early childhood education and care
Data and monitoring can be a powerful lever to encourage quality in early childhood education and care (ECEC) by establishing facts, trends and evidence about whether children have equitable access to high-quality ECEC, as well as to inform of measures to achieve improvements. This chapter provides a review of findings on one single feature of data and monitoring systems, i.e. Quality Rating Improvement Systems or QRIS, and its relationship with to process quality and child outcomes. Overall, the implementation of quality monitoring and rating improvement systems was associated with better staff-child interactions, in particular for centres for children aged 3 to 5 and for children aged 0 to 2. The associations and applicability of the QRIS for family daycare settings are more uncertain. There are also mixed findings on whether the use of QRIS is associated with higher levels of children’s development and learning.
Introduction
Data and monitoring can be a powerful lever for encouraging quality in early childhood education and care (ECEC), by establishing facts, trends and evidence about whether children have equitable access to high-quality ECEC, as shown in the Starting Strong IV report (OECD, 2015[5]). Monitoring is understood as the process of systematically tracking aspects of ECEC services, staff, child development and curriculum implementation, with a view to data collection, accountability, enhancing effectiveness or quality (OECD, 2015[5]). For example, research suggests that monitoring can help inform planning, contribute to more efficient resource allocation and increase cost-effectiveness (Bennett, 2002[181]).
In ECEC, the expansion of evidence has played a key role in explaining the success or failure of ECEC programmes, prioritising important areas for ECEC investment and informing ECEC practices through evidence. Gaining an understanding of the performance of ECEC systems through monitoring is important not only for purposes of accountability, but also for policy design and implementation, as well as for informing parents about the level of quality being offered (Levitt, Janta and Wegrich, 2008[182]). Most importantly, monitoring quality can play a key role in determining whether and how provision of ECEC is supporting children’s development and well-being – and what can be done to improve it.
Several studies have found that the collection and monitoring of quality data can lead to increased programme quality, as reflected in the adoption of higher standards, improved classroom environment ratings and teachers with a higher level of credentials (Office of Child Development and Early Learning, 2010[183]; Zellman et al., 2008[184]). For example, in New Jersey (United States) the introduction of a quality rating score for ECEC centres allowed practitioners and management to improve their practices, and positive differences were also observed in children’s literacy skills (Frede et al., 2007[185]; 2009[186]).
Monitoring curriculum implementation may offer insights into what can be improved in curriculum and pedagogical practices, or training for curriculum, which can then enhance quality and child outcomes. Furthermore, family satisfaction is often monitored in surveys. Monitoring such aspects of ECEC helps create a greater understanding of what constitutes quality ECEC (OECD, 2015[5]).
A careful selection of indicators can help improve programmes and the workforce, increase access (especially in underserved communities), and improve practice and child outcomes (Early Childhood Data Collaborative, 2011). Information on structure and process indicators contributes to increased knowledge about the level of quality provision; while information on the demographic and background characteristics of children served can be included in data systems to determine programme effects on target groups and the current state of play of ECEC.
Monitoring quality and measuring effectiveness can be a daunting challenge (OECD, 2015[5]). Data collection requires the capability to co-ordinate a strategic collection of data and maintain high standards of reliability over time across multiple data collectors and geographical regions (Zaslow et al., 2010[160]). In turn, unco-ordinated efforts of documentation make it difficult to manage trade-offs in policy development, and understand how workforce policies or professional development investments are related to children’s learning and development. For example, a United States review of ECEC data systems reveals that, while states are collecting a lot of early education-related data, their efforts are often uncoordinated. US states also struggle to determine which children are simultaneously enrolled in multiple ECEC programmes. This can lead to duplication of services and present obstacles for the co-ordination of the efforts of ECEC programmes working with the same children (Early Childhood Data Collaborative, 2011).
Many countries monitor the service quality of ECEC settings using external evaluation practices and tools (e.g. inspections using rating scales, or surveys and questionnaires with checklists) or internal evaluation practices and tools (e.g. self-assessments with evaluation reports or portfolios) (OECD, 2015[5]). Studies have been conducted, mainly in the United Kingdom and the United States, on the impact certain monitoring tools have had on the quality of ECEC services; but it is often challenging to separate and identify the impact of a single tool or method. In addition, there is very little research on whether one monitoring instrument for ECEC used in a given country or context would result in similar findings or effects in other countries. In general, further research is necessary to create a better understanding of the impact of certain tools or instruments, and whether they are valid and effective.
An analysis of the areas monitored in process quality in different jurisdictions (OECD, 2015[5]) shows that they assess relationships and interactions between staff and children; collaboration between staff and parents; collaboration between colleagues (ECEC staff); sensitivity (refers to child-responsive actions and practices); responsiveness to children’s individual needs; age-appropriateness of practices; pedagogy (the ECEC staff’s methods of teaching and care of the ECEC staff) and the implementation of curriculum.
Summary of findings
The only feature of data and monitoring systems examined in the literature is the use of Quality Rating Improvement Systems, or QRIS (see Box 4.1). No other associations between indicators of monitoring and assessing and staff-child interactions were reported in the literature review.
Quality rating and improvement systems (QRIS) are used in many countries. The majority focus on basic standards concerning structural characteristics, such as staff qualifications, group size or ratio; and other examine aspects of curriculum, process quality or child outcomes. In the United States, the stated goals of a QRIS are generally to improve ECEC quality to enhance children’s development, well-being and learning. QRIS are defined at the state level, and participation is on a voluntary basis. Most QRIS in the United States include the following elements: i) quality standards; ii) accountability measures (monitoring or assigning ratings); iii) support for providers in quality improvement; iv) financial incentives; and v) dissemination of ratings to inform parents or other stakeholders, e.g. (Boller et al., 2015[187]; Zellman and Perlman, 2008[188]).
Overall, the implementation of quality monitoring and rating improvement systems was associated with higher-quality staff-child interactions across all settings. It is important to note that for family daycare, QRIS seem to be particularly important in supporting staff with lower pre-service qualifications to achieve higher quality, as illustrated in one of the studies reported below.
However, the literature review also noted that monitoring and rating systems provided only rough indicators of quality; i.e. the QRIS seemed to be most accurate in distinguishing between low levels and high levels of quality, rather than useful for making more fine-grained quality assessments (Slot, 2017[16]). One exception is when QRIS systems are designed based on other valid observation measures of quality, such as the Environment Rating Scale (ERS), and applied to scale for monitoring purposes. In these cases, there is stronger alignment between the QRIS and process quality measures.
Links between QRIS and children’s development and learning are mixed, with some studies finding significant associations between QRIS ratings and children’s developmental outcomes, and others not finding linkages.
Finally, and although many countries now have quality monitoring systems in place (see (OECD, 2015[5]) for a comprehensive overview), the majority of the research is from the US, and restricted to state-level QRIS systems that are voluntary. There may be thus considerable self-selection of relatively higher-quality centres into these studies that are not representative of the average quality prevalent in the state or country.
This chapter provides an overview of the evidence linking structural mechanisms in data and monitoring to staff-child interactions as well as child development, learning and well-being. To build a solid knowledge base on this theme, it draws on a literature review and meta-analysis that update conceptual knowledge and empirical evidence base for the strength of these associations, while keeping a cross-national focus. The chapter first summarises these two pieces of research, and examines the importance of these mechanisms for process quality in ECEC. Each mechanism is considered in turn, integrating the evidence for centres for children aged 3 to 5, centres for children under the age of 3, and finally family daycare settings. Finally, the chapter examines the evidence for the links between quality mechanisms and child development, learning and well-being.
What does research tell us about the importance of data and monitoring for staff-child interactions in early childhood education and care?
Quality rating and improvement systems (QRIS) are associated with higher-quality staff-child interactions in centres for 3- to 6-year-olds
In US centres for children aged 3 to 5, staff participating in QRIS demonstrated higher emotional and instructional support, as well as better language and literacy environment and curriculum, than centres not participating in QRIS (Jeon, Buettner and Hur, 2014[189]). A recent review of studies that investigated the use of QRIS in the United States showed that overall, there appear to be associations between higher QRIS ratings and alternative measures of quality that were usually based on the environmental rating scale and sometimes the CLASS (Karoly et al., 2016[190]). Some correlations were reported for all eight studies, but the magnitude of the associations appeared weak. In another recent overview (Tout et al., 2017[191]) analysing findings from 10 validation studies examining quality ratings of ECE programmes participating in state QRIS, the results were the same, i.e. QRIS ratings appear to be a helpful tool for state early childhood systems to differentiate programmes at lower and higher levels of quality. Overall, QRIS ratings reflect differences in environments, interactions, and activities between ECE programmes at higher and lower rating levels. Although statistically significant, the differences in observed quality scores between QRIS rating levels were generally small. Findings for family daycare programmes had mixed results.
Evidence of the association between participation in a QRIS and process quality has also been demonstrated by intervention studies. In a US randomised controlled trial in which the intervention group was provided with grants and funding for quality improvement and professional development, as well as on-site coaching, the intervention group showed improvements in the quality of staff-child interactions, when compared to the control group (Boller et al., 2015[187]).
Other US studies have examined specifically the importance of star ratings or quality levels in determining process quality. In these studies, the existing state-level QRIS implementation was related to process quality, as observed with commonly used measures, such as the ECERS, CLASS Pre-K or Caregiver Interaction Scale (CIS), although it mainly distinguished the lowest quality centres from the highest quality centres; i.e. there were no differences across other star or quality levels (Hestenes et al., 2015[192]; Lahti et al., 2015[70]; Lipscomb et al., 2017[193]). In studying which staff profiles were more conducive to higher process quality, there is some indication that the director’s qualifications, rather than the qualifications of classroom staff, were related to higher process quality (Lipscomb et al., 2017[193]). In Australia and China, similar relations between the use of QRIS and process quality were observed. In Australian centres, ratings on the Quality Improvement and Accreditation System were moderately associated with process quality, as measured by the Infant/Toddler Environment Rating Scale (ITERS) or Early Childhood Environment Rating Scale (ECERS) (Fenech, Sweller and Harrison, 2010[68]), particularly for centres providing the lowest quality. In China, quality levels as determined by the QRIS were related with observed process quality (using the ECERS), but these did not distinguish between centres at the lowest levels of quality in one study (Hu, Vong and Mak, 2015[194]) and at the highest levels of process quality in another (Pan, Liu and Lau, 2010[195]).
A study conducted by the Rand Corporation (Zellman and Perlman, 2008[188]) assesses the validity of a QRTIS as a tool for improving child care quality. The QRIS assessed was implemented in 1999, was one of the first of its kind and was created by Qualistar Early Learning, a Colorado (United States) based non-profit organisation. The rating system includes components generally agreed to contribute to high-quality care: classroom environment, child-staff ratios, staff and director training and education, parent involvement and accreditation. The study found that among providers using the QRIS, service quality did improve over time. However, it is not possible to unequivocally attribute improvements to the QRIS: improvements could have been a response to being monitored, for example. Difficulties in measuring the effect of this particular intervention include participant self-selection, the lack of a comparison group and limited data on the implementation of the intervention. The study notes the importance of validating a tool such as the QRIS, particularly as it is sometimes linked to rewarding higher-quality services with, for example, higher per-child subsidies. Tout et al. (2009[196]) find that while QRIS potentially serve as a hub for quality improvement, attaining this goal requires extensive co-ordination across agencies, services and data systems.
QRIS also seem to support staff-child interactions in centres for the youngest children
Regarding centres for children under the age of 3, the QRIS seems to be able to differentiate between the lowest and the highest quality centres in terms of their support for children’s development, well-being and learning (Lipscomb et al., 2017[193]). Moreover, in a randomised controlled trial of US centres offering services to children under the age of 3 in infant and toddler classrooms, intervention centres that used a QRIS received grants and funding for quality improvement and professional development, as well as on-site coaching, and demonstrated improvements in the quality of staff-child interactions by comparison with the control group (Boller et al., 2015[187]). The observed changes occurred largely in overall environmental quality and the quality of the curriculum and learning environment, and smaller changes were also observed in the quality of interactions. Interestingly, intervention centres also presented lower child-to-staff ratios, than the control group of centres.
It is less clear whether the star systems implemented as part of the QRIS also differentiate across other levels of quality. In a state-wide study in the United States (Lipscomb et al., 2017[193]), the levels of the five-star system did not differentiate centres of different quality, namely in terms of the classical measures of observation (e.g. CLASS scores). However, in another US study conducted in two states, observed quality (measured by the ERS and CIS) seemed to be a good indicator of the rated quality level across all four levels of the star system implemented (Lahti et al., 2015[70]). In a small-scale study specifically examining the different aspects of process quality (using the CLASS Toddler), the QRIS star ratings for four out of six quality dimensions seem to be related to observed process quality; however, aspects of the negative climate in the classroom and staff’s behaviour guidance were unrelated to the star rating (Thomason and La Paro, 2009[38]).
Linkages between QRIS and staff-child interactions in family daycare settings are less clear
The United States’ family daycare settings with the highest star rating seem to also present the highest levels of process quality (as observed with the CLASS), concerning the organisation of the environment, the support of children’s development, behaviour, well-being and learning and the provision of (learning) activities (Lipscomb et al., 2017[193]).
However, this association between QRIS and process quality for family daycare seems sometimes to be limited to the content areas covered in the professional development programme embedded in the QRIS: for example, health and safety, or practices in teaching (Hallam, Bargreen and Ridgley, 2013[167]). Moreover, when inspired by a particular measure of quality, the association between QRIS quality levels and process quality also seems to be limited to the measure the QRIS is largely based on, and does not generalise beyond that measure. For example, a QRIS system inspired in the ERS scales demonstrated an association between the star system and process quality as measured by the ERS, but not as measured by the CIS (Lahti et al., 2015[70]).
QRIS systems need to be used with caution when comparing centre-based and home-based types of settings, since this system places strong emphasis on requirements that are harder to meet by home-based providers, such as formal policies, written procedures, specific furnishings and materials. These requirements are considered beyond the scope of measures such as the CLASS that focus exclusively on the quality of the interactions. In the United States, home-based providers reported lower QRIS ratings than centre-based care providers, although there were no significant differences in the observed quality (i.e. staff-child interactions as measured with the CLASS) (Lipscomb et al., 2017[193]).
Links between QRIS and children’s development and learning are mixed.
The outcomes of the use of the QRIS are mixed, with some studies finding significant linkages between QRIS ratings and children’s developmental outcomes and others not finding linkages. In Missouri (United States), children who participated in programmes with higher-quality ratings and especially low-income children showed significantly higher gains on measures of social-emotional development than children in programmes with lower ratings (Thornburg et al., 2009[197]) In contrast, in an evaluation of Colorado’s Qualistar programme, linkages between the ratings and children’s outcomes were not found (Zellman et al., 2008[184])
A recent review of findings from 10 validation studies (Tout et al., 2017[191]) examining quality ratings of ECE programmes participating in state QRIS yielded inconsistent evidence of small positive associations between ratings and patterns of children’s development. Some selective positive associations were found in some states, but not across all developmental domains examined, nor across all measures within a domain. Three out of six studies found evidence that QRIS ratings were associated with some measures of executive function, and four found selective associations between ratings and measures of social-emotional development.
References
[2] Bennett, J. (2002), “Economics as a tool of environmental management”, Ecological Management and Restoration, Vol. 3/2, pp. 79-80, https://doi.org/10.1046/j.1442-8903.2002.00097.x.
[10] Boller, K. et al. (2015), “Impacts of a child care rating and improvement system on child care quality”, Early Childhood Research Quarterly, Vol. 30, pp. 306-315, https://doi.org/10.1016/j.ecresq.2014.10.001.
[18] Fenech, M., N. Sweller and L. Harrison (2010), “Identifying high-quality centre-based childcare using quantitative data-sets: What the numbers do and don’t tell us”, International Journal of Early Years Education, Vol. 18/4, pp. 283-296, https://doi.org/10.1080/09669760.2010.531615.
[6] Frede, E. et al. (2007), The Abbbott Preschool Program Longitudinal Effects Study (APPLES) Interim Report, http://nieer.org/wp-content/uploads/2016/12/APPLES.pdf (accessed on 02 February 2018).
[7] Frede, E. et al. (2009), “The APPLES Blossom: Abbott Preschool Program Longitudinal Effects Study (APPLES) Preliminary Results through 2nd Grade Interim Report”, http://www.state.nj.us/education/ece/research/apples2.pdf (accessed on 2 February 2018).
[23] Hallam, R., K. Bargreen and R. Ridgley (2013), “Quality in family child care settings: The relationship between provider educational experiences and global quality scores in a statewide quality rating and improvement system”, Journal of Research in Childhood Education, Vol. 27/4, pp. 393-406, https://doi.org/10.1080/02568543.2013.822950.
[15] Hestenes, L. et al. (2015), “Comparisons among quality in child care settings: Understanding the use of multiple measures in North Carolina’s QRIS and their links to social-emotional development in preschool children”, Early Childhood Research Quarterly, Vol. 30/Part B, pp. 199-214, https://doi.org/10.1016/j.ecresq.2014.06.003.
[19] Hu, B., K. Vong and M. Mak (2015), “An evaluation of China’s kindergarten quality rating system through the Chinese early childhood environment rating scale - The Zhejiang case”, Early Years, Vol. 35/1, pp. 50-66, https://doi.org/10.1080/09575146.2014.978266.
[12] Jeon, L., C. Buettner and E. Hur (2014), “Examining pre-school classroom quality in a statewide quality rating and improvement system”, Child & Youth Care Forum, Vol. 43/4, pp. 469-487, https://doi.org/10.1007/s10566-014-9248-z.
[13] Karoly, L. et al. (2016), Evaluation of Delaware Stars for Early Success: Final Report, Rand Corporation, Santa Monica, CA.
[16] Lahti, M. et al. (2015), “Approaches to validating child care quality rating and improvement systems (QRIS): Results from two states with similar QRIS type”, Early Childhood Research Quarterly, Vol. 30, pp. 280-290, https://doi.org/10.1016/j.ecresq.2014.04.005.
[3] Levitt, R., B. Janta and K. Wegrich (2008), “Accountability of Teachers Literature Review”, http://dera.ioe.ac.uk/14020/1/1009_Accountability_of_teachers_Literature_review.pdf (accessed on 02 February 2018).
[17] Lipscomb, S. et al. (2017), Oregon’s quality rating improvement system (QRIS) validation study one: Associations with observed program quality, Portland State University and Oregon State University.
[1] OECD (2015), Starting Strong IV: Monitoring Quality in Early Childhood Education and Care, OECD Publishing, Paris, https://doi.org/10.1787/9789264233515-en.
[4] Office of Child Development and Early Learning (2010), “Keystone Stars: Reaching Higher for Quality Early Education”.
[20] Pan, Y., Y. Liu and E. Lau (2010), “Evaluation of the kindergarten quality rating system in Beijing”, Early Education and Development, Vol. 21, pp. 186-204, https://doi.org/10.1080/10409281003631134.
[9] Slot, P. (2017), Literature review on Early Childhood Education and Care quality: Relations between structural characteristics at different levels and process quality, Internal document, OECD, Paris.
[22] Thomason, A. and K. La Paro (2009), “Measuring the quality of teacher–child interactions in toddler child care”, Early Education and Development, Vol. 20, pp. 285-304, https://doi.org/10.1080/10409280902773351.
[24] Thornburg, K. et al. (2009), “The Missouri Quality Rating System School Readiness Study”, http://www.elcmdm.org/Knowledge%20Center/reports/MOQRSreport.pdf (accessed on 16 February 2018).
[14] Tout, K. et al. (2017), “Validation of the Quality Ratings Used in Quality Rating and Improvement Systems (QRIS): A Synthesis of State Studies”, https://www.acf.hhs.gov/sites/default/files/opre/cceepra_qris_validation_report_b508.pdf (accessed on 14 February 2018).
[21] Tout, M. et al. (2009), “Issues for the Next Decade of Quality rating and Improvement systems”, Vol. 3, https://www.acf.hhs.gov/sites/default/files/opre/next_decade.pdf (accessed on 16 February 2018).
[8] Zaslow, M. et al. (2010), Quality, dosage, thresholds, and features in early childhood settings: A review of the literature, Mathematica Policy Research, Washington, DC.
[5] Zellman, G. et al. (2008), ''Assessing Child-Care Quality: How Well Does Colorado's Qualistar Quality Rating and Improvement System Work?'', Rand Corporation, CA, https://www.rand.org/content/dam/rand/pubs/research_briefs/2008/RAND_RB9343.pdf (accessed on 16 February 2018).
[11] Zellman, G. and M. Perlman (2008), Child-care Quality Rating and Improvement Systems in Five Pioneer States: Implementation Issues and Lessons Learned, Rand Corporation, CA, https://www.rand.org/content/dam/rand/pubs/monographs/2008/RAND_MG795.pdf.