Chapter 2. The design and implementation of the study

The International Early Learning and Child Well-Being Study (IELS) was designed in conjunction with both participating and interested countries. This chapter outlines key elements of the design and implementation of the study. This includes the objectives and guiding principles for the study, as well as the target group of children sampled, the aspects of children’s early learning that were assessed, how the assessments were carried out, and the contextual information gathered on each child.

    

The International Early Learning and Child Well-being Study (IELS) has been developed to inform countries’ efforts to improve children’s early learning and well-being. To achieve this goal, the study was designed in conjunction with participating and interested countries, to ensure the approach responded well to their policy needs. The fundamental objective of the study is to provide countries with reliable, valid and comparable data they can use to benchmark and monitor the performance of their systems in giving all children a strong early start.

This chapter outlines the key features of the overall design and implementation of the study, including:

• objectives and guiding principles

• the conceptual design of the study

• the target group of children sampled in the study

• the aspects of children’s early learning that were assessed

• how children were assessed

• contextual information gathered on each child

• how the study was implemented in the participating countries.

Objectives and guiding principles

The overarching objective of IELS is to support countries in their efforts to improve children’s early learning experiences, to better foster their early development and overall well-being (Box 2.1). Children’s early learning is a strong determinant of their later success in life, across a range of outcomes. As outlined in Chapter 1, sound early learning correlates with positive benefits in later educational achievement and attainment, employment and earnings, mental and physical health, citizenship, well-being and life satisfaction.

Box 2.1. Policy and research questions addressed by the study

IELS was designed to answer the following policy and research questions for each participating country:

• How well are children developing the skills they need, for their well-being and ongoing positive outcomes?

• To what extent are children developing a sound balance of cognitive and social-emotional skills?

• How much variation is there in children’s outcomes?

• How are different groups of children faring, such as:

  • boys versus girls?

  • children from different socio-economic backgrounds?

  • children from migrant families?

  • children who speak a home language that is different from that spoken in their ECEC centre or school?

• What factors are positively or negatively associated with children’s early learning and development?

The vast majority of children who have poor cognitive and social-emotional skills at the age of seven will not catch up with their peers who had a better start in their early years and will face poorer outcomes in schooling and in later life (Stiles and Jernigan, 2010[1]). For many of these children, these poorer outcomes could have been averted if they had received tailored, high-quality interventions in their early years and in the first year or two of early schooling. As children move through primary school, it is increasingly difficult to significantly ameliorate or eliminate learning gaps through, for example, remedial education or other interventions (Shuey and Kankaraš, 2018[2]).

Education systems with a larger proportion of children with poor early development are limited in the extent to which they can achieve success for these children and can perform well overall, as a system. Ensuring as many children as possible develop well in their early years will provide system-level benefits for society as a whole, as well as improving outcomes for individual children and their families.

IELS enables countries to understand how well different groups of children are developing, including any significant differences between girls and boys, socio-economic groups, and children from migrant or linguistically diverse backgrounds.

IELS helps countries to consider where it might be possible to make improvements to enhance children’s early years experiences and outcomes. International comparative data helps policy makers, education leaders and practitioners, and parents to see what can be achieved for children in the early years. This includes key system goals such as mitigating disadvantage and ensuring children are well-positioned to succeed in school. Thus, IELS provides countries with a common language and framework to learn from each other, to improve the relative effectiveness and equity of their approaches and systems.

IELS also provides countries with insights into the focus needed in early primary school. Much can still be done to enhance five-year-olds’ cognitive and social-emotional development, but this will depend on the extent to which early schooling is oriented towards children’s actual development needs and can effectively meet them.

As noted above, the study was developed in conjunction with countries interested in children’s early learning and well-being. As well as setting the objectives for the study, countries agreed on a set of guiding principles to steer the design, development and implementation of the study. These were that the study would be:

• Ethical - ensuring the well-being of children in the study was paramount in all decisions, including developmentally appropriate assessments

• Policy relevant - responding to the policy questions above and enabling changes in policy and/or practices

• Feasible - straightforward to implement

• Reliable, valid and comparable - across countries, languages, cultural contexts and over time

• Efficient - limiting the burden on practitioners and parents, as well as on children

• Cost effective – affordable for a range of countries

• Sustainable – establishing strong foundations to enable multiple cycles.

The conceptual design of the study

The study focuses on those aspects of children’s early learning that have been found to best predict positive later outcomes. These later outcomes include educational achievement and attainment, mental and physical health, employment and earnings, citizenship, wider well-being and life satisfaction.

The areas of early learning that best support positive later outcomes are inter-related and mutually reinforcing. This core set of early learning encompasses emergent literacy, emergent numeracy, self-regulation, and social-emotional skills.

The study also captures relevant contextual information relating to children’s individual characteristics, family and home environments, and their early childhood education and care (ECEC) experiences.

These elements are set out diagrammatically in Figure 2.1.

Figure 2.1. Conceptual framework for the study

The target group of children sampled in the study

The children included in the study were all five years old and in regulated ECEC settings or in school. The survey took a representative sample of children in each participating country. To achieve nationally representative samples, a two-stage probability design was used. In the first stage, a random sample of ECEC centres or schools was selected in each country. In the second stage, children were randomly selected from the list of children who met the age requirements within each of the selected settings. Overall, just under 7 000 children took part in the study.

The assessments were carried out among children at a common age, rather than at a particular stage of education, in order to provide comparability across countries. A stage of education refers to a level or step within an education system, such as the point of entry into school or the last year of ECEC. Stages of education vary across countries and even within countries in some cases. For example, the age of school entry can vary by as much as three years across education systems. Given that the policy and research questions the study is addressing relate to children’s outcomes, from a system perspective, focusing on a common age rather than a common stage gives a better basis for comparison across diverse systems.

Five years is the age at which there is near-universal participation in some form of formal setting in most OECD countries. While some countries have not yet achieved universal participation at this age, most are progressing towards universality as a result of increasing rates of participation in ECEC and younger ages of entry into schooling.

It is simpler and more reliable to carry out the assessment process with five-year-olds than with children who are two years younger or even one year younger. At the age of five, children are able to follow directions and complete straight-forward activities with little assistance.

From a system perspective, five years is often the age when children are about to enter the school system. Comparative assessment information at this point will therefore help countries to later gauge the impact their schooling systems are having on children’s later educational outcomes.

The children were assessed in the regulated ECEC centre or school they attended, to ensure the assessment was carried out in a setting the child was familiar with and for reasons of practicality and cost. Accessing children in formal settings is easier and less costly than locating children and undertaking the assessments in the child’s home or in another type of setting.

The aspects of children’s early learning that were assessed

The early learning domains included in IELS are comprehensive and reflect the critical skills children need for their ongoing learning and well-being. As noted above, children’s learning in different domains is inter-related and mutually reinforcing. Children who are developing well in one area, such as self-regulation, are likely to also be developing in other areas such as emergent numeracy and prosocial skills. While all areas of children’s development matter, the inter-related and overlapping nature of early learning domains means it is not necessary to measure every skill to have an accurate indication of how well a child is developing. Nonetheless, some early learning domains are more predictive than others in terms of children’s later outcomes and some also have greater independent effects on children’s later development than other domains.

The three key aspects of children’s early learning selected for the study each have significant effects on children’s development trajectories and also have independent effects. They were:

• cognitive skills: emergent literacy and emergent numeracy

• self-regulation, and

• social-emotional skills.

In the early years, emergent literacy is predictive of children’s later cognitive and social-emotional development (Shuey and Kankaraš, 2018[2]). IELS has therefore focused on key components of emergent literacy, including listening comprehension, vocabulary and phonological awareness.

Key early numeracy skills reflect children’s ability to identify and understand numbers, an understanding that things can be counted, measured and compared, and an ability to detect patterns and shapes.

Children’s early skills in self-regulation - generally known as executive function - involve remembering and applying information, regulating impulsive behaviours, persisting in tasks and being able to adapt to different rules or circumstances.

Critical early social-emotional skills are those that enable children to form close relationships with others, manage their emotions and actions, and take others’ perspectives. Of particular importance in the early years are children’s levels of emergent empathy, trust and prosocial behaviour.

The aspects of children’s early learning that are included in the study are set out in Figure 2.2.

Figure 2.2. Children’s early learning included in IELS

Children who are progressing well across the above areas will be well positioned to form positive relationships with others, succeed in school and experience positive well-being.

How children were assessed

Three sources of information provided a triangulated perspective on each child’s learning and development:

• a direct assessment of each child’s skills

• feedback from each child’s parents on the child’s learning, development and behaviour

• feedback from each child’s teacher on the child’s learning, development and behaviour.

An explanation of each of these three sources of assessment information is provided below.

Direct assessment of each child’s skills

Children completed the direct assessment on tablets, with one-to-one support from a trained study administrator. The children listened to stories and engaged with cartoon-like characters in these stories by touching or moving items on the screen. The children were able to go at their own pace, for example when moving to a new screen or to a new activity.

The process was simple and intuitive, and was trialled with different groups of children before the study was implemented in the participating countries. Children did not need any prior experience with tablets or other digital devices to successfully complete the assessment.

The study administrators ensured each activity was ready before children started and that the children could navigate their way through the activities. The study administrator remained with the child throughout the assessment.

Each assessment activity took approximately 15 minutes. Two assessment activities were administered per day, across two days. The two-day format worked well for children and there was little attrition of children across the two days.

The stories and other activities the children engaged in during the assessment were interesting, fun and developmentally appropriate for this age group. Two characters – Tom and Mia – guided the children via audio through the activities (Figure 2.3). The names and physical characteristics of these lead characters were adapted to the context of each participating country. There was no reading or writing involved in the direct assessment activities, only visual and audio materials.

Figure 2.3. Tom and Mia, the lead characters from the children’s stories in IELS

As well as being easy to navigate and engaging for children, the digital design of the direct assessment made the results more accurate than they would have been with paper based or observational models of assessment. It also meant the assessment process was more efficient to administer.

Information from parents and teachers

The children’s parents and the teacher or ECEC staff member who knew them best were asked to provide information on their learning and development. This enabled the study to triangulate the information gathered from the direct assessments and indirect assessments, and also gauge children’s development across a broader scope of domains than is possible solely through a direct assessment.

Parents and teachers provided information on the same skills covered by the direct assessment as well as on a broader set of skills and behaviours. They answered questions on the children’s capacity to complete a series of cognitive tasks (assessing their emergent literacy and emergent numeracy), the social-emotional skills and behaviour that they observed at home or at the ECEC centre or school the children attended.

For example, they were asked how often each child:

• joins in with other children playing (prosocial behaviour)

• approaches familiar adults for comfort when upset (trust)

• fights with other children (disruptive behaviour).

Parents and teachers could answer “never”, “rarely”, “sometimes”, “often” or “always”.

Contextual information on each child

Contextual information on children’s home environments and ECEC history is critical for understanding how children’s skills develop and how they can be better supported. This information was collected in addition to information on the individual characteristics of each child. The study gathered contextual information about three key areas.

The individual characteristics of each child

The study collected information about the individual characteristics of the children participating in the study, including their exact ages, gender, any special learning or behavioural needs, and whether they had a low birthweight or had been premature.

Home environment

Children’s home environments also affect their development and well-being. For this reason, the study collected information from parents on:

• the socio-economic status of the family, i.e. parental occupation, parents’ level of education, household income

• household composition, e.g. whether it was a one or two parent household and number of siblings

• the immigration background of the family1

• the language/s spoken in the home

• the activities parents undertake with the children such as reading to them from books and having back-and-forth conversations

• access to and frequency of use of digital devices.

Early childhood education and care experiences

The children’s parents were also asked whether their child had previously participated in ECEC. If so, they were then asked for further details about the types of ECEC their child had participated in, the age the child started ECEC, and whether the child attended more or fewer than 20 hours a week, for each year the child attended ECEC.

How the study was implemented

Three OECD countries participated in this study: England (United Kingdom), Estonia and the United States. The study was implemented in each country by a national study centre set up for this purpose. Each country appointed a national project manager to oversee all aspects of implementation. These included:

• recruiting centres/schools and children to meet sample requirements

• translations, e.g. into Estonian and Russian

• adaptations, to reflect the context of the country or adding additional questions of particular interest to an individual country

• recruiting and training study administrators

• providing resources for the study, e.g. tablets.

Each child in the study was supported on a one-to-one basis throughout the direct assessment by a trained study administrator. The primary role of the study administrators was to ensure the well-being of each child throughout the assessment. They showed children how to navigate the tablet and took them through practice exercises at the beginning of the process to ensure they were ready to start the assessment.

Study administrators remained with the children until they had completed the assessment activities. They ensured the children were happy to continue at each stage of the assessment process and that they took a break between assessment activities, if they wanted to. Study administrators did not guide children on how to respond to the activities, but did help any child who was unsure what to do next, such as how to move to the next screen.

The implementation of the study was monitored in each participating country by international quality monitors. These experts attested that implementation complied with the international quality standards for the study. All three participating countries met these standards.

Participation rates

Participation rates in the study among children, teachers and parents were exceptionally high, compared to similar studies2. Parents readily provided their consent for their children to participate and also completed the survey questionnaire in high numbers.

Response rates from teachers were also high, at 90% or higher in each country, and there appeared to be no concerns amongst teachers about their participation or the time required to complete the questionnaire. In visits the OECD Secretariat made to each participating country at the end of 2018, a number of heads and teachers from participating schools and centres told them that they believed more attention needs to be given to this age group and that an international study of this nature would achieve this goal. They also noted their support for the balanced focus of the study across children’s cognitive and social-emotional skills.

References

[2] Shuey, E. and M. Kankaraš (2018), “The Power and Promise of Early Learning”, OECD Education Working Papers, No. 186, OECD Publishing, Paris, https://dx.doi.org/10.1787/f9b2e53f-en.

[1] Stiles, J. and T. Jernigan (2010), The basics of brain development, https://doi.org/10.1007/s11065-010-9148-4.

Notes

← 1. Defined has having two parents who were born in a country or economy other than that in which the child participated in IELS (or one parent, in the case where information on only one parent was provided).

← 2. Further information on participation rates can be found in the Technical Annex to this report.