Annex A. Technical notes

This technical note provides additional background information on technical aspects relating to the International Early Learning the Child Well-being Study (IELS). It sets out the rationale for the types of assessment used in the study, response rates and other factors influencing the robustness, reliability and comparability of the data. More information on the conceptual and technical aspects of the study can be found in the Assessment Framework and Technical Standards for the study.

Assessment methods

The study used two types of assessment: direct assessment of children’s skills through developmentally-appropriate, interactive stories and games delivered on a tablet device and indirect assessment through reports on children’s skills from parents and educators. The key benefit of direct assessment is that it provides countries with a common basis for comparing children’s early learning. Through careful development, testing and analysis,1 any cultural or other biases are minimised so that readers can have confidence that the results are comparable across countries. Furthermore, delivery of the assessment through a tablet device enhances the reliability of the results through the avoidance of transcription and coding errors.

The indirect assessment provides benefits in triangulating the results from the direct assessment and in providing a fuller picture of children’s development and skills. Parents have knowledge of their child over time and in a range of settings, whereas educators have a comparative group of children at the same age on which to base their assessments. Thus, gaining information from parents as well as from educators provides greater breadth and depth on children’s early learning and development while the direct assessment provides a stronger basis for comparability across countries.

Participation rates

A critical factor influencing the reliability of the results from any survey is the response rates, particularly for any form of direct assessment. The quality standard for child participation rates for IELS was set at 75 %, meaning this level of participation rate provides confidence that the sample is representative of children at that age in that country. Each participating country exceeded this standard. Teacher response rates were also very high, 90 % or higher in each country. While parent response rates were somewhat lower, these were still higher than is generally expected.

Table A.1. Response rates for IELS, by informant and country
	Participation rates
	England (%)	Estonia (%)	United States (%)
Child	94.9	84.1	92.7
Parent	67.5	86.0	71.2
Educator	89.7	94.1	96.4
Note: The participation rates are weighted and based on participating centre/schools and children.

Quality assurance

Standards for administration and assessment procedures – to achieve standardised implementation procedures – were set out in comprehensive manuals, applicable to each participating country. Precise instructions were provided for centre and school coordinators and scripts were provided to study administrators, in addition to the provision of mandatory training.

National and International Quality Assurance Monitors (IQAMs) were appointed to attest that the implementation in each country complied with the standards for the study. These Quality Monitors were independent and observed the administration of the assessments in each participating country in order to attest that the required standards were met. Across all quality assurance activities, the observations showed that all three participating countries generally followed the standardized procedures as outlined in the IELS Technical Standards.

Note

← 1. The types of analysis used for this study included differential item functioning by gender, country, and language, item-level analysis, latent trait-level analysis, and convergent and predictive validity analysis.