Annex A. Technical notes on sampling procedures, response rates and adjudication for TALIS Starting Strong 2018
Sampling procedures and response rates
The objective of the Starting Strong Teaching and Learning International Survey (TALIS Starting Strong) 2018 was to obtain a representative sample in each participating country of staff and leaders providing early childhood education and care (ECEC) for each level of ECEC (settings for children under age 3 and ISCED 02) in which the country participated. This report includes only settings for children under age 3.
The international sampling plan for TALIS Starting Strong used a stratified two-stage probability sampling design. This means that staff (second-stage units, or secondary sampling units) were randomly selected from the list of in-scope staff in each of the randomly selected ECEC settings (first-stage units, or primary sampling units). The leader at each setting (i.e. the person with the most responsibility for administrative, managerial and/or pedagogical leadership) was automatically selected for participation as well. For countries with integrated ECEC systems that participated in data collection for both pre-primary education and settings for children under age 3, settings serving both age groups were split between the two samples so that each setting could be selected for participation in only one level of ECEC. A more detailed description of the survey design and its implementation can be found in the TALIS Starting Strong 2018 Technical Report (OECD, 2019[1]). Staff for children under age 3 are those who, as part of their regular duties in their centre, provide learning opportunities for children in this age group. There is no minimum cut-off for how much time staff need to be engaged with children under age 3 to be included in the TALIS Starting Strong universe.
The international target population of TALIS Starting Strong restricts the survey to those staff and leaders who work in officially registered settings providing ECEC. ECEC settings exclusively for children with special educational needs are deemed out of scope. Also considered out of scope are: short-term substitute educators (to replace staff on sick leave); nannies and other people involved in informal arrangements; volunteers who occasionally came in to provide a special activity; auxiliary staff (e.g. cleaners, cooking staff) who did not interact regularly in a pedagogical manner with the children; and medical and therapeutic staff (e.g. speech therapists, occupational therapists) whose work was primarily non-pedagogical.
For national reasons, participating countries could choose to restrict the coverage of their national implementation of TALIS Starting Strong. For example, Norway decided to exclude home-based settings (within the homes of the respective staff) from their sample, while Denmark, Germany and Israel included these settings in the universe of settings providing services for children under age 3. Participating countries were asked to keep sample exclusions to a minimum by keeping the national survey population to at least 95% of ECEC staff. The national project manager for each country was required to document the reasons for any exclusions.
Sample size requirements
To allow for reliable estimation and modelling while permitting some amount of non-response, TALIS Starting Strong 2018 set the minimum number of ECEC settings per country for each population of interest (pre-primary education and settings for children under age of 3) at 180. Within each setting, the minimum number of staff members selected was eight. If there were fewer than eight staff members in a setting, then all staff members were selected. Participating countries could choose to augment their national sample by selecting more settings, or by selecting more staff within each selected setting, or by increasing both. In some cases, because the average number of staff in the settings was lower than the number expected in the international plan, the number of settings sampled was increased.
Adjudication process
The basic principle that guided the adjudication was to determine, for each participating country and for each level of ECEC, whether the data released to the countries are fit to provide policy-relevant, robust international indicators and analysis on staff and leaders. To establish fitness for use, a number of quality-assurance processes were designed and activated throughout the survey process. Some processes relied on expert advice and opinion, some on qualitative information and learned judgement, some on quantitative information. More detailed information is available in the TALIS Starting Strong 2018 Technical Report (OECD, 2019[1]).
During the adjudication session, each individual dataset (i.e. the combination of participating countries and levels of ECEC) was submitted to the same examination. In addition, both staff participation and leader participation were adjudicated for each combination of participating countries and levels of ECEC.
The issues evaluated concerned the questionnaire’s adaptation to the national context, translation and verification, quality of the sampling frame, handling of out-of-scope and refusal units (i.e. staff and/or centres), within-centre sampling, data collection, data cleaning, quality observers’ reports, participation rates, and overall compliance with the technical standards (see OECD (2019[1])). Once each survey process was assessed, a recommended rating was formulated, accounting for the participation rates and for any unresolved issues. The adjudication rules, based on participation rates for leaders and staff, are shown in Tables A A.1 and A A.2.
The following is a guide to help data users appreciate the limitations on use or quality:
Good: The participating country’s data can be used for all reporting and analytical purposes and should be included in international comparisons.
Fair (A): National and subnational estimates can be produced. Some staff characteristics may suffer from larger standard errors (s.e.), hence the warning “Fair”. No additional warnings to users appear necessary.
Fair (B, only for staff data adjudication): National and subnational estimates can be produced. Some subnational estimates may be of lower precision (larger s.e.) if sample size is locally low, hence the warning “Fair”. No additional warnings to users appear necessary.
Fair (C):
National and subnational estimates can be produced.
Some subnational estimates may be of lower precision (larger s.e.) if sample size is locally low, hence the warning “Fair”. But a note on data quality could appear, pointing to the outcome of the non-response bias analysis.
Since centre participation is somewhat lower than under (B), comparing subnational estimates should be done with care, as some of those results are based on few centres.
Comparing small subnational estimates with similar groups from other participating countries is unlikely to uncover any statistically meaningful differences as s.e. are likely too large.
Poor (D):
In addition to the warnings issued for the previous category, a note should warn users of indications of non-response biases in some estimates.
Comparisons of subnational estimates should be limited to groups with larger sample sizes.
At this point, the sample represents 37-56% of the workforce, from a rather small sample of settings.
Comparisons with similar groups in other participating countries would not be encouraged.
Poor (E, only for staff data adjudication): Subnational estimates would not be recommended. There should be a note pointing out the difficulty of obtaining a representative sample of settings.
Poor (F, only for staff data adjudication): Limitations similar to those of (E), but there should be a note pointing out the difficulty of obtaining at least 50% participation of the selected sample of settings. There are risks of having a non-representative sample of settings.
Insufficient: Weights should not be calculated for any official tabulations. Hence, data should not be incorporated in international tables, models, averages, etc.1
The participation rates and the adjudication rating per participating country for settings for children under age 3 are presented in Tables A A.3 and A A.4. These tables display the participation rate estimates that were the most favourable for the adjudication rating. The most favourable estimates could have been weighted or unweighted depending on the characteristics of the country and the staff and leader population.
Notes regarding use and interpretation of the data
This section lists issues to be noted regarding the sampling or field operations that should be considered when interpreting the data reported for the following countries:
Denmark
Low response rates in the survey may have resulted in bias in the estimates reported and limit comparability of the data.
The data collection period was extended due to a public strike.
Germany
The data collection period was reopened for a few weeks to encourage additional settings and staff to participate.
Non-response bias analysis failed to show that there is not a high risk of setting and staff non-response bias.
In two settings for children under age 3, staff listings were found to be incorrect; these settings were considered as “non-participant”.
A private company provided a list of settings that did not include all home-based settings. These home-based settings were excluded from the sample coverage and estimates of the number of excluded settings are based on national statistics.
Israel
The data collection period was extended for settings serving children under age 3 to accommodate the split system in Israel.
Norway
Home-based ECEC settings were excluded. The exclusion rate therefore exceeds 5%.
References
[1] OECD (2019), TALIS Starting Strong 2018 Technical Report, OECD Publishing, Paris, http://www.oecd.org/education/talis/TALIS-Starting-Strong-2018-Technical-Report.pdf.
Note
← 1. At its November 2018 meeting in Paris, the TALIS Starting Strong 2018 Technical Advisory Group recommended that data from participating countries that had not reached 50% participation should nonetheless be weighted and displayed in tables.
This document, as well as any data and map included herein, are without prejudice to the status of or sovereignty over any territory, to the delimitation of international frontiers and boundaries and to the name of any territory, city or area. Extracts from publications may be subject to additional disclaimers, which are set out in the complete version of the publication, available at the link provided.
https://doi.org/10.1787/99f8bc95-en
© OECD 2020
The use of this work, whether digital or print, is governed by the Terms and Conditions to be found at http://www.oecd.org/termsandconditions.
