Annex A. Technical notes on TALIS 2018
The objective of the Teaching and Learning International Survey (TALIS) in 2018 was to obtain, in each participating country or territory, a representative sample of teachers for each International Standard Classification of Education (ISCED) level in which the country or territory participated. TALIS 2018 identified policy issues that encompass the classroom, teachers, schools and school management, so the coverage of TALIS 2018 extends to all teachers of each concerned ISCED level and to the principals of the schools where they teach. The international sampling plan prepared for TALIS 2018 used a stratified two-stage probability sampling design. This means that teachers (second-stage units, or secondary sampling units) were to be randomly selected from the list of in-scope teachers in each of the randomly selected schools (first-stage units, or primary sampling units). A more detailed description of the survey design and its implementation can be found in the TALIS 2018 Technical Report (OECD, 2019[1]).
A teacher of ISCED level 1, 2 or 3 is one who, as part of his or her regular duties in their school, provides instruction in programmes at that ISCED level. Teachers who teach a mixture of programmes at different ISCED levels in the target school are included in the TALIS universe. There is no minimum cut-off for how much teaching these teachers need to be engaged in at any of the three ISCED levels.
The international target population of TALIS 2018 restricts the survey to those teachers who teach regular classes in ordinary schools and to the principals of those schools. Teachers teaching adults and teachers working in schools exclusively devoted to children with special needs are not part of the international target population and are deemed out of scope. Unlike in TALIS 2008, however, teachers working with special needs students in a regular school setting were considered in scope in TALIS 2013 and 2018. When a school is made up exclusively of these teachers, the school itself is said to be out of scope. Teacher aides, pedagogical support staff (e.g. guidance counsellors and librarians) and health and social support staff (e.g. doctors, nurses, psychiatrists, psychologists, occupational therapists and social workers) were not considered to be teachers and, thus, not part of the TALIS international target population.
For national reasons, participating countries/territories could choose to restrict the coverage of their national implementation of TALIS 2018 to parts of the country. For example, a province or state experiencing civil unrest or in an area struck by a natural disaster could be removed from the international target population to create a national target population that does not include these provinces, states or areas. Participating countries were invited to keep these exclusions to a minimum by keeping the national survey population to at least 95% of the teachers.
TALIS 2018 recognised that attempting to survey teachers in very small schools can be inefficient and difficult. For each ISCED level, surveying teachers in schools with no more than three teachers at a specific ISCED level and those teaching in schools located in geographically remote areas could be a costly, time-consuming and statistically inefficient exercise. Therefore, participating countries were allowed to exclude those teachers for TALIS 2018 data collection, thus creating a national survey population different from the national target population. The national project manager (NPM) for each country/territory was required to document the reasons for exclusion, the size, the location, the clientele, etc., of each excluded school. This documentation was required for each ISCED level in which a country/territory participated.
Within a selected in-scope school, the following categories of teachers were excluded from the sample:
teachers teaching in schools exclusively serving special needs students
teachers who also act as school principals: no teacher data collected, but school principal data collected
For each ISCED level, the same requirements for sample size and precision of estimates were established. To allow for reliable estimation and modelling, while allowing for some amount of non-response, the minimum sample size was set at 20 teachers within each participating school. A minimum sample of 200 schools was to be drawn from the population of in-scope schools. Thus, the nominal international sample size was a minimum of 4 000 teachers for each ISCED level in which a country or territory participated. Participating countries and territories could choose to augment their national sample by selecting more schools, by selecting more teachers within each selected school or by increasing both. Some countries and territories were asked to increase the within-school sample to counterbalance the effect of selecting too many schools with fewer than 20 teachers. The sample size requirement was reduced for some participating countries and territories because of the smaller number of schools available for sampling. In a few cases, because the average number of teachers in the schools was fewer than expected in the international plan, the number of schools sampled was increased to maintain a minimum total number of participating teachers.
In many countries/territories, the separation of grades in ISCED levels does not correspond to a physical separation of school buildings or administrations: schools that offer grades 8 to 12 straddle ISCED levels 2 and 3, but all of ISCED level 2 would not be covered by those schools. Arrangements were made with the NPM and their team to optimise the selection of the school sample either by minimising the overlap of the respective samples (one school is selected for participation in only one ISCED level) or maximising the sample overlap (a selected school contributes to all concerned ISCED levels). However, in the case of maximised overlap, teachers who taught at more than one level would be asked to participate in only one.
As in previous cycles, TALIS 2018 followed the INES (Indicators of Educational Systems) data collection definition of a teacher for sampling and analysis: “A classroom teacher (ISCED 0-4) is defined as a person who plans, organises and conducts a group of activities with the aim of developing students’ knowledge, skills and competencies as stipulated by educational programmes.” (OECD, 2018, p. 43[2]).
The basic principle that guides the adjudication is to determine, for each participating country or territory and for each of the TALIS options, whether the data released to the countries and territories are fit to provide policy relevant, robust international indicators and analysis on teachers and teaching in a timely and cost effective manner.
To establish fitness for use, a number of quality assurance processes were designed and activated throughout the survey process. Some processes relied on expert advice and opinion; some relied on qualitative information and learned judgement; some relied on quantitative information. For more detailed information, please refer to the TALIS 2018 Technical Report (OECD, 2019[1]).
During the adjudication session, each individual dataset – that is, the combination of participating countries/territories, survey options and questionnaire types – was submitted to the same examination. For the first time in a TALIS cycle, principal data were evaluated on their own. In other words, principal and teacher data received separate adjudication evaluations per TALIS option and per country/territory.
The issues evaluated concerned the questionnaire adaptation to national context, translation and verification, quality of the sampling frame, handling of out-of-scope and refusal units (i.e. teachers and/or schools), within-school sampling, data collection, data cleaning, the reports of quality observers, participation rates and overall compliance with the technical standards. Once each survey process had been assessed, a recommended rating was formulated, accounting for the participation rates, and for any unresolved issues.
The adjudication rules, based on participation rates for principals and teachers, are displayed in Table A A.1 and Table A A.2. An explanation of the codes used is given below.
The following bulleted list is a simple guide to help data users appreciate the limitations on use or quality:
Good: The participating country’s/territory’s data can be used for all reporting and analytical purposes and can be included in international comparisons.
Fair (A): National and sub-national estimates can be produced; some teacher characteristics may suffer from a larger standard error (S.E.), hence the warning “Fair”, and no additional warnings to users appear necessary.
Fair (B, only for teacher data adjudication): National and sub-national estimates can be produced; some sub-national estimates may be of lower precision (larger S.E.) if sample size is locally low, hence the warning “Fair”, and no additional warnings to users appear necessary.
Some sub-national estimates may be of lower precision (larger S.E.) if sample size is locally low, hence the warning “Fair”, but a note on data quality could appear, pointing to the outcome of the non-response bias analysis (NRBA).
Since school participation is somewhat lower than under (B), comparing sub-national estimates should be done with care, as some of those results are based on few schools.
Comparing small sub-national estimates with similar groups from other countries/territories is likely to uncover any statistically meaningful differences, as the S.E. are likely too large.
In addition to the warnings issued for the previous category, a note should warn users of indications of non-response biases in some estimates.
Comparisons of sub-national estimates should be limited to the groups with the larger sample sizes.
At this point, the sample represents between 37% and 56% of the teaching workforce, from a rather small sample of schools.
Comparisons with similar groups in foreign countries would not be encouraged.
Poor (E, only for teacher data adjudication): Sub-national estimates would not be recommended; there should be a note pointing out the difficulty of obtaining a representative sample of schools.
Poor (F, only for teacher data adjudication): Limitations similar to those of line E, but there should be a note pointing out the difficulty of obtaining at least 50% participation of the selected sample of schools; risks of having a non-representative sample of schools.
Insufficient: Weights should not be calculated for any official tabulations; hence, data should not be incorporated into international tables, models, averages, etc.
The participation rates and the adjudication rating per participating country/territory at ISCED level 2 are presented in Table A A.3 and Table A A.4.1
This section lists issues to be noted regarding the sampling or field operations that should be considered when interpreting the ISCED level 2 data reported for these countries.
The data collection window for both teachers and principals was extended from the end of the academic year in 2017 to the beginning of the following academic year in 2018.
For principals, data from Australia are located below the line in selected tables in this report and not included in the calculations for the international average. This is because Australia did not meet the international standards for participation rates, as shown in Table A A.3 and Table A A.4.
Colombia: Non-response bias analysis shows no evidence of high risk of school non-response bias on the investigated variables for teachers or principals and, as such, their rating was upgraded from “poor” to “fair”.
Denmark: Non-response bias analysis shows no evidence of high risk of school non-response bias on the investigated variables for teachers or principals and, as such, their rating was upgraded from “poor” to “fair”.
Flemish Community of Belgium: Entries on the sampling frame are administrative units and not “schools” as they are usually defined; a “school” may be comprised of one or several administrative units and the principal would be reporting for the school and not only the selected administrative unit. Therefore, users should exercise care when analysing and comparing school level statistics.
French Community of Belgium: Items regarding the share of students with special needs should be interpreted carefully due to complications that could arise from the interpretation of the definition of special needs. Students studying for a differentiated first degree, which is organised for students who did not pass their primary certificate and who receive extra support and resources, are formally identified as having learning difficulties but most of them do not suffer from any kind of disability.
Georgia: Some translation issues could still exist in the Georgian and Azerbaijani version of the questionnaires.
Israel: Coverage falls below 95%, after post facto exclusion of ultraorthodox schools for low response rates, making coverage identical to that of TALIS 2013.
Latvia: Some translation issues could still exist in the national instruments that could affect the data.
Korea: In four schools, teacher listings were found to be incorrect; those schools were set to “non-participant”.
The Netherlands had a six-week early start and extended collection window.
The Netherlands had an unapproved collection protocol that resulted in the inclusion of some 50 “national” schools that were not included in the international dataset but were left on the national dataset; participation rates were computed on the international dataset.
New Zealand: Coverage was extended to small schools (four or fewer teachers). While the impact of this action on the target population of teachers was negligible, the impact on the target population of principals is important because, compared to TALIS 2013, the target population for principals nearly doubled in size.
Russian Federation: Coverage falls below 95% after the exclusion of Moscow.
Saudi Arabia: Coverage falls below 95% after the sampling excluded two provinces bordering Yemen.
Singapore: Coverage included both privately and publicly managed schools. Nevertheless, private schools were excluded from the sample in TALIS 2013 due to confidentiality issues.
United Arab Emirates: Because of the selection of multi-level schools, the principal data were copied from the original ISCED level 2 principal questionnaire to the corresponding ISCED level 1 and ISCED level 3 forms, except for Question 17 in the principal questionnaire.
References
[1] OECD (2019), TALIS 2018 Technical Report, OECD, Paris, http://www.oecd.org/education/talis/TALIS_2018_Technical_Report.pdf.
[2] OECD (2018), OECD Handbook for Internationally Comparative Education Statistics 2018: Concepts, Standards, Definitions and Classifications, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264304444-en.
Note
← 1. Table A A.3 and Table A A.4 display the participation rate estimates that were the most favourable for the adjudication rating. The most favourable estimates could have been weighted or unweighted depending on the characteristics of the country/territory, the teacher and principal populations and the educational level.