copy the linklink copied! 5. Relationship of the Survey of Adult Skills (PIAAC) to other international skills surveys
--42-29000093.png)
This chapter examines the relationship between the Survey of Adult Skills (PIAAC) and previous international skills surveys, notably the International Adult Literacy Survey (IALS) and the Adult Literacy and Life Skills Survey (ALL). It also discusses the differences and similarities between the Survey of Adult Skills and the Literacy Assessment and Monitoring Programme (LAMP) of UNESCO and the STEP Measurement Study, conducted by the World Bank.
Prior to the Survey of Adult Skills (PIAAC), two international assessments of adult skills were conducted in OECD countries: the International Adult Literacy Survey (IALS) of 1994-98 and the Adult Literacy and Life Skills Survey (ALL) of 2003-07.1 In total, 18 countries/economies participating in the Survey of Adult Skills also participated in one or both of its predecessors. In addition, both UNESCO (the Literacy Assessment and Monitoring Programme – LAMP) and the World Bank (the STEP Measurement Study) have also conducted adult literacy and skills surveys in recent years.
This chapter describes the relationship between the Survey of Adult Skills and these other international adult skills surveys. Its objective is to help readers understand the links between the surveys and the factors that need to be taken into account when comparing results. It focuses on the Survey of Adult Skills, and IALS and ALL given the fact that many countries/economies participating in the Survey of Adult Skills also participated in IALS and/or ALL, and given the ultimate objective of providing comparable measures of proficiency in the domains of literacy and numeracy. Specifically, the discussion covers the factors that affect the degree to which valid comparisons may be made among the literacy and numeracy scores from the Survey of Adult Skills and the other assessments (see, for example, Mislevy, 1992), in particular:
-
The comparability of the constructs measured and the content of the instruments used
-
The comparability of the populations assessed
-
The degree of similarity of the methodology used when conducting the survey.
The first four sections of the chapter cover the relationship between the Survey of Adult Skills and IALS and ALL, including information on the countries/economies for which repeated measures of literacy and/or numeracy proficiency are available; links between the surveys, in terms of the constructs, assessment instruments and background questionnaires; and the operational aspects of the three surveys. The final section describes the relationship between the Survey of Adult Skills and LAMP and STEP, respectively.
copy the linklink copied! Countries and economies participating in the survey of adult skills (PIAAC) and ials and/or all
In total, 20 of the countries or economies participating in the first two rounds of the Survey of Adult Skills participated in IALS, and six participated in both IALS and ALL (Table 5.1). IALS was undertaken in three separate waves with data collection occurring in 1994, 1996 and 1998. ALL was undertaken in two waves with data collection taking place in 2003 and 2006-08. Table 5.1 lists the countries/economies participating in each of IALS, ALL and PIAAC together with the dates of data collection.
copy the linklink copied! Constructs and instruments: The survey of adult skills, all and ials
The domains of skills assessed in the Survey of Adult Skills and its predecessors are presented graphically in Table 5.2. Shading indicates links between assessments in terms of the constructs measured and the content of the assessment instruments.
The domains of literacy, including reading components, and problem solving in technology-rich environments, as assessed in the Survey of Adult Skills, represent new domains of assessment, notwithstanding the close links between literacy as conceived and measured in the Survey of Adult Skills and prose and document literacy as assessed in IALS and ALL. Reading components is also a new domain. The conceptualisation of numeracy in the Survey of Adult Skills is very close to that used in ALL.
Literacy
As defined in the Survey of Adult Skills, literacy is conceived more broadly than in IALS and ALL. Literacy encompasses the domains of prose and document literacy,2 which were assessed separately in IALS and ALL. In addition, literacy includes the reading of digital texts in addition to the reading of print-based texts (see Chapter 1). Apart from including digital texts and mixed-format texts (i.e. texts containing both continuous and non-continuous elements) in the corpus of texts defining the domain, there is, by design, considerable overlap between the concept of literacy and those of prose and document literacy (see OECD/Statistics Canada, 2005, pp. 277-290, for a description of the conceptualisation of prose and document literacy). The conceptualisation of the cognitive processes used in gaining meaning from text, the definition of the contexts in which reading takes place and the factors affecting the difficulty of test items are very similar.
In addition, the Survey of Adult Skills is linked to IALS and ALL through the use of a number of common test items. Twenty-nine of the 52 literacy items included in the computer-based version of the literacy assessment were linking items (i.e. items that had been used in the assessments of prose and document literacy in IALS and/or ALL). In the paper-based versions, 18 of the 24 items administered were linking items. Reading components represents a new element of the assessment of literacy that was not included in either IALS or ALL.
The reading-components assessment in the Survey of Adult Skills should not be confused with the identically named reading-components assessment of the International Study of Reading Skills (ISRS) (Grenier et al., 2008), administered in 2005 to a sample of respondents to ALL in Canada and to a sample of just over 1 000 adults (for the most part enrolled in adult literacy centres) in the United States (Strucker, Kirsch and Yamamoto, 2007). The ISRS tested word recognition, vocabulary, basic text processing and spelling.3 The only direct point of convergence between the ISRS and the Survey of Adult Skills is in the area of vocabulary, where a broadly similar approach was used.
Numeracy
The conceptualisation of numeracy in the Survey of Adult Skills is similar to that used in ALL. As can be seen in Table 5.2 above, the domain of numeracy was introduced in ALL to replace that of quantitative literacy, which had been measured in IALS. Quantitative literacy covered the skills needed to undertake arithmetic operations such as addition, subtraction, multiplication, and division, either singly or in combination, using numbers or quantities embedded in printed material.
Numeracy is conceived as a broader domain than quantitative literacy, covering a wider range of quantitative skills and knowledge, not just computational operations. It also covers a broader range of situations in which actors have to deal with mathematical information of different types, not just situations involving numbers embedded in printed materials (Gal et al., 2005, p. 151). As in the case of the literacy assessment, a number of numeracy items are common to both the Survey of Adult Skills and ALL. Of the 52 literacy items included in the computer-based version of the numeracy assessment, 30 were taken from ALL. In the paper-based versions, 19 of the 24 items administered had been previously used in ALL.
Problem solving in technology-rich environments
The domain of problem solving in technology-rich environments is one that has not previously been assessed. In particular, its emphasis on “information problems” and the solution of problems in an ICT context, rather than on analytic problem-solving skills per se, distinguishes it from previous conceptualisations of problem solving.4
Mode of delivery
A major difference between the Survey of Adult Skills and IALS and ALL is that it was designed as a computer-based assessment (with a pencil-and-paper option for respondents who did not have sufficient computer skills to take the assessment in computer-based mode). In contrast, both IALS and ALL were exclusively paper-and-pencil-based assessments in which respondents received printed booklets in which they responded to questions in writing.
Despite the similarity in the skills measured and the use of common items, the difference in the delivery mode adopted for the Survey of Adult Skills compared to IALS and ALL had the potential to negatively affect the comparability of results in the domains of literacy and numeracy. It was possible that response patterns could be affected by the mode of delivery of test items; and the difficulty and degree of discrimination of some items could vary according to whether they were answered in computer-based or paper-based format.
The existence and extent of mode effects was explored in the field test, which was implemented from March to July 2010. A proportion of respondents undertaking the field test in each country was randomly assigned to either the computer-based or paper-based version of the assessment.5 The results for the two randomly equivalent samples were compared. Overall, no significant mode effects were identified.6
copy the linklink copied! Comparability of background questions
The extent to which comparisons can be made between the Survey of Adult Skills and its predecessors depends not only on the psychometric links between the assessments. For the results for subgroups of the population to be reliably compared between surveys, the definitions of the relevant subgroups must be similar between the surveys.
In areas such as the personal characteristics of respondents, language background, immigration status, educational attainment and participation, and labour-force status, there is high degree of similarity between the questions and response categories used in the Survey of Adult Skills and those used in IALS and ALL. Comparable information is also collected regarding literacy, numeracy and ICT use at work. Where there are differences in response categories, derived variables were created to facilitate comparisons between assessments; these have been included in published data files with full documentation for analysts. Annex B provides a list of the background variables common to the Survey of Adult Skills and one or both of IALS and ALL.
A revised version of the International Standard Classification of Occupations (ISCO) – ISCO-08 – was adopted in 2007, replacing the former ISCO-88 (ILO, 2007). This has necessitated the mapping of the ISCO-88 categories used in the coding of occupations in IALS and ALL to the ISCO-08. As a consequence, comparisons can only be made at the one digit level between the occupational information contained in the Survey of Adult Skills and that available from IALS and ALL.
copy the linklink copied! Survey methods and operational standards and procedures
Other things being equal, differences in design, methodology and operational procedures may have a potentially significant effect on the comparability of different assessments. This section presents a comparison of the extent of comparability between IALS, ALL and the Survey of Adult Skills in terms of:
-
the target population
-
sample design and procedures
-
survey operations
-
response rates.
The target population
The target population defined for both IALS and ALL is identical to that of the Survey of Adult Skills, i.e. civilian, non-institutionalised persons aged 16-65. In each of the three surveys, participating countries/economies were required to use sampling frames that covered the target population. Exclusions of up to a maximum of 5% of the target population were permitted.7 The estimated coverage of the target population in each of the three surveys is presented in Table 5.3.
Sample design
In the Survey of Adult Skills, ALL and IALS, participating countries/economies were required to use a probability sample representative of the target population. Of the countries/economies participating in the Survey of Adult Skills and one or both of IALS or ALL, there is only one documented case of deviation from this requirement. In IALS, Germany employed a non-probability selection method at the second stage of its three-stage sample design (Murray et al., 1998, p. 28). However, the extent of deviation from strict probability sampling was assessed to be “relatively minor” and was not believed to have “introduced significant bias into the survey estimates” (Murray et al., 1998, p. 39).
Survey operations
Both the degree of standardisation of survey procedures and the effort put into monitoring compliance with these standards have been greater in the Survey of Adult Skills than was the case in either IALS or ALL. An external review of the implementation of the first round of IALS8 conducted in the second half of 1995 (Kalton, Lyberg and Rempp, 1998) concluded that while there were no concerns regarding the development of instrumentation: “The variation in survey execution across countries is so large that we recommend that all comparative analyses across countries should be interpreted with due caution” (Kalton, Lyberg and Rempp, 1998, p. 4). In particular, while guidance on survey procedures was provided to the participating countries, the reviewers found that little was done to “enforce adherence to specific procedures” (Kalton, Lyberg and Rempp, 1998, p. 4). Quality-assurance procedures were subsequently improved for the second and third rounds of IALS (OECD/Statistics Canada, 2000, p. 129) and in ALL.9
Maximising standardisation in processes and procedures and, therefore, minimising any differentials in error resulting from variation in implementation was a central objective of the Survey of Adult Skills. The quality-assurance and quality control procedures put in place are among the most comprehensive and stringent ever implemented for an international household-based survey. The standards that participating countries/economies are required to meet in implementing the Survey of Adult Skills were set out in a comprehensive set of Technical Standards and Guidelines (PIAAC, 2014). These were accompanied by a quality-assurance and quality-control process that involved review of and sign-off by the international consortium at key stages of implementation (e.g. sampling designs) and data collection throughout the project. The results of the quality-control activity fed into an assessment of the overall quality of the data from each participating country.
Survey response
Non-response is a potentially significant source of error in any sample survey. In comparing results across the Survey of Adult Skills, IALS and ALL, it is important to be aware of the response rates for the different surveys. Table 5.4 presents the response rates of the three surveys for those countries/economies for which repeated observations are available.
copy the linklink copied! Educational attainment in IALS
For four countries participating in IALS (the Czech Republic, Germany, Poland and the United Kingdom), the proportion of the adult population classified as having educational attainment at lower secondary level (ISCED 0-2)10 is considerably lower and the proportion with secondary attainment (ISCED 3-4) is considerably higher than is found in other statistics on educational attainment for the years as IALS data was collected (1994 or 1996 depending on the country) such as those published by the OECD in Education at a Glance (Gesthuizen, Solga and Künster, 2009). Analysts should bear this in mind when comparing results between IALS and ALL and the Survey of Adult Skills for these countries. Gesthuizen, Solga and Künster (2009) propose a method to correct the attribution of respondents to levels of educational attainment in the IALS data set that provides distributions in line with other attainment statistics.
copy the linklink copied! Summary of the relationship between the survey of adult skills (PIAAC), IALS and ALL
In summary, the Survey of Adult Skills was designed to be linked psychometrically with IALS and ALL in the domain of literacy and ALL in the domain of numeracy. Analysis of data from the field trial and from the main data collection confirmed that results from IALS, ALL and the Survey of Adult Skills could be placed on the same scale in literacy and that the results from the Survey of Adult Skills and ALL could be placed on the same scale in numeracy. At the same time, caution is advised in comparing the results of the Survey of Adult Skills and previous surveys, particularly IALS, due to possible variations in operational procedures and low response rates in some countries/economies.
copy the linklink copied! The relationship between the survey of adult skills (PIAAC), LAMP and STEP
Two other international surveys of adults that have been administered since 2003 – UNESCO’s Literacy Assessment Monitoring Programme (LAMP) and the World Bank’s STEP measurement study11 – have assessed either the same (STEP) or related (LAMP) skills as the Survey of Adult Skills. Table 5.5 provides an overview of the common skills assessed in the three studies; the relationship of these studies to the Survey of Adult Skills is addressed in more detail below.
LAMP
UNESCO’s Literacy Assessment and Monitoring Programme (LAMP) was an assessment developed by the UNESCO Institute for Statistics (UIS). Its aim was “to provide policymakers with robust information on population profiles in terms of literacy and numeracy” (UNESCO Institute for Statistics, 2009, p. 7). LAMP assessed the proficiency of adults in the domains of prose literacy, document literacy and numeracy. In addition, it involved an assessment of reading components (recognition of letters and numbers, word recognition, print vocabulary, sentence processing and passage fluency). The design of LAMP owed much to that of IALS and ALL. In particular, the conceptualisation of prose and document literacy and numeracy was based on the assessment frameworks developed for these studies and the assessment instruments contained items from IALS and ALL. Ten countries undertook the field testing of the LAMP instruments. Four countries (Jordan, Mongolia, Palestinian Authority and Paraguay) implemented the assessment in 2011. Some results have been published in UIS (2017). The implementation of LAMP followed a rather different model from that adopted in the Survey of Adult Skills. In particular, the timing of implementation was left to the discretion of participating countries, and process of quality assurance and control was far lighter.
Despite its relationship to IALS and ALL (and, by virtue of this, to the Survey of Adult Skills) at the level of the assessment frameworks, LAMP was not designed to have psychometric links to either of these surveys in any of the domains measured. In the presentation of results, the distinct nature of the LAMP scales was emphasised by using scales with values from 0-2 000 with a mean of 1 000 (as opposed to a 0-500 point scale) and by defining three (as opposed to five) proficiency levels.
STEP
The World Bank’s STEP measurement study was launched in 2010 with the aim of enhancing the information available to policy makers regarding the level and distribution of skills relevant to the labour market in the adult populations of developing countries (Gaëlle et al., 2014). STEP involved both a household survey and an employer survey. A total of seventeen countries have participated in the study over the period 2010-17. Of these, thirteen participated in the household survey (Armenia, Plurinational State of Bolivia, Colombia, Georgia, Ghana, Kenya, Lao People’s Democratic Republic, Republic of North Macedonia, Sri Lanka, Ukraine, Viet Nam, and Yunnan province of the People’s Republic of China (hereafter “China”). Five countries participated in the employer survey only: Albania, Azerbaijan, Bosnia and Herzegovina, Kosovo and Serbia.
The household study contained modules focused on cognitive skills, technical skills and socio-emotional skills. In addition to collecting self-reported information regarding certain cognitive skills, the cognitive module involved administering a direct assessment of reading literacy based on the Survey of Adult Skills instruments.
The STEP literacy assessment (ETS, 2014) involved two versions. The first used an extended version of the paper-based literacy assessment administered by the Survey of Adult Skills as well as the latter’s reading components assessment. This was implemented in Armenia, Bolivia, Colombia, Georgia, Ghana, Kenya, Ukraine and Viet Nam. The second used the literacy core test from the Survey of Adult Skills only, and was implemented in North Macedonia, Lao People’s Democratic Republic, Sri Lanka and Yunnan province in China. The STEP literacy assessment was designed with the objective of recording results on the literacy scale of the Survey of Adult Skills.
There are important differences between STEP and the Survey of Adult Skills. First, the target population for STEP was not the resident adult population of the participating country or region as a whole but the population of urban centres. Second, although similar technical standards for the literacy assessment were followed in both surveys, the operational standards applied (including the quality-assurance and control processes) followed protocols established by each data collection agency. Both these factors need to be taken into account when comparing results from STEP and the Survey of Adult Skills.
References
Educational Testing Services (ETS) (2014), A guide to understanding the literacy assessment of the STEP Skills Measurement Survey. Accessed at: http://microdata.worldbank.org/index.php/catalog/2010.
Gaëlle, P., M.L. Sanchez Puerta, A. Valerio and T. Rajadel (2014), “STEP Skills Measurement Surveys - Innovative Tools for Assessing Skills”, Social Protection & Labor Discussion Paper 1421, World Bank Group.
Gal, I., M. van Groenestijn, M. Manly, M.J. Schmitt and D. Tout (2005), “Adult Numeracy and its Assessment in the ALL Survey: A Conceptual Framework and Pilot Results”, in S. Murray, Y. Clermont and M. Binkley (eds) (2005), Measuring Adult Literacy and Life Skills: New frameworks for Assessment, Statistics Canada, Ottawa, Catalogue No. 89-552-MIE, No. 13.
Gesthuizen, M., H. Solga and R. Künster (2009), “Context Matters: Economic Marginalization of Low-educated Workers in Crossnational Perspective”, in European Sociological Review, Vol. 27, No. 2, 2011, pp. 264-280.
Grenier, S., S. Jones, J. Strucker, T.S. Murray, G. Gervais and S. Brink (2008), Learning Literacy in Canada: Evidence from the International Survey of Reading Skills, Statistics Canada, Ottawa, Catalogue No. 89-552-MIE, No. 19.
International Labour Organization (ILO) (2007), “Resolution Concerning Updating the International Standard Classification of Occupations”, www.ilo.org/public/english/bureau/stat/isco/docs/resol08.pdf.
Kalton, G., L. Lyberg and J.-M. Rempp (1998), “Review of Methodology”, in T.S. Murray, I. Kirsch and L. Jenkins (eds) (1988), Adult Literacy in OECD Countries: Technical Report on the First International Adult Literacy Survey, National Center for Education Statistics, Office of Educational Research and Improvement, Washington, D.C.
Mislevy, R.J. (1992), Linking Educational Assessments: Concepts, Issues, Methods, and Prospects, Policy Information Center, Educational Testing Service, Princeton.
Murray, T.S., I. Kirsch and L. Jenkins (eds) (1998), Adult Literacy in OECD Countries: Technical Report on the First International Adult Literacy Survey, National Center for Education Statistics, Office of Educational Research and Improvement, Washington, DC.
OECD (2019), Technical Report of the Survey of Adult Skills, Third Edition, www.oecd.org/skills/piaac/publications/PIAAC_Technical_Report_2019.pdf.
OECD/Statistics Canada (2011), Literacy for Life: Further Results from the Adult Literacy and Life Skills Survey, OECD Publishing, Paris, https://doi.org/10.1787/9789264091269-en.
OECD/Statistics Canada (2005), Learning a Living: First Results of the Adult Literacy and Life Skills Survey, OECD Publishing, Paris, https://doi.org/10.1787/9789264010390-en.
OECD/Statistics Canada (2000), Literacy in the Information Age: Final Report of the International Adult Literacy Survey, OECD Publishing, Paris, https://doi.org/10.1787/9789264181762-en.
PIAAC (2014), PIAAC Technical Standards and Guidelines, OECD Programme for the International Assessment of Adult Competencies, www.oecd.org/site/piaac/PIAAC-NPM%282014_06%29PIAAC_Technical_Standards_and_Guidelines.pdf.
Strucker, J., I. Kirsch and K. Yamamoto (2007), The Relationship of the Component Skills of Reading to IALS Performance: Tipping Points and Five Classes of Adult Literacy Learners, NCSALL Reports #29, National Center for the Study of Adult Learning and Literacy, Cambridge, www.ncsall.net/fileadmin/resources/research/report_29_ials.pdf.
UNESCO Institute for Statistics (UIS) (2017), Implementation in Diverse Settings of the Literacy Assessment and Monitoring Programme (LAMP): Lessons for Sustainable Development Goal 4 (SDG 4), UNESCO Institute for Statistics, accessed at: http://uis.unesco.org/sites/default/files/documents/implementation-diverse-settings-lamp-2017-en.pdf.
UNESCO Institute for Statistics (UIS) (2009), The Next Generation of Literacy Statistics: Implementing the Literacy Assessment and Monitoring Programme (LAMP), Technical Paper No. 1, UNESCO Institute for Statistics, Montreal, http://uis.unesco.org/sites/default/files/documents/the-next-generation-of-literacy-statistics-implementing-the-literacy-assessment-and-monitoring-programme-lamp-en_0.pdf.
World Bank (n.d.), STEP Skills Measurement Study, http://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1281723119684/STEP_Skills_Measurement_Brochure_Jan_2012.pdf.
Notes
← 1. See OECD/Statistics Canada (2000), OECD/Statistics Canada (2005) and OECD/Statistics Canada (2011) for information on the methods and results of IALS and ALL.
← 2. In IALS and ALL, prose literacy was defined as the knowledge and skills needed to understand and use continuous texts – information organised in sentence and paragraph formats. Document literacy represented the knowledge and skills needed to process documents (or non-continuous texts) in which information is organised in matrix structures (i.e. in rows and columns). The type of documents covered by this domain included tables, signs, indexes, lists, coupons, schedules, charts, graphs, maps, and forms.
← 3. Word recognition was assessed with the Test of Word Recognition Efficiency (TOWRE) – real words (TOWRE-A) and pseudo-words (TOWRE-B). Vocabulary was assessed with the abridged Peabody Picture Vocabulary Test (PPVT-m), general processing skills were assessed with the Rapid Automatized Naming (RAN) test and the Digit-Span test, and spelling with an abridged version of a test developed by Moats (Grenier, et al., 2008, p. 94).
← 4. In ALL, problem solving was defined as “goal-directed thinking and action in situations for which no routine solution procedure is available” (OECD/Statistics Canada, 2005, p.16).
← 5. Of the respondents who passed the ICT core, 27% were directed to the paper-based assessment and 63% to the computer-based assessment.
← 6. A complete description of the field test design and analysis of mode effects can be found in Chapters 18 and 19 of the Technical Report of the Survey of Adult Skills, Third Edition (OECD, 2019).
← 7. Exclusions were permitted for “practical operational” reasons in ALL (OECD/Statistics Canada, 2005, p. 216). Murray, Kirsch and Jenkins (1998, p. 26) provides a list of exclusions in participating countries for the first wave of IALS.
← 8. The first round involved nine countries: Canada, France, Germany, Ireland, the Netherlands, Poland, Sweden, Switzerland, and the United States. France withdrew from the study in 1995 citing concerns regarding data quality.
← 9. A technical report covering the first wave of IALS was published in 1998 (Murray, Kirsch and Jenkins [eds], 1998). Some information on the implementation of the 2nd and 3rd rounds of IALS and the implementation of ALL is available in the methodological appendices of OECD/Statistics Canada (2000), OECD/Statistics Canada (2005), and OECD/Statistics Canada (2011). However, technical reports covering the 2nd and 3rd rounds of IALS and the two rounds of ALL have not been released.
← 10. ISCED is the International Standard Classification of Education.
← 11. Information regarding LAMP can be found at: www.uis.unesco.org/literacy/Pages/lamp-literacy-assessment.aspx and information regarding STEP in Gaëlle et al. (2014).
Metadata, Legal and Rights
https://doi.org/10.1787/f70238c7-en
© OECD 2019
The use of this work, whether digital or print, is governed by the Terms and Conditions to be found at http://www.oecd.org/termsandconditions.
