Executive summary
Advances in artificial intelligence (AI) are ushering in a large and rapid technological transformation. Understanding how the capabilities of AI relate to human skills and how they develop over time is crucial for understanding this ongoing process. Knowing what AI can do compared to humans can help predict which skills may become obsolete and which skills may become more significant in the years ahead. This knowledge base can help policy makers reshape education systems in ways that best prepare students for the future and provide opportunities to adult learners to renew their skills.
This report follows up an earlier pilot study, collecting expert evaluations of how well AI can do the literacy and numeracy tests of the OECD Survey of Adult Skills of the Programme for International Assessment of Adult Competencies (PIAAC). It shows how AI capabilities in these domains have evolved since the pilot assessment in 2016, up until mid-2022 (shortly before the release of ChatGPT). Assessing AI capabilities in literacy and numeracy is indicative of AI’s potential impact on work and life since these skills are relevant in most social contexts and work situations.
The study is part of a comprehensive ongoing project for assessing computer capabilities and their implications for work and education. The AI and the Future of Skills (AIFS) project at OECD’s Centre for Educational Research and Innovation (CERI) uses several information sources to develop measures of AI capabilities that are understandable, comprehensive, repeatable and policy relevant.
Both the pilot and this follow-up asked computer scientists to rate AI’s capacity to answer the questions on PIAAC’s literacy and numeracy tests. AI’s likely performance on the tests was determined by looking at the majority expert opinion on each question. The use of standardised education tests enables the comparison to human capabilities, allows tracking AI progress across time and provides understandable AI measures. However, experts did not always agree in their evaluations. The study aimed at improving the methodology for eliciting expert knowledge on AI with standardised tests to address this challenge.
Experts expect AI to perform well on both the literacy and numeracy tests of PIAAC.
According to experts, AI can answer around 80% of the PIAAC literacy questions. It can solve most of the easy questions, which typically involve locating information in short texts and identifying basic vocabulary. It can also master many of the harder questions, which require navigating across larger chunks of text to formulate responses. This evaluation rests on high consensus among experts.
According to experts, AI can solve around two-thirds of the PIAAC numeracy test. However, there is disagreement behind this result. Some experts imagined narrow AI solutions for separate numeracy questions. Others considered general systems that can reason mathematically and process all kinds of numeracy questions similar to those in PIAAC. This led to diverging evaluations, with the latter experts giving lower ratings than the former.
AI capabilities in literacy have increased substantially since 2016.
A comparison to the pilot assessment reveals considerable improvement in AI’s literacy capabilities since 2016. The expected success rate of AI in the literacy test has increased by 25 percentage points since then. This reflects the technological breakthroughs in natural language processing (NLP) in the period, related to the introduction of pre-trained language models, such as GPT.
The discussion with experts suggested that numeracy capabilities of AI are unlikely to have changed much between 2016 and 2021. While formal mathematics underlying numeracy problems are easily automatable, extracting formal models from tasks that require general knowledge and are expressed in language and in images has received less research attention.
According to experts, AI will be able to solve the entire literacy and numeracy tests by 2026.
Given recent technological advancements and the heavy investment and research in NLP, experts judged that AI’s literacy capabilities will continue to develop.
More recently, large language models have been fine-tuned and applied for mathematical problems. The field has produced important benchmark tests as well as systems that perform well on these tests. These trends led experts to expect that AI will advance considerably in numeracy over the next few years.
AI can potentially outperform large shares of the population in literacy and numeracy.
PIAAC assesses respondents’ proficiency in literacy and numeracy on several levels – from low (Level 1 and below) to high (Levels 4-5). Following the evaluation of experts, AI’s potential performance in literacy is close to that of adults with proficiency at Level 3. Across the OECD countries in PIAAC, on average, 90% of adults are at or below Level 3 in literacy and only 10% perform better than Level 3.
The AI numeracy performance assessed by experts is close to that of adults at proficiency Level 2 on the easier and intermediate PIAAC questions, and similar to that of Level 3 adults on the harder questions. Across OECD countries with data, on average, 57% of adults are at or below Level 2 in numeracy, and 88% are at Level 3 or below that level.
Despite its limitations, this study suggests that advancing AI capabilities with respect to literacy and numeracy may have important implications for employment and education. Most workers use these skills every day at work. At the same time, these skills have not improved in most countries in the last decades. By contrast, AI capabilities in literacy and numeracy are developing quickly.
Across countries in PIAAC, on average, 59% of the workforce uses literacy skills daily at a proficiency comparable to or below that of computers. Between 27% and 44% of workers daily perform numeracy tasks at work, having numeracy proficiency below or at the level of AI. AI could affect the literacy- and numeracy-related tasks of these workers.
Even the best-ranking countries to date cannot supply more than a quarter of their workforce with the literacy and numeracy skills needed to outperform AI. In this context, the focus of education may need to shift more towards teaching students to use AI systems to perform literacy and numeracy tasks more effectively.