1. Frontiers of smart education technology: Opportunities and challenges
This chapter serves as an introduction to the book and presents some of its findings and policy implications. After highlighting the importance of digitalisation as a societal trend for education, it introduces the main focus of the book: exploring the frontiers of education technology. Artificial intelligence and learning analytics are transforming (or have the potential to transform) educational practices, and so have other smart or advanced technologies such as robotics and blockchain. How can they improve classroom instruction and the management of educational establishments and systems? After presenting the objectives and chapters of the book, the chapter highlights the opportunities of smart technologies for education systems and points to some emerging policy issues and dimensions to consider before making some forward-looking concluding remarks.
Digitalisation opens new possibilities for education. While education has always been a sector rich in data such as grades or administrative information, the use of this data to help students learn better, teachers to teach better, and inform decision-making in educational administrations is recent. That said, education stakeholders have always paid attention to new technologies and their potential to revolutionise education. This was true with the invention of radio, television, and more recently computers and the Internet. However, most uses of innovative technology have been to conserve existing educational practice and sometime enrich it, but rarely transform it. Might digital technology, and, notably, smart technologies based on artificial intelligence, learning analytics, robotics, and others, transform education in the same ways they are transforming the rest of society (OECD, 2019[1]; OECD, 2019[2])? If so, how might this look like?
There are two important aspects to the “digitalisation” discussion in education.
The first aspect relates to the changes that technology could induce in the delivery of education, from early childhood to adult learning. This is what this book explores. In this book we ask: how does and how could digitalisation transform education as a sector in the short, medium and long term? How does or may the rapid advances in artificial intelligence, learning analytics, robotics, etc., change how teachers and students teach and learn? What tasks do teachers perform that computers or robots might take over? Those technological advances can also translate into new work and management processes at the establishment or sector levels, sometimes in quest of cost-efficiency and productivity enhancement, sometimes to improve the effectiveness of the sector in reaching its traditional objectives (learning outcomes, equity, completion, etc.). Is digitalisation going to change schooling, higher education or lifelong learning? Will the educational processes be different, with more automated or computer-based tasks throughout the learning process? Will the digital infrastructure available to students, teachers, administrators and policy makers be different? Will an increased use of computers, data, smart devices, robots (and the technology that powers them) translate into better learning outcomes, more equity, more efficiency and productivity in education? What are the new possibilities, the opportunities and the challenges to be expected? Those questions have become more strategic for education policy makers in the past years. Between 2015 and 2019, 17 OECD countries published a digital strategy for education (and 16 others included an education chapter in their new national digital strategy) (van der Vlies, 2020[3]).
The second important question about digitalisation relates to how adequately education is responding to emerging societal and labour-market needs. This points to the 21st-century skills discussion in education circles, and the increasing importance of skills that are more difficult to automate and which foster innovation, such as creativity, critical thinking, communication and collaboration (Vincent-Lancrin et al., 2019[4]). The digitalisation of society and future shifts in labour market demand make the question of the content and nature of education more important: what are the knowledge, skills, attitudes and values people need in a highly digitalised, AI-inflected world? While this is not the primary focus of this book, some of the analysis will show how smart technology can also support the acquisition and assessment of those skills, for example through gamification or new forms of assessments.
After presenting the objectives and chapters of the book, the chapter highlights the opportunities of smart technologies for education systems and points to some emerging policy issues and dimensions to consider before making some forward-looking concluding remarks.
As the objective is to get close to the “technology frontiers” of education technology and take stock of what technology in education can already do, this book limits itself to technology that is demonstrably possible and currently used in some jurisdictions, establishments or laboratories. Whenever possible, evidence about their effectiveness is provided.
Artificial intelligence (AI): An AI system is a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments. AI systems are designed to operate with varying levels of autonomy. AI system lifecycle phases involve: i) ‘design, data and models’; which is a context-dependent sequence encompassing planning and design, data collection and processing, as well as model building; ii) ‘verification and validation’; iii) ‘deployment’; and iv) ‘operation and monitoring’. These phases often take place in an iterative manner and are not necessarily sequential. The decision to retire an AI system from operation may occur at any point during the operation and monitoring phase (OECD, 2019[1]).
Learning analytics: Learning analytics is one of the new young disciplines in data science. It studies how to employ data mining, machine learning, natural language processing, visualisation, and human-computer interaction approaches among others to provide educators and learners with insights that might improve learning processes and teaching practice.
Internet of Things (IoT) / Smart devices: The Internet of Things includes all devices and objects whose state can be altered via the Internet, with or without the active involvement of individuals. “Smart” devices, equipment, machines and infrastructure are creating opportunities for automation and for interaction in real time. Applications and services built for the Internet of Things, with insights provided by data analytics, are expected to become pervasive and educational establishments and classrooms may become “connected”.
Robots: A robot is a physical machine with sensing, computing and actuating capabilities, able to carry actions automatically. Often robots can make autonomous decisions and can adapt these decisions based on prior knowledge and sensor input. In education, most robots used are “social robots” that interact with learners.
Blockchain: Fundamentally, blockchain is a combination of already existing technologies that together can create networks that secure trust between people or parties who otherwise have no reason to trust one another. The marriage of these technologies gives blockchain networks key characteristics that can remove the need for trust, and therefore enable a secure transfer of value and data directly between parties. Specifically, it utilises distributed ledger technology (DLT) to store information verified by cryptography among a group of users, which is agreed through a pre-defined network protocol, often without the control of a central authority. You can think of a ledger as a record book: it records and stores all transactions between users in chronological order. Instead of one authority controlling this ledger (like a bank), an identical copy of the ledger is held by all users on the network, called nodes. Along with its own hash, each block stores the hash of the block before it. A hash is a unique string of letters and numbers created from text using a mathematical formula. Blocks are therefore “chained” together, making the ledger (almost) immutable or unable to be changed (OECD, 2017[5]).
Source: OECD (2017[5]; 2018[6]; 2019[2]) ; Lang et al. (2017[7])
The book is organised by educational objective or issue, rather than technology, thereby acknowledging that several technologies can be used to address similar issues (as alternatives or supplements). Roughly speaking, it covers three main types of technology fields (or families): artificial intelligence (in the all-encompassing meaning it currently has) and learning analytics; robotics (which adds a physical embodiment to artificial intelligence); and blockchain. Box 1.1 provides some initial definitions for those technologies. The book focuses on two areas where technology has (and will have) a transformative effect: teaching and learning in the classroom, and managing educational establishments and systems.
The chapters present how smart technologies are addressing (or could address) a number of educational issues, how they work, what they do well, what their shortcomings currently are, and what role they may play in the future in countries’ education systems. The selection of applications was made in areas where technology is either sufficiently mature and its benefits appear as low-hanging fruits, or where recent breakthroughs may be less well-known by policy makers and a wider audience. The analyses focus on formal education, from primary education to higher education, and leave out all applications for informal and non-formal education, applications focusing on the teaching and learning of specific subjects (e.g. foreign language, mathematics, reading, etc.) as well as the teaching and learning of technology itself (coding, etc.).Chapter 2 by Ryan Baker (University of Pennsylvania, United States) provides a general overview of artificial intelligence in education. After clarifying different terms and definitions to help readers understand the relations and sometimes overlaps between different technological techniques and terms, Baker provides an overview of the technologies currently being used in education, their core applications and their potential to bring education forward. The overview introduces some of the core applications that are explored in more depth in the report, and which could transform teaching and learning (e.g. personalisation of learning). It also highlights the potential of smart technologies in other areas such as formative assessment, digital games and simulations or just the provision of data to inform pedagogy. Likewise, beyond highlighting some of the technology applications used in managing education establishments and systems (e.g. early warning systems), it also points to a vast array of other possible applications such as real-time reporting to parents, admission systems or proctoring systems.
Part I of the book mainly focuses on the use of smart technologies in the classroom. It covers a variety of applications of smart technologies, from the most common ones (intelligent tutoring systems) to new developments (class orchestration, social robots, learning engagement).
Chapter 3 by Inge Molenaar (Radboud University, the Netherlands) discusses state-of-the-art personalisation of learning. Adaptive learning technology has arguably been one of the oldest lines of work for technology in education and is a mature field in the area. Existing learning technologies have a strong focus on diagnosing students’ knowledge and adjusting feedback or problems at the task level (which task next?); at the step level (what is the next step in a given task?); or at the curriculum level (what topic or curriculum unit next?). The frontiers in this area constitute computers taking into account a broader range of learner characteristics such as self-regulation, motivation and emotion. The state of the art is described using a “6 levels of automation of personalised learning” model that articulates the different roles of AI, teachers and learners, and shows how hybrid human-AI solutions combine the strengths of human and artificial intelligence to implement personalised learning.
Chapter 4 by Sidney D’Mello (University of Colorado Boulder, United States) zeroes in on a frontier area of personalisation, which is learning engagement. It gives a broad overview of some promising avenues for measuring students’ level of engagement during learning in an automated way with digital technologies. It also discusses how these technologies can be designed to improve engagement on the onset of learning or when learners’ disengagement sets in. After discussing why engagement matters for learning, it presents different types of approaches to improve students’ engagement and learning by using data and technology, for example, facial feature analysis, gaze analysis or eye tracking, sometimes in online but often in in-presence environments.
Chapter 5 by Pierre Dillenbourg (Ecole Polytechnique Fédérale de Lausanne, Switzerland) shifts the focus to the use of learning analytics and artificial intelligence to support teachers in orchestrating the teaching and learning of students in their classrooms. Here, the classroom as a whole and what is happening within it is the unit of analysis. Equipped with sensors, cameras or connected devices, classrooms become a hybrid physical-digital space in which computers analyse the behaviours of both students and teachers, and give teachers feedback on different parameters. Through different types of dashboards and displays, teachers get real-time information, for example, about when to move on to the next sequence of the lesson, or receive feedback after the class for their professional development or the planning of their next lessons.
Chapter 6 by Judith Good (University of Sussex, United Kingdom, and University of Amsterdam, the Netherlands) shows how technology provides specific services to the learning and engagement of students with physical impairments and mental health issues in education. Some of these technologies help bypass some of the obstacles to learning (e.g. text-to-speech or speech-to-text for blind and visually impaired students). Some simple applications assist adults in providing a first diagnosis of special needs, as is the case with dysgraphia. The chapter emphasises the importance of human-AI systems, both in the diagnosis and learning process. Because smart technologies are still limited compared to human beings, they sometimes help connect students with special needs with teachers or other people (e.g. the operator of a digital learning environment helping students with autism spectrum disorder). It also shows the value of engaging students with special needs in the development of the technologies designed to support them.
Chapter 7 by Tony Belpaeme (Ghent University, Belgium) and Fumihide Tanaka (Tsukuba University, Japan) presents possible roles for robots as educators. While it is based on a different type of technology (robotics), some of the smart technologies presented in the previous chapters can be embedded in the robots. Two main roles for robots are presented. Robots can be educators and tutors (typically one-on-one) or peer learners (with students teaching robots about something they are learning). Typically, social robots are designed for a learning environment supervised by teachers. Robots can also be telepresence devices, allowing teachers to teach a class (or students to attend class) remotely, offering more opportunities than videoconference systems. While research shows that robots are rather effective in the narrow tasks they perform, it seems unlikely that they will have the capability to replace teachers in the foreseeable future. Their cost is also a limit to their mainstreaming in education.
Part II of the book mainly focuses on the use of smart technologies for the management of educational institutions and systems.
Chapter 8 by Dirk Ifenthaler (University of Mannheim, Germany and Curtin University, Australia) starts with an overview of the different possible uses of learning analytics to manage higher education institutions and provide information to decision-makers on a variety of governance and organisational processes from forecasting future education to productivity enhancements. In spite of interesting cases of change management through learning analytics at the organisational (or faculty) level, few cases of holistic approach to learning analytics are documented. Some of the challenges at the organisational level apply to the system level as well and several general guidelines for a stronger adoption of learning analytics within organisations and system-wide are presented. Most of the research on learning analytics for the management of organisations concerns higher education institutions so that specific challenges for schools remain to be explored.
Chapter 9 by Alex Bowers (Columbia University, United States) shows how smart data and technology helps understand and tackle an important problem in most OECD countries: high-school dropouts. This is one of the most immediate and increasing uses of administrative data and the application of smart technologies here is relatively common, at least in some countries. A first important step is to have good predictors, and while the chapter notes how a variety of early warning indicators (and systems using them) fall short of their claimed objective, it also presents advanced data techniques that allow for a much more accurate identification of “at risk” students. Open data and open algorithms should allow third parties to verify the quality and fairness of algorithms. Recent analysis shows that students who drop out have different profiles, which require a greater variety of policy interventions than those currently provided to students considered “at risk” of dropping out.
Chapter 10 by Jack Buckley, Laura Colosimo, Rebecca Kantar, Marty McCall and Erica Snow (Imbellus Inc., United States) discusses how recent advancements in digital technology could lead to a new generation of game-based standardised assessments and provide education systems with feedback on students’ higher-order or socio-emotional skills that are difficult to assess via traditional standardised tests, be they computerised or not. Game-based tests may analyse eye-tracking data and audio recording, and process natural language in addition to analysing information such as time on task or use simulations. Given their cost, the complexity of their development and also some of their intrinsic limitations, they will supplement rather than replace traditional standardised tests, which have their advantages for assessing certain knowledge and skills.
Chapter 11 by Natalie Smolenski (Hyland, United States) focuses on the use of blockchain technology to make the credentialing process more efficient – and also possibly some other educational administrative processes requiring verification. The chapter starts with a history of blockchain in cryptocurrencies to make the functionalities of the technology more easily understandable. Credentials are a good use case for blockchain technology and many blockchain initiatives around credentialing are underway worldwide. Blockchain allows for secure and transparent sharing of qualifications, credits and badges. At the national (but even more so at the international level), it could help stop fake degrees and certifications, facilitate the transfer of educational records as well as the credentialing of small units of learning such as MOOCs or corporate professional development provided by companies. The human and legal infrastructure for their full mainstreaming remains to be developed, including open standards and interoperability. Compared to the current processes in place, the chapter argues it would be a cost-efficient solution.
Smart technologies can improve education systems and education delivery in different ways. They can enhance access to education, improve its quality for learners, and enhance its cost-efficiency for societies. This section highlights how smart technology contributes (or could contribute) to the achievement of these goals.
Effectiveness
Attending school or university does not always translate into as much academic learning as one would hope for. The OECD Programme for International Student Assessment (PISA) has shown that attending school may actually lead to very different levels of learning outcomes across countries. While there is no similar evidence at the higher education level as yet, this is arguably the same at that level too. One of the key promises of smart technologies is to enhance the effectiveness of teaching and learning for better student learning.
In the classroom, applications that directly support student learning show early promise. Personalised learning aims to provide all students with the appropriate curriculum or task, and scaffold them within a task, based on a diagnosis of their knowledge and knowledge gaps. This is not only done at the academic level, focusing on the “what”, but increasingly takes into account how students learn and factors such as self-regulation, motivation or effort (Molenaar, 2021[8]). Engagement is key for learning, and solutions to keep students engaged within digital or physical learning environments are being developed to identify their affective states during learning and nudge them towards re-engagement when they seem to disengage (D’Mello, 2021[9]). Social robots perform similar tasks in different ways: they can use adaptive learning to tutor students with natural language, but they can also teach, or motivate them to learn by playing the role of a peer student. They support teachers by enabling the implementation of different types of teaching and learning strategies (Belpaeme and Fumihide, 2021[10]). Finally, smart technologies give students with impairments and special needs access to curriculum materials and allow those students to participate in learning activities to an extent that was not possible before, here again increasing the effectiveness of education (Good, 2021[11]).
Those solutions can be used and remain helpful outside of the classroom too, either for homework, as automated private tutoring or practice solutions, and for lifelong learning. In fact, the largest market for educational technology companies is the consumer market targeting students and parents directly, either for recreational learning activities or for tutoring or test preparation.
A second promise of learning effectiveness comes from classroom analytics that support teachers in providing more effective teaching. This is still a work in progress but many applications already show how a variety of solutions could support teachers in better using their time in class, for example, by suggesting when it is a good time to shift to the next teaching or learning activity, who would require their attention the most, how they could engage the whole class in collaborative learning activities. While classroom orchestration solutions can help teachers in real time, they also provide feedback on their own practice, for example, how much they talk, to whom, or how they divide their time between different types of activities (Dillenbourg, 2021[12]). Both real-time and post-hoc feedback are akin to personal professional learning opportunities for the teacher in question, and have the significant advantage of being about the specific teacher who was (digitally) observed rather than about theoretical or general teaching practice. In that sense, smart technology has real potential to improve the teaching practice of all individual teachers, and subsequently the learning outcomes of their students.
Increasingly, school buildings will be equipped with sensors, cameras, and computers to fulfil certain administrative as well as teaching and learning functions. Some schools are already experimenting and developing innovative ways to integrate smart technologies in their operations. Here are a few examples from Shanghai (China).
The Luwan No 1 Central Primary School (Huangpu District, Shanghai) is a public school integrating AI in its school resource management as well as its teaching and learning – a digital model that may then be extended to other schools. The management of the campus, and the teaching and learning all rely on smart technologies. Using IoT sensing technology, the “digital campus” consists of collecting and analysing campus data to automatically control and manage environmental factors such as security, lighting, water quality and air quality, but also to collect campus activity data; for example, people density in corridors etc. Combined with wearable devices, the school also collects physiological data such as students’ body temperature and heart rate as well as academic data and learning process data in order to support teachers and learners. The “digital students” application analyses student data to create a detailed, holistic portrait of students. The collection of data increases the understanding of students’ learning status and growth, and provides teachers with data to tailor their teaching to their needs. The data cover discipline, academic level, physical and mental health, aesthetic taste and social practice. Socio-emotional aspects such as learning engagement and affective states are measured by voice and face-recognition technology. Finally, a “digital teaching” system "provides teachers with support on five aspects of teaching: lesson preparation, classroom orchestration, homework, tutoring and evaluation – with functionalities such as “classroom orchestration”, “intelligent assessment” and “intelligent homework review”. The intelligent tutoring system supports students directly in accessing resources, tools, pathways and personalised guidance. As of June 2021, this model has been studied and adopted by more than 250 schools in Shanghai, Qinghai, Shaanxi, Guizhou, etc.
Note: The left panel shows a screenshot of the campus management system. The right panel shows a student using the classroom handwriting board collecting data about this progress.
Source: Courtesy of the municipal government of Shanghai
The demonstration high school affiliated to Tongji University (China) is also implementing a new “digital classroom” system in English, geography and biology. Students’ learning data collected in the system is the basis for teaching and further learning. Before the lessons, teachers use digital interactive “practice” tools to assess students’ learning; they also conduct short and concise in-class tests from time to time to obtain real-time student learning data. This allows them to change their teaching strategies during class, and to develop individualised after-class strategies. Based on this information, teachers will set online assignments, which are automatically marked by the system and provide the basis to the generation of personalised “knowledge analysis” reports Figure 1.2). Based on these individual cards, the system proposes micro-tutoring video resources and exercises to meet individual learning needs, and teachers receive guidance to propose targeted after-class assignments and counselling and to customise their teaching to students’ needs. The system also allows for collaborative learning with students and teachers being able to see and comment on students’ work.
Note: The left picture shows the visualisation of a student’s acquired knowledge in a chemistry curricula unit. The right picture shows how the system can be used for cooperative learning, with both students and teacher being able to view and assess how all students responded to a given assignment.
Source: Courtesy of the municipal government of Shanghai
Other demonstration schools focusing on digital technology explore other possible aspects of technology.
Shanghai Xuhui Middle School has a traditional emphasis on science education and 22 engineering science and innovation labs (18 kinds). After developing 5G in the school campus, it developed a “holographic” science education model based on Mixed Reality in order to make difficult knowledge more directly understandable and intuitive, and to enhance students’ attention and enthusiasm for the subjects. As of June 2021, two lessons (“Exploring the Mysteries of the Solar System” and “Understanding the Bones of the Human Body”) were developed and delivered with real-time interaction with Yuanyang No. 1 Middle School (Honghe Prefecture, Yunnan Province).
The Shanghai Industrial Technology School provides its students with advanced mixed reality and simulation technology to learn manufacturing. Simulation-based training projects are carried out in a 3D virtual environment, containing a series of workstations such as graphic drawing, workpiece handling and work units such as loading and unloading of computer numerical controlled (CNC) machine tools.
At the organisational and system levels, smart technologies also hold promise in making education more effective. While this remains relatively rare (Ifenthaler, 2021[13]), smart technologies can be integrated in most dimensions of school activities, providing administrators, teachers and learners with feedback to manage school resources as well as improve the effectiveness of teaching and learning (see Box 1.1). The rise of a new generation of assessments powered by AI also opens new avenues for recognising and evaluating competences that were hard to assess through paper and pencil tests. This could accompany most education systems in their shift towards emphasising skills (in addition to the traditional emphasis on knowledge). Game-based assessments and simulations allow assessments to be designed to be more realistic but also to assess skills such as complex problem solving, creativity or collaboration in new ways (Buckley et al., 2021[14]).
Finally, the emergence of longitudinal education data systems that follow students through the course of their studies also allows for more effective policy and organisational interventions and a better design of educational offerings. For example, in the United States, the analysis of community college graduation rates, the success of their student placement strategies in “remedial courses” and students’ study patterns as part-time or full-time enrolment led to revisiting what the educational experience of community college students actually is and “redesigning” community colleges (Bailey, Smith-Jaggars and Jenkins, 2015[15]). As in other sectors (OECD, 2019[16]), the use of data supports policy design and interventions.
Equity
Smart technologies can help education systems provide more equitable learning opportunities. In this respect, smart technologies are more ambivalent. One the one hand, they clearly do or could help reduce inequity both by increasing access to learning opportunities for all and improving learning effectiveness for those who need it the most. On the other hand, without a widespread and equitable availability of smart technologies, inequity could also rise. They may also leave achievement gaps unchanged or even widened, depending on their differential impact on different learners.
Let us start with the difficulties. There are at least two reasons why technology may have a negative effect on equity. The first, obvious reason lies in the difference in access to devices and connectivity by students from different groups, notably students from lower socio-economic backgrounds. These students may not have the devices, the connectivity or the resources that allow accessing and using smart technologies either at the school they attend or at home. A second reason is that, if technology (e.g. personalised learning) works the same for everyone, those who start with stronger prior knowledge can maintain their advantage or even make faster progress than those with less prior knowledge. In spite of supporting students with less prior knowledge, it is thus possible that technology could help more advanced students more. This would widen rather than reduce the achievement gap.
There are also many reasons to believe that smart technologies can advance the equity agenda.
First, learning technology can expand access to learning opportunities. Educational platforms proposing open educational resources (Orr, Rimini and van Damme, 2015[17]) or massive open online course (MOOC) platforms are good examples. They allow learners to access learning materials with a quality that may be superior to what they can access locally. While many studies have shown that this increased access has not decreased inequity at scale given the low take-up and the fact that most users are already well educated, a recent systematic review of their effect on equity provides a more optimistic perspective, notably for non-English MOOCs or open educational resources (Lambert, 2020[18]).
As importantly, smart technologies can reduce inequity by facilitating the inclusion of students with special needs and by adapting learning to different learning styles. Technology has, for example, made it much easier to support the diagnosis of learning difficulties such as dysgraphia, and remedial digital responses have also been developed. A variety of smart technologies applied to learning solutions also make it easier for blind or visually impaired students as well as deaf or hard-of-hearing students to access learning materials and easily perform the educational tasks required from other students. Artificial intelligence enabling speech to text (and vice versa) or automatic subtitles are the most obvious examples. Learning technologies also tackle more difficult issues and support the socio-emotional (and thus the subsequent academic) learning of autistic children. They increasingly propose ways to help children with attention deficit hyperactivity disorder (ADHD) to self-regulate and better benefit from their schooling. One caveat here is that inclusion is not just about the individual “fitting in” but also for society to be more inclusive and open to differences (Good, 2021[11]). Technology encourages that by enabling students with special needs to study in a traditional (and inclusive) learning environment, which also changes peoples’ view on disability and special needs.
Second, solutions such as early warning systems are entirely focused on reducing inequity by helping students at risk of dropping out from high school (or university) to graduate – students who drop out typically come from disadvantaged and minority backgrounds. Early warning systems also allow designing appropriate interventions by identifying the factors or indicators most likely to predict dropout (Bowers, 2021[19]). Some use of learning analytics within institutions, for example, to monitor student engagement or redesign study programmes, could also have the same effects, should the educational institution pay particular attention to inequity (Ifenthaler, 2021[13]).
Third, the use of learning analytics as exemplified by personalisation at the individual level, be it using intelligent tutoring systems or learning analytics to keep students engaged in learning, all hold promise in reducing inequity, notably by supporting students with less prior knowledge to learn at the right pace. Box 1.3 gives an example of an online solution that reduced the learning gap between the strongest and weakest students in mathematics at the beginning of the intervention. There is, however, little evidence that adaptive learning generally reduces achievement gaps between students. Classroom analytics can also give feedback to teachers on how they could improve their teaching; specifically how and when to pay more attention to different group of students in their class, based on their academic level, gender, ethnicity, etc. Adaptive learning technology can also help students practice and make progress at home, outside of the classroom, supported by intelligent tutoring systems. This may be particularly important for students coming from households where parents can support their students less effectively with their school work, be it directly or indirectly.
Few studies show that adaptive technology (or personalised learning) reduces the achievement gap between students with more and less prior academic knowledge. And yet, in order for intelligent tutoring systems to reduce achievement gaps, this would indeed be the objective. Evaluated through a randomised control trial, an intervention in the state of Maine (United States) showed that this may become the case (Murphy et al., 2020[20]). Teachers in the intervention were asked to use ASSISTments software to provide students with mathematics homework. The system provides feedback to students as they solve mathematics homework problems and automatically prepares reports for teachers about student performance on daily assignments. Teachers received training and coaching on formative assessment. The study found that students in the schools that were assigned to ASSISTments learned more compared to their peers in the control schools, with large effect sizes, and that the impact was greater for students with lower prior mathematics achievement. The evaluation confirms initial results by Roschelle et al. (2016[21]), which found both evidence of strong maths learning outcomes when using the platform and also a reduction of the achievement gap.
Efficiency
In most other sectors than education, smart technologies are used as a tool to enhance the cost-efficiency of operations, notably by automating a number of tasks and processes, making services faster and often cheaper (OECD, 2019[16]). While education might be behind most other sectors in this respect, digitalisation is also making many educational processes more efficient as interactions between stakeholders and educational institutions become increasingly automated. As noted above, also in teaching and learning, some degree of automation is gaining ground. To what extent will digitalisation allow for enhanced cost-efficiency and productivity in education?
Any discussion of cost-efficiency should keep in mind that technology incurs investment and maintenance costs, which have to be compared with the costs of current arrangements. Digital technology has not always delivered on its cost-efficiency promises in the past because one has sometimes forgotten that, beyond the initial investment, it needs to be continuously upgraded, maintained, etc.
Nevertheless, as in other sectors, there are good reasons to believe that smart technologies could increase cost-efficiency in education.
One example lies in the application (and admission) process to educational institutions. They are increasingly undertaken through digital platforms, especially in higher education, where a “matching” (or selection) process is often necessary. In cases of open admission institutions, when no selection is required, implementing seamless automated processes is even easier. The implementation of the National Education Information System (NEIS) in Korea, an e-government system allowing, inter alie, for the digital transfer of students’ academic records from one school to the other (as well as from school to university) was estimated to save USD 237 million a year in 2010 (KERIS, 2010[22]).
A second area where digitalisation could possibly lead to cost-efficiency is the provision of verifiable degrees and other credentials on blockchain. The gradual development of an infrastructure for digital credentials and the adoption of open standards will gradually lead to a different way of certifying and holding degrees, with individuals being able to manage their qualifications themselves. This is one of the strongest and most immediate arguments of cost-efficiency in the different digital solutions examined in this book.
A third area where cost-efficiency is underway is the collection of system-level statistical information. While statistical information often relied on the establishment of statistical panels (of representative samples of individuals or institutions) and often involved multiple handlings of the same data, the use of administrative data combined with the interoperability of diverse systems has made it much easier to get statistical information from operational services in almost real time (González-Sancho and Vincent-Lancrin, 2016[23]; n.d.[24]).
At the end of the day, a cost-benefit analysis comparing the benefits of smart technology, including the non-financial ones mentioned above, to that of an existing solution, will determine how cost-efficient it is for a given service (or educational goal).
The emergence of smart education technologies or solutions powered by artificial intelligence, learning analytics, algorithms and other technologies presents many opportunities. At the same time, they raise a number of policy questions. How can governments best harness the benefits of technology in education while limiting its possible risks? This involves a good understanding of the opportunities and risks, both from a technical and political dimension. One success factor lies in the social willingness to adopt those technologies. This section will draw some key lessons from the book from a policy perspective and highlight some key features of smart technologies that matter for policy making and for the effective deployment of smart technologies in education.
Smart technologies as socio-technical systems
Whether they are already available or under development, most of the smart education technologies for teaching and learning covered in this book do not aim to replace teachers or human beings. They were in fact developed with the current education model in mind. One common thread of all analyses in the book is that most teaching and learning solutions are designed as hybrid human-AI systems and require teacher-student interactions and human oversight of the machine at different points. Molenaar (2021[8]) offers a model to better understand the continuum between fully automated and teacher-only education. Most advanced personalisation solutions require teacher intervention or alert teachers when they should intervene, for example, because students are still struggling or they need to move on to another step of their learning process. Most solutions to support classroom orchestration are also hybrid solutions that merely scaffold teachers in implementing rich learning scenarios for their students. As Dillenbourg (2021[12]) puts it, “there is a teacher in the loop” and classroom analytics are designed to support teachers in orchestrating the teaching and learning of students in the class and in providing them effectively with rich learning scenarios – not to replace them.
Contrary to how robots are often presented in other sectors, the social robots presented by Belpaeme and Tanaka (2021[10]) are not meant to replace teachers either, but to support their students for specific learning tasks, in the same spirit as personalisation tools. Not that it could not be possible one day, in a distant future. As of 2021, social robots are mainly effective in accomplishing narrowly defined tasks. They play the role of a teacher assistant, as computers do in their different way. As for telepresence robots, they enable human teachers to be present from a distance. Good (2021[11]) provides an excellent case showing how smart technologies for students with special needs may actually create new social relationships between learners and the humans in charge of providing them with appropriate learning tasks – rather than suppress them.
At the system and organisational level, the use of smart technologies follows the same pattern. Early warning systems help predict dropout, but they require a human intervention for “at risk” students not to drop out (Bowers, 2021[19]). Other types of learning analytics used within educational institutions to support decision-making also provide information that needs to be acted upon; they do not make final decisions in the place of administrators and teachers (Ifenthaler, 2021[13]).
This is not to say that smart technologies never make decisions or are not designed for full automation. Personalisation systems, classroom analytics, and early warning systems all make some decisions to enact their next step or recommend one to human beings. But they typically only provide input to decisions. Game-based standardised assessments do more than provide a suggestion: they automatically score the test-takers and assess their skills – as is already the case with traditional computer-based standardised assessments. Blockchain technology does not make decisions, it just records trustfully what a variety of (usually) human actors have done, building on different social processes: accrediting institutions, awarding a qualification or credential, storing the credential on some blockchain, sharing the credential with other parties, verifying the authenticity of the credential, etc. (Smolenski, 2021[25]). But both cases also highlight the relevance of thinking of smart technologies as socio-technical systems, that is, systems in which social and technical features interact and are shaped together.
One of the challenges of game-based standardised assessments will be to create social acceptance, if not full trust, as was the case for traditional standardised assessments. Buckley et al. (2021[14]) note that building valid, reliable, and fair game-based assessments is considerably more complex and challenging than designing traditional standardised tests. While it solves a problem with a clear level of efficacy, one challenge to the widespread use of blockchain for credentialing also relates to social change and legal adaptations: social processes to certify credentials already exist and not everyone may be willing to change those social habits – or cope with the uncertainties attached to any new solution.
There are different ways to acknowledge the fact that most smart technologies are hybrid AI-human systems or that, more generally speaking, smart technologies are better designed and understood as socio-technical systems. One is to clearly communicate that while technology could play a bigger role in the future, it currently needs to be supplemented and controlled by human actions in most cases. The ways to recognise those current realities would be as follows:
Involving teachers, students and other end users as co-designers in the research and development process would ensure the usefulness and use of smart digital solutions. It also help the people involved in understanding and shaping the social context in which smart education technologies would best be used (the classroom, home, etc.). This should be an aim even when it is challenging to involve end users, for example, students with special needs.
Public-private partnerships between government, technology researchers within universities and companies, and the education technology industry should be a key characteristic of most research and development projects in this area. This research should go beyond the technology functionalities to analyse how it is used in context and also work on the social and legal adjustments that would be required for their widespread adoption.
Algorithm accuracy
Smart technologies have made and are making very rapid progress, and this book illustrates their potential benefits in a wide range of educational areas both in terms of teaching and learning, and administrative efficacy. Smart technologies often outperform traditional data analysis and technology thanks to more powerful algorithms. Game-based assessments allow skills to be assessed that are difficult to assess through traditional computer-based or paper-and-pencil tests. Personalisation adapts to learner characteristics in ways that pre-existing personalisation methods could not permit – and they possibly do it as well as human teachers. The new algorithms in early warning systems outperform traditional regressions in terms of predictive power and bring visibility to dropout patterns that were not traditionally acknowledged by human administrators or teachers.
Nevertheless, many of the smart technologies presented in the book are not fully mature yet. For example, while some early warning systems now approach good predictive power, Bowers (2021[19]) shows that most early warning systems rely on predictors that are no better than a random guess. In the areas of student engagement, D’Mello (2021[9]) points to new approaches that are developed to better measure students’ engagement in learning using facial image analysis and other ways but also notes the inaccuracy of many of the measures used in the field of learning engagement. In the area of classroom analytics, some solutions manage to identify whether learners are working individually or in groups with a very high level of accuracy (90%) but identifying the type of teaching and learning activity remains more challenging (67% of accuracy). Those are just three examples, which are encouraging as accuracy levels can be very high, but show that this is not guaranteed for any AI-powered education application.
In the current state of technology, one policy challenge is to ensure that the developed technology solutions perform their tasks with accuracy – or to get a clear sense of that level of accuracy. In spite of the very rapid advancement of smart technologies, computers and smart education technologies still remain imperfect – though not necessarily more imperfect than humans. Some of those imperfections are expected to be rapidly solved while it may take much longer for others. The contrary would be surprising when looking at technologies considered to be at the education technology frontier. But it is even possible that for some tasks, smart technologies will never become perfectly accurate and remain confounded by false positives, etc. However, the real question is how they compare to humans. After all, human beings perform those tasks with some levels of imperfection too (when they do) as the problems that they are addressing are usually complex. Whether full accuracy should be the expected standard is an open question. Probably this should depend on the social stakes related to the task. In many cases, an “accurate enough” diagnosis or decision should be sufficient.
Given that smart technologies are not fully mature yet and still have some intrinsic limitations, it is important for users and governments to remain aware of those limitations without preventing those technologies from continuing to improve and grow thanks to their actual use. Some possible policy pointers to mitigate the limits of smart technologies while embracing their potential are as follows:
While smart education technologies can be useful before they are fully accurate (or even without being fully accurate), they should demonstrate a certain level of accuracy in their predictions and diagnoses when they support decision-making – or just be effective enough when they have a different role. Education technology companies could be asked to demonstrate the level of accuracy or effectiveness of their technology solutions with different accuracy requirements depending on the stakes of the supported decision (when they support a decision). Those accuracy requirements should ideally be compared with the current performance of human teachers and administrators.
When still imperfect in terms of accuracy, smart education technologies should merely inform human decision-making and reflection rather than make fully automated decisions or support a decision process that will rarely deviate from their recommendations, especially for high-stakes solutions. Those technology requirements should have a risk-mitigation rather than no-risk policy, acknowledging that smart technologies can be beneficial even when they are not fully accurate. This may just imply keeping humans in control in the final stages when social stakes are high.
Designing for use
Sometimes, education technology solutions are designed and proposed because they are possible rather than because they are useful and provide clear benefits to end users in education. Most education technology products are mere educational derivatives of solutions designed for other sectors. Even when technology applications are useful and beneficial, some teachers, learners and users may have no interest in using them. Instances of lack of use and lack of usefulness of education technology have given rise to several critiques of education technology (Cuban, 1986[26]; Reich, 2020[27]) even though the increased use of technology in classroom instruction represents one of the biggest changes in classrooms of the 2010s (Vincent-Lancrin et al., 2019[28]).
How can one overcome this problem? Several chapters in this book discuss smart technology solutions that may not be sufficiently useful to be used at scale (given stakeholders’ usual ways of working). One reason for this lack of use lies in the design of the smart technology solutions or in an insufficient understanding of how teachers can use them in their professional practice in ways that support rather than distract them. For example, classroom analytics are useful when they make visible to teachers what is either invisible or not easy to see (either in real time or after class) and when they provide information that they can act upon and interpret (Dillenbourg, 2021[12]).
One aspect of smart technologies that makes them more or less useful to their users relates to how they display the final information to end users. The interface between technology and humans is essential. For example, research shows that different types of dashboards can be more or less effective to support teachers and learners, or more appropriate in certain contexts than others (Molenaar, 2021[8]; Dillenbourg, 2021[12]). Dashboards typically display the final output of the analytics: they can take different forms (e.g. centralised, distributed, ambient) and use different display devices. While the mere appearance of social robots does not seem to matter much, social robots work better than virtual agents as users may relate to them in different ways than with a virtual image or a computer (Belpaeme and Fumihide, 2021[10]). The effectiveness of some solutions such as the ECHOES learning environment, which scaffolds autistic children’s exploration and learning of social communication skills, partly lies in a setting that fosters communication between the autistic child and the adult monitoring the software and learning environment. This important dimension was actually discovered as the tool was tested and improved rather than as a preconceived use case, showcasing the importance in designing and adapting technology solutions with end users (Good, 2021[11]).
In some cases though, the usefulness of education technology may go beyond offering a technical solution to a specific problem. This is why usefulness and algorithmic accuracy are not always related: a solution with an algorithm accurately performing its task may not be so useful, whereas algorithms imperfectly performing their task may be. Changing the stakeholders’ mindset or catalysing some broader change within an institution or an education system may be its usefulness. Generally speaking, innovation is a driver of professional learning and change (Avvisati et al., 2013[29]; Vincent-Lancrin, 2016[30]; Vincent-Lancrin et al., 2019[28]). Smart technologies play the same role.
Ifenthaler (2021[13]) shows that many universities and institutions introduce learning analytics at the institution-wide level in order to change their organisational culture or processes, and perhaps sometimes to foster new collaborations and ways of working between different stakeholders within the institution. Providing a solution to a specific problem or automating their processes may only be a secondary objective. Regardless of their effectiveness in reducing student dropout, one of the benefits that early warning systems (and the related research) has already delivered lies in a better and broader understanding of the circumstances leading students to drop out.
Bowers (2021[19]) shows that the traditional conception of students at risk of dropping out (as students with low and declining grades who do not like school) corresponds to only 38% of actual dropouts in the United States – so that traditional interventions miss out on the majority of students who actually drop out. Beyond providing real-time information, several of the functionalities of classroom analytics provide feedback to teachers on what happened in their class and one of their virtues can be to trigger professional reflection and learning, hopefully followed by behavioural change and improved teaching proficiency (Dillenbourg, 2021[12]).
Beyond design, including the final display of information to learners and other users, cost-benefit analysis is a final way to think about this “use” problem: if cheaper alternatives are available, in budget, time, cognitive load (or whatever relevant metric), new smart technology solutions may remain unattractive compared to existing human or older technology solutions. This is easily understandable when learning analytics predict, diagnose or act with little accuracy, but this can also be true for digital solutions with fully accurate or effective algorithms. Many of the digital tools that support teachers and administrators help them to solve very specific problems and sometimes the existing solutions may outperform the new ones. For example, at the system level, game-based assessments and simulations are likely to supplement rather than replace traditional standardised assessments based on batteries of questions: they are much more expensive to design, have less generalisability, and are only better suited to assess complex competences that are more difficult (or impossible) to assess through cheaper traditional alternatives (Buckley et al., 2021[14]). Smolenski (2021[25]) shows that blockchain has the potential to make the credentialing process more cost-efficient and simple for individuals. The technology is superior to some other technologies when fraud is possible, but not necessarily an adequate solution in all situations.
One last important point on usefulness relates to the affordability of smart technologies for public establishments and individuals. In the context of education systems, digital technology usually has to be very affordable to be bought – and thus be useful and have a chance to be actually used. As noted by Good (2021[11]) in the case of students with special needs, smart technologies should be designed to run on low-cost and widely available platforms (or devices). This is not always the case, but remains one of the conditions for them to be used – and to make their benefits widely accessible. Smolenski (2021[25]) also highlights the importance of open standards in the case of blockchain, partly as a way to make long term costs lower and ensure the solution is more sustainable for the end users (both institutions and individuals). This is true with many other technologies: open standards allow for greater interoperability, more sustainability over time, more competition among vendors, and often lower user costs. Technology solutions that run on widely available platforms are also more affordable and useable than specialised devices.
Several key messages emerge in terms of enhancing the usefulness and usage of smart technologies in education:
Cost-benefit analysis should typically guide the design and adoption of smart digital solutions for different types of problems, acknowledging that benefits and costs are not just pecuniary.
While the benefits of any solution may go beyond immediate student academic learning gains, and the costs, beyond merely financial ones, both the expected costs and benefits of smart technologies should be clearly identified and estimated either through research evidence when available and possible or a good theory of action (or theory of change).
The display (or communication) of the information provided by learning analytics and other technology matters in making smart technologies useful to students, teachers and decision makers. More broadly speaking, the design of the interface between human and smart education is often a key aspect of the useability and impact of the digital solutions on learning or other targeted goals.
Smart technology solutions should aim to be low cost and run on widely available platforms/devices to be as affordable as possible, possibly using open standards (and interoperability standards). Governments can support the development of those standards, preferably at the international level. Attention to affordability is essential to making smart technologies accessible to all so as not to reinforce the digital divide. Ensuring smart technology solutions can benefit all learners or institutions is key to equity and inclusiveness.
Smart technology and data governance: transparency, fairness and ethics
One key element of any socio-technical system is the broader social context in which the system operates, including its values and principles. Because they rely on large amounts of education data, including, sometimes, personal data such as biological markers, face recognition or expression, etc., or require a permanent monitoring and tracking of learners, classrooms or institutions, common concerns about the development and use of smart technologies relates to data protection and privacy, but also to ethical and political concerns. Could (or should) education establishments and systems become a new version of “Big Brother” for the sake of improved learning outcomes? Can governments and other parties be trusted to use this information for the mere sake of educational improvement – and to enforce strong data protection regimes? What could be the adverse consequences of this use, if not done properly, either in the present or in the future. Could data-rich education technologies, for example, perpetuate and reinforce biases and inequity? Embracing smart technologies implies some trust in how they are used, credible safeguards, and some level of understanding and acceptance of their processes and outputs.
Most OECD countries have strong data protection regulation that ensures that personal education data cannot be shared with (or used by) third parties beyond the educational processes for which they are collected unless certain privacy conditions are met. This is the case in the Europe Union with the General Data Protection Regulation (GDPR) and in the United States with the Family Educational Rights and Privacy Act (FERPA), which have both influenced many other data protection laws across countries. Much of the data concerns administrative micro-data (González-Sancho and Vincent-Lancrin, 2016[23]; n.d.[24]). The data protection regime usually extends to vendors providing technology solutions to schools and education administrations. It is noteworthy that the enforcement and implementation of data protection regulations can vary from one country to the other (or even one place to the other within a country). Given these arguably strong safeguards, the fact that privacy and data protection issues remain central in public discussions may point to a lack of trust in how the data are used (or could be used) within the education system and beyond.
Data protection is just one aspect of data governance though. One important question relates to the relationship between governments, the data subjects (who are usually the users of (sometimes mandatory) education services), and the private sector that usually develops the smart education technologies. This relates to questions of data ownership and of competition policy in the digital world: how should administrative and other education data be shared across education technology companies and public researchers to allow for progress and for a sufficient amount of competition in the sector of smart education technologies? How can the interests of learners and other individuals be best preserved in this context? The different solutions proposed for other sectors than education could probably be adapted to education (OECD, 2019[31]; 2019[1]).
Ethical discussions should normally concern what is not regulated within a country and thus aspects for which individuals and authorities have more freedom of action. This is how we interpret the question of “ethics” in educational AI.
The regulation of algorithms is usually not as strong as regulation for data protection. A major concern about algorithms is that they could be biased and have an undesirable social impact for some population groups (based on gender, ethnicity, socio-economic status, etc.) – but also that they could be flawed or just reinforce past human biases rather than reflect current societal values. A usual requirement is to ensure they are transparent and open, and that their “decisions” can be explained and challenged when automated. In the case of the EU General Data Protection Regulation, the regulatory language about algorithms is ambiguous (articles 13-15, 22, and recital 73) and lawyers are still debating what those articles imply in terms of “right of explanation” (transparency) and of the possibility to opt out of or challenge “automated decisions” for citizens. In the European Union, only the French and Hungarian laws have an explicit law about the “right of explanation” (with French law requiring both ex-ante and ex-post explanation in an intelligible way, and Hungarian law, some level of explanation) (Malgieri, 2019[32]). In the United States, there is no regulation about algorithms and their requirements as part of FERPA (or other regulation). Most OECD countries do not have clear regulatory requirements about them as of 2021.
Because algorithms based on machine learning are trained with historical data, many observers are concerned they will reproduce past biased (human) practices, as has apparently been the case in some countries in domains other than education (finance, justice, etc.) (O’Neil, 2016[33]). Several guidelines have been developed to avoid these pitfalls, which can happen at different steps of the process: measurement (data collection or labelling), model learning (when machine learning is involved), and action (when the algorithms detect, diagnose and act, for example). Different possible measures of fairness are also possible, which makes the issue even more complex (Kizilcec and Lee, 2020[34]; Baker and Hawn, 2021[35]). Ifenthaler (2021[13]) mentions several check lists of good practice and ethics for learning analytics. Bowers (2021[19]) points to the “open algorithm” movement in the area of early warning systems and notably to two overlapping sets of criteria to ensure transparency, verification and replicability of algorithms: the AAAA (accurate, accessible, actionable and accountable) and FAIR (findable, accessible, interoperable and reproducible) principles. Molenaar (2021[8]) also points to the importance of transparency to govern learning analytics and algorithms. As was shown in 2019 with some difficulties around exams and grade assignment for university admission, when it comes to high-stakes automated decisions, transparency is also about initiating an early dialogue about the criteria, expected social outcomes, relevance and acceptability of smart algorithms with diverse stakeholders from experts through to final users and other social bodies (Box 1.3). In some cases, the algorithms can be human-coded rather than involve AI techniques.
For example, given that accurate predictors for early warning systems can rely on minimal information that does not include information about gender, race and socio-economic status (Bowers, 2021[19]), it may not be considered ethical (or even necessary) to include these kinds of indicators to diagnose dropout in early warning systems unless they improve significantly the performance of the algorithms. On the other hand, results that do not include any personal information about learners may still lead to biased or socially/politically undesirable outcomes. Ethical concerns should therefore include a verification and discussion of the effects of smart technologies on different groups and ensure they are aligned with countries’ social and political principles. As few people are, in practice, able to verify the effects and impacts of algorithms, some independent groups of stakeholders may be responsible for or even assigned this task. While anyone should be allowed to do it in the frame of an open algorithm culture (at least when algorithms lead to a decision or a quasi-decision), education researchers, non-governmental organisations, but also, possibly, independent governmental agencies, could play an enhanced role in this area.
In the case of the most advanced applications of learning analytics based on a continuous monitoring of individuals (e.g. engagement, self-regulation, classroom orchestration, game-based assessments), another question is whether stakeholders feel comfortable with some aspects of the applications even if they are legal. While the tracking and data collection necessary to power learning analytics focusing on student engagement, self-regulation or classroom orchestration have to comply with domestic data protection regulations (and algorithm regulation, if any), the question is how to make them compatible with the political values of the country where they are implemented. This may require some imagination in terms of data protection arrangements (such as deleting immediately the data once processed). As in the exam case mentioned above, this also requires a social negotiation with all stakeholders including transparency about the data collection and how they are used. This is not just a matter of regulation or even perceived ethics. Even within the same country, what can be acceptable in some communities may not be in others depending on how the smart technologies were introduced (Box 1.5).
In summer 2020, the controversies that can arise around the use of algorithms in education were placed into sharp focus for both the International Baccalaureate Organization (IBO) and the Office of Qualifications and Examinations Regulation (Ofqual) in England. Lockdown measures and school building closures put in place to curb the spread of COVID-19 led to high-stakes end-of-high-school exams (English A level and International Baccalaureate) being cancelled. The results of these exams were used to allocate university admission places, creating the need to find ways of assigning grades to students for the same purpose.
Both IBO and Ofqual opted to develop algorithms to standardise grades on the basis of data from teacher assessments, previous performance, and a variety of other factors. In the case of IBO, historical assessment data from previous exam sessions as well as individual school data was used. In the case of England’s A levels, teachers were asked to provide a centre assessment grade representing the grade students would have been most likely to achieve if teaching and learning had continued as usual rather than being disrupted by COVID-19. Teachers were also asked to provide a rank order of students for each student and grade. When centre assessment grades were considered, Ofqual found that their compound effect was likely to lead to an increase of up to 13% percentage points for some grade points compared to 2019. A statistical standardisation model was therefore developed and tested to produce grades based on the historical performance of the school or college in particular subjects as well as take into account factors such as changes in the prior attainment of candidates, centre assessment grades and rankings, and size of the cohort (Ofqual, 2020[36]).
In the case of both A levels and the IB, this process of standardisation led to substantial differences between predicted and assigned grades, meaning that offers of university places could be revoked, especially where they were conditional on students achieving a particular grade. When A-level results were released to students in England on 13 August 2020, for example, media reports suggested that around 40% of results were downgraded from teacher predictions meaning that many students did not meet the requirements of their offers for study at their first or second-choice universities (BBC, 2020[37]). Some alleged this disproportionately affected high-performing students in low-attaining schools, often in disadvantaged areas, because the algorithm had used the average of the previous performance of the school as part of the measures used to avoid grade inflation. This resulted in petitions, including from scientific associations such as the UK Royal Statistical Society, protests, and media articles complaining of a lack of transparency regarding the algorithms, models, and processes used and claiming compounded disadvantage for a cohort of students who were already having to cope with the effects of the pandemic (e.g. (Studemann, 2020[38]; Adams and McIntyre, 2020[39]).
By mid-August 2020, it was announced that both A-level and IB results would be adjusted to reflect teacher estimates rather than results produced by the algorithms, ensuring in both cases that students would receive the highest grade of the two methods.
These examples highlight the socio-technical nature of smart technologies and the need for authorities to engage in a political dialogue with stakeholders to make the outcomes of smart technologies socially acceptable. In both cases, no one contested the accuracy of the algorithms, which did what they were coded to do, but rather the design parameters for predicting or adjusting grades, the observable or perceived outcome, especially for some sub-populations, and the lack of transparency of the process. Exams and grade assignment are social institutions that have been built over decades and centuries to become acceptable and part of the “meritocracy” construct (Sandel, 2020[40]). A big challenge ahead for the use of smart technologies will be to develop similar negotiated social acceptance. This is particularly true when algorithms lead to high-stakes decisions, as was the case in these two examples.
Measuring and monitoring students’ attention, behaviour or emotions in the classroom may help teachers to keep them engaged in learning. However, privacy protection and engagement with parents and other stakeholders are key dimensions for success. Two examples from China show the importance of social acceptability, stakeholder engagement and transparency in the deployment of such technologies.
In 2019, the Jinhua Xiaoshun Primary School (Zhejiang Province, China) trialled the FocusEDU headband. In combination with a software platform, these brainwave-tracking headbands used electroencephalography (EEG) technology to measure the extent to which students paid attention in class. Three hydrogel electrodes – one at the forehead and two behind the ears – detected electrical brain signals that an AI algorithm then converts into an attention score. The FocusEDU software provided teachers real-time access to individual attention levels in the class through a dashboard. In addition, lights on the headband’s front showed different colours for different attention levels, signalling to teachers which students were identified as not paying attention. Local authorities suspended the trial in October 2019 due to privacy concerns.
Another example of monitoring students based on their behaviour and emotions was piloted in Hangzhou No. 11 Middle School (Zhejiang Province, China). Hikvision, a manufacturer of video surveillance equipment based in Hangzhou, developed cameras equipped with facial recognition technology that monitored students’ in-class behaviour and facial expression under the name of “smart classroom behavioural management system”. An AI algorithm classifies behaviour into six categories (reading, writing, listening, standing up, and lying on the desk) and distinguishes seven facial expressions (neutral, happy, sad, disappointed, angry, scared and surprised). An overall attention score was computed from these classifications that teachers could access in real-time through a screen. Following parents’ concerns, the use of the technology to evaluate facial expressions was suspended in May 2018. Since then, Hangzhou No. 11 middle school mainly uses the facial recognition cameras for monitoring attendance and for on-campus payments.
Those two examples show the difficulties that come with some monitoring aspects of smart technologies. On the research side, one difficulty lies in the quality of the theoretical models used to identify emotions and link them to learning outcomes. (There is no published research about those pilots and the underlying models to measure engagement to our knowledge.) As the proposed solutions are not so different from those that are in use at some other Chinese schools (see Box 1.1), the interruption of both pilots points more to difficulties related to local acceptance and communication, which matter also where regulation on privacy and data protection might not be as restrictive as in some OECD countries.
Source: Focus EDU (Standaert, 2019[41]; Wang, Hong and Tai, 2019[42]); Hikvision (Li and Jourdan, 2018[43]; Yujie, 2019[44]; Lee, 2018[45])
Another pragmatic and ethical issue relates to the use of the information generated by data analytics about teachers and other staff. Typically, staff within an organisation do not benefit of the same data protection and privacy regulation as students and other users. While smart technologies and learning analytics have the potential to provide feedback and support to teachers and other education stakeholders to make better decisions and improve their professional practices, they could also be used against them and unintentionally lead to undesirable social behaviours. Classroom analytics can be used to monitor teachers’ professional behaviour, and sometimes identify shortcoming in how they orchestrate learning in their classroom (Dillenbourg, 2021[12]). Should this information be used to sanction or support them? One possible intervention for students at risk of dropping out also consists of expelling or pushing them out of the school detecting this risk, which eventually enhances their likelihood to drop out eventually (Bowers, 2021[19]). Given their possible intrusive surveillance nature, the adoption of smart technologies relies on some level of trust in their positive, human-empowering ethical use. Should their voluntary use have adverse effects on teachers, school principals and decision-makers, they may appear as less acceptable and be resisted. Ethics about their use may require either full confidentiality about their results or a discussion and clear disclosure of how they can affect the staff using them. (This may also depend on their expected accuracy and effectiveness). As in the case of the information provided by longitudinal information systems, two different philosophies are possible. Some argue that their information should be used to reward and sanction stakeholders as an accountability mechanism, which is also a way to make them pay attention to the provided information. Others argue that the information should not be used to reward or sanction stakeholders as this may lead to their opposing the use of the information or just incentivise them to try to “game the system” – and thus lead to unethical behaviour. The jury is still out on what the best strategy is.
Building on the fact that smart technologies are socio-technical systems, their adoption and use will typically require some level of trust in teachers, schools, governments, and other stakeholders – and some attention paid to their possible adverse effects. Building trust should build both on regulation and the establishment of ethical practices, including different governance mechanisms:
Regulation on data protection and privacy, as well as practical guidance on how to implement those provisions with different types of data and different levels of resources and competence;
Regulation or good practice on data governance, and notably a strategy that makes data collected through smart technologies available to researchers and possibly to competitors to the vendors whose technology solution collects them;
Regulation or guidelines about the transparency, openness and replicability of algorithms, as well as funding and support for the verification of the design and the final results of algorithms by independent parties;
A risk-management approach to data protection and algorithm supervision that strikes an appropriate balance between risk-taking and data/usage sensitivity, acknowledging that regulations (or ethical guidance) that are too risk-averse will hinder the development of smart education technologies within a country or region and prevent reaping their possible benefits. The regulations above should be updated as smart technologies continue their rapid progress, and a close collaboration between researchers on data protection and on smart technologies may help co-design good solutions.
Infrastructure and public good
Smart technologies do usually require a strong Internet, computer and data infrastructure. Artificial intelligence systems, adaptive learning systems providing real-time information to teachers and learners, game-based assessments, blockchain, social robots, all require sufficient computer hardware in schools and universities (including students’ hardware under “bring your own device” policies), but also increasingly at home, as well as a bandwidth and networking capacity capable of stable and acceptable data transfer speeds. This also implies some level of investment in IT staff to deal with the maintenance of the hardware in (or across) educational establishments. Hardware is a necessary basis, but there is more to what a digital learning infrastructure should be.
Policy makers need to think about infrastructure also in terms of digital resources, both content and tools, that should be publicly provided to citizens, students, and educational institutions – either directly or indirectly through subsidies or funding to institutions. Budget constraints can limit these aspirations, but it remains important to define what should be the core of digital resources for all, and what should be accessible privately. A new question is the extent to which smart technologies should be part of the core digital education infrastructure. Is it enough to make some educational resources available to the public or should they also have the personalised learning features that some intelligent tutoring systems propose?
A final aspect of a digital infrastructure lies in people’s “digital skills”, that is, the ability to use digital resources as part of one’s professional practice. In the case of teachers and professors, digital skills are less about mastering the technology than about integrating technology tools, resources and outputs in their pedagogy. Unless fully automated, technology solutions are indeed mere tools for human teaching, learning or managing education systems. Professional learning opportunities for staff, both through training and organised continuous professional learning opportunities, are thus a final aspect that should be an integral part of a strong digital infrastructure.
While university researchers and researchers in public technology agencies may have the skills to develop some of those smart technologies, governments, schools and universities will typically rely on private education technology companies to provide and maintain them (hopefully through public-private partnerships, as mentioned above).
Beyond this investment in its digital infrastructure and connectivity for all, governments still have two important responsibilities in dealing with the private sector:
Ensuring through its procurement policies and other incentives that publicly funded or purchased smart technologies are available to schools with affordable costs.
Ensuring that some key techniques or discoveries of those smart education technologies become or remain a public, international good and allow more actors internationally to develop new interoperable solutions that will help improve education for all across the globe.
Ensuring that staff have the learning opportunities to properly use the smart technologies and digital resources at their disposal.
Research and development
This book shows how smart technologies can be beneficial to education in various ways. At the same time, a common thread through the different chapters of the book is the insufficient amount of strong evidence about the effectiveness of the various uses of smart technologies. As noted above, there is a relative lack of evidence and transparency about the actual algorithmic accuracy of the technology solutions but more evidence on their effective use in real-life educational settings should also be produced. Social robots are one of the few areas where meta-analyses and some strong evidence results could be established about some specific use of technology to improve learning outcomes (Belpaeme and Fumihide, 2021[10]).
While education research and development is needed, new approaches may be warranted.
Policy makers and research funders should remain aware that technology is not to be tested in itself. As a mere tool that is part of a socio-technical system, the appropriate research question is rarely whether some specific technological application is “effective” but more about identifying and testing whether some specific uses yield positive outcomes. Usually, when it comes to instruction, the research question will be about pedagogy and how technology supports this pedagogy rather than about the technology itself. When it comes to administrative processes, the question may be about the interventions supported or induced by the smart technology rather than the technology itself.
Some specific technological applications of smart technology have been (and can be) researched through robust experimental research designs. However, as the development of smart technologies (and computing power) advances quickly, the evidence about their effectiveness may be quickly outdated and irrelevant as the technology gets upgraded.
As other digital technologies, smart technologies allow for new types of speedy research when they are done online: A/B research. Typically, this consists of trying two different designs (A and B) of a given technology with two different groups to identify quickly which one works best (when there are large numbers of use/exposure online). This approach can be extended to classrooms to evaluate the effectiveness in using different designs of some given smart technology or of different smart technologies pursuing similar objectives in a classroom or at the system level, factoring in the complex human behaviours that are attached to their use.
Should one wait for strong evidence before using technology in education? In practice, this may be too restrictive an approach. Requirements for strong evidence before the introduction of any innovation are often a way to protect the status quo of practices whose effectiveness is rarely proven. Disciplined innovation can happen in other ways too, notably through a proportionate use of effectiveness evidence. In the case of some smart technologies, usual evaluation processes may not be the most appropriate and sometimes it may be difficult to carry them out. It would, for example, be difficult to evaluate the effectiveness of early warning systems through a randomised control trial – one reason being that systems may have to be different from one site to the other to accommodate local circumstances. The speed of development of the technology will also typically mean that experimental results will either validate or invalidate technology solutions that are already outdated by improved versions of the solution (or new technologies altogether). Perhaps another approach would be to evaluate families of technology solutions used in certain ways.
The evidence standards required for different types of solutions should probably vary depending on the possible stakes and harm coming from the solution. Smart technologies supported by a good “theory of action” or good “underlying theory” may be acceptable for use even without a strong evidence base for low-stakes applications. When there is little underlying theory, no clear theory of action, no related basis of evidence, policy makers and stakeholders should be more reluctant to allow their use in public settings. Finally, in the case of high-stakes situations, the highest evidence standards should be in place, although algorithmic effectiveness should be compared to evidence of human effectiveness when performing similar tasks.
To further address this question, governments could:
Invest in educational research about the use of technology in education in real life settings with a focus on pedagogy or administrative processes rather than the technology itself;
Develop domestic and possibly international repositories of evidence on different types of uses of technology in education and different types of families of education technology;
Develop common standards about good educational research and development in educational technology, acknowledging some of the specificities of the field.
Smart technologies that could transform education are already available. Some are more mature than others, but a range of solutions could and will make education systems and establishments operate differently in the future. Some of these tools are about the personalisation of learning (intelligent tutoring systems), about keeping students motivated and engaged in their learning, about allowing students with special needs to benefit fully from the curriculum. Technology can also support teachers. Smart technologies based on classroom analytics allow them to orchestrate teaching and learning in their class in real time and post-hoc while social robots can support them as teaching assistants, instructors and even peer students. Technology has also made big strides in supporting the management of education systems, with a host of solutions at the system and organisational level to manage budget, study paths, relationships with external stakeholders, etc. The development of early warning systems to help prevent students from dropping out of high school (or university) is a good example of this progress.
Some of the promises of smart technologies relate to the effectiveness of education. They can support students to achieve better learning outcomes, and teachers to teach (and also learn) better. Another promise lies in enhancing equity: technology helps to make education more inclusive and can provide additional learning opportunities to students from more disadvantaged groups – assuming that they are widely accessible and used. Cost-efficiency through automation is one aspect that digitalisation has brought to many sectors of society: this is also gradually happening in education. At the same time, developing and maintaining technology can be costly, and the public cost has to be balanced against its benefits.
Even if none of these promises of digitalisation materialised, digitalisation could still open new avenues for formal education and make it more convenient, more enjoyable, or just… different and aligned with modern life. Innovation is in itself a source of professional learning for teachers and also for students: it is a means to create new capacity within a system just because people have to adjust to the new requirements it promotes (Cohen and Moffitt, 2009[46]; Vincent-Lancrin, 2016[30]). Introducing digital tools in schools and universities may not have a narrow objective but be a tool to trigger change and improvement efforts. It is also a way for formal education to be part of its digital time. Should schools and universities resist or embrace digitalisation, which is gaining ground in all OECD societies, regardless of what happens in education? While formal education systems should empower everyone to enjoy, access and learn from all the knowledge and experiences that have been developed by humanity, education should be more than a museum.
Several scenarios are possible (and several have been developed: see (OECD, 2020[47]) for general schooling scenarios, and (HolonIQ, 2018[48]) for scenarios on digitalisation).
One scenario would be for education to change minimally and continue to have little adoption of technology and digital resources in teaching and learning. This may mean that most smart technologies would be available privately for out-of-school learning for those who can afford it. The education technology market would continue to mainly target its supply to the informal education market and corporate training. One long-term question in this scenario is whether education systems will remain relevant and whether out-of-school learning could become as important or more important than in-school learning.
A second scenario would be for education to look similar, on the surface, but become quite different, exactly as cars or planes look more or less the same as 40 years ago, but have become quite different now that they are fully equipped with sensors and computing devices. Education establishments may also become connected buildings with cameras, sensors and digital devices supporting students, teachers and administrators to make decisions to improve their teaching, learning and management practices. Technology may also become more prevalent for learning at home, with more intelligent tutoring resources etc. available for everyone to use.
A third scenario could be for education to build on smart technologies and other social trends related to digitalisation to reshape as a social institution. People may increasingly telework, more schoolwork may happen at home, sometimes with more involvement of parents and communities, and social time in school may become used mainly for individual tutoring and collective learning. For example, students can choose to go to school to do some tasks individually or perform them at home while other activities must be done at school with peers and under the guidance of their teachers.
The two latter scenarios would have implications for teachers and the main aspects of teaching, but also for what it means to be a student and how parents can support their children. Similar scenarios could be envisaged when it comes to the management of education systems and organisations. For example, many administrative processes could be fully automated, from assessments to the allocation of students to different educational institutions.
Of course, the future might hold completely different scenarios or any combination of them. But now is the time to think about what is possible and how digital technology can best support the improvement of education.
References
[39] Adams, R. and N. McIntyre (2020), England A-level downgrades hit pupils from disadvantaged areas hardest (The Guardian), https://www.theguardian.com/education/2020/aug/13/england-a-level-downgrades-hit-pupils-from-disadvantaged-areas-hardest (accessed on 19 May 2021).
[29] Avvisati, F. et al. (2013), “Review of the Italian Strategy for Digital Schools”, OECD Education Working Paper, Vol. 90, http://www.oecd.org/education/ceri/Innovation%20Strategy%20Working%20Paper%2090.pdf.
[15] Bailey, T., S. Smith-Jaggars and D. Jenkins (2015), Redesigning America’s Community Colleges: A Clearer Path to Student Success, Harvard University Press.
[35] Baker, R. and A. Hawn (2021), Algorithmic Bias in Education, Center for Open Science, https://doi.org/10.35542/osf.io/pbmvz.
[37] BBC (2020), A-levels and GCSEs: U-turn as teacher estimates to be used for exam results, https://www.bbc.com/news/uk-53810655 (accessed on 19 May 2021).
[10] Belpaeme, T. and T. Fumihide (2021), “Social Robots as Educators”, in OECD Digital Education Outlook 2021: Pushing the frontiers with AI, blockchain, and robots, OECD Publishing.
[19] Bowers, A. (2021), “Early Warning systems and indicators of dropping out of upper secondary school: the emerging role of digital technologies”, in OECD Digital Education Outlook 2021: Pushing the frontiers with AI, blockchain, and robots, OECD Publishing.
[14] Buckley, J. et al. (2021), “Game-based Assessment for Education”, in OECD Digital Education Outlook 2021: Pushing the frontiers with AI, blockchain, and robots, OECD Publishing.
[46] Cohen, D. and S. Moffitt (2009), The ordeal of equality: Did federal regulation fix the schools?, Harvard University Press, Cambridge, MA.
[26] Cuban, L. (1986), Teachers and Machines: The Classroom of Technology Since 1920, Teachers College Press.
[9] D’Mello, S. (2021), “Improving Student Engagement in and with Digital Learning Technologies”, in OECD, Digital Education Outlook: Pushing the frontiers with AI, blockchain, and robots, OECD Publishing.
[12] Dillenbourg, P. (2021), “Classroom analytics: Zooming out from a pupil to a classroom”, in OECD Digital Education Outlook 2021: Pushing the frontiers with AI, blockchain, and robots, OECD Publishing.
[23] González-Sancho, C. and S. Vincent-Lancrin (2016), “Transforming education by using a new generation of information systems”, Policy Futures in Education, Vol. 14/6, pp. 741-758.
[24] González-Sancho, C. and S. Vincent-Lancrin (n.d.), Transforming education by using a new generation of information systems.
[11] Good, J. (2021), “Serving students with special needs better: How digital technology can help”, in OECD Digital Education Outlook 2021: Pushing the frontiers with AI, blockchain, and robots, OECD Publishing.
[48] HolonIQ (2018), Building Scenarios for Education in 2030, https://www.holoniq.com/2030/building-scenarios-education-in-2030 (accessed on 19 May 2021).
[13] Ifenthaler, D. (2021), “Learning Analytics for School and System Management”, in OECD Digital Education Outlook 2021: Pushing the frontiers with AI, blockchain, and robots, OECD Publishing.
[22] KERIS (2010), “The Future of Education: NEIS (National Education Information System)”, Presentation at OECD/SSRC/Stupski Workshop on Educational Information Systems for Innovation and Improvement, https://www.oecd.org/education/ceri/46182781.ppt (accessed on 19 May 2021).
[34] Kizilcec, R. and H. Lee (2020), “Algorithmic Fairness in Education”, in Holmes, W. and K. Porayska-Pomsta (eds.), Ethics in Artificial Intelligence in Education (forthcoming), Taylor & Francis. ArXiv E-Prints, https://arxiv.org/abs/2007.05443.
[18] Lambert, S. (2020), “Do MOOCs contribute to student equity and social inclusion? A systematic review 2014–18”, Computers & Education, Vol. 145, p. 103693, https://doi.org/10.1016/j.compedu.2019.103693.
[7] Lang, C. et al. (eds.) (2017), Handbook of Learning Analytics, SOLAR, https://solaresearch.org/wp-content/uploads/2017/05/hla17.pdf.
[45] Lee, D. (2018), Must Reads: At this Chinese school, Big Brother was watching students — and charting every smile or frown (Los Angeles Times), https://www.latimes.com/world/la-fg-china-face-surveillance-2018-story.html (accessed on 19 May 2021).
[43] Li, P. and A. Jourdan (2018), Sleepy pupils in the picture at high-tech Chinese school (Reuters), https://www.reuters.com/article/us-china-surveillance-education/sleepy-pupils-in-the-picture-at-high-tech-chinese-school-idUSKCN1II123 (accessed on 19 May 2021).
[32] Malgieri, G. (2019), “Automated decision-making in the EU Member States: The right to explanation and other “suitable safeguards” in the national legislations”, Computer Law & Security Review, Vol. 35/5, p. 105327, https://doi.org/10.1016/j.clsr.2019.05.002.
[8] Molenaar, I. (2021), “Personalisation of learning: Towards hybrid human-AI learning technologies”, in OECD Digital Education Outlook 2021: Pushing the frontiers with AI, blockchain, and robots, OECD Publishing.
[20] Murphy, R. et al. (2020), “Investigating Efficacy, Moderators and Mediators for an Online Mathematics Homework Intervention”, Journal of Research on Educational Effectiveness, Vol. 13/2, pp. 235-270, https://doi.org/10.1080/19345747.2019.1710885.
[33] O’Neil, C. (2016), Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, Penguin Books.
[47] OECD (2020), Back to the Future(s) of Education, OECD, https://doi.org/10.1787/178ef527-en.
[2] OECD (2019), Artificial Intelligence in Society, OECD Publishing, Paris, https://dx.doi.org/10.1787/eedfee77-en.
[31] OECD (2019), Enhancing Access to and Sharing of Data: Reconciling Risks and Benefits for Data Re-use across Societies, OECD Publishing, Paris, https://dx.doi.org/10.1787/276aaca8-en.
[1] OECD (2019), Going Digital: Shaping Policies, Improving Lives, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264312012-en.
[16] OECD (2019), “Using digital technologies to improve the design and enforcement of public policies”, OECD Digital Economy Papers, No. 274, OECD Publishing, Paris, https://dx.doi.org/10.1787/99b9ba70-en.
[6] OECD (2018), “IoT measurement and applications”, OECD Digital Economy Papers, No. 271, OECD Publishing, Paris, https://dx.doi.org/10.1787/35209dbf-en.
[5] OECD (2017), OECD Blockchain Primer, https://www.oecd.org/finance/OECD-Blockchain-Primer.pdf.
[36] Ofqual (2020), Research and analysis: Awarding GCSE, AS & A levels in summer 2020: interim report, https://www.gov.uk/government/publications/awarding-gcse-as-a-levels-in-summer-2020-interim-report (accessed on 19 May 2021).
[17] Orr, D., M. Rimini and D. van Damme (2015), Open Educational Resources: A Catalyst for Innovation, Educational Research and Innovation, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264247543-en.
[27] Reich, J. (2020), Failure to Disrupt: Why Technology Alone Can’t Transform Education, Harvard University Press.
[21] Roschelle, J. et al. (2016), “Online Mathematics Homework Increases Student Achievement”, AERA Open, Vol. 2/4, p. 233285841667396, https://doi.org/10.1177/2332858416673968.
[40] Sandel, M. (2020), The Tyranny of Merit: What’s Become of the Common Good?, Allen Lane.
[25] Smolenski, N. (2021), “Blockchain for education: a new credentialing ecosystem”, in OECD Digital Education Outlook 2021: Pushing the frontiers with AI, blockchain, and robots, OECD Publishing.
[41] Standaert, M. (2019), Chinese primary school halts trial of device that monitors pupils’ brainwaves (The Guardian), https://www.theguardian.com/world/2019/nov/01/chinese-primary-school-halts-trial-of-device-that-monitors-pupils-brainwaves (accessed on 19 May 2021).
[38] Studemann, F. (2020), A-level results chaos will have lasting impact on class of 2021 (Financial Times), https://www.ft.com/content/0e86d8c1-cb47-4965-bef2-6f694352235a (accessed on 19 May 2021).
[3] van der Vlies, R. (2020), “Digital strategies in education across OECD countries”, OECD Education Working Papers, Organisation for Economic Co-Operation and Development (OECD), https://doi.org/10.1787/33dd4c26-en.
[30] Vincent-Lancrin, S. (2016), “Innovation, Skills and Adult Learning: two or three things we know about them”, European Journal of Education, Vol. 51/2, pp. 146-153.
[4] Vincent-Lancrin, S. et al. (2019), Fostering Students’ Creativity and Critical Thinking: What it Means in School, Educational Research and Innovation, OECD Publishing, Paris, https://dx.doi.org/10.1787/62212c37-en.
[28] Vincent-Lancrin, S. et al. (2019), Measuring Innovation in Education 2019: What Has Changed in the Classroom?, Educational Research and Innovation, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264311671-en.
[42] Wang, J., S. Hong and C. Tai (2019), China’s Efforts to Lead the Way in AI Start in Its Classrooms (The Wall Street Journal), https://www.wsj.com/articles/chinas-efforts-to-lead-the-way-in-ai-start-in-its-classrooms-11571958181 (accessed on 19 May 2021).
[44] Yujie, X. (2019), Camera Above the Classroom (Sixth tone), https://www.sixthtone.com/news/1003759/camera-above-the-classroom (accessed on 19 May 2021).