1. Introduction
This project, led and implemented by the Organisation for Economic Co-operation and Development (OECD), was carried out with financial support provided by the European Commission’s Directorate-General for Structural Reform Support (DG REFORM), and in close collaboration with the Hungarian Ministry of Culture and Innovation (KIM) and the Hungarian Accreditation Committee (MAB).
This chapter presents the project’s context and objectives, defines key concepts and starting points, and presents the analytical approach and methodology underpinning the project.
This section summarises key findings from the project “Supporting the Digital Transformation of Higher Education in Hungary” (OECD, 2021[1]) and describes how the current project builds on the achievements of this project to further support the digital transformation and quality of higher education in Hungary.
Supporting the digital transformation of higher education in Hungary
“Supporting the Digital Transformation of Higher Education in Hungary” (OECD, 2021[1]) reviewed the state of higher education digitalisation in Hungary, analysing three dimensions of higher education digitalisation: digital readiness, digital practices, and digital performance. The project found that the Hungarian higher education system made a successful transition to emergency remote instruction in response to the COVID-19 pandemic, using digital technologies to ensure that learners had continued access to their study programmes. However, according to students and academic staff, the quality of the digital higher education provided during the pandemic varied. When surveyed as part of the project, among other issues they highlighted significant deficiencies in access to digital infrastructure and digital resources, and insufficiently tailored course design, delivery and assessment practices in digital study environments.
The project also identified Hungary’s existing regulatory and external quality assurance (QA) frameworks for higher education as one of the main barriers to further strengthening the digitalisation and quality of teaching and learning in higher education. Recommendation 2 of the project states (OECD, 2021, p. 54[1]):
The government should use feedback from higher education stakeholders to develop a system change plan designed to remove obstacles to the adoption of digitally enhanced learning, make legislative or regulation changes as necessary, and use funding incentives to encourage change in particular areas. This could involve […] reviewing accreditation and QA practices and requirements (in the legislation and rules of the Hungarian Accreditation Committee, as necessary) to ensure they are neutral between different modes of delivery; and providing guidance to institutions on how to implement internal QA processes in a digital environment.
Ensuring quality digital higher education in Hungary
“Ensuring Quality Digital Higher Education in Hungary” has sought to build on the findings and recommendations of the project “Supporting the Digital Transformation of Higher Education in Hungary” (OECD, 2021[1]), It was launched to support:
The adoption of new quality standards as a basis for government policymaking and a revised external quality assurance framework; and
The development of new external and internal QA services and support mechanisms by the government and implemented by HEIs.
Its recommendations and policy options for Hungary encompass three areas:
Standards. Policy recommendations and policy options to support the development of QA standards and procedures that can be adopted into Hungarian legislation by KIM and implemented by MAB and HEIs to assure and improve digital higher education.
Practices. Policy recommendations and options to strengthen HEIs’ autonomy and capacity to adopt practices to effectively manage the quality of their digital education offerings.
Supports. Policy recommendations and options to develop relevant institutional supports for the enhancement of digital and learning infrastructures to assist HEIs in meeting quality standards for digital education.
This section defines key concepts underpinning the review: digital education and quality assurance. It then outlines the importance of quality assurance in the context of digital higher education.
Understanding digital education
Three types of digital education
Based on differences in time and location of instruction, three broad types of digital education can be identified: online, hybrid and blended education (see Box 1.1). The location of instruction considers the amount of time spent learning online (remotely) versus in-person; the time of instruction refers to whether learning takes place synchronously (i.e., “learning in which learner(s) and instructor(s) are in the same place, at the same time, in order for learning to take place”) or asynchronously (i.e., “different times and spaces particular to each learner […] instructors usually set up a learning path, which students engage with at their own pace”) (Finol, 2020[3]).
Online education
All instruction is delivered online, either synchronously or asynchronously, or a combination of both. While instruction is delivered solely at a distance, learners may have the option to meet in person with peers or instructors, or to make use of on-campus facilities and learning materials. It is different from “distance education”, which describes all forms of education where learner and instructor are physically separated (e.g., internet, radio, television, and print-based instruction).
Hybrid education
Education is delivered through a mix of online and on-campus instruction, with the online components taking place synchronously, asynchronously, or a combination of both. The online components replace, and therefore reduce the frequency of, in-person instruction.
Blended education
Instruction takes place fully in-person and is blended with or enhanced by online materials and activities, such as a virtual learning environment or learning management system (VLE/LMS), open educational resources (OER), simulations, or gaming. In contrast to hybrid education, the online components are intended to build upon classroom instruction rather than replace it.
Source: Adapted from Staring et al. (2022[2]), “Digital Higher Education: Emerging Quality Standards, Practices and Supports”, OECD Education Working Papers, No. 281, OECD Publishing, Paris, https://www.oecd-ilibrary.org/education/digital-higher-education_f622f257-en.
Blended education as the new normal
The definition above recognises that all education is – or soon will be – “blended” or enhanced by some form of digital technology. Gourlay (2021[4]) explains that there is almost no in-person instruction that is not supported, at least to some extent, by digital tools or a virtual learning environment or learning management system (VLE/LMS) to structure and support teaching and learning. For example, even in cases where instruction takes place fully face-to-face between instructors and students, most instructors make use of presentational tools (such as PowerPoint) and other forms of digital technology to accompany and structure their lectures or use email to facilitate communication and collaboration with and between their students for the preparation, submission, and assessment of assignments. Likewise, adopting a socio-material perspective, Gourlay (2021[4]) argues that – especially since the COVID-19 pandemic – our human connection with technology has become such an integral part of our everyday lives that the very notion of “virtual learning” is flawed. Even when studying at home and alone in front of a screen, Gourlay argues, learning has an in-person dimension (Gourlay, 2021, p. 57[4]).
Understanding quality assurance in higher education
The development of QA systems in higher education is a relatively recent phenomenon (OECD, 2019[5]). It is only during the last three decades that most governments across the OECD have introduced some form of external QA for higher education and that HEIs have started to adopt internal processes to ensure that the educational content, teaching and learning practices, student support services, and outcomes of their courses and programmes meet national and international quality standards. These developments represent a change in direction from the historic tradition of leaving the QA of learning and teaching in higher education to qualified academic staff with an established record of scholarship. A high degree of academic autonomy in universities has also meant that it has traditionally been very difficult for governments and HEI management to intervene in the teaching activities of higher education staff members.
The emergence of quality assurance
Increased government intervention to assure the quality of higher education provision is particularly linked to the fact that HEIs have had to adapt their course offerings and instructional practices to an increasingly diverse student population and a competitive higher education market. On the one hand, HEIs are being asked to rapidly adapt and diversify their course offer to meet the demands of an increasingly wide range of higher education audiences, or “clients”, ranging from students from disadvantaged socio-economic backgrounds to (working) adults in need of upskilling or reskilling. HEIs are also being challenged to diversify their course offer in response to an expanded and increasingly competitive higher education market, filled with private providers offering many of their courses and programmes online. For example, Allied Market Research, estimated the total value of the e-learning market to be at USD 197 billion in 2020 and to reach USD 840 billion by 2030, with a compound annual growth rate (CAGR) of 17.5% from 2021 to 2030 (Allied Market Research, 2020[6]). Coupled with shrinking public budgets and investments in higher education, it has become more important than ever for governments and HEIs to ensure that higher education teaching and learning remains of the highest possible quality, and delivers the outcomes needed by students and society.
The purpose of quality assurance
Within the higher education community, there exists a distinction between external QA for the purpose of accountability (assurance) and external QA for the purpose of improvement (enhancement) (ENQA, 2015[7]; CHEA, 2016[8]). Quality assurance (QA) can be described as “regulatory” or formal activities aimed at providing information to assure the public beneficiaries or “clients” of higher education (students, employers, governments, civil society) of the quality of HEIs’ activities or “the process of establishing stakeholder confidence that provision (input, process and outcomes) fulfils expectations and measures up to threshold minimum requirements” (Harvey, n.d.[9]). Quality enhancement (QE) refers to “enabling” activities that seek to build institutions’ capacity for the development of their own internal QA processes by providing them with advice, recommendations and supports (OECD, 2018, p. 53[10]). The relationship between QA and QE – and the role of QA agencies in both – has dominated the international QA debate for several decades (see Box 1.2).
Looking at the relationship between quality assurance (QA) and quality enhancement (QE), Williams (2016[11]) identifies a spectrum of views. Firstly, there are those who believe that QA and QE “must be conceptually and practically distinct, with separate resourcing” (Williams, 2016, p. 98[11]). This creates the perception that QA and QE can work in isolation from each other, which has often been the case in the past, with national QA processes not leading to any significant QE outcomes. Several critics see QA and QE in opposition to each other, with QE presented in a much more positive light than QA. QA, according to this group, is seen as a negative and burdensome “naming and shaming” practice which undermines the academic integrity and expertise of scholars. Other perspectives view QA and QE on a linear scale, where “quality enhancement is dependent on QA. This implies a need for good QA data that is then used to inform enhancement” (Williams, 2016, p. 100[11]). Finally, there are those who view QA and QE as an integral part of the same process, with the results of each process feeding into the other. According to this last view, external QA carried out by national or government agencies should not only encourage but also be informed by QE activities, including the practices of HEIs themselves.
Source: Williams (2016[11]), “Quality assurance and quality enhancement: is there a relationship?”, Quality in Higher Education 22 (2), pp. 97-102, https://doi.org/10.1080/13538322.2016.1227207
Quality assurance of digital higher education
There is a lack of sound evidence on the risks and benefits of online learning, however, and the few studies to date that have evaluated the quality of online and hybrid instruction are not always conclusive and often focus on comparing online instruction with face-to-face instruction. For example, one paper, which reviews several US studies on the quality of online instruction during the pandemic, highlights that most studies to date show mixed results, have been carried out on a single institution (or even a single course within that institution) and that “the content, instructor, assignments, and other course features might differ across online and in-person modes as well, which makes apples-to-apples comparisons difficult” (Riegg and Friday, 2021[12]). The negative impacts of digital education are most often experienced among bachelor-level and disadvantaged students. Potential positive impacts include lower time-to-degree completion for more advanced students, highlighting potential efficiency gains for higher education, although there is also a high number of online programmes with high drop-out rates.
Despite the lack of conclusive evidence – so far – on the quality of online and hybrid instruction, the COVID-19 pandemic has made it clear that digital higher education in its fully online, hybrid and blended formats is here to stay. It is therefore important to guide, assess and support institutions to enhance the quality of their digital provision. As we are moving towards a “post-digital understanding of teaching and learning environments” (Nørgård, 2021, p. 12[13]) in which digital and online has become part of our everyday actions, interactions and experiences – including education – it should therefore be within the scope of QA agencies. Not covering digital education would entail maintaining an “implicit bias […] towards the ‘presential’ learning found in classrooms and seminar rooms” (Bacsich et al., 2015, p. 7[14]). Delivering high-quality digital higher education requires HEIs and instructors to put in place a range of additional considerations, which should therefore form part of the quality indicators monitored by QA agencies and the supports offered by public authorities. One study estimates that the provision of quality and equitable digital education requires almost doubling the human and financial resources of institutions (EDUCAUSE, 2021[15]).
A comprehensive view of quality assurance
Building on the view that QA and QE are an integral part of the same process, ensuring the quality of digital higher education requires three mutually reinforcing mechanisms (see Figure 1.1).
National quality standards, enhanced for digital education, embedded in national legislation, and monitored by an independent external QA agency, to steer and guide institutional practices, while guaranteeing their autonomy and flexibility to develop innovative (and digital) study programmes.
Meaningful accreditation processes that enable HEIs to take responsibility for the development of internal quality management practices to ensure their (digital) courses and programmes provide students with relevant learning outcomes and labour market opportunities.
Supports and incentives to build the capacity of HEIs to effectively manage the quality of their (digital) course offerings and support the enhancement of (digital) teaching and learning practices.
At the heart of these three mechanisms – and, ultimately, at the heart of higher education quality – should be the shared ambition of all stakeholders to enhance the quality of teaching, learning and research.
This section presents the analytical framework and methodology underpinning the project. It describes the research questions guiding the analysis, as well as the primary and secondary research methods used to conduct the analysis and engage higher education stakeholder organisations and institutions in the project, both across Hungary and internationally.
Analytical framework
The project was guided by three overarching research questions (see Figure 1.2):
Research question 1. What is the impact of Hungary’s regulatory framework and external quality assurance system for higher education on the development and quality of digital higher education?
Research question 2. What is the capacity of higher education institutions in Hungary to manage the quality of their digital study programmes and courses?
Research question 3. What supports are available to higher education institutions and instructors in Hungary to enhance the quality of their digital teaching and learning practices and internal quality management practices?
Regulatory framework and external quality assurance of digital higher education
The first research question was broken down into two areas of analysis. First, Hungary’s regulatory framework for higher education, or how teaching and learning in higher education is organised at institutional and programme level. Specific consideration was given to the degree structure and study formats in Hungarian higher education, and how they shape the student learning experience (e.g. student admission, course selection, progression and certification), as well as how national regulation defines the flexibility of institutions and instructors to develop innovative and labour market relevant (digital) study programmes.
The second area of analysis concerned Hungary’s external quality assurance system of higher education, focusing more specifically on the role of MAB as the designated independent higher education QA agency in Hungary. This included looking at MAB’s activities for assuring and reviewing the quality of HEIs and their operations at institution and programme level (i.e. quality assurance), as well as how MAB supports HEIs with the quality enhancement of their institutional quality management practices (i.e. quality enhancement).
Quality assurance. Looking at the QA activities typically carried out by QA bodies across the OECD, a distinction is commonly made between agencies that conduct reviews and accreditation at institution or programme level, or a combination of both. There are also differences in terms of the stage at which accreditation takes place. While in many jurisdictions, there is still a strong focus on ensuring that institutions and programmes meet a number of minimum requirements set out in national regulation prior to operation (i.e. ex ante accreditation), many jurisdictions have also introduced processes to monitor and assure the quality of the processes and outputs of higher education (i.e. ex post accreditation) (Krcal, Glass and Tremblay, 2014[16]).
Quality enhancement. A review of common policies and practices for the QE of digital higher education across QA agencies in the OECD (Staring et al., 2022[2]) identified three main mechanisms that agencies are using to support institutions with the QE of their digital teaching and learning practices and institutional quality management systems: the development of a common taxonomy of guidelines for the QE of digital higher education; the collection and dissemination of resources and good practice for digital teaching and learning; and training and support for instructors and QA staff.
Table 1.1 presents the analytical framework guiding the review of Hungary’s regulatory framework and external quality assurance system for digital higher education.
Institutional quality management of digital higher education
As the principal responsibility for quality rests with HEIs, the second research question focused on how HEIs in Hungary are managing the quality of their digital study programmes. This involved looking at how the overall structure, governance, and management of HEIs in Hungary affects the development of internal quality cultures. The specific institutional practices for the quality management of digital courses and study programmes were analysed following the key principles for effective institutional QA of digital higher education identified in Staring et al. (2022, p. 26[2]):
Planning and investment. The institutional quality management of digital higher education requires clear digitalisation objectives and indicators for monitoring their implementation in all areas of institutional activity, including policies and processes for QA and development, supported by investments in the necessary digital education infrastructure. Decisions on digital education policies and infrastructure investments should be evidence-based, developed in consultation with relevant stakeholder groups, set out in writing and communicated transparently.
Implementation. The implementation of an institutional digitalisation and QA strategy should be carried out on a decentralised basis, by the directly responsible unit(s). The institution should support the QA and development processes centrally, through professional services and the provision of the necessary resources, and should pay specific attention to supporting students and instructors with the effective use of digital technologies for pedagogical purposes.
Monitoring. Institutional strategies and implementation processes should be embedded in a system of monitoring and feedback loops to assess the performance and quality of digital higher education (and QA processes) on an ongoing basis. Monitoring practices should seek to collect data and feedback from stakeholder groups across the institution through a variety of both qualitative and quantitative data collection mechanisms (e.g. interviews, surveys, learning analytics data), as well as specific internal or external reviews and/or benchmarking exercises of digital teaching and learning practice.
Table 1.2 presents the analytical framework guiding the review of institutional practices for the QA of digital higher education in Hungary.
Supporting the quality enhancement of digital teaching and learning
The third research question focused on analysing Hungary’s wider institutional support landscape for the quality enhancement of digital higher education. Building on the four “phases of action” identified as part of the OECD project “Supporting the Digital Transformation of Higher Education in Hungary” (i.e. setting the direction, building the foundation, developing the processes, and delivering benefits to users) (OECD, 2021, p. 48[1]) and on a mapping of institutional supports for digital higher education across the OECD in Staring et al. (2022, pp. 53-55[2]), this was done by focusing on three key questions:
Who are the actors supporting the quality enhancement of (digital) teaching and learning in Hungary? The government and other publicly funded national bodies have a key role to play in supporting institutions to enhance the quality of their digital practices and develop sound internal QA systems. In addition to this, a wide range of sectoral stakeholder associations (can) also play a role in supporting the quality enhancement of digital higher education, including student and staff associations, and national academies of science.
How are different actors in Hungary supporting the quality enhancement of digital teaching and learning in higher education? Institutional support can be provided through four main policy levers or mechanisms: strategy setting and guidance; financial support and incentives; stakeholder capacity building and collaboration; and national performance monitoring and evidence collection.
What are the key areas in which HEIs require support? Four main areas of institutional support can be targeted through these mechanisms: the development of institutional policies for the QA of digital higher education; the development and effective use of digital resources; the professional development of instructors; and the development of institutional performance monitoring processes.
Table 1.3 presents the analytical framework guiding the review of Hungary’s institutional support landscape for digital higher education.
Source: Based on a review of emerging quality standards, practices and supports for digital higher education in Staring et al. (2022[2]), “Digital Higher Education: Emerging Quality Standards, Practices and Supports”, OECD Education Working Papers, No. 281, OECD Publishing, Paris, https://www.oecd-ilibrary.org/education/digital-higher-education_f622f257-en.
Methodology
The project included two main activities undertaken between November 2021 and March 2023: a national and international review of policies and practices for ensuring quality digital higher education, and a wide range of stakeholder engagement activities (see Figure 1.3 overleaf).
National and international policy and practice review. The project collected and analysed evidence – through desk research and expert interviews – on policies and practices for the quality assurance of digital higher education in Hungary and internationally. The review of policies and practices in Hungary – carried out in collaboration with national experts – led to the preparation of an internal report Analysis of the Hungarian Quality Assurance Landscape for Higher Education, submitted to the project Steering Committee in July 2022. The international mapping of policies and practices across the OECD and European Higher Education Area (EHEA) – carried out in collaboration with international experts – led to the publication of an OECD Working Paper on Digital Higher Education: Emerging Quality Standards, Practices and Supports in November 2022 (Staring et al., 2022[2]). The findings included in both reports form the basis for the analysis, policy options and recommendations presented in this report.
Stakeholder engagement activities. As part of the project, the OECD review team organised a range of stakeholder engagement activities to involve policymakers, higher education stakeholder organisations, institutions, practitioners and students across the sector in a national dialogue reflecting on how Hungary’s regulatory, quality assurance and institutional support frameworks for higher education could be revised to enhance the capacity of institutions and instructors to offer high-quality digital study programmes. As part of these activities, the OECD review team engaged more than 200 higher education stakeholders across Hungary. An overview of the stakeholder engagement activities is presented in Annex A.
The report is structured as follows:
Chapter 2 presents a review of Hungary’s regulatory framework and external quality assurance system for higher education, and presents policy options and recommendations on how both can be revised to support a modernisation of Hungary’s higher education system that embeds flexibility and digitalisation at the heard of its system.
Chapter 3 reviews institutional practices for the quality management of digital higher education in Hungary, including recommendations and policy options on how Hungary’s existing accreditation processes can be revised to incentivise greater institutional responsibility for assuring the quality of (digital) course offers.
Chapter 4 analyses Hungary’s institutional support landscape for digital higher education and proposes recommendations and policy options on how Hungary can support institutions in the quality enhancement of their digital teaching and learning practices.
Annex A provides an overview of the stakeholder engagement activities carried out as part of the project, the research tools used and participants in each activity.
Annex B presents an analysis of the assessment frameworks used by MAB for the accreditation of institutions, bachelor’s, master’s and doctoral programmes.
Source: The authors, based on the Detailed Project Description (DPD) agreed between the European Commission’s DG REFORM, KIM, MAB and the OECD
References
[6] Allied Market Research (2020), Global E-Learning Market: Opportunities and Forecast, 2021-2030, https://www.alliedmarketresearch.com/e-learning-market-A06253 (accessed on 19 December 2022).
[14] Bacsich, P. et al. (2015), The Showcases Reports Quality Agencies in Europe and their approach to e-learning and open learning, European Association of Distance Teaching Universities (EADTU), Brussels, https://www.sequent-network.eu/ (accessed on 19 December 2022).
[8] CHEA (2016), The CIQG International Quality Principles: Toward a Shared Understanding of Quality, Edited by Stamenka Uvalić-Trumbić. Council for Higher Education Accreditation (CHE), https://www.chea.org/sites/default/files/pdf/Principles_Papers_Complete_web.pdf (accessed on 19 December 2022).
[15] EDUCAUSE (2021), 2021 EDUCAUSE horizon report. Teaching and learning edition., Educause, Louisville, https://library.educause.edu/-/media/files/library/2021/4/2021hrteachinglearning.pdf?la=en&hash=C9DEC12398593F297CC634409DFF4B8C5A60B36E (accessed on 21 July 2022).
[7] ENQA (2015), Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG), European Association for Quality Assurance in Higher Education (ENQA), Brussels, http://www.enqa.eu/wp-content/uploads/2015/11/ESG_2015.pdf (accessed on 19 December 2022).
[3] Finol, M. (2020), “Asynchronous vs. Synchronous Learning: A Quick Overview”, Bryn Mawr College, https://www.brynmawr.edu/blendedlearning/asynchronous-vs-synchronous-learning-quick-overview (accessed on 19 December 2022).
[4] Gourlay, L. (2021), “There Is No ’Virtual Learning’: The Materiality of Digital Education”, Journal of New Approaches in Educational Research, Vol. 10/1, p. 57, https://doi.org/10.7821/naer.2021.1.649 (accessed on 19 December 2022).
[9] Harvey, L. (n.d.), Analytic Quality Glossary, Quality Research International, http://www.qualityresearchinternational.com/glossary/ (accessed on 19 December 2022).
[16] Krcal, A., A. Glass and K. Tremblay (2014), “Monitoring and enhancing quality in higher education: developing a quality framework”, in Glass, A. (ed.), The State of Higher Education 2014, OECD, Paris, http://www.oecd.org/edu/imhe (accessed on 19 December 2022).
[13] Nørgård, R. (2021), “Theorising hybrid lifelong learning”, British Journal of Educational Technology, Vol. 52/4, pp. 1709-1723, https://doi.org/10.1111/bjet.13121 (accessed on 19 December 2022).
[1] OECD (2021), Supporting the Digital Transformation of Higher Education in Hungary, Higher Education, OECD Publishing, Paris, https://doi.org/10.1787/d30ab43f-en (accessed on 19 December 2022).
[5] OECD (2019), Benchmarking Higher Education System Performance, Higher Education, OECD Publishing, Paris, https://doi.org/10.1787/be5514d7-en (accessed on 19 December 2022).
[10] OECD (2018), Rethinking Quality Assurance for Higher Education in Brazil, Reviews of National Policies for Education, OECD Publishing, Paris, https://doi.org/10.1787/9789264309050-en (accessed on 19 December 2022).
[12] Riegg, S. and C. Friday (2021), How does virtual learning impact students in higher education? Evaluating online instruction in higher education, https://www.brookings.edu/blog/brown-center-chalkboard/2021/08/13/how-does-virtual-learning-impact-students-in-higher-education/.
[2] Staring, F. et al. (2022), “Digital Higher Education: Emerging Standards, Practices, and Supports”, OECD Education Working Papers , No. 281, OECD Publishing, Paris, p. 97, https://www.oecd-ilibrary.org/education/digital-higher-education_f622f257-en (accessed on 19 December 2022).
[11] Williams, J. (2016), “Quality assurance and quality enhancement: is there a relationship?”, Quality in Higher Education, Vol. 22/2, pp. 97-102, https://doi.org/10.1080/13538322.2016.1227207 (accessed on 19 December 2022).