3. Institutional quality management of digital higher education
This project, led and implemented by the Organisation for Economic Co-Operation and Development (OECD), was carried out with financial support provided by the European Commission’s Directorate-General for Structural Reform Support (DG REFORM), in close collaboration with the Hungarian Ministry of Culture and Innovation (KIM) and the Hungarian Accreditation Committee (MAB).
This chapter analyses trends in institutional practice for the quality management of digital higher education in Hungary and provides recommendations on how accreditation processes can be revised to incentivise institutions to take greater responsibility for the quality management and innovation of their (digital) education offerings.
This section starts by analysing the general development of institutional quality cultures for teaching and learning in Hungarian higher education institutions (HEIs), followed by trends in how HEIs in Hungary have responded to the challenge of managing the quality of digital higher education more specifically. It then presents three key barriers to the further development of institutional quality cultures in Hungary.
Slow development of institutional quality cultures for teaching and learning
It is a well-known principle, articulated in international quality circles (e.g. the International Network of Quality Assurance Agencies in Higher Education (INQAAHE) or the European Association of Quality Assurance in Higher Education (ENQA), that responsibility for assuring the quality of teaching and learning rests principally with higher education providers, while quality assurance (QA) agencies, in their capacity as independent expert bodies, are responsible for ensuring the inputs, processes and outcomes of programmes offered by HEIs meet quality standards set out in national law and regulation. The European Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG), for example, state that “higher education institutions have primary responsibility for the quality of their provision and its assurance” (ENQA, 2015[1]). Similarly, INQAAHE’s Guidelines of Good Practice state that “institutional and programmatic quality and quality assurance are primarily the responsibility of the higher education institutions (HEIs) themselves, and [this] respects the academic autonomy, identity and integrity of the institutions and programmes” (INQAAHE, 2018, p. 7[2]).
From as early as 1993, Hungarian HEIs were required to put in place regulations and processes for the quality management of their internal operations, programmes, staff and student support services, in line with Part 1 of the ESG. The National Act on Higher Education stipulates that the Hungarian Accreditation Committee (MAB), in its external reviews of HEIs, should “tak[e] into account the Standards and Guidelines in the European Higher Education Area” (Government of Hungary, 2011[3]). Institutional quality cultures for teaching and learning are still developing in Hungary, slowed by three main conditions.
Perceptions of quality assurance as administratively burdensome
The first reason relates to the wider political history of the country. Stakeholders interviewed by the OECD review team explained that before the regime change in 1989, the tradition and practice of QA was not common among Hungarian HEIs, as it was seen as a control mechanism exercised by the ruling communist party. This has significantly shaped how QA is perceived in Hungary today, i.e. as a “regulatory” administrative process to exert control over the practice of individual institutions and instructors rather than an “enabling” process to support quality enhancement (DiMaggio and Powell, 1983[4]). In this context, HEIs mentioned the ex ante programme accreditation process as an example of a highly burdensome administrative procedure that hinders the development of institutional quality cultures. However, international evidence also shows that the perception of QA as an administratively burdensome or “box-ticking exercise” purely to satisfy external expectations is common across many higher education systems (Greere, 2022[5]). One higher education stakeholder interviewed by the OECD review team described the issue as follows:
“Quality assurance should not be seen as necessary or a burden. It should provide helpful and competitive services and information to students and staff” (Higher education stakeholder, February 2022)
Lack of shared national guidance, training, or support
The second reason highlighted by higher education stakeholders interviewed by the OECD review team is the lack of nationally shared guidance, support and resources on “why” or “how” to embed the ESG in institutional contexts. The only resources currently available to HEIs in Hungary are the highly detailed application and evaluation sheets used by MAB as part of accreditation procedures. However, several stakeholders felt that these templates could not be used as guidance materials to support quality enhancement.
In other OECD jurisdictions, QA agencies have developed specific guidance to support institutions with the implementation of national and international quality standards. An example is Malta, which in addition to its national standards and guidelines for institutional accreditation, has developed a Step-by-Step Guide to Internal Quality Assurance. The guide is “aimed mainly at providers that are still developing their IQA [internal quality assurance] policy” (National Commission for Further and Higher Education Malta, 2017a, p. 5[6]) and addresses all standards included in the national QA framework for further and higher education (National Commission for Further and Higher Education Malta, 2017b[7]), as well as the ESG. Other systems have developed specialised training programmes for institutional QA staff, as evidence shows that these actors are often appointed “with minimum preparation or training; and only external quality assurance requirements to guide internal action” (Greere, 2022, p. 2[5]). In Spain, for example, the national QA agency runs a specific programmes to support HEIs with the development of their internal QA systems and teacher performance assessment systems (ANECA, 2022a[8]; ANECA, 2022b[9]). Other agencies, such as the Quality Assurance Agency (QAA) in the United Kingdom, regularly organise (online) training for institutional QA staff. Based on a review of trainings organised by several QA agencies, Greere (2022[5]). has developed a framework of potential topics to be considered in the design of QA training (see Table 3.1).
Limited involvement of institutional stakeholders
The third and related reason interviewees highlighted for the slow development of institutional quality cultures in Hungarian higher education is the challenge of developing institutional QA systems that are able to successfully engage actors across the entire institution in a process of continuous quality enhancement. International evidence shows that the development of institutional quality cultures requires both centralised guidance and decentralised implementation (Staring et al., 2022[10]). In other words, it requires institutions “to move from the existing control framework to a culture creation framework and integrate QA activities into their institutional cultures and everyday practices” (Jung, 2022, p. 12[11]). However, in institutions where QA is still developing or has only recently been introduced, “a centralised system may be the most effective when an institution first introduces the QA system” (Jung, 2022, p. 7[11]).
Efforts to steer institutional QA practice are being made in several Hungarian HEIs. For example, Eötvös Loránd University (ELTE), a large public university located in Budapest, adopted an institution-level Quality Manual in 2016 (Eötvös Loránd University, 2016[12]) and the institution-level Academic Regulations for Students also include some “provisions pertaining to certain faculties” (ELTE, 2022[13]). Based on the guidelines included in these documents, each faculty is responsible for formulating its own quality goals, have them approved by a Faculty Quality Council, and report annually on the actions taken to meet institution-level quality goals. Implementation of these central level QA guidelines however is still developing. For example, the Faculty of Education and Psychology’s QA website states: “although the university has created quality improvement documents, the development of a faculty quality improvement system requires more than a mechanical adoption of these documents. It requires shared thinking, shared goals, and joint commitment” (ELTE, 2022[13]). Similarly, at Budapest Metropolitan University (METU), a private institution, centralised coordination of QA processes is seen as key for the development of an institutional quality culture: “quality management processes are under continuous monitoring and control co-ordinated by the Strategic and Quality Management Directorate” (Budapest Metropolitan University, n.d.[14]). At the University of Debrecen (DE), a large foundation university with 14 Faculties, a Quality Manual was first developed in 2004. The eighth version of the manual states that it aims at “co-ordinating the operation of the university’s quality assurance system” (University of Debrecen, 2017, p. 7[15]).
There are significant differences between institutions in terms of how institutional QA is organised, and a lack of evidence as to whether a centralised or decentralised approach leads to better outcomes (Jung, 2022[11]), (EUA, 2022[16]). However, the institutional site visits and interviews carried out by the OECD review team as part of this project reveal that those HEIs with more developed QA systems in Hungary are typically organised as follows. This structure can provide a potential model for HEIs in Hungary that are either just starting or still in the process of developing their internal QA systems (see Figure 3.1):
Rector or Vice Rector. In Hungarian HEIs, teaching, learning and research matters typically fall under the responsibility of the Rector, and this includes the QA of teaching and learning. In many institutions, the responsibility for QA is delegated to the Vice Rector for Educational Affairs. Along with the Senate, the (Vice-)Rector is responsible for formulating quality goals at institutional level, along with drafting the institution’s development plan, in which the institution is required by law to outline its strategic goals and priorities for the next five years (OECD, 2021, p. 92[17]).
Institution and Faculty-Level Quality Assurance Office. Depending on the size of the institution, the (Vice-)Rector will appoint a Quality Assurance Officer, who is responsible for co-ordinating the activities of a dedicated Quality Assurance (QA) Office. The QA Office is typically responsible for formulating the institution’s rules and procedures for internal QA in line with the institutional development plan. It is also responsible for co-ordinating the internal and external monitoring and evaluation activities across the institution. Depending on the size of the institution, the QA Office will either play a more co-ordinating role (i.e. compiling and analysing data collected by faculty-level QA Offices) or a more active role (i.e. central data collection, for example through institution-wide surveys of students and staff). Most often, a combination of both is present in HEIs.
Institution and Faculty-Level Quality Assurance Committee. HEIs and faculties typically also have a Quality Assurance Committee, which at institution level is often chaired by the Head of the QA Office and includes student and senior staff members (e.g. Deans or Vice-Deans) involved in managing, supporting or monitoring the quality of teaching and learning at faculty level. In some cases, the Committee also includes representatives from the labour market. However, a recent OECD review on the labour market relevance and outcomes of doctoral education in Hungary (OECD, 2022[18]) shows that the inclusion of feedback from labour market stakeholders in the development of study programmes is not common. The QA Committee is typically responsible for reviewing and voting on the QA rules, procedures and reports prepared by the QA Office, and for advising the Senate and/or (Vice-)Rector on quality-related issues. In some institutions, the QA Office, QA Committee and Senate also review the quality and performance of study programmes (and instructors) on an annual basis, based on administrative data and stakeholder feedback.
Institution and Faculty-Level Support Centres. To support implementation and bridge the QA activities at institution, faculty and individual student/instructor level, some HEIs have established dedicated centres to support students, instructors and administrative support staff with specific quality issues (e.g. centres for digital teaching and learning). Other institutions have expanded the scope of the supports provided by existing centres to these specific issues (e.g. student union, student information centre, library, IT support centre, faculty administration). A smaller number of institutions has started pooling the supports provided by different centres into one dedicated centre for (digital) teaching and learning. Depending on the size of the institution, these support centres either operate as “middleware” organisations, providing supports across the institution, or at faculty level. Often, a combination of both types are present in institutions.
Source: Based on stakeholder interviews and institutional site visits, as well as a review of emerging quality standards, practices and supports for digital higher education in Staring et al. (2022[10]), “Digital Higher Education: Emerging Quality Standards, Practices and Supports”, OECD Education Working Papers, No. 281, OECD Publishing, Paris, https://www.oecd-ilibrary.org/education/digital-higher-education_f622f257-en.
Emerging practices for the quality management of digital higher education
This section describes trends in how HEIs in Hungary have responded to the challenge of managing the quality of their digital course offerings. It starts by describing how digitalisation is embedded in the strategy and investment plans of institutions. Next, it describes how institutions are supporting the implementation of quality practices, focusing specifically on the teaching and learning practices of instructors and students. Finally, it looks at how institutions are monitoring the performance of digital higher education.
Strategy and investments for the development of digital higher education
A major survey of 368 institutions from 48 countries in Europe, carried out by the European Universities Association (EUA) in 2020 (Gaebel et al., 2021[19]), found that 95% of HEIs saw digitalisation as a strategic priority over the next five years. In 51% of HEIs, digitally enhanced teaching and learning was already included in their internal QA systems, and in 41% this was under development. This represents a significant increase compared with 2014, when the figures were 29% and 35% respectively. In Hungary, too, several HEIs have included the expansion of their fully online and hybrid course offer as an explicit priority in their institutional development plans, as well as the investments in digital technology to strengthen the quality of pedagogical practices.
Integration of digitalisation in institutional vision, mission and strategy
Higher education stakeholders interviewed by the OECD review team explained that digitalisation is not a new issue in Hungary. Digitalisation has already been on HEIs’ agenda for several years, and they are increasingly aware of the many benefits it offers. The most frequently cited benefits are that digitalisation has the potential to support greater inclusion, sustainability, internationalisation, quality, flexibility, and openness. Tolnai (2021[20]) confirms that “during the pandemic, institutions lagging behind in digital development have, by necessity, significantly improved their digital services, which will lead to strong competition in the Hungarian higher education market for courses that exploit the potential of online space” (Tolnai, 2021, p. 173[20]). One higher education stakeholder interviewed by the OECD review team noted:
“The digital transformation is not a requirement that comes from inside [the institution] or the government. It is a driver that comes from society itself. It is difficult to be competitive in the European scene without up-to-date teaching methodologies, up-to-date digital infrastructure or without improved competences of teachers” (Higher education administrator, March 2022)
However, the consultations carried out by the OECD review team reveal that there are differences in the way in which higher education leadership are seeking to embed digitalisation across their course offers. At one end of the spectrum, there are those institutions that wish to fully embrace the opportunities offered by digitalisation and develop fully online and hybrid courses across their entire academic offer. At the other end, there are those which take a more reticent approach and would prefer to maintain an emphasis on place-based education (see Box 3.1). This renewed emphasis on place-based education is present in many OECD jurisdictions and is a “reaction to the pandemic and the far from ideal experience of emergency remote teaching” (Ó Caollaí, 2022[21]). It highlights the need to strengthen commitment and alignment at the level of institutional leadership, staff and students around the benefits and potential of digitalisation to support programme innovation, international collaboration, and to strengthen the quality of pedagogical practices in general in higher education.
Institutions fully embracing the opportunities offered by digitalisation
Interviews with stakeholders from Budapest Metropolitan University (METU) revealed that there is a desire among leadership to support the development of fully online and hybrid programmes across virtually the entire institution’s course offer. The main reasons cited are to attract more international students, and to provide students with greater flexibility and a high-quality learning experience. More specifically, the expansion of digitalisation is seen as a means to strengthen the implementation of the MyBRAND pedagogical model (Budapest Metropolitan University, n.d.[22]), which encourages students to approach their studies as a “portfolio-building exercise” to prepare for their future career. The pedagogical model is based on self-paced study, engagement with the labour market and personal learning projects in addition to the core curriculum. A partnership with Coursera is one of the strategies used to expand the institution’s online course offer (Budapest Metropolitan University, n.d.[23]).
The University of Szeged (SZTE) has developed a dedicated digital education strategy. Like METU, SZTE is seeking to expand its digital course offer to attract more international students and meet student demands for greater flexibility. More specifically, SZTE is actively exploring the further development of its hybrid course offer, in which students would only have to be physically present for some courses or semesters (e.g. practical training), while theoretical courses would be delivered primarily online. The university hopes that this could attract more Hungarian and international students. During the pandemic, the institution also set up a partnership with Coursera to provide students with free access to courses from the world’s leading universities and industry educators (University of Szeged, 2020[24]).
Institutions placing a renewed emphasis on place-based education
Interviews with higher education leadership at the University of Debrecen revealed a more reticent approach towards the development of digital higher education. One of the main reasons cited is the fact that in-person instruction and student life in the city of Debrecen are seen as key features of the student experience. This view is also based on the results emerging from two surveys carried out among students and staff during the COVID-19 pandemic, co-ordinated by the institution’s Directorate for Quality Policies and Developments (University of Debrecen, 2017[15]). The results from the first survey (carried out in spring 2020), showed that 63% of students and instructors wanted future study programmes to be delivered in hybrid format, 33% wanted to return to fully in-person courses and 4% favoured fully online instruction. A second survey (carried out in the autumn of 2020) showed a significant decrease for hybrid delivery (49%) and an increase of students and staff in favour of returning to traditional delivery (41%). Fully online instruction also increased to 10%.
Similarly, at Károli Gáspár University of the Reformed Church (KRE) and Tomori Pál College (TPF), there is a desire to return to on-campus education. In the case of KRE, interviewees underlined that, as a church-owned institution, the “humanistic values” of the institution required a continued commitment to in-person instruction. The view of leadership, however, seems to contrast with that of instructors and students, who are in favour of expanding the institution’s digital course offer and building on the lessons learned during the pandemic. At TPF, a small and relatively young private college (founded in 2004), interviewees explained that the main focus is on offering practical higher vocational education and training (VET), bachelor’s and postgraduate specialisation programmes for adults, which require on-campus instruction. Several students and instructors agreed with this view and highlighted many challenges related to online learning, particularly a lack of digital skills.
Source: Based on stakeholder interviews conducted as part of virtual site visits carried out by the OECD review team in March 2022.
While there are significant differences between HEIs in terms of the extent to which they envisage digitalising their course offerings, there is an almost universal commitment among HEIs to move towards e-administration (Tolnai, 2021[20]). The pandemic has pushed institutions to digitise virtually all administrative processes, which has highlighted benefits for internal and external collaboration with students and instructors, as well as attracting (and retaining) more international students. Higher education stakeholders indicated that international developments such as the European Commission’s Erasmus Without Papers initiative will drive all HEIs to move their administration online (European Commission, n.d.[25]). Box 3.2 provides details on a mobile application developed by the University of Debrecen (DE) in 2020, specifically designed to support students with the organisation and administration of their studies.
Available in Hungarian and English, the Studyversity mobile application developed by the University of Debrecen (DE) provides students with access to up-to-date information on the organisational and administrative aspects of their studies. Integrated with NEPTUN, it allows students to easily consult their calendar and courses, which the application can synchronise with their personal calendar. The application also offers a platform for initiating and completing certain administrative procedures and reminds students about major university events or scholarship opportunities.
Source: Adapted from DE (2020b[26]), Studyversity – University in your pocket, University of Debrecen (DE), Debrecen, https://mad-hatter.it.unideb.hu/promo/studyversity/en.html.
The integration of specific standards and indicators to support and monitor the implementation of institutional quality goals for digital higher education is, however, still developing in most Hungarian HEIs. For example, the latest version of the University of Debrecen’s (DE) Quality Manual (8th version) does not include any specific e-learning considerations (University of Debrecen, 2017[15]). Similarly, in the QA policy at the Eszterházy Károly Catholic University (EKKE) reference to digitalisation is only made at the organisational policy level in relation to a Centre for Distance Learning under the Vice Rector, responsible for faculty development for distance learning and teaching, training students in using the LMS, and developing pedagogical and accreditation support for distance learning programmes (Eszterházy Károly Catholic University, 2022[27]). At Gábor Dénes College (GDF), a private institution with longstanding experience in offering distance learning programmes, digital learning is fully embedded in the institution’s QA processes, including a definition of some broad implementation goals and indicators (see Box 3.3).
Gábor Dénes College (GDF) has a well-developed internal QA system for distance learning programmes that takes into account Hungarian higher education law, MAB guidelines and the ESG. The institution’s QA documentation includes clearly defined process descriptions for various aspects of its operations. As an example, the quality goals for 2020 included the following areas:
Distance learning is understood as an individual form of instruction where students are mostly studying from home. Students therefore require various supports, including:
Digitally available teaching materials that support self-directed learning
Access to the institution’s virtual learning environment or learning management system (VLE/LMS), which includes teaching materials, self-assessments, glossaries, animated and interactive content that makes learning engaging
Qualified tutors providing professional support in using the digital materials and resources.
GDF also has an online database of teaching materials and a prize for the best digital materials. The award is based on detailed process regulations for the QA of distance learning programmes and courses. The institution’s quality standards for digital materials are:
Sources: Adapted from GDF (2022a[28]), Minőségbiztosítás (Quality Assurance), Gábor Dénes College, http://gdf.hu/nyilvanos-adatok/minosegbiztositas/; GDF (2022b[29]), Távoktatás (Distance education), Gábor Dénes College, http://gdf.hu/felvetelizoknek/tavoktatas/.
Strong investments in digital education infrastructure
In addition to embedding digitalisation in the institution’s overall vision, mission and strategy, to date, HEIs in Hungary have focused primarily on strengthening their physical digital education infrastructure. In some cases, this digital transformation was already under way before the COVID-19 pandemic. Although the pandemic highlighted that some challenges remain – for example, connectivity issues for some students and institutions (OECD, 2021[17]) – HEIs’ digital infrastructure is overall quite well-developed. This is confirmed by the speed with which institutions and instructors were able to respond to the challenge of moving education entirely online during the COVID-19 pandemic (DSN/DHECC, 2020[30]). Some institutions visited by the OECD review team have invested in professional video recording equipment to support instructors to develop online courses. Students entering higher education also have good access to digital tools and internet connectivity. An OECD survey carried out as part of the project Supporting the Digital Transformation of Higher Education in Hungary (OECD, 2021[17]) confirmed that 93% of students have access to an adequate (or better) computer at home and have adequate internet access. There are however indications that disadvantaged groups such as Roma and students with disabilities, who are already under-represented in Hungarian higher education, may be at risk of further disadvantages due to the digitalisation of higher education (KIM, 2016[31]; KIM, 2021[32]).
HEIs in Hungary are free to choose which LMS/VLE they use for the organisation and management of teaching and learning activities. Many institutions use Moodle or Blackboard (both widely used systems internationally) or the Hungarian system CourseGarden (DSN/DHECC, 2021[33]). The delivery of online courses themselves, however, differs significantly between individual departments and instructors (e.g. the most used online course delivery tools are Microsoft Teams, Zoom or Google Meets). In addition to this, while private institutions are free to select their own student information system (SIS), public HEIs are required to use the NEPTUN system to collect and store student and course data (OECD, 2021[17]). Stakeholders interviewed by the OECD team, especially students, mentioned that the large variety of digital tools and systems used across institutions, departments and individual instructors means that they have to use multiple usernames and passwords to log in to different systems. This proliferation of accounts not only creates time management challenges, but it also increases cybersecurity risks for the institution. Stakeholders also noted that many instructors were insufficiently trained to effectively use digital technologies for pedagogical purposes, and that HEIs face challenges into linking their institutional software and platforms to central systems such as NEPTUN (Tolnai, 2021, p. 172[20]).
Supporting the quality enhancement of teaching and learning practices
Varying levels of quality in online instruction have refocused attention on previously documented concerns in national and international studies about the need to modernise pedagogical practices in Hungary (KIM, 2016[31]; KIM, 2021[32]). For example, one OECD survey (OECD, 2021[17]) shows that 45% of Hungarian students found the online learning offered as an emergency response during the COVID-19 pandemic to be less engaging than in-person instruction. Despite digital breakthroughs globally, improvements in digital pedagogy are lagging in Hungary (Eurydice/EACEA/EC, 2019[34]; Hülber, Papp-Danka and Dringó-Horváth, 2020[35]). Recent empirical studies on the competencies of Hungarian academics confirm that instructors’ digital and pedagogical skills are underdeveloped and considered to be less important by HEIs in Hungary (Kálmán, 2019[36]; Redecker and Punie, 2017[37]). The pandemic, however, has required all instructors to move their instruction online and experiment with digital tools. Likewise, the shift to online learning has required students to develop their digital and self-directed learning skills. One higher education student interviewed by the OECD review team said:
“Suddenly, due to the COVID-19 pandemic, there were expectations for teachers and students to improve how they were teaching and learning” (Higher education student, March 2022).
Emergence of supports for the professional development of academic staff
Some HEIs in Hungary have set up staff professional development centres to support the professional development of academic staff in their institutions. Table 3.2 shows that, in 2021, eight HEIs in Hungary had set up a staff professional development centre, representing only a small proportion of the total of 64 accredited HEIs in the country. However, stakeholder interviews carried out by the OECD review team revealed that this list is not up-to-date, and that more institutions are considering setting up such units (e.g. University of Debrecen). Other institutions either do not publish up to date public information on the activities of their teaching and learning centres, or the centres operate more at faculty level (e.g. Hungarian Dance Academy). Nevertheless, compared with other OECD and European Higher Education Are (EHEA) jurisdictions, the number of centres remains small. A recent EUA report found that institutions in 28 European countries are organising continuous professional development (CPD) for their teaching staff, typically through a teaching and learning centre (Zhang, 2022, p. 36[38]). The study found that in The Netherlands, for example, all universities have teaching and learning centres that offer basic and senior teaching qualifications, as well as leadership development. In some countries (e.g. Lithuania, The Netherlands, Norway, Sweden and Switzerland), the teaching enhancement offer is often shared between HEIs, to the benefit of smaller institutions that either do not have the resources to run such centres or cannot cover all their training needs independently.
The supports typically provided to instructors by these staff professional development centres include: the development of information guides and teaching materials, including YouTube videos and podcasts (e.g. the University of Pannonia information page on online teaching (University of Pannonia, 2020[39])); the organisation of training programmes; the creation and maintenance of informal support structures, such as individual counselling or peer learning groups; and the provision of prizes and awards. Most of these services focus on improving the digital skills and methods of teaching staff. In terms of governance, the centres usually sit under the responsibility of the Rector, Vice-Rector or Chancellor. In many cases they are also linked to a specific faculty or department with expertise on education and/or staff professional development. For example, at Eötvös Loránd University (ELTE), the Education Development and Talent Support Department is linked to the Faculty of Education and Psychology. At Károli Gáspár University of the Reformed Church (KRE), the ICT Research Centre has strong links to the Faculty of Humanities and Social Sciences.
An increasing number of HEIs in Hungary has also started to conduct performance assessments of instructors’ pedagogical skills and to include these in appraisal procedures. A recent survey conducted as part of a benchmarking study on the landscape of higher education teacher performance assessments (PROFFORMANCE, 2022[42]) found that 88% of HEIs in Hungary have a dedicated framework or process in place for the assessment and appraisal of academic staff. The study compared practices in six countries (Austria, Croatia, Czech Republic, Georgia, Hungary, and Serbia) and found that teaching, research and student feedback/learning outcomes were the three most common types of evidence included in performance assessments. Table 3.3 presents an overview of the priorities included in the appraisal procedures of HEIs in the six participating countries.
As in many other OECD systems, one of the main challenges faced by HEIs in Hungary is getting staff other than the “digital frontrunners” to engage in professional development (Tømte et al., 2019[44]; Staring et al., 2022[10]). As noted by Tolnai (2021[20]), “due to the isolated development, general digital developments covering the whole higher education or a specific field, level or type of education have not been implemented” (Tolnai, 2021, p. 173[20]), Many instructors remain hesitant about the benefits offered by digital technology, with interviews revealing that senior academics and staff teaching more practical disciplines have the greatest reservations, and that career structures prioritise research excellence. HEIs are, however, introducing several incentives for professional development (see Box 3.4).
Hungarian HEIs have introduced various incentives to support the engagement of academic staff in professional development activities:
Prizes and awards. Some HEIs have launched prizes and awards for the best online teaching materials, for example Gábor Dénes College (GDF) and Eötvös Loránd University (ELTE). The University of Nyiregyhaza (NYF) also publishes a yearly top ten of those teachers rated highest in student evaluations (Horváth, 2021[43]).
Mandatory training and skills assessment. Some institutions are introducing digital skills assessments or staff professional development as a mandatory requirement in recruitment and staff appraisal processes. For example, at Eötvös Loránd University (ELTE), some job advertisements (e.g. for Assistant Professor) explicitly ask applicants to demonstrate practical knowledge/experience of digital tools and platforms such as MS Office, MS Teams, Zoom, Outlook, Canvas and Moodle (Közigállás, 2022[45]). At Corvinus University (BCE), academics who receive a sub-standard performance evaluation are required to participate in a coaching programme with a teaching and learning expert (Horváth, 2021[43]).
Institutional platforms to support best practice sharing. Several instructors interviewed by the OECD review team said that they use digital platforms, such as the institutional LMS, MS Teams or Facebook, to create groups to store and exchange digital resources and methods. There is however a lack of coordination at faculty and institutional level to more widely disseminate the best practices shared in these informal discussion channels.
Source: Based on stakeholder interviews conducted as part of virtual site visits carried out by the OECD review team in March 2022.
The Hungarian Ministry of Culture and Innovation (KIM), in collaboration with Tempus Public Foundation, recently launched a higher education teacher performance self-assessment tool as part of the PROFFORMANCE project (PROFFORMANCE, 2022[42]). The tool was piloted in HEIs from six participating countries and is structured around three main dimensions and four horizontal dimensions, one of which is digitalisation. For each of these dimensions, sample questionnaires have been developed to support the self-assessment, peer review, student assessment and appraisal of staff’s pedagogical skills. The questionnaires focus on six thematic areas, representing the core tasks of academic staff: teaching and learning; curriculum design and development; teaching performance and student support; assessment; professional development; teaching-related research, innovation and social impact; and organisational and administrative tasks.
In 2020, experts from Károli Gáspár University of the Reformed Church, Budapest Business School, the University of Pécs and the Hungarian Dance Academy collaborated on the development of a handbook to promote and support the conscious use of digital tools among Hungarian higher education instructors. The handbook follows the six dimensions included in the EU’s DigCompEdu framework (Redecker and Punie, 2017[37]) and provides guidance on how each of these dimensions can be implemented in practice by instructors.
The chapters explore the topics of professional engagement (how to use digital technologies to promote communication, collaboration and professional development, and scientific visibility), digital resources (how to find, create and share digital resources effectively), teaching and learning (good practices and useful applications to support the effective use of digital technologies in teaching and learning), assessment (how to increase the effectiveness of assessment by using digital technologies or strategies), supporting learners (how to use digital tools to support inclusion, personalisation and student engagement), and the acquisition of digital competencies (how to help students use digital technologies creatively and responsibly to obtain information, communicate, create different types of content, and solve problems).
Source: Dringó-Horváth et al. (2020[46]), Az oktatásinformatika módszertana a felsőoktatásban (Educational Technology in Higher Education – Methodological Considerations), Károli Gáspár University of the Reformed Church, Budapest, https://btk.kre.hu/images/ikt/oktatasinformatika_a_felsooktatasban.pdf.
Finally, some HEIs in Hungary have taken the lead in organising annual conferences on the topic of digital learning to support inter-institutional collaboration and peer learning on digital higher education. For example, in 2020, the ICT Research Centre and the Centre for Continuing Education in Educational Informatics at Károli Gáspár University of the Reformed Church (KRE) launched an annual conference series on digitalisation in higher education. The first conference, in November 2020, focused on dialogue and co-operation for the identification and development of good practices in digital teaching and learning (Pintér, 2021[41]). The second conference, in October 2021, focused on the organisational, regulatory and infrastructural changes in Hungarian higher education that have occurred during the COVID-19 pandemic (Károli Gáspár University of the Reformed Church, 2021[47]). As a result of the inter-institutional collaboration on digital teaching and learning, experts from four HEIs in Hungary have developed a handbook to promote and support the conscious use of digital tools among Hungarian HEIs (see Box 3.5).
Increased focus on student support for digital learning
Many higher education stakeholders interviewed by the OECD review team mentioned that the COVID-19 pandemic had raised institutions’ awareness of the need to strengthen both their student services in general, and to prepare students specifically for digital learning. As noted by Tolnai (2021[20]), the emergence of digital higher education in Hungary has underlined the need to strengthen the “link between student needs and programme development” (Tolnai, 2021, p. 176.[20]).
In response to the COVID-19 pandemic, institutions in Hungary have implemented various practices to (better) prepare and support students for digital learning. First, several institutions have started offering students online consultation opportunities, which has significantly increased their accessibility. Next, both during and following the pandemic, many HEIs have strengthened their online presence and communication with students. Finally, several institutions have developed manuals and training courses to teach students “how to learn online” (see Box 3.6), with a particular focus on self-directed and autonomous learning skills. As Hungary’s higher education system is characterised by a high number of weekly student-teacher contact hours (see Chapter 2), as well as a primarily lecture-, knowledge- and teacher-based instructional model, stakeholders felt that these skills are particularly underdeveloped among students in Hungary.
In response to the emergency remote instruction during the COVID-19 pandemic, several institutions in Hungary have developed (online) courses – often in collaboration with Massive Open Online Course (MOOC) providers – to support the development of students’ digital and self-directed learning skills. Szeged University (SZTE), for example, collaborates with Coursera and international online learning experts to offer a MOOC on autonomous learning (Coursera, 2022[48]). The MOOC has over 8 000 enrolled students and is also being used by other institutions in Hungary Eötvös Loránd University (ELTE), for example, refers to the course on its info page for distance learning students and teachers (Eötvös Loránd University, 2022[49]). Likewise, at Budapest University of Technology and Engineering (BME), the Directorate for Student Services offers online courses to both first-year and more advanced students to prepare them for digital learning (Budapest University of Technology and Economics, 2022[50]). At the Budapest Business School (BBS), a specific remedial e-learning course has been created for mathematics (Budapest Business School, 2022[51]).
Source: Based on stakeholder interviews conducted as part of virtual site visits carried out by the OECD review team in March 2022.
Almost all higher education stakeholders interviewed by the OECD review team highlighted the urgent need to strengthen mental health support for students. Although several HEIs and instructors have started to provide some form of online (mental wellbeing) support, the number of consultations is usually limited. Szeged University (SZTE), for example, employs multiple full-time psychologists to provide individual and group sessions online as well as in person. However, the university only subsidises five therapy sessions per student (University of Szeged, 2022[52]). The University of Debrecen (DE) has a separate Mental Health Centre, which offers counselling to students and specific supports to students with special educational needs (University of Debrecen, 2022[53]). The university also has a student-mentoring programme managed by the Distance Education Learning Centre (Hungarian Insider, 2021[54]).
The COVID-19 pandemic has also driven some institutions to move their student feedback surveys online, and to pay greater attention in general to students’ (digital) learning experience in programme development and QA. As stated by one interviewee: “Learning about quality is best done through learners themselves”. However, this is not the case for all HEIs in Hungary. Many institutions still carry out paper-based feedback surveys. Digital education is not yet embedded as a regular topic in institution- and faculty-level data collection exercises, and it is much less common for HEIs to collect feedback from PhD students. Some interviewees also mentioned that student feedback surveys are rarely carried out more than once or twice per year, and that response rates are often low and insufficiently representative, especially in HEIs and courses with low student numbers where anonymity cannot always be guaranteed. Tolnai (2021, p. 175[20]) further notes that “respondents may be either only the unsatisfied or only the highly satisfied students”.
“Learning about quality is best done through learners” (Higher education stakeholder, February 2022)
At Szeged University (SZTE), first-year full-time student can have their skills assessed upon entry (University of Szeged, 2021[55]). Eötvös Loránd University (ELTE) conducts an end-of-first-year survey, end-of-course evaluations, occasional student and employee satisfaction surveys, as well as other more ad hoc thematic surveys (Eötvös Loránd University, 2022[56]). By contrast, in the spring semester of 2020 Károli Gáspár University of the Reformed Church (KRE) carried out weekly surveys to rapidly identify and respond to online learning issues faced by students. At Semmelweis University (SE), a QR code system has been developed to collect student feedback after each lecture (see Box 3.7).
In 2020, Semmelweis University introduced a QR code-based student feedback system in response to the high demand for immediate student feedback and educational development. The system allows instructors to gather immediate and anonymous student feedback at the end of each lecture to help them reflect on changes to be made for their next lesson. By scanning a QR code with their mobile phone at the end of lectures or practical seminars, the system asks students to answer a small number of fixed-response questions (nine questions for lectures, ten for practical seminars). Students also have the option not to answer questions or to expand on their answers. The system is run by the Centre for Educational Development, Methodology and Organisation and seeks to encourage a culture of continuous feedback and collaboration between students and teachers and support the overall quality enhancement of teaching and learning at the university.
Source: Adapted from Kiss (2022[57]), “QR code system helps student feedback on teaching at Semmelweis University”, Semmelweis News, https://semmelweis.hu/english/2022/01/qr-code-system-helps-student-feedback-on-teaching-at-semmelweis-university/.
Feedback and performance monitoring of digital higher education
As mentioned at the start of this section, QA is still seen by many institutions and instructors in Hungary as a compliance or “box-ticking exercise”, rather than an opportunity for critical and open self-reflection or dialogue to inform continuous quality enhancement. Higher education stakeholders also mentioned that, as Hungary currently does not have any ex post programme review procedures (see Chapter 2), HEIs and instructors have limited incentives to focus on the development of their internal programme review and monitoring procedures. Stakeholders flagged this as one of the main barriers to the further development of institutional quality management in Hungary. Moreover, at present “quality assurance measurement in higher education is mainly optimised for contact learning” (Tolnai, 2021, p. 176[20]).
Limited institutional self-assessment of digital higher education
In Hungarian higher education, there is a lack of comprehensive and institution-wide self-assessment and benchmarking exercises for digital learning that consider the institution’s entire digital learning ecosystem. Exceptions are the University of Debrecen (DE), which has carried out a self-assessment of its digital education infrastructure and human resources (University of Debrecen, 2020a[58]), and Károli Gáspár University of the Reformed Church (KRE), which has conducted a self-assessment of its digital readiness using the DigCompEdu framework (Dringó-Horváth et al., 2020[46]).
There are several reasons why only a few institutions to date have carried out comprehensive reviews of their digital practices at institution or programme level. The first reason is that the self-evaluations carried out by HEIs as part of the five-yearly institutional accreditation process are based on the ESG, which do not include an in-depth reflection of digital education (see Chapter 2). The second reason is the limited capacity and expertise of HEIs on how to conduct specific reviews of their digital capacity, especially in smaller HEIs. This is a common challenge among institutions in many OECD jurisdictions (Staring et al., 2022[10]). In some OECD jurisdictions, public authorities have (co-)funded the development of self-assessment toolkits and guidelines to support specific institutional, programmatic and course level reviews of digital education. In Germany, for example, the Leibniz Institute for Knowledge Media has developed a Digital Benchmarking Toolkit in collaboration with several German universities for application in the German context (Leibniz Institute for Knowledge Media, 2022[59]). In New Zealand, funding from Ako Aotearoa (via two major grants) and later the Tertiary Education Commission (one grant) has supported the development of the E-Learning Maturity Model, led by experts across New Zealand (Marshall, 2012[60]).
Limited variety and digitalisation of data collection tools and methods
The higher education stakeholder consultations carried out by the OECD review team also highlighted a need for institutions to diversify their methods of data collection and analysis to support more comprehensive and in-depth quality reviews. Stakeholders also noted the potential offered by digital technologies to strengthen data collection and analytical processes. One instructor said:
“Digital education can provide an evidence-rich and adaptable framework for quality development” (Higher education instructor, March 2022).
Student and staff satisfaction surveys are the approach most commonly used by HEIs to assess the quality of digital practices, with some institutions carrying out institution-wide surveys to obtain a more comprehensive view of the challenges facing students and teachers. For example, the 2020 and 2021 annual student surveys at Eötvös Loránd University (ELTE) were expanded to include a section on digital teaching and learning, while the end-of-semester course evaluations in NEPTUN were updated to include questions related to digital aspects of courses (Eötvös Loránd University, 2022[56]). National- and institution-level administrative data, while strong in Hungary, are not widely used by institutions as part of their internal QA systems for digital learning. One reason for this might be the limited amount of information related to digitalisation included in these datasets (OECD, 2021[17]). For example, the Higher Education Database and Information System (FIR) does not include any data on the delivery mode of study programmes (i.e. online, hybrid or in person/blended) (DSN/DHECC, 2021[33]).
Learning analytics data generated through the institutional LMS/VLE is also used by only a small number of HEIs for QA purposes (DSN/DHECC, 2020[30]). The Society for Research in Learning Analytics (SoLAR) defines learning analytics as "the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs" (SoLAR, n.d.[61]). Triangulated with survey and administrative data, learning analytics “can generate rich insights into student engagement in learning and can be used to support student success” (OECD, 2021, p. 13[17]). At Szeged University (SZTE) and Corvinus University (BCE), however, stakeholder interviews carried out by the OECD review team revealed that there are plans to increase the use of learning analytics data to track student performance in real time.
Stakeholders also underlined the importance of qualitative feedback to supplement survey, administrative and learning analytics data. This is confirmed by international research, which states that qualitative research methods can help institutions understand the “context and illuminate the ‘why’ behind patterns encountered in institutional assessment” (Sillat, Tammets and Laanpere, 2021, p. 11[62]). Finally, higher education stakeholders underlined the importance of finding better mechanisms to capture employer feedback on students’ labour market outcomes and performance. While most Hungarian HEIs participate in the national Graduate Career Tracking Survey (DPR), carried out by the Educational Authority (OH) (Educational Authority, 2020[63]), the inclusion of employer feedback in institutional QA systems is not common. At present, labour market feedback is primarily collected at faculty level and through informal feedback mechanisms. At Eötvös Loránd University (ELTE), for example, the institution-level QA guidelines recommend that faculties consult with employers on required and acquired learning outcomes (ELTE, 2022[13]), but stakeholders from the university interviewed by the OECD review team explained that the practice of regularly collecting feedback differs significantly from faculty to faculty. At Károli Gáspár University of the Reformed Church (KRE), employer feedback is primarily collected informally as part of study programmes that have a work-based learning component, such as teacher training programmes. Similarly, at the University of Debrecen, the way in which employer feedback is collected is “partly formal and partly informal by nature” (University of Debrecen, 2017, p. 61[15]).
Key barriers to the further development of institutional responsibility for the quality management and innovation of (digital) education
The COVID-19 pandemic has pushed institutions and instructors across Hungarian higher education to reflect on their internal quality management systems and pedagogical practices, with some institutions putting in place policies and practices to support and monitor the quality of digital teaching and learning specifically. For example, several institutions and faculties have established dedicated teaching and learning centres to support the professional development of academic staff, provided additional supports to students for online learning or collected feedback from students and instructors on the quality of fully online and hybrid courses. However, compared to other OECD systems, institutional quality cultures in general are still developing in Hungarian HEIs. Institutional policies and processes to support the professional development of instructors remain limited to date, as does the regular collection of data and feedback from students, instructors and employers on the quality of (digital) programmes, including through learning analytics data generated from the LMS/VLE.
Stakeholder consultations carried out by the OECD review team point to three key barriers for the further development of institutions’ responsibility for the quality management and innovation of their provision:
Accreditation procedures do not sufficiently incentivise institutional responsibility for quality
Ex ante accreditation procedures focus on compliance with input requirements rather than programme performance
Ex ante accreditation procedures are burdensome for HEIs and MAB, diverting attention and resource away from quality enhancement.
Accreditation procedures do not sufficiently incentivise institutional responsibility for quality
In recent years Hungary has introduced several reforms to its accreditation procedures for institutions, doctoral schools, and medical training to provide HEIs with greater incentives to take responsibility for the quality management of their educational offerings. More recently, legislation was passed that grants all accredited institutions the freedom to launch new master’s programmes in disciplines within which they are already offering bachelor’s programmes. The introduction of this self-accreditation status for HEIs in Hungary will be an important stimulus for the further development of institutional quality management in Hungarian higher education (see Chapter 2).
Despite all these reforms, stakeholders interviewed by the OECD review team said that quality cultures are still developing in many Hungary HEIs, for three main reasons: historical resistance to QA as an administratively burdensome “box-checking exercise” rather than an “enabling” process supporting quality enhancement; limited guidance and support offered by MAB to institutions to support the implementation of national quality standards in institutional contexts; and challenges facing HEIs to engage the wider stakeholder community across their institution in quality enhancement processes.
Ex ante accreditation procedures focus on compliance with input requirements rather than programme performance
Another key barrier to the development of institutional responsibility for quality management is the limited capacity of institutions to monitor and assess the performance and quality of their (digital) study programmes. One reason for this is that current programme accreditation procedures focus exclusively on ensuring compliance with a wide range of input requirements, and therefore do not incentivise institutions to pay attention to ensuring the quality of programme outputs. Once a new programme proposal has been successfully evaluated by MAB and formally included in the National Qualifications Register by the OH, there is no incentive or requirement for institutions or instructors to update programmes or courses in line with the latest international developments in their scientific field, innovate teaching and assessment practices or experiment with the various opportunities offered by digital technologies (such as descriptive or predictive learning analytics) to support greater student success and learning outcomes. This lack of an ex post programme review procedure was mentioned by HEIs as one of the main barriers to incentivising greater institutional responsibility for quality (see Chapter 2).
Ex ante accreditation procedures are administratively burdensome for HEIs and MAB, diverting attention and capacity from quality enhancement
The third key barrier mentioned by higher education stakeholders is the heavy cost, low success rate and high administrative burden associated with the formal quality assurance of higher education programmes (see Chapter 2). This has limited the capacity of both HEIs and MAB to focus on the quality enhancement of the (digital) education offer. Stakeholders interviewed by the OECD review team highlighted a desire for MAB to take a more proactive and supporting role in building the capacity of HEIs to develop their internal quality management policies and procedures through the organisation of more quality enhancement-oriented activities. However, MAB’s capacity to expand such activities remains limited, especially in relation to digital education.
Hungary has already taken several steps to devolve greater responsibility for the QA of higher education to institutions and strengthen MAB’s capacity to organise quality enhancement activities, and additional reforms are being planned to further support this process. However, several barriers remain – especially in relation to the current programme accreditation procedures – that are preventing institutions from taking greater responsibility for programme QA and MAB from taking greater responsibility in relation to quality enhancement. These barriers are also preventing institutions and instructors from fully experimenting with and exploiting the potential offered by digital technologies to innovate teaching and learning practices and improve student success and outcomes.
This section presents examples of international practice that Hungary could learn from, as well as three proposed policy recommendations. The main message for Hungary, as it seeks to implement these proposals, is to ensure a careful balance between processes that encourage institutional experimentation and innovation alongside the need for public accountability and transparency.
Grant self-accreditation status to institutions with demonstrated capacity to manage study programmes at a high level of quality
In several OECD jurisdictions, institutions with demonstrated capacity to manage their study programmes at a high level of quality are granted self-accreditation status and are not required to undergo programme accreditation. This is the case in England (the United Kingdom), for example, where all higher education providers are granted self-accreditation status upon successful initial registration with the Office for Students (OfS) as the designated quality body for English higher education. When a provider first registers with the OfS, they are assessed upon seven conditions1 (OfS, 2022a[64]). In Ireland, publicly funded providers have self-accreditation status and are allowed to independently launch new study programmes. Private and independent providers have to meet a number of sector specific guidelines if they wish to offer recognised qualifications (QQI, 2016[65]), in addition to the Core Statutory Quality Assurance Guidelines applicable to all providers (QQI, 2016b[66]). In Norway, HEIs are granted self-accreditation status based on their legal status and training profile. Universities are allowed to self-accredit study programmes at all levels. Specialised university institutions and accredited university colleges can self-accredit study programmes at bachelor’s level, as well as all levels in which they have been granted the right to award doctoral degrees. For all other master’s and PhD programmes, these institutions must apply for accreditation (NOKUT, 2022[67]).
In Australia, providers can apply to the Tertiary Education Quality Standards Agency (TEQSA) for two types of self-accrediting authority. Institutions can either be granted unlimited self-accrediting authority (i.e. the provider is allowed to self-accredit programmes in any level or field of education) or limited self-accrediting authority (i.e. the provider may self-accredit programmes in a specific set of levels and/or fields) (TEQSA, 2022[68]). The criteria applied by TEQSA for the evaluation of applications for self-accrediting authority are presented in Box 3.8. Institutions without self-accrediting authority must apply for new programme accreditation and renewal. However, for new undergraduate- (i.e. bachelor’s) and postgraduate- (i.e. master’s) level programmes, a simplified, or “short course assessment” is provided drawing together four units from existing accredited undergraduate/graduate programmes. All other programmes are required to meet the scope and evidence requirements described in a detailed assessment framework for the launch of new programmes (TEQSA, 2020[69]).
In Australia, higher education providers applying for self-accrediting authority are required to meet the criteria set out in section B2 of the Higher Education Standards Framework (Threshold Standards) 2021. If a provider wishes to apply for unlimited self-accrediting authority, it must demonstrate it has “mature and advanced processes for the design, delivery, accreditation, monitoring, institutional quality assurance, review and improvement of courses of study, and the maintenance of academic integrity across at least three (2 digit) fields of education” (Australian Government, 2021[70]).
Providers seeking limited self-accrediting authority must demonstrate:
A track record of consistent compliance with Part A of the Higher Education Standards (HES) Framework (Threshold Standards), including a five-year track record of compliance of the programme (or programmes) for which self-accrediting authority is sought
That there are no unresolved compliance matters or conditions outstanding from its most recent registration with TEQSA or a recognised registration or accreditation authority
Completion of at least one review and improvement cycle in relation to the study programme(s) for which self-accrediting authority is sought
Successful implementation of evidence-based improvements arising from reviews
The existence of course review and improvement activities that cover the programme(s) for which self-accrediting authority is sought
Course review and improvement activities as effective features of their operations across all courses of study.
Source: Adapted from Australian Government (2021[70]), Higher Education Standards Framework (Threshold Standards) 2021, Australian Government, Melbourne, https://www.legislation.gov.au/Details/F2022C00105.
Recommendation 3: Grant self-accreditation status to institutions with demonstrated capacity to manage study programmes at a high level of quality
As Hungary seeks to revise its existing accreditation procedures to enable greater institutional autonomy for quality, the OECD team advises that it give consideration to granting self-accreditation status to HEIs with a demonstrated capacity to manage study programmes at a high level of quality in line with the ESG (ENQA, 2015[1]) and national key performance indicators (see Recommendation 5). A small number of exceptions to programme self-accreditation could be established for study fields such as medical education, with a special process of external accreditation.
To ensure a streamlined process that is meaningful to HEIs, the granting of self-accreditation status should be embedded in a revised institutional accreditation process. The revised institutional review should ensure that HEIs have adequate processes in place to monitor and support the quality enhancement of study programmes in different fields, modes and levels of study. Depending on their performance, HEIs could be granted “unlimited” or “limited” self-accreditation status, as per the Australian model (see Table 3.4). HEIs without self-accreditation status would be required to undergo cyclical quality reviews of their programmes (see Recommendation 5); non-accredited HEIs would be required to undergo ex ante programme accreditation (see Recommendation 6). These exemptions could serve as a strong incentive for HEIs to put in place sound internal QA systems.
Introduce a performance and outcomes-based programme monitoring and review procedure
In international quality circles. there is widespread agreement that in addition to assuring the quality of inputs to higher education programmes, it is important to also ensure the quality of teaching, learning and assessment processes, as well as student outcomes (i.e. time-to-completion and drop-out rates, graduate employment rates) (ENQA, 2015[1]; CHEA, 2016[71]; OECD, 2018[72]; OECD, 2019[73]). In this context, the opinions of the main “beneficiaries” of higher education are becoming increasingly important in the assessment of the relative success or failure of institutions and their programmes. This includes employers, civil society and students (Braun et al., 2020[74]; Egloffstein and Ifenthaler, 2021[75]).
An increasing number of higher education systems across the OECD has therefore introduced a cyclical ex post programme review procedure, focused on the performance of study programmes against a limited set of national key performance indicators (KPIs) and quality standards. Higher education systems are also increasingly introducing monitoring practices to track the performance of higher education providers and programmes on an ongoing basis to inform more focused quality reviews.
In Denmark, all higher education programmes are subject to review by the Danish Accreditation Institution every six years. The review asks HEIs to provide written documentation on the programme and complete a self-assessment report, in which they are required to answer questions related to five criteria, which are also used for the ex ante approval of new study programme proposals (programme demand and relevance, knowledge base, goals for learning outcomes, organisation and completion, and international quality assurance and development). In addition to this, institutions are required to provide key figures on the programme’s outcomes: graduate employment rates, student completion and attrition rates, research publications, ratio of full-time and part-time academic staff, and ratio of students to full-time academic staff. The Application Guide states “if a key figure indicates that there could be problematic circumstances, this will initially be regarded as a sign of potential problems […] you [the institution] will be asked […] to explain which special circumstances you believe influence the key figures” (Danish Accreditation Institution, 2019, p. 9[76]). The self-assessment report and written documentation are prepared by the institution followed by an institutional site visit and accreditation report, which are conducted and prepared by an external review panel. Based on the report, the Accreditation Institution decides whether to grant a positive, conditional, or negative decision. A negative decision means that the programme will no longer be allowed to take new student enrolments, and will eventually have to shut down.
England (United Kingdom) uses a similar outcomes-based approach to assuring the quality of higher education providers and programmes. Once an HEI is registered, the OfS monitors, on an ongoing basis, whether it meets the initial registration conditions, adopting a risk-based approach rather than reviewing the quality of institutions and programmes on a cyclical basis. This means that the OfS only “monitor[s] a provider more closely where [they] have information that the quality or standards of its courses may be of concern” (OfS, 2022a[64]). Importantly, as part of its monitoring arrangements for Condition B3 (student outcomes), the OfS has set numerical thresholds for continuation, completion, and progression, which came into effect on 3 October 2022 and represent “the percentage of students achieving positive outcomes” (OfS, 2022, p. 6[77]). The numerical thresholds were set based on an analysis of overall sector performance (i.e. anonymised sector distributions for the indicator, the sector overall rate, and the median performance of providers in the sector) to identify a “starting point value” for each indicator.2 This was complemented by an analysis of the impact of student and course characteristics on continuation, completion, and progression rates to inform whether a downward adjustment to the sectoral starting point value is necessary for certain modes or levels of study (see Table 3.5). For example, a downward adjustment is proposed for part-time undergraduate programmes and programmes with a high proportion of students for which there is evidence that they are at a higher risk of underperformance (e.g. students aged 51 years old or above, students from a migrant background, students with a mental health condition or other impairment).
In addition to disaggregating performance in relation to specific indicators by time, subject, course type or student characteristics, when monitoring institutional performance against numerical thresholds, the OfS considers policy or contextual factors that might explain why a certain provider or programme is performing below a relevant numerical threshold before launching a more in-depth investigation into potential quality issues. This includes external factors that are beyond the provider’s control (e.g. COVID-19 pandemic or local issues), course or profession-specific attributes (e.g. courses designed to provide access to a particular profession that is not classified as managerial or professional in the way the indicator has been constructed) and actions already taken or planned by the institution to address underperformance (e.g. the institution has already decided to stop offering the course or has introduced actions to improve performance) (OfS, 2022[77]). Going forward, the OfS will decide each year which student outcome measures, modes, and levels of study to prioritise as part of its performance monitoring, to be able to identify providers and programmes with performance below a relevant threshold indicator in a more targeted way.
In New Zealand, there is a more focused approach to assuring the quality of higher education. Through a regular cycle of academic quality audits, the Academic Agency for New Zealand Universities (AQA) provides external QA for all New Zealand universities. Each academic audit is linked to a specific “Enhancement Theme”, i.e. “a topic in which universities collectively address an issue which is important to individual universities and of national significance” and around which Te Pokai Tara (Universities New Zealand) organises quality enhancement activities (Te Pokai Tara, 2022[78]). The current Enhancement Theme is “Access, outcomes and opportunities for Māori students and for Pasifika students”. Each university has been required (and supported) to develop specific objectives and actions to address this theme, and will be required to demonstrate progress against the Enhancement Theme as part of the Cycle 6 academic audit (2017-24) conducted by AQA (AQA, 2020[79]).
Recommendation 4: Introduce a performance and outcomes-based programme monitoring system, coupled with a targeted cyclical programme review procedure
A proposed recommendation for Hungary is to introduce a performance and outcomes-based programme monitoring system for all HEIs and programmes, based on a limited number of KPIs, differentiated by study level, mode and intensity. This could be complemented by a cyclical and targeted programme review procedure for those HEIs that have not obtained self-accreditation status, as well as those programmes from HEIs with self-accreditation status for which data indicates there may be a concern with quality.
The development of minimum thresholds for national KPIs as part of a sectoral performance monitoring system should be carried out in close consultation with HEIs and informed by a careful analysis of sector performance on each indicator, based on available data in national datasets for higher education (i.e., the national Higher Education Database and Information System, Felsőoktatási Információs Rendszer (FIR), and the national Graduate Career Tracking Survey (DPR)). Table 3.6 provides a grid that can be used by Hungary as a basis to develop numerical thresholds to monitor programme performance by study level, mode and intensity, building on the potential study formats presented in Recommendation 1. The proposed areas are based on data used for the development of institutional performance agreements as part of the model change process (see Chapter 2), provided to the OECD review team by KIM, for which trends and baselines can be accurately defined at national level. The advantage of developing national KPIs for higher education programmes is that MAB has an evidence base to monitor performance on an ongoing basis in between cyclical reviews of institutions and programmes and can carry out ad hoc reviews in cases where quality issues are observed. For institutions, national KPIs do not only provide clear targets and incentives to improve performance and implement QA processes, but they can also serve as a basis to inform evidence-based intra- and inter-institutional benchmarking and peer learning.
For those HEIs and programmes that have not obtained self-accreditation status (as well as programmes from institutions with self-accreditation status for which data indicates that there might be a concern with quality), MAB could consider introducing a cyclical and targeted quality review procedure. The WFME-based programme review procedure for medical training programmes (MAB, 2021[80]) could be used as a basis for the development of such a targeted and cyclical programme review procedure (in disciplinary clusters). The process consists of the preparation of a self-assessment report by the institution based on the WFME standards, followed by an institutional site visit and accreditation report, which are conducted and prepared by an external review team, co-ordinated by MAB. However, to manage the workload associated with these reviews, MAB should reflect carefully on the regularity and focus of programme reviews for different types of HEIs, programmes, and disciplines, possibly in disciplinary clusters.
Table 3.7 presents a potential model for a performance and outcomes-based programme monitoring and review system in Hungary.
Simplify ex ante programme accreditation procedures
Many QA agencies across the OECD and EHEA have simplified their ex ante programme launch requirements for HEIs with demonstrated capacity to manage programmes at a high level of quality, giving them greater independence and flexibility to establish innovative (digital) study programmes.
In Denmark, the Accreditation Act of 2013 stipulates that all new programmes must be approved by the Danish Accreditation Institution. Like Hungary, Denmark uses two stages for ex ante programme accreditation: prequalification, to assess the demand and relevance of the proposed new programme, and accreditation, to assess the educational content, learning outcomes, organisation and QA provisions of the programme. Providers are required to complete different stages depending on their accreditation status. HEIs that have been granted a positive institutional accreditation decision are only required to obtain prequalification to launch new programmes. Institutions with conditional accreditation status, or those who have not yet begun the institutional accreditation process, must obtain both prequalification and accreditation for new programmes. Institutions with negative accreditation status cannot establish new programmes.
Table 3.8 below provides an overview of the procedures and criteria for the launch of new higher education programmes in Denmark.
In Ireland, as mentioned earlier in this section, the ex ante accreditation of study programmes only applies to private providers. As well as having to comply with the Core Statutory Quality Assurance Guidelines (QQI, 2016b[66]) and the sector specific guidelines (QQI, 2016[65]), private providers are also required to meet four “prerequisites for programme validation […] Applications will not be accepted from providers who do not meet these four prerequisites” (QQI, 2017, p. 9[83]). As in Denmark, the programme validation criteria focus on the programme’s proposed educational content, learning outcomes, organisation, and QA provisions (see Table 3.9).
Recommendation 5: Increase institutional autonomy for the establishment of new programmes, depending on accreditation status
To give institutions and instructors increased autonomy and flexibility to develop innovative (and digital) study programmes, as well as free up MAB’s capacity to conduct cyclical quality reviews at programme level and support the quality enhancement of institutional quality management practices, Hungary could consider simplifying its ex ante programme accreditation procedures. Table 3.10 below presents a model of what a revised programme launch procedure in Hungary might look like, with progressive responsibility for institutions depending on their accreditation status.
Institutions with self-accreditation status would be allowed to establish new programmes directly with the OH, providing basic information such as the relevance and need for the new programme and the institution’s own account of the programme’s proposed educational content and learning outcomes (rather than conformity to a National Qualifications Register).
Accredited institutions without self-accreditation status would also be allowed to establish new programmes directly with the OH, except in the case of programmes launched in certain study fields, modes or levels within which the institution is not yet offering degree programmes. For these programmes, MAB would conduct a light, desk-based review of the institution’s proposed QA arrangements for the programme, prior to registering the programme with the OH. For example, if a university wanted to offer a master’s programme in a new discipline, the proposed programme would need to be reviewed by MAB.
Non-accredited institutions would require all new programme proposals to undergo an in-depth quality review by MAB prior to the programme being registered with the OH.
References
[8] ANECA (2022a), AUDIT, National Agency for Quality Assessment and Accreditation (ANECA), Madrid, http://www.aneca.es/eng/Evaluation-and-reports/Institutional-evaluation-procedure/AUDIT (accessed on 19 December 2022).
[9] ANECA (2022b), DOCENTIA - Teaching Performance Assessment, National Agency for Quality Assessment and Accreditation (ANECA), Madrid, http://www.aneca.es/eng/Evaluation-and-reports/Institutional-evaluation-procedure/DOCENTIA (accessed on 19 December 2022).
[79] AQA (2020), Guide to Cycle 6 Academic Audit Second Edition (V 1.1), Academic Quality Agency for New Zealand Universities (AQA), Wellington, https://www.aqa.ac.nz/sites/all/files/Guide%20to%20Cycle%206%20Academic%20Audit_June%2021%20reprint.pdf (accessed on 19 December 2022).
[70] Australian Government (2021), Higher Education Standards Framework (Threshold Standards) 2021, Australian Government, Melbourne, https://www.legislation.gov.au/Details/F2022C00105 (accessed on 19 December 2022).
[74] Braun, A. et al. (2020), Rethinking education in the digital age, European Parliament, Brussels, https://doi.org/10.2861/84330 (accessed on 19 December 2022).
[51] Budapest Business School (2022), Egyetemi felzárkóztató matematika e-learning képzés [University catch-up mathematics e-learning training], Budapest Business School, Budapest, https://uni-bge.hu/en/office-for-teaching-and-learning (accessed on 19 December 2022).
[23] Budapest Metropolitan University (n.d.), METU and Coursera partnership: study abroad with free access to leading university courses, Budapest Metropolitan University (METU), Budapest, https://www.metubudapest.hu/metu-coursera-free-courses (accessed on 19 December 2022).
[22] Budapest Metropolitan University (n.d.), MyBRAND, Budapest Metropolitan University (METU), Budapest, https://mybrand.metubudapest.hu/ (accessed on 19 December 2022).
[14] Budapest Metropolitan University (n.d.), Official documents and quality assurance, Budapest Metropolitan University (METU), Budapest, https://www.metubudapest.hu/official-documents (accessed on 19 December 2022).
[50] Budapest University of Technology and Economics (2022), BME Directorate of Student Services, https://hszi.bme.hu/en/ (accessed on 19 December 2022).
[71] CHEA (2016), The CIQG International Quality Principles: Toward a Shared Understanding of Quality, Center for Higher Education Accreditation (CHEA), https://www.chea.org/sites/default/files/pdf/Principles_Papers_Complete_web.pdf (accessed on 19 December 2022).
[48] Coursera (2022), A tanulás tanulása: Hatékony mentális eszközök, melyek segítenek megbirkózni a nehéz tantárgyakkal [Learning How to Learn: Effective mental tools to help you cope with difficult subjects (Learning How to Learn], Coursera, https://www.coursera.org/learn/a-tanulas-tanulasa#instructors (accessed on 19 December 2022).
[82] Danish Accreditation Institution (2022), Vejledning til prækvalifikation af nye uddannelser og nye uddannelsesudbud af videregående uddannelser [Guide to prequalification of new programmes and new offers of higher education], Danish Accreditation Institution, Kopenhagen, http://www.ufm.dk (accessed on 14 October 2022).
[76] Danish Accreditation Institution (2019), Guide to programme accreditation - New programmes and local provision of programmes, Danish Accreditation Institution, Kopenhagen, https://akkr.dk/wp-content/filer/akkr/Vejl-til-uddannelsesakkred-Nye-uddannelse-og-udbud-oktober-2019_eng.pdf (accessed on 14 October 2022).
[81] Danish Accreditation Institution (2022a), New programmes, Danish Accreditation Institution, Kopenhagen, https://akkr.dk/en/accreditation-in-denmark/new-programmes/ (accessed on 14 October 2022).
[4] DiMaggio, P. and W. Powell (1983), “The Iron Cage Revisited: Institutional Isomorphism and Collective Rationality in Organizational Fields”, American Sociological Review, Vol. 48/2, pp. 147-160, https://www.jstor.org/stable/2095101 (accessed on 19 December 2022).
[46] Dringó-Horváth, I. et al. (2020), Az oktatásinformatika módszertana a felsőoktatásban (Educational Technology in Higher Education – Methodological Considerations), Károli Gáspár Református Egyetem IKT Kutatóközpontja, Budapest, https://btk.kre.hu/images/ikt/oktatasinformatika_a_felsooktatasban.pdf (accessed on 13 July 2022).
[40] Dringó-Horváth, I., J. T. Nagy and A. Weber (2022), “Felsőoktatásban oktatók digitális kompetenciáinak fejlesztési lehetőségei [Measurement and complex development of digital competence of teachers in higher education]”, Educatio, Vol. 30/3, pp. 496-507, https://doi.org/10.1556/2063.30.2021.3.9 (accessed on 19 December 2022).
[33] DSN/DHECC (2021), An Analysis of Current Higher Education Data Collected in Hungary and the Value of This Data to Assess Digital Readiness and Digital Practices, Digital Success Nonprofit Ltd. (DSN)/Digital Higher Education Competence Centre (DHECC), Budapest, document provided to OECD for the project "Supporting the Digital Transformation of Higher Education in Hungary".
[30] DSN/DHECC (2020), Position Paper on Digitalisation of Hungarian Higher Education, Digital Success Nonprofit Ltd. (DSN)/Digital Higher Education Competence Centre (DHECC), Budapest, document provided to OECD for the project "Supporting the Digital Transformation of Higher Education in Hungary".
[63] Educational Authority (2020), Graduate Career Tracking System, https://www.diplomantul.hu/ (accessed on 19 December 2022).
[75] Egloffstein, M. and D. Ifenthaler (2021), “Tracing Digital Transformation in Educational Organizations”, Digital Transformation of Learning Organizations, pp. 41-57, https://doi.org/10.1007/978-3-030-55878-9_3 (accessed on 19 December 2022).
[13] ELTE (2022), Academic Regulations for Students (Eötvös Loránd University Organisational and Operational Regulations – Volume 2), ELTE, Budapest, https://www.elte.hu/en/regulations (accessed on 2 July 2022).
[1] ENQA (2015), Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG), European Association of Quality Assurance in Higher Education (ENQA), Brussels, http://www.enqa.eu/wp-content/uploads/2015/11/ESG_2015.pdf (accessed on 19 December 2022).
[49] Eötvös Loránd University (2022), Distance education, Eötvös Loránd University (ELTE), Budapest, https://www.elte.hu/en/content/distance-education.t.1266?m=442 (accessed on 19 December 2022).
[56] Eötvös Loránd University (2022), Minőségfejlesztés [Quality Development], Eötvös Loránd University (ELTE), Budapest, https://www.elte.hu/minosegbiztositas/minosegfejlesztes (accessed on 19 December 2022).
[12] Eötvös Loránd University (2016), Quality Improvement, Eötvös Loránd University (ELTE), Budapest, https://www.elte.hu/minosegbiztositas/minosegfejlesztes (accessed on 19 December 2022).
[27] Eszterházy Károly Catholic University (2022), Minőségbiztosítás [Quality Assurance], Eszterházy Károly Catholic University (EKKE), Eger, https://uni-eszterhazy.hu/egyetem/m/egyetem/minosegbiztositas (accessed on 19 December 2022).
[16] EUA (2022), Learning and teaching thematic peer groups: Strategy and organisational culture group report, European Universities Association (EUA), Brussels, https://www.eua.eu/resources/publications/1010:strategy-and-organisational-culture-group-report.html (accessed on 19 December 2022).
[25] European Commission (n.d.), Erasmus Without Paper, https://erasmus-plus.ec.europa.eu/european-student-card-initiative/ewp (accessed on 19 December 2022).
[34] Eurydice/EACEA/EC (2019), Digital Education at School in Europe, Publications Office of the European Union, Luxembourg, https://op.europa.eu/en/publication-detail/-/publication/d7834ad0-ddac-11e9-9c4e-01aa75ed71a1/language-en.
[28] Gábor Dénes College (2022a), Minőségbiztosítás [Quality Assurance], http://gdf.hu/nyilvanos-adatok/minosegbiztositas/ (accessed on 19 December 2022).
[29] Gábor Dénes College (2022b), Távoktatás [Distance education], http://gdf.hu/felvetelizoknek/tavoktatas/ (accessed on 19 December 2022).
[19] Gaebel, M. et al. (2021), Digitally enhanced learning and teaching in European higher education institutions, European University Association (EUA), Brussels, https://eua.eu/resources/publications/954:digitally-enhanced-learning-and-teaching-in-european-higher-education-institutions.html (accessed on 19 December 2021).
[3] Government of Hungary (2011), Act CCIV of 2011 on National Higher Education, Government of Hungary, Budapest, https://net.jogtar.hu/jogszabaly?docid=A1100204.TV (accessed on 19 December 2022).
[5] Greere, A. (2022), “Training for quality assurance in higher education: practical insights for effective design and successful delivery”, Quality in Higher Education, pp. 1-27, https://doi.org/10.1080/13538322.2021.2020978.
[43] Horváth, L. (2021), The landscape of higher education teachers’ performance. Final report on the results of the benchmarking exercise., Tempus Public Foundation, Budapest, https://tka.hu/docs/palyazatok/proff_kiadv_final_op.pdf (accessed on 19 December 2022).
[35] Hülber, L., A. Papp-Danka and I. Dringó-Horváth (2020), “A felsőoktatás digitális oktatási kultúrájának korrajza szakértői interjúk alapján [Era of digital education culture in higher education based on expert interviews]”, Opus et Educatio, Vol. 7, pp. 302-330, https://doi.org/10.3311/ope.401. (accessed on 19 December 2022).
[54] Hungarian Insider (2021), “The University of Debrecen has a hypermodern “Learning Center””, Hungarian Insider, https://hungarianinsider.com/the-university-of-debrecen-has-a-hypermodern-learning-center-8808/ (accessed on 19 December 2022).
[2] INQAAHE (2018), Guidelines of Good Practice, International Network for Quality Assurance Agencies in Higher Education (INQAAHE), Barcelona, https://www.inqaahe.org/guidelines-good-practice (accessed on 19 December 2022).
[11] Jung, I. (2022), “Quality Assurance in Online, Open, and Distance Education”, in Handbook of Open, Distance and Digital Education, Springer Singapore, Singapore, https://doi.org/10.1007/978-981-19-0351-9_39-1.
[36] Kálmán, O. (2019), “Oktatói kompetenciák – ahogyan azt a hazai oktatók látják [Teacher competences - as perceived by national teachers]”, Pedagógusképzés 1-2, pp. 14-42, https://doi.org/10.37205/TEL-hun.2019.1-2.01 (accessed on 19 December 2022).
[47] Károli Gáspár University of the Reformed Church (2021), Oktatásinformatika a felsőoktatásban II. – A digitális oktatás átmentett értékei 2021. október 22. (Educational Technology in Higher Education II - The Transferable Values of Digital Education 22 October 2021.), Károli Gáspár University of the Reformed Church (KRE), Budapest, https://btk.kre.hu/konf/oktinfkonf_felsooktatas/2021/fooldal (accessed on 19 December 2022).
[32] KIM (2021), Implementation of the Shifting of Gears in Higher Education Mid-Term Policy Strategy 2016 Action Plan 2016-2020, Hungarian Ministry of Culture and Innovation (KIM), Budapest, document provided to OECD for the project "Supporting the Digital Transformation of Higher Education in Hungary".
[31] KIM (2016), Shifting of Gears in Higher Education Mid-Term Policy Strategy 2016: Action Plan 2016-2020, Hungarian Ministry of Culture and Innovation (KIM), Budapest, https://2015-2019.kormany.hu/download/9/19/d1000/Hungarian%20Higher%20Education%20Mid-Term%20Policy%20Strategy%20-%20Action%20Plan%202016-2020.pdf (accessed on 19 December 2022).
[57] Kiss, V. (2022), “QR code system helps student feedback on teaching at Semmelweis University”, Semmelweis News, https://semmelweis.hu/english/2022/01/qr-code-system-helps-student-feedback-on-teaching-at-semmelweis-university/ (accessed on 19 December 2022).
[45] Közigállás (2022), Pályázat egyetemi adjunktus munkakör betöltésére [Job advertisment for assistant professors], https://kozigallas.gov.hu/pages/jobviewer.aspx?ID=id7g4wjxj9 (accessed on 19 December 2022).
[59] Leibniz Institute for Knowledge Media (2022), HFD Strategie-Benchmark - Home, https://benchmark.hfd.digital/ (accessed on 19 December 2022).
[80] MAB (2021), Az orvosképzés akkreditációs eljárásainak dokumentumai - Értékelő lap [Documents on accreditation procedures for medical training - Evaluation sheet], Hungarian Accreditation Committee (MAB), Budapest, https://www.mab.hu/eljarasok/ (accessed on 19 December 2022).
[60] Marshall, S. (2012), “E-learning and Higher Education: Understanding and Supporting Organisational Change in New Zealand”, Journal of Open, Flexible and Distance Learning, Vol. 16/1, pp. 141-155, https://files.eric.ed.gov/fulltext/EJ1079905.pdf (accessed on 19 December 2022).
[7] National Commission for Further and Higher Education Malta (2017b), National Quality Assurance Framework Standards Amended to also cover Online/Blended Learning, Malta Further and Higher Education Authority (MFHE), Valletta, https://mfhea.mt/wp-content/uploads/2021/07/Annex-1-NQAF-Standards-with-amendments-to-cover-online-and-blended-learning.pdf (accessed on 19 December 2022).
[6] National Commission for Further and Higher Education Malta (2017a), Step by Step Guide to Internal Quality Assurance, Malta Further and Higher Education Authority (MFHE), Valletta, https://mfhea.mt/wp-content/uploads/2021/03/Step-by-step-guide-to-Internal-Quality-Assurance%E2%80%8B.pdf (accessed on 19 December 2022).
[67] NOKUT (2022), Self-accreditation, Norwegian Agency for Quality Assurance in Education (NOKUT), Oslo, https://www.nokut.no/en/quality-enhancement/self-accreditation/ (accessed on 19 December 2022).
[21] Ó Caollaí, É. (2022), “Will online education become the ‘next new normal?’”, The Irish Times, https://www.irishtimes.com/ireland/education/2022/06/28/will-online-education-be-the-next-new-normal/ (accessed on 19 December 2022).
[18] OECD (2022), “Enhancing labour market relevance and outcomes of doctoral education: Country note Hungary”, in OECD Education Policy Perspectives, OECD Publishing, Paris, https://www.oecd.org/publications/enhancing-labour-market-relevance-and-outcomes-of-doctoral-education-country-note-hungary-e3a2aafc-en.htm (accessed on 19 December 2022).
[17] OECD (2021), Supporting the Digital Transformation of Higher Education in Hungary, Higher Education, OECD Publishing, Paris, https://doi.org/10.1787/d30ab43f-en (accessed on 19 December 2022).
[73] OECD (2019), The Future of Mexican Higher Education: Promoting Quality and Equity, Reviews of National Policies for Education, OECD Publishing, Paris, https://doi.org/10.1787/9789264309371-en (accessed on 19 December 2022).
[72] OECD (2018), Rethinking Quality Assurance for Higher Education in Brazil, Reviews of National Policies for Education, OECD Publishing, Paris, https://doi.org/10.1787/9789264309050-en (accessed on 19 December 2022).
[77] OfS (2022), Setting numerical thresholds for condition B3, Office for Students, https://www.officeforstudents.org.uk/media/1206417b-9b11-402c-9706-d88c080b58fc/setting-minimum-numerical-thresholds-for-condition-b3.pdf (accessed on 11 October 2022).
[64] OfS (2022a), How we regulate quality and standards, Office for Students (OfS), https://www.officeforstudents.org.uk/advice-and-guidance/quality-and-standards/how-we-regulate-quality-and-standards/ (accessed on 19 December 2022).
[84] OfS (2022b), Regulatory advice 20 - Regulating student outcomes, Office for Students, https://www.officeforstudents.org.uk/media/cc83740f-7252-4484-81b8-45471fee0b4f/regulatory-advice-20-regulating-student-outcomes-revised.pdf (accessed on 19 December 2022).
[41] Pintér, T. (2021), “Oktatásinformatikai helyzetkép a magyarországi felsőoktatásban [State of play of educational technology in higher education in Hungary]”, Új Pedagógiai Szemle (New Pedagogical Review), Vol. 71/3-4, pp. 54-74, https://upszonline.hu/index.php?article=710304009 (accessed on 19 December 2022).
[42] PROFFORMANCE (2022), PROFFORMANCE - Assessment Tool and Incentive Systems for Developing Higher Education Teachers’ Performance, https://profformance.eu/.
[83] QQI (2017), Policies and criteria for the validation of programmes of education and training, Quality and Qualifications Ireland (QQI), Dublin, https://www.qqi.ie/sites/default/files/2021-11/qp-17-policies-and-criteria-for-the-validation-of-programmes-of-education-and-training.pdf (accessed on 13 October 2022).
[65] QQI (2016), Statutory Quality Assurance Guidelines developed by QQI for Independent/Private Providers coming to QQI on a Voluntary Basis, Quality and Qualifications Ireland (QQI), Dublin, https://www.qqi.ie/sites/default/files/2021-11/qg-2-sector-specific-qa-guidelines-for-private-and-independent-providers.pdf (accessed on 14 October 2022).
[66] QQI (2016b), Statutory Quality Assurance Guidelines developed by QQI for use by all Providers, Quality and Qualifications Ireland (QQI), Dublin, https://www.qqi.ie/sites/default/files/2021-11/qg-1-core-statutory-quality-assurance-guidelines.pdf (accessed on 14 October 2022).
[37] Redecker, C. and Y. Punie (2017), European Framework for the Digital Competence of Educators. DigCompEdu., Publications Office of the European Union, Luxembourg, http://publications.jrc.ec.europa.eu/repository/bitstream/JRC107466/pdf_digcomedu_a4_final.pdf (accessed on 19 December 2022).
[62] Sillat, L., K. Tammets and M. Laanpere (2021), “Digital competence assessment methods in higher education: A systematic literature review”, Education Sciences, Vol. 11/8, https://doi.org/10.3390/EDUCSCI11080402 (accessed on 19 December 2022).
[61] SoLAR (n.d.), What is Learning Analytics?, https://www.solaresearch.org/about/what-is-learning-analytics/ (accessed on 19 December 2022).
[10] Staring, F. et al. (2022), “Digital Higher Education: Emerging Standards, Practices, and Supports”, OECD Education Working Papers , No. 281, OECD Publishing, Paris, p. 97, https://www.oecd-ilibrary.org/education/digital-higher-education_f622f257-en (accessed on 19 December 2022).
[78] Te Pokai Tara (2022), New Zealand Universities Enhancement Theme, https://www.enhanceunz.com/ (accessed on 17 October 2022).
[68] TEQSA (2022), Application Guide for Self-Accrediting Authority, Tertiary Education Quality Standards Agency (TEQSA), https://www.teqsa.gov.au/sites/default/files/saa-application-guide_2022-_v0.6.pdf?v=1663547701 (accessed on 19 December 2022).
[69] TEQSA (2020), Scope and evidence requirements for an application for course accreditation of Undergraduate and Graduate Certificates Scope Evidence requirements, Tertiary Education Quality Standards Agency (TEQSA), https://www.teqsa.gov.au/sites/default/files/scope-evidence-requirements-course-accreditation-undergraduate-graduate-certificates.pdf?v=1588669468 (accessed on 19 December 2022).
[20] Tolnai, Á. (2021), Quality assurance issues of blended learning courses, J. Selye University, Komárno, Slovakia, pp. 171-177, https://doi.org/10.36007/4133.2022.171 (accessed on 19 December 2022).
[44] Tømte, C. et al. (2019), “Digitalisation in higher education: mapping institutional approaches for teaching and learning”, Quality in Higher Education, Vol. 25/1, pp. 98-114, https://doi.org/10.1080/13538322.2019.1603611 (accessed on 19 December 2022).
[53] University of Debrecen (2022), Mental Health Centre, https://demek.unideb.hu/en (accessed on 19 December 2022).
[15] University of Debrecen (2017), Quality assurance manual, University of Debrecen (DE), Debrecen, https://www.edu.unideb.hu/tartalom/downloads/Quality%20Book%202018-0116%203.20%20UD.pdf (accessed on 19 December 2022).
[58] University of Debrecen (2020a), Online felmérés az online oktatásról [Online research about online education], University of Debrecen (DE), Debrecen, https://btk.unideb.hu/hu/online-felmeres-az-online-oktatasrol (accessed on 19 December 2022).
[26] University of Debrecen (2020b), Studyversity - University in your pocket, University of Debrecen (DE), Debrecen, https://mad-hatter.it.unideb.hu/promo/studyversity/en.html (accessed on 19 December 2022).
[39] University of Pannonia (2020), COVID, University of Pannonia (PE), Veszprém, https://uni-pannon.hu/covid (accessed on 19 December 2022).
[52] University of Szeged (2022), University of Szeged - Foreign Language Programs - Psychological Counseling for medical Students, Albert Szent-Györgyi Medical School Foreign Students’ Secretariat, http://www.med.u-szeged.hu/fs/psychological-counseling/psychological-counseling (accessed on 19 December 2022).
[55] University of Szeged (2021), Bemeneti kompetenciamérés elsőéves hallgatóknak [Measurement of input competences for first-year students], https://arts.u-szeged.hu/sztehirek/2021-szeptember/bemeneti?objectParentFolderId=24891 (accessed on 19 December 2022).
[24] University of Szeged (2020), “University of Szeged and Coursera Partner to Support Students with Online Learning During COVID-19”, University of Szeged News, https://u-szeged.hu/news-and-events/2020/university-of-szeged-and (accessed on 19 December 2022).
[38] Zhang, T. (2022), National Developments in Learning and Teaching in Europe: A report from the Leadership and Organisation for Teaching and Learning, European University Association asbl, Brussels, https://eua.eu/downloads/publications/lotus%20report_2022_fin2.pdf (accessed on 19 December 2022).
Notes
← 1. Access and participation for students from all backgrounds (Condition A); Quality, reliable standards and positive outcomes for all students (Condition B); Protecting the interests of all students (Condition C); Financial sustainability (Condition D); Good governance (Condition E); Information for students (Condition F); and Accountability for fees and funding (Condition G).
← 2. A starting point value refers to “a judgement about the point at which we consider there be to minimal risk that a provider is not delivering positive outcomes” (OfS, 2022b, p. 11[84]).