2. Measuring innovation through surveys: Main considerations and applications to education
This chapter presents the research underpinnings for the survey chapters that follow. It starts with exploring existing research on measuring innovation that is relevant to the education sector, in the form of experimental surveys in the public sector and Oslo Manual guidelines. After, the chapter evaluates 13 relevant surveys on innovation or organisational change in the education sector, covering the four most relevant surveys in-depth. The last sections discuss survey methods to ensure the surveys obtain the representative data or data valuable for self-reflection and conclude.
Technological change from new technologies such as digitalisation, genomics and artificial intelligence offer substantial economic, social, and environmental benefits as well as significant challenges to equality, governance, and social inclusion. Obtaining the benefits from technological change, while minimising the costs, requires current and future citizens to acquire skillsets that enable them to actively participate in and benefit from a changing world. For many, this could require continually learning new skills and adapting to the need to switch jobs multiple times throughout their careers. These skillsets include a variety of digital skills, but also, importantly, the “4Cs” of creativity, critical thinking, communication and collaboration.
The education sector plays a prominent role in assisting both adults and children to acquire, strengthen, and maintain digital and 4C skills. Moreover, governments are encouraging education systems to adopt digital applications for school and system management, in-class teaching, and home learning. To ensure that these changes are implemented successfully, the education sector also needs to re-skill and up-skill teachers, which requires continuing investments in the teaching profession. The goal is to increase the productivity of teachers, the efficiency and pleasure of learning for students, and the ability of students to enter the workforce after graduation. These initiatives for both students and teachers are expected to improve learning outcomes and assist societies to benefit from the digital economy.
Improving the productivity of teachers and the efficiency and pleasure of learning for students requires innovations in the administration of education and the methods of teaching and engaging students in learning. A common concern is that innovation is complex and challenging for governments (Mulgan and Albury, 2003[1]; Potts and Kastelle, 2010[2]). Innovation is new, unknown, and can entail risks, whereas governments have a statutory duty, democratic responsibility, and political mandate to deliver public services in consistent and equitable ways. Managing the tension between government duties and innovation can be difficult if the risk of innovating appears far greater than the risk of maintaining the status quo. Nor does innovation sit well with the control function of hierarchies which, while they ensure stewardship and accountability over the use of resources, can discourage risk-taking (OECD, 2017[3]). Nevertheless, research using representative surveys has found that the incidence of innovation in the public sector often exceeds that of the private sector (European Commission, 2013[4]; APSC, 2011[5]). This is partly because public sector organisations are larger than many businesses and have the resources to invest in innovation through purchasing new technologies. In addition, research has found that public sector managers are capable of innovating within risk averse environments (Kay and Goldspink, 2012[6]; Torugsa and Arundel, 2017[7]).
Over the last few decades, digital technologies have been a major driver of innovation in the education sector, but many other factors also play important roles. For example, alterations in funding or personnel, the changing needs of students or parents, or system shocks (such as the COVID-19 pandemic) can cause or require innovation. The private sector is also an important source of innovations with applications in education. A recent publication at the OECD has assessed how new technologies could assist pedagogical or organisational conditions in education (OECD, 2021[8]).
It is entirely possible, however, that specific administrative or pedagogical innovations can fail or only succeed for some types of students or have unintended consequences. Furthermore, the staff of educational institutions can lack the necessary know-how to design innovations or adapt good practices to their own circumstances. Measuring innovation processes, the extent and variety of innovations, and the outcomes of innovation, can provide governments and education practitioners such as teachers with useful knowledge on how to improve their innovation capabilities and outcomes. Relevant information on processes includes whether the organisation’s management has cultivated an environment conducive to innovation, for instance through incentives for staff to participate in innovation and support for collaboration and communication within the organisation and with external partners. Information on outcomes include the effects of an innovation on different users, non-users, and on other processes and services.
The measurement of innovation in education is a necessary complement to work focused on cultivating 21st century skills, improving teaching and learning, and the use of technology in the education sector. Schools frequently innovate through introducing new processes and services. Ensuring that school leaders have timely information about these new initiatives is a key part of the feedback mechanism to improve the ability to innovate and innovation outcomes. Moreover, asking educators questions about the innovation activities of their own organisation encourages reflection on what might be necessary to achieve better outcomes.
However, there is little focus in the education sector on measuring innovation processes and identifying factors that can improve outcomes. Conversely, both governments and businesses measure innovation in the private sector. Governments in Europe and in many OECD countries outside Europe conduct innovation surveys to obtain statistically representative data on innovation expenditures, activities, and outcomes. The act of completing innovation survey questionnaires can also inform managers about the value of specific activities such as collaboration to innovation (Gault, 2018[9]). This aspect of surveys provides a ‘self-reflective’ function, whereby managers are required to think about the range of innovation activities within their organisation, which could also encourage them to think about how to improve these activities.
Previous OECD efforts to measure innovation in education have focused on using existing surveys to highlight possible proxies about the intensity of innovation in education. The first attempt explored two approaches: using surveys of tertiary-educated professionals that included some questions inspired by innovation surveys and using existing international surveys of educational practices to measure how they have changed over time (OECD, 2014[10]). The second attempt systematised and improved the second approach and inferred innovation from the intensity of change that could be observed in the “most important” school practices at the primary and secondary levels – while casting light on what had actually changed in terms of pedagogical and institutional practices (OECD, 2019[11]). Measuring innovation with data that were not specifically collected for that purpose comes with limitations. Should institutions or jurisdictions be able to implement surveys on innovation in education, this would be the first best option.
In order to inform model questionnaires of how to measure important dimensions of the innovation process and outcomes in education, at the level of educational institutions such as schools or jurisdictions, this chapter presents and evaluates existing research on innovation (surveys), including relevant questionnaire surveys in education. The next two sections provide an overview of previous experience with measuring innovation that is relevant to the education sector, notably commonly accepted definitions as well as factors and dimensions of innovation. In addition to recent experimental surveys of innovation in the public sector, those sections draw on the Oslo Manual guidelines for measuring innovation (OECD/Eurostat, 2018[12]). The next section evaluates 13 relevant questionnaire surveys on innovation or organisational change in the education sector, plus one survey of the private sector. Along with expert reviews, the material covered in this chapter has been used to develop several model questionnaires for measuring innovation in primary, secondary and tertiary educational establishments (chapters 3, 4, 5). The four most relevant surveys are discussed in-depth. A section discusses survey methods to ensure that the questionnaires obtain representative data or data of value to self-reflection before the conclusions.
Measuring innovation is primarily about measuring processes (OECD/Eurostat, 2018[12]) instead of measuring innovation outcomes, although innovation surveys can provide limited data for some types of outcomes. Nevertheless, some processes and practices are more likely to lead to better outcomes, for instance collaborating with external sources of expertise or conducting pilot tests. Collecting this data can help governments and educators to determine if best practices are in use and if improvements are needed. In addition, innovation data can be linked to external outcome data of interest, such as student academic performance or satisfaction with their learning environment, to determine if there is a significant positive or negative relationship between innovation and outcomes. However, while this can be useful as an indicator of the possible success or failure of an innovation, a comparison of different types of interventions on outcomes requires a different methodology based on policy evaluation methods that are generally unsuited to an innovation survey (OECD/Eurostat, 2018, pp. 229-237[12]). This is particularly important for pedagogical innovations or evaluating social inclusion or equity effects, where self-selection bias is likely.
Definitions
Definitions are obtained from the 4th edition of the Oslo Manual (OECD/Eurostat, 2018[12]), which includes universal definitions that apply to all sectors, including education provided by governments or businesses.
The term “innovation” can refer to the process of developing an innovation or an output, such as a specific type of innovation. To avoid confusion, this paper uses the term “innovation activities” to refer to innovation as a process, and the term “innovation” for outputs. The term “outcome” is used to refer to the effects of innovations on the innovating organisation itself (as with process innovations) or on the users of innovations (as with the users of educational services).
Innovation activities include all developmental, financial, and commercial activities undertaken by an organisation to create an innovation. The organisation of innovation activities can vary for each innovation and between institutions. An innovation can be developed through dedicated projects with an allotted budget, through ad-hoc “back of the desk” activities without a dedicated budget, or as part of regular operations to continuously improve processes or services.
The Oslo Manual’s general definition of an innovation for all types of organisations (units), including educational establishments is as follows: “An innovation is a new or improved product or process (or combination thereof) that differs significantly from the unit’s previous products or processes and that has been made available to potential users (product) or brought into use by the unit (process)”. A product includes goods and services.
The minimum requirement for an innovation is that it must have one or more characteristics that are significantly different from the process or product that the organisation previously offered or used. The requirement for significantly different characteristics applies to product and business process innovations that an organisation develops itself and innovations that were developed by other organisations, such as a business or a different educational establishment, with little or no additional modification. One implication of the definition is that a significant difference is from the organisation’s perspective. A teaching method could have been in widespread use by other educational establishments, but it is an innovation for a school that never used the method before.
Innovation is not necessarily an improvement over existing processes, goods, or services. It is possible for an innovation to make matters worse, for instance a new teaching method could reduce student performance or make learning less pleasurable. Some of the problems caused by innovation are due to conflicting goals, for instance an innovation that successfully reduces costs could have detrimental effects on learning, or back-office innovations could increase instead of decrease the workload of educators.
Definitions of products and processes are provided by the UN’s System of National Accounts (SNA) and are as follows1:
Goods are physical or virtual objects that can be transferred from one owner to another.
Services change the psychological conditions of users and are consumed at the time of their production. Changes in the psychological condition of a person include the acquisition of education, information, advice, entertainment, or experiences. Services can be delivered through a physical interaction or digitally.
Processes are all activities, under the control of an institutional unit, that use inputs of labour, capital, goods, and services to produce outputs of goods and services.
In education, which is a service, process and service innovations are likely to be more common than goods innovations.
Educational organisations (e.g. schools, universities, training centres, education publishers) contribute to product innovation when they introduce new or significantly different products and services, such as new syllabi, textbooks or educational resources, or new pedagogies or educational experiences (for example e-learning or new qualifications). They contribute to process innovation when they change significantly their organisational processes for producing their educational goods or services. For example, they may change how teachers work together, how they group students and manage other aspects of their learning experience; they may collaborate with other entities, use new marketing and external relations methods, new forms of communication with students and parents, etc. In the case of services such as education, products and processes may also be difficult to tell apart. For example, an innovation in education delivery (a service) can also require a new delivery method via tablet computers (a good) and use new software to automatically track attendance and grades (a process) (OECD, 2014[10]; OECD, 2019[11]; Halász, 2018[13]; Halász and Ágnes, 2021[14]).
Another defining characteristic of an innovation is that it is implemented within a defined period of time, defined as the observation period. The Oslo Manual recommends that the observation period should be no shorter than one year and no longer than three years. Consequently, a new teaching method that was implemented four years before the start of the observation period would not be defined as an innovation.
Object and subject-based approaches
Three main methods are used to measure innovation: an object-based approach, a subject-based approach, and a hybrid approach that combines these two methods. The object method collects data on specific innovations (the object), while the subject method collects general data on the innovation activities of an organisation (the subject) (OECD/Eurostat, 2018, pp. 205-212[12]). Instead of using survey questionnaires, object-based methods usually identify innovations of interest through case studies, newspaper reports, innovation awards, etc. and are therefore not discussed further. Subject-based methods usually collect data via questionnaire surveys, although it is possible to collect data from websites or other sources. The hybrid method commonly collects data through a questionnaire survey that is divided into two main sections. The first ‘subject’ section uses general questions to collect data on all the organisation’s innovation activities, while a second ‘object’ section asks questions about a single, focal innovation. Respondents are asked to think about a single innovation, provide a brief written description, and answer all subsequent questions with this innovation in mind. The purpose of the object-based questions is primarily to collect data for analytical and research purposes, whereas subject-based questions are used both for this purpose and to produce indicators for benchmarking.
The inclusion of the object method within a subject-based innovation survey has two notable advantages. First, respondents can provide more accurate responses for a single innovation, particularly for expenditures in personnel time or currency units, outcomes, and some inputs such as the source of the idea for the innovation. Second, it provides a direct link between innovation activities and outcomes.
Education can be provided by public, private, and non-profit schools or universities, plus the education sector more broadly includes private businesses that produce educational materials such as textbooks and software and administrative and other process innovations are often sourced from or developed in collaboration with private businesses. Consequently, some of the extensive literature on innovation in the private sector is relevant to how innovation occurs in education. However, this report draws on research on innovation in the public sector, where available, for two reasons. First, all education providers are usually regulated by government and therefore face similar requirements for student performance. Second, the sector is dominated by public or non-profit providers of education, with the latter often receiving substantial financial contributions from government. The result is that the education sector primarily functions as a public-sector provider of services, where market incentives for innovation play only a minor role (Bloch and Bugge, 2013[15]; Gault, 2012[16]).
The study of innovation within government and the public sector more broadly has attracted a growing body of empirical research, motivated in part by the increasing demand for benchmarking the efficiency and quality of public services as well as identifying the factors that contribute to desirable innovation outputs and outcomes. Several surveys have adapted the Oslo Manual guidelines for measuring innovation in the private sector to the public-sector context (APSC, 2011[5]; Arundel and Huber, 2013[17]; Bloch and Bugge, 2013[15]; OECD, 2015[18]), and recent innovation surveys have added questions that are explicitly designed for the Government sector (European Commission, 2013[4]). This shift was driven by the need to collect data to support public sector innovation policy (Arundel, Bloch and Ferguson, 2019[19]). In addition, several surveys have focused specifically on the education sector (see below, including Table 2.1).
Case studies and interviews have also been widely used to examine innovation in education, health, and social care services (Windrum and Koch, 2008[20]; Osborne and Brown, 2013[21]). This research is relevant to measurement because it identifies innovation activities and barriers that differ from those covered by the Oslo Manual for the private sector.
Environments conducive to innovation
The key foundations for innovation in the public sector can be summarised in four areas that government policies need to address to strengthen the abilities of public-sector organisations to innovate. Within each of these areas, there are policies and practices that can foster innovation.
Pro-innovation culture: Leadership needs to motivate and empower staff to explore new ideas and experiment with new approaches to their work and ensure that their staff have the knowledge and capabilities to develop, implement and evaluate innovations.
Knowledge and capabilities: Knowledge is essential to innovation and can be obtained from multiple sources within and external to the organisation. The challenge is to build the capacity to collect, identify, and apply knowledge to improve decisions about innovative solutions and to evaluate outcomes.
Innovation management: This includes the development of teams to guide innovation activities and work in partnerships across organisations and even sectors. Good management is also required to address obstacles and manage risks.
Resources and drivers: Innovation requires resources, both financial and time, and goals that can drive the innovation process.
Pro-innovation culture
New processes and services are generated by civil servants, political leaders, service users and members of the broader community, and the efforts of various professionals and stakeholders at different stages of the innovation process ensure they are developed and brought to scale. Civil servants and public employees play a key role at every stage, which means that the way in which the innovation process is managed is fundamental to successful innovation in public organisations. Public-sector staff are instrumental to implementing reforms as well as putting forward innovative ideas and contributing to their development at every stage of the innovation process (OECD, 2015[22]). Ideas for innovations often stem from middle managers or front-line staff, according to research on public-sector innovation (Borins, 2014[23]), which could support the conclusion that, “increasingly, innovation is as much a ‘bottom-up’ and ‘sideways-in’ process as a ‘top-down’ process” (Hartley, 2006[24]). Eurofound (Eurofound, 2012[25]), moreover, posits that employee-driven innovation “depends strongly on employees contributing their knowledge, expertise, creativity and commitment to the process”.
The challenge for management, therefore, is to harness the potential for creative problem-solving of their workforce, which should enable more employee-driven innovation. To this effect, managers and leaders should invest in fostering a ‘pro-innovation culture’ that begets the conditions needed to support employees to innovate, champion and lead employee-driven innovations, and rally the necessary resources.
Building a pro-innovation culture in a public-sector organisation requires a governance structure that permits managers to make decisions on innovation and encourage staff, at all levels, to participate in innovation activities. The trend in governance since the 1980s has moved towards giving managers greater discretionary power within the boundaries set by the political arm of government. For instance, New Public Management gave senior managers decision making power over efficiency improvements, but this has changed over time to a networked government structure that encourages a broader range of innovation activities (Torfing, 2019[26]; Crosby, ‘t Hart and Torfing, 2017[27]). However, governance structures can vary considerably within governments and even within government agencies.
A pro-innovation culture is defined as the behaviours and practices, shared by staff, that support innovation (OECD, 2017[3]). Relevant behaviours include open-mindedness, willingness to change, diversity of profiles, collaboration, and learning from failure. Relevant practices include involving employees in innovation decisions, providing staff with sufficient time and resources for innovation activities, training employees in innovation methods, including design thinking, co-creation, and pilot testing (European Commission, 2013[4]; Christiansen and Bunt, 2012[28]); recognising innovators through awards or incentives, and evaluating innovation outcomes.
Working with the diversity of an organisation’s workforce can help to create a pro-innovation culture. A more diverse workforce could affect innovation activities by means of communication and interaction among employees, and such forms of knowledge exchange can both be stimulated and hampered by diversity (Østergaard, Timmermans and Kristinsson, 2011[29]). Employee diversity could comprise of dimensions like age, gender, nationality, and sociocultural background.
Ensuring that employees can take the time and effort to innovate and are recognised for their work via awards or promotion is a key part of creating an innovative workplace culture. Financial incentives, however, are at risk of providing perverse effects and system gaming (Amabile, 1997[30]).
Employee motivation and empowerment
Many of the behaviours to support innovation require employee motivation. While ability determines what the workforce is capable of, motivation determines what the workforce will try to do when given the opportunity. Motivation can at times even make up for an initial skill deficiency, because highly motivated employees will invest more effort into acquiring such necessary skills (Amabile, 1997[30]). Ex ante motivation levels of employees are heterogeneous, yet this motivation can also be fostered or deteriorated by the organisational environment (Mumford, 2000[31]). Various studies have corroborated that motivated employee engagement is significantly positively correlated with organisational outcomes, such as performance and innovation (OECD, 2017[3]).
Motivation can be intrinsic to the person or created by external factors. Intrinsic motivation occurs when a person obtains pleasure from an activity or its completion, or when an activity meets internal values and standards, such as community service or ethical fairness (Frey and Osterloh, 2002[32]). With extrinsic motivation, people act because of a financial or other reward that is separate from the act itself. Intrinsic motivation is much more important in fostering creativity and innovation than extrinsic motivation. Conversely, extrinsic motivation, particularly when stemming from rewards for short-term performance, may result in a narrower definition or view of the task and thus cause employees to steer clear of more innovative approaches to such tasks (Amabile, 1997[30]; Fernandez and Moldogaziev, 2012[33]).
A pro-innovation culture also needs to give employees the discretionary power or opportunity to innovate, which requires some level of autonomy (Shalley and Gilson, 2004[34]). Employees need a level of freedom in how they plan their time and approach their tasks in order to address their tasks creatively. Clear goals and expectations can frame this autonomy to provide some structure which can benefit both managers (to regulate action), and employees (to structure their time and work).
Employees, in short, require the ability, motivation and the opportunity to do well at their jobs (Boxall and Purcell, 2011[35]). A pro-innovation culture addresses the abilities and motivation of employees to innovate and provides them with the opportunities they need to put their abilities and motivation to work.
Resources and drivers
Adequate resources are essential for innovation. Sufficient time is critical, as developing an innovation requires an iterative process of evaluating alternatives, testing them, learning from the test results, and revising the innovation as necessary.
Central budget agencies often fund major innovations in response to policy initiatives, but innovations developed from staff ideas often receive little or no dedicated funding. For these, the main input is staff time. Centrally funded innovations are often given targets and goals, as well as resources for performance management and evaluation. Other resources such as skills and expertise are discussed below.
Drivers for public sector innovation include government policies and legislation (Borins, 2014[23]), for instance to digitise many government services or to improve service outcomes, Businesses and citizens can also create demand for services or regulations to improve well-being (address different forms of pollution such as noise, airborne particulates, etc.) (Agolla and Van Lill, 2016[36]).
Political and demand drivers are frequently external factors over which public sector managers have little or no influence. Conversely, some aspects of innovation demand are created internally, within the organisation. These include staff demands for process innovations that improve working conditions or process efficiency and many “bottom-up” innovations that result from interactions between service users and the front-line staff of government organisations (Andersen and Jakobsen, 2018[37]; Simmons and Brennan, 2017[38]).
Drivers are often closely linked to innovation objectives, such as plans to digitise a specified percentage of government services within a defined time period, or an educational policy to improve student outcomes.
Innovation management
Innovation management covers all activities to initiate, develop, and implement an innovation, including systems for recognising innovation opportunities, managing their development, and managing obstacles. These tasks require dynamic managerial capabilities to foresee and react effectively to internal and external challenges (Helfat and Martin, 2014[39]; Helfat, C. et al., 2007[40]).
The management of innovation activities affects the type of innovations that are undertaken, the extent to which employees are involved in and invested in innovation, and how easily knowledge about the innovation is diffused across and beyond the organisation. Good innovation management can also require assigning responsibility for an innovation or constructing a team of staff to develop an innovation. Another role for management is monitoring outcome performance.
Use of innovation teams
Innovation teams of employees with different skill sets can be created to run complex innovation projects. Their use has a significant positive effect on innovation outcomes (Arundel, Casali and Hollanders, 2015[41]) and can manage tensions due to continuing business-as-usual work at the same time as experimenting and introducing new approaches. Innovation teams can bring together different innovation methods and skills and facilitate collaboration with external partners.
Obstacles to innovation
There are two types of obstacles to innovation. The first is common in organisations that do not innovate, that experience an obstacle as a barrier that prevents innovation from occurring. The second creates problems for innovation, but the organisation is able to solve the problem or work around it, although this can be costly in terms of time and funds or require the organisation to downgrade its goals for an innovation.
The types of innovation obstacles that affect the public sector have been examined in multiple studies, including questionnaire surveys (Cinar, Trott and Simms, 2019[42]). In a survey of over 3 000 European public sector managers from innovation organisations, the most frequently reported ‘high importance’ obstacles were insufficient human or financial resource, cited by 55% of managers, followed by legal requirements (38%), lack of management support (29%) and staff resistance (21%) (Arundel, Casali and Hollanders, 2015[41]).
Internal regulations or staff resistance are common obstacles (de Vries, Bekkers and Tummers, 2014[43]; Osborne and Brown, 2013[21]), that can also act as barriers to innovation. Public sector improvements through innovation can also be hampered by bureaucratic obstacles, which no longer successfully serve the purpose for which they were designed – according to several reports (European Commission, 2013[4]; Lunn, 2014[44]; OECD, 2015[22]). By applying such outdated or redundant rules rigidly and having inadequate will to change existing processes and services, the production of public value through innovation could be stifled.
Research from the Netherlands has highlighted that conservative interpretation of laws and regulations by civil servants may cause a barrier rather than the rules and regulations themselves (Kruiter et al., 2008[45]; Cels, de Jong and Nauta, 2012[46]). This could stem from a lack of imagination or factors hindering civil servants in taking initiative for more liberal interpretations. For example, the culture at the organisation may undervalue innovation or the manner in which accountability for failure is structured may discourage employees to take risks.
Several methods can reduce bureaucratic obstacles or barriers to innovation: engage stakeholders who are or will be subject to regulations to help identify solutions to hurdles at an early stage in the innovation process (OECD, 2015[22]); obtain the assistance of central innovation units who are familiar with managing regulations; and, in the longer term, use behavioural insights, either from a dedicated innovation unit or separate consultations, to embed experimentation into policy design (Lunn, 2014[44]; OECD, 2015[47]).
Risks
The risk of failure due to technical or organisational causes can create reputational or political damage and consequently senior civil servants can be reluctant to engage in ambitious innovation projects. Osborne and Brown (2011[48]) argue that this could cause innovation projects to focus on minor improvements, instead of making major changes that can offer substantially greater benefits. It may also be difficult to evaluate the payoffs from taking risks, since there is a lack of comparable quantitative metrics for outcomes (Townsend, 2013[49]).
One of the pillars to an innovative public sector is to challenge perceptions of risk (Mulgan, 2009[50]). Such perceptions are found to be responsive to change by senior management taking responsibility for failure (Potts and Kastelle, 2010[2]; Townsend, 2013[49]) to alleviate employees of such concerns and feel safe to experiment and push the envelope with riskier approaches. Other means to change perceptions are to install reward structures that are linked to the potential benefits of innovation, recognise staff successes, improve the protection of staff when innovation efforts fail, build narratives around successful risk taking, and train staff to manage risk rather than avoid it. Managers can also minimise risk through obtaining the assistance of external experts, for instance through collaboration and co-creation with users, and developing innovations carefully (Torugsa and Arundel, 2017[7]; Kay and Goldspink, 2012[6]).
Knowledge and skills
Theories of innovation such as Kline and Rosenberg’s (1986[51]) chain-link model and innovation systems theory (Freeman, 1987[52]; Lundvall, 1992[53]; Nelson, 1993[54]; OECD, 1997[55]) posit that innovation is not a linear activity, but an iterative process that draws on multiple knowledge inputs to solve a problem. Knowledge and skills can be acquired through learning from outside sources, including through collaborating on the development of an innovation, or by activities within the organisation, such as design thinking, co-creation with users, experimentation, and pilot-testing. The goal is to create value, either for the organisation itself, as with processes, or for the users of services.
External sources of knowledge and skills
Knowledge of relevance to innovation is generated, distributed, and used by multiple actors, such as firms, universities, public research institutions, public sector organisations, customers as users of goods and service innovations, and other individuals. The innovation activities of both firms and public sector organisations rely on external sources of knowledge (Chesbrough, 2003[56]; Dahlander and Gann, 2010[57]; Demircioglu and Audretsch, 2020[58]; Sørensen and Torfing, 2011[59]). Information can also be exchanged, but unless it is understood and processed to become knowledge it is not useful.
Knowledge of value for innovation activities can be obtained from external sources in many ways. It can be purchased, for instance by hiring the services of consultants or buying new technology; or acquired at little cost from reading reports or articles that describe innovations, attending conferences, or through contacts with the staff of other organisations. Employees can also be seconded to work in an academic institution or other organisation as part of a collaboration project.
Collaboration on innovation is a defining feature of how public sector organisations innovate, with up to 80% of European public agencies reporting its use (Arundel, Casali and Hollanders, 2015[41]; Bugge, Mortensen and Bloch, 2011[60]). The most frequently reported collaboration partners are other government organisations. Collaboration can reduce risks by drawing on the expertise or experience of collaboration partners with similar innovations. In this respect, collaboration can support the diffusion of good practices.
An innovative public sector organisation can also share its expertise with valuable innovations with other public sector organisations, leading to the ongoing diffusion of good ideas. The sharing of knowledge among different public sector organisations can be supported through innovation-oriented networks that connect multiple organisations that provide different types of services, for instance education and health. Such networks can support multidisciplinary perspectives, pool knowledge of different parts of an innovation process (e.g. co-creation, pilot testing and post-implementation evaluation), and integrate potential spillovers. Nodal organisations in these networks, such as the OECD Observatory of Public Sector Innovation, can share evidence, case studies, and examples of good practice that buttress the adoption and diffusion of innovation and innovation methods within the public sector as a whole (Bellefontaine, 2012[61]; Carstensen, Bason and Vibeke, 2012[62]; Puttick, Baeck and Colligan, 2014[63]; Torjman, 2012[64]). Mobility programmes can diffuse knowledge through the movement of skilled employees across different divisions, directorates, or government ministries. As a result, innovative activities in one workplace can be transferred to another, and fresh perspectives can be integrated into innovation teams on a frequent basis.
Internal knowledge and skills
Theoretically, public sector organisations could contract out all work required to develop an innovation. In practice, this is neither efficient nor good practice. Public sector staff need to fully understand the problem before drawing up an innovation contract for an external provider, estimate costs, timelines, and unexpected contingencies; and evaluate the outcomes. All of these tasks require skills that are used for innovation, such as design thinking and co-creation with users to identify problems and solutions, and the ability to evaluate outcomes. For activities such as problem identification and co-creation of services with users, front-line staff in a public sector organisation are likely to possess considerably more relevant knowledge than external consultants. Public sector organisations can benefit from drawing on external expertise, but this is more effective if combined with internal expertise, particularly in areas where the public sector staff has hands-on knowledge.
Relevant internal knowledge is likely to be held by multiple people within a public sector organisation and consequently support for co-operation and mutual learning within the organisation is necessary. This is one of the functions of an innovation team. Several other methods can be used to support communication between different functional areas within the organisation, including the joint development of innovation strategies across functional areas, exchanging innovation ideas openly across the organisation, regular meetings of heads of functional areas to discuss innovation issue, and temporary involvement in innovation projects of personnel from different functional areas.
Public sector staff do not always need specialised skills to generate new ideas for improving work processes or services, but several types of skills are valuable for developing an idea into an innovation. OECD research identifies three types of skills for innovation (OECD, 2011[65]; Vincent-Lancrin et al., 2019[66]):
Technical subject-specific skills include procedural and content knowledge associated with the type of innovation, such as computer skills for developing administrative processes using software, or expertise in learning and teaching methods for educational innovations.
Thinking and creativity skills include the ability to ask the right questions and develop creative solutions and approaches to solve problems. This includes the ability to look across seemingly disparate data, cases, problems, and processes to identify common threads and connect the dots. Imagination and curiosity are drivers.
Behavioural and social skills include the ability to work in partnerships, communicate, negotiate, network, and collaborate within and across organisational boundaries.
The novelty or improved characteristics of an innovation are often due to the use of new or modified technology, particularly digital technologies that support the provision of online services and back-office automation. The ability of public sector organisations to take advantage of these technologies depends on its own technological capabilities combined with its expertise in sourcing technical assistance from external sources.
A central skill set for innovation by public sector organisations concerns the nuts and bolts of how to develop an idea into an innovation. Two methodologies, developed in the private sector, can be highlighted: design thinking and co-creation. Design thinking is an iterative methodology that spans the innovation process from identifying the characteristics of a problem, developing possible solutions, producing prototypes, and conducting pilot tests of prototypes (McGann, Blomkamp and Lewis, 2018[67]). Co-creation obtains the input of the potential users of the innovation. The users of processes are civil servants, while the users of many service innovations are citizens or residents (Alves, 2013[68]; Christiansen and Bunt, 2012[28]). User input can be obtained non-interactively, for instance through surveys, or interactively, with users included in brainstorming sessions, focus groups, or one-on-one conversations with service designers (Osborne et al., 2021[69]). Public sector organisations should have access to design thinking and co-creation expertise in-house, or access to external sources of these skills, such as a government or private sector Living Lab, Innovation Lab, or Service Design Centre.
Research from the OECD and its Observatory of Public Sector Innovation finds that data and information are building blocks for innovation. Their free flow within and across public sector organisations is an important condition for building individual and organisational capacity to innovate (OECD, 2015[22]) and is essential for generating new ideas. Public sector organisation can have a lot of data of relevance to innovation, but the ability to extract value from data requires appropriate analytic capacities (OECD, 2013[70]; Ubaldi, 2013[71]) to extract useful knowledge from vast amounts of information (Speier, Valacich and Vessey, 1999[72]).
Assessment of outcomes
A necessary skill to obtain good quality outcomes from an innovation is the capability to assess the innovation after implementation. For many government innovations, outcomes either need to be specific to the type of innovation, for instance student learning outcomes after the introduction of a new teaching method, or general outcomes that need to be measured. General outcomes rely on subjective, self-reported measures, such as an increase in efficiency or improved user satisfaction or user access to information (Bloch and Bugge, 2013[15]). These types of outcomes can be collected through online or other types of surveys of the users of service innovations. Of note, assessment provides information on whether an innovation is underperforming, meeting or exceeding expectations and identifies problems that require fixing. It is not equivalent to an evaluation that compares the efficacy of different methods of providing a service, such as different teaching methods.
Summary of the innovation process in the public sector
Innovation often occurs through several stages, although one or more of these stages could be skipped, depending on the innovation. The following descriptions outline the innovation process for substantial innovations that contain a high level of novelty. Each of these stages is more likely to succeed given a pro-innovation culture, sufficient resources, competent innovation management, and appropriate knowledge and skills.
Identifying problems. Understanding the nature and characteristics of a problem is a first step towards triggering innovative ideas to respond to it. Public sector organisations often lack the capacity to identify risks and opportunities coming from their environment and to effectively capture and interpret demand from the users of their services.
Generating ideas. Ideas that fuel innovation can be generated from the bottom up by civil servants in the front-line or initiated by executive leadership. Supporting the creation of ideas often involves incentives and rewards, creating opportunities to share experiences, and support for mobility so that civil servants can gain a broad understanding of issues and the tools to respond to them. For many public sector leaders, the rewards on offer from successful innovation are low, even if the innovation could create huge gains for the public sector and citizens, while the impact of failure can be significantly higher. This can be a major obstacle to innovation.
Developing proposals. Proof of concepts, pilot testing and trials are important steps towards translating ideas into workable projects with potential for implementation. This means creating space for public sector organisations to experiment and try new things. Innovation, by definition, entails novelty and therefore requires organisations to accept a certain level of uncertainty and transform it into manageable risk. However, the very nature of the public sector’s role, with its statutory and moral responsibilities to ensure the basic safety and welfare of its citizens and be accountable for the use of public funds, means that any practice that can pose risks to meeting these responsibilities must be viewed with caution. Supporting this phase involves developing tools to better navigate uncertainty and creating the conditions for experimentation.
Implementing projects. Financial rules and controls can impede the investments needed to bring a project to scale. Budgeting can stimulate innovation through financial incentives, promoting greater flexibility, aligning budgeting and investment frameworks to scale up innovation and diffuse its benefits through the system, and promoting methodologies to ensure successful outcomes. Innovation is also likely to emerge from interactions between different groups, so appropriate frameworks are needed to allow these interactions to happen. Government organisations need opportunities to think about how their interventions interact with those of other groups, and how they can collaborate more effectively to solve common challenges.
Evaluating projects. Innovative projects need to be monitored and evaluated to determine whether they are resolving the problems they are trying to address. Evaluating innovative projects can be a non-linear process – for example fast iteration allows assessments to be conducted during development phases. Yet few countries have developed systematic approaches to evaluating the success of innovative projects. Countries’ experiences suggest that information from project data and social media could be used to evaluate the effectiveness of a project and assess whether it should be iterated, scaled up, or cancelled. Innovation requires evidence, but often at a faster and more agile pace than the traditional policy cycle.
Diffusing lessons. Sharing ideas and experiences are a constituent part of the innovation process and allow successful approaches to be replicated in different contexts. Understanding what went wrong is a powerful source of learning, given the level of risk inherent in innovative projects. At the same time, political and media scrutiny can reduce tolerance for failure, making the uncertainty and risk of innovation seem unduly expensive for the public sector. These factors may create the perception that the public sector is risk averse, but internal learning can reduce risk and uncertainty by pooling experience and results.
Innovation in education
This section zeroes in on innovation in education, one of the largest public sectors in most OECD countries. The education sector poses various opportunities for further innovation. Examples include changes in curricula, new or improved pedagogies in traditional subjects, changes in how learning is assessed (e.g. to measure a broader skillset), and new delivery methods for education, such as online learning. As in other sectors, innovations in education can drive productivity and welfare gains in society. On average, countries spend 6% of their national income on educational institutions. However, despite progress in some countries, not all education systems have taken advantage of opportunities to innovate to improve learning outcomes, enhance equity and equality, improve efficiency, and adapt to societal needs (OECD, 2019[11]).
In many countries, the awareness that a co-ordinated innovation policy for education could be beneficial is just emerging. Yet, contrary to common belief, there is a fair level of innovation in education, both relative to other sectors of society and in absolute terms. While education is below average in terms of the speed of adoption of innovations, 58% of tertiary-educated professionals in education hold a highly innovative job, that is, a job contributing to the innovation process, slightly above the 55% average in the economy (OECD, 2013[73]; 2014[10]). Within education, higher education is the most innovative sub-sector, but examples of innovation exist at all levels. The COVID-19 pandemic has also clearly shown that in some specific circumstances, education systems could innovate at a very fast pace (Vincent-Lancrin, Cobo Romaní and Reimers, 2022[74]; Vincent-Lancrin, 2022[75]). However, a good level of innovation does not necessarily imply that the education sector has a strong innovation ecosystem. Co-ordination of the different levers of innovation policy is often lacking, and the knowledge generated by past innovative pilots or experiments is not always shared and used in a cumulative way (OECD, 2013[73]; 2014[10]).
Teaching innovation
The education sector also plays a role in equipping people with the skills required to innovate, such as creative and critical thinking and social skills. Education systems increasingly include these skills in their educational objectives, but there is no evidence that they are systematically developed or evaluated. Nevertheless, several observations are relevant here:
A broad curriculum exposes students to different knowledge content and ways of thinking. This could directly contribute to innovation by enhancing the ability to make connections between different bodies of knowledge.
Revisiting pedagogies in traditional subjects could be valuable. For example, in mathematics education, metacognitive pedagogies that integrate an explicit reflection about students’ learning and thinking, generally by using self-questioning, have been shown to lead to better learning outcomes. Not only do students improve their mathematical reasoning, but they also develop stronger skills for solving complex, unfamiliar and non-routine problems (Mevarech and Kramarski, 2014[76]). Metacognitive pedagogies are also effective in disciplines other than mathematics.
While countries have changed curricula to broaden the skills that they want students to acquire, many of these skills are not assessed at either the school or system level. The development of new tools to assess such skills, or at least to ensure that teachers pay explicit attention to them, is critical to ensuring that students acquire skills used in innovation (OECD, 2014[77]; Lucas, Claxton and Spencer, 2013[78]; Vincent-Lancrin et al., 2019[66]).
Metacognitive pedagogies are also effective in higher education, even though their effects tend to be smaller than in school education (Mevarech and Kramarski, 2014[76]). Collaborative learning, problem-based learning, game-based learning, real time formative assessment, and the use of online laboratories have been shown to improve students’ understanding, reasoning and creativity in science education (Kärkkäinen and Vincent-Lancrin, 2013[79]). This suggests that tertiary education institutions could enhance innovation-related skills through a variety of pedagogical models.
Policies to support innovation in education
There are four policy areas that are relevant to supporting innovation in education (Vincent-Lancrin, 2017[80]).
First, the education regulatory framework needs to be conducive to innovation. For example, curriculum and assessment policies have an impact on the scope for innovation, and most countries have checks and balances to ensure that grass-roots innovation is possible, but controlled (Kärkkäinen, 2012[81]). As in other sectors, quasi-markets have been used with the objective to foster innovation, and they have been found to help diffuse a variety of alternative models of schooling; however, they do not seem to lead to the emergence of new ones (Lubienski, 2009[82]). Access to finance for innovation, dissemination strategies and staff development policies are also key elements of this regulatory framework.
Second, policies to invest in R&D can support innovation. Given the significance of the sector, public spending on educational research is likely to be below what is needed. In 2012, research on education was the least funded of all socio-economic objectives for which information is available (Foray and Raffo, 2012[83]). In addition, there is often a lack of incentives for companies that produce educational resources and devices to invest in R&D, even though there has been a rise in specialised and innovative educational companies (Foray and Raffo, 2012[83]; OECD, 2016[84]).
Third, policies need to support forms of work organisation that support individual, organisational and sectoral learning of relevance to innovation. The role of learning organisations and professional learning communities is often highlighted, as is the importance of leadership (OECD, 2013[85]).
Fourth, the use of appropriate technologies, notably information and communication technology ICT, can be supported by policy, which has many applications in the education sector. Technology can also be used to transform and enhance pedagogy (Kärkkäinen and Vincent-Lancrin, 2013[79]; OECD, 2021[8]) or modify how education is delivered to students, for example through open educational resources or online courses. Technology can also transform education through data-driven innovation, which is increasingly facilitated by the establishment of administrative longitudinal information systems that follow students throughout their school and university years (González-Sancho and Vincent-Lancrin, 2016[86]).
A few countries already have an innovation policy for education. Italy has developed digital plans for education (Avvisati et al., 2013[87]). France has a chapter on innovation in its education law. The United States has had several programmes to support innovation in education at the federal level, following the model of the Education Innovation and Research (EIR) Program (powercy i3) of the US Department of Education. What is still missing in countries is a holistic and explicit strategy to create an innovation-friendly ecosystem for the education sector.
One aspect that is missing for this ecosystem to emerge are measures of innovation that could guide policy makers in their innovation support. This section evaluates 12 survey instruments that used questionnaires to measure innovation in education, plus one survey of the private sector on work climate (included because of its relevance to measuring a pro-innovation culture). The surveys differ by the part of the education sector covered (primary, secondary, tertiary), region (local, national, or international) and purpose. After a review of each survey and its relevance to innovation in the education sector, we discuss four of them in depth.
Overview
The survey name, source, and relevance to innovation in education are summarised in Table 2.1. These surveys are from several sources, including governmental, academic, private sector, and non-governmental organisations. Most focus on digitalisation rather than innovation on a whole.
Table 2.2 summarises the relevance of the design of each survey questionnaire to innovation in education. All questionnaires are relevant to self-reflection by the respondent. The following factors are evaluated:
1. General/focused: Questions cover all innovation activities (General), or focused on a specific area of innovation (Focused, usually on digitalisation);
2. Innovation activities: Includes questions on innovation activities;
3. Personal characteristics: Includes questions about the characteristics of the respondent or employees;
4. Innovation capacity: Assesses the capacity of the organisation to innovate/digitise;
5. Learning/teaching: Includes questions on teaching and learning and relevant innovations.
The main insight from this overview is that 8 (or 62%) of the 13 surveys on innovation in the education sector are limited to digitalisation, with only five (including two slightly different INNOVA surveys) addressing innovation in a more general way.
A large majority of surveys contains questions on innovation activities though. These include incentives, the type of employees involved in innovation, collaboration, evaluation, etc. Fewer (4, or 31%) collect data on the personal characteristics of respondents or employees, which suggests either more limited attention to workforce skills and attitudes, or the influence of privacy or ethics approval in limiting the collection of personal data. Almost all tools ask participants about the organisation’s capacity for innovation, whether it concerns financial resources, digital technology, etc. The teaching and learning component is only included when the questionnaire focuses on providing education (10 out of 13).
Table 2.3 highlights that most of the questionnaires ask about a pro-innovation culture and governance, implementation, and evaluation. The least frequent topics are data and innovation management. Interestingly, many of the surveys that cover employee skills do not collect data on the creation and diffusion of knowledge, and vice versa. In terms of the surveys, LIKA, DigiPeegel, Innova Individual and NAACE SRF cover the most factors.
Question types
There are also differences across the surveys in the response categories for questions, which can influence accuracy and how the results can be used. Scenarios and open questions, for example, can provide details and information that are missed in a closed question2, but are difficult to turn into indicators. Binary (yes/no) response categories and ordinal importance categories (Likert scales) can be used to construct indicators, but give less information. Multiple choice questions ask the respondent to select one option out of several options. List questions are similar to binary questions, but do not provide a ‘yes or no’ option. Instead, they ask respondents to tick all listed options that apply. Table 2.4 gives an overview of which types of questions are asked in each survey. The most frequently used question types are multiple choice, lists, and scenarios. Most questionnaires use several types of questions, with the Australia and New Zealand university survey using all types except for scenarios.
Summary
The analysis of questionnaires on innovation in the education sector highlights a variety of topics and question types. The majority cover digitalisation and factors that are part of a pro-innovation culture. The latter could be commonly covered because of widespread concerns that bureaucratic conditions hinder innovation in the public sector. Most surveys use more than one question type, which could reflect an interest in avoiding common method bias3.
Details on four questionnaire surveys
This section provides an in-depth discussion of four surveys on innovation in the education sector. Two surveys use multiple questionnaires: SELFIE uses customised questionnaires for different types of schools and respondents, and Innova includes organisation-level and employee-level questionnaires. This fact, alongside their broad coverage of innovation factors (only missing data management) and differing question types, makes them relevant candidates for further analysis. In addition, the KEYS survey is also evaluated in detail because it covers the work environment and is applicable to both public and private sector settings and can thus be relevant to educational institutions. Lastly, Edu Week RC is selected for further analysis because it focuses on inequality.
SELFIE
SELFIE, an initiative by the European Commission, is a self-reflection questionnaire that anonymously gathers the views of students, teachers, and school leaders on how technology is used in their primary, secondary, or vocational school. The questionnaire was developed in co-operation with schools, education ministries and research institutions across Europe. The survey covers leadership, infrastructure, teacher training, and students’ digital competencies. The tool is specific to the education sector and is largely focused on publicly funded schools. It is a reasonably long survey, featuring 72 distinct items.
Each participating school receives a tailor-made, interactive report which provides both in-depth data and quick insights into the school’s strengths and weaknesses and areas needing improvement. The results for specific schools are not shared unless the school leadership chooses to do so. Participation in the survey is relatively high in the European Union, with several countries, such as Spain, moving towards representative samples.
The questionnaire covers leadership, infrastructure and equipment, continuing professional development (availability and accessibility, plus experience with it), teaching and learning (teacher digital competence and practices), assessment practices, student digital competence, and personal characteristics. Given the self-reflective nature of the questionnaire, many of the questions are statements that are answered with a 5-point Likert scale (from “strongly disagree” to “strongly agree”), and a “not applicable” option. The statements focus on the presence of specific processes or assets. There is little focus on how processes or assets are implemented or used, and whether they are deemed sufficient by the teachers and students. Several questions are optional, such that schools can adapt the “standard” survey to fit their needs. Moreover, a limited number of questions of particular interest can be added to the survey in the design phase.
Although all versions of the survey cover most topic areas, different questions are used for school leaders, teachers and students and for primary, secondary, and vocational schools. Statements about school practices are consistent for school leaders and teachers, but questions on support for teachers are reworded to match the reflective nature of the tool. Several questions are only asked of teachers, such as questions about their confidence in using digital technologies in several areas, and the percentage of teaching in the last three months that used digital technologies in class. Questions for students, on the other hand, differ significantly from those for teachers and school leaders and differ for primary school and secondary/vocational students.
The disadvantages of SELFIE are lack of specificity in most of the questions. For example, the section on continuing professional development does not collect information on if or how often teachers attend, whether the opportunities provided are used, and whether it is of good quality and provides sufficient training. Moreover, there is only one question on students in need of special support. In respect to innovation factors, SELFIE does not cover skills or innovation management.
There are several advantages to innovation measurement of the SELFIE design, including covering a broad range of topics and using questions that are appropriate for different groups. There is also a lot of interest from schools to participate, likely aided by the dashboard function, anonymous participation, and non-public results. In terms of factors related to innovation, SELFIE is particularly strong on the application of technology and work organisation, with reasonably good coverage as well on governance and implementation, knowledge creation and diffusion, and rules and processes.
The SELFIE survey also points to the need to customise surveys on innovation in the education sector to ensure that questions are appropriate for different types of schools, students (primary or older) and between school leaders and teachers. It also provides a clear emphasis on implementing a survey to nurture stakeholders’ discussion (and hopefully improvement initiatives).
Innova
The Innova survey covers the national education sector in Hungary and was developed by The Higher Education and Innovation Research Group of the Institute of Education of ELTE University, Budapest (Halász, 2018[13]; Halász and Ágnes, 2021[14]). It targets teachers and school leaders, with an individual-level survey for the former and an organisational survey for the latter. Hence, the tool has two separate versions for organisations and individuals. The survey is longer than the SELFIE tool, with 37 and 34 questions in the individual and organisational versions respectively (160 and 146 items).
The organisational questionnaire covers the innovation environment, innovation practices and activities, the effectiveness of the organisation, and includes a section on a single innovation. The survey includes questions about the specific nature of the organisation, for example, who provides maintenance, the size of the school, whether it is public or private, has a special education focus, and what share of the organisation participates in various staff activities. These questions provide an impression of the ability of the organisation to adapt to various innovations. The questionnaire covers innovation outcomes, including increased effectiveness, whether innovations are permanent, tested elsewhere, and whether knowledge about innovation has been shared.
Other questions cover participation in development programmes (funding, courses, etc.), the organisational work culture, and organisational performance in various areas. Moreover, there are questions about organisational challenges, environmental influences, performance relative to peers, and changes in the organisation’s effectiveness over the last 10 years.
The questions on a single innovation ask when it was created, how, and if relevant, why it was discontinued There are also questions about how much the innovation differs from previous practice, its level of success, and which areas or processes it affects through questions on what the innovation is, for example a method or tool concerning planning and implementation of lessons, improving competences and abilities of students, or special education and the education of disadvantaged children.
The individual-level questionnaire includes questions on innovation practices and activities, and questions about a single selected innovation. It differs in several respects from the organisational version. For instance, it includes questions about the respondent’s experience, age, foreign language skills, education level, gender, years of experience in education, and whether the respondent belongs to an internal community or professional organisation. Other questions cover the nature of innovations, potential collaborators, the organisational culture, and obstacles to improvement or success. The section on a selected innovation largely covers the same topics as the organisational survey.
The Innova questionnaires strike a balance between a questionnaire applicable to school leaders and to employees, providing complementary perspectives on innovation. Although meant for the Hungarian context, the questionnaires are applicable to other contexts, except for some questions on networks and collaboration. The questionnaire covers innovation practices on a general level and delves into a single innovation in-depth. The instrument places a strong emphasis on governance and implementation, as well as on innovation management. There are few questions on the organisation’s capacity to innovate, skills, and incentives for employees to innovate, despite the separate organisational questionnaire. Finally, the questionnaire does not collect specific information of relevance to digital innovations. The questionnaire is very long and at times repetitive. This has the advantage of collecting a lot of information but could also be discourage respondents to answer. This remains a good example to draw on to design questionnaires on innovation in education.
The KEYS survey
KEYS, developed by Teresa Amabile and colleagues, surveys the climate for innovation in private sector work groups or organisations, but many of the questions are also applicable to the public sector (Amabile, Burnside and Gryskiewicz, 1995[95]; Amabile, 1997[30]). The questionnaire aims to distinguish corporate work environments that produce better results from innovation from others, in both technical and non-technical work.
KEYS was designed to help managers gain a clear picture of the climate for innovation within a work group or organisation. The questionnaire measures creativity within the organisation, support for new ideas, the ability to innovate and take risks, and three aspects of the work climate: management practices, organisational motivation, and resources and work pressure. It has 78 items on Likert response scales. The innovation climate greatly influences an employee’s ability to be creative. Work environments that produce highly creative projects and are seen as innovative are generally given higher ratings on KEYS dimensions.
There is a strong focus on employee perceptions. For example, the questionnaire asks if employees can decide what work to do and how to do it, if they feel a sense of control over their work, if their work is challenging, and if they feel that they do important work. Regarding managers, questions ask if the boss is a good work role model, sets goals appropriately, supports the work group, values individual contributions, and shows confidence in the work group. Workload pressure is covered through questions on the absence of extreme time pressures, unrealistic expectation for productivity, and distractions from creative work. Questions on the employee’s work group address the presence of diverse skillsets, good communication, openness to new ideas, constructive challenges to each other’s work, trust and helpfulness towards each other, and commitment to the work. Questions on the organisational context cover the organisational culture and impediments for creativity, such as harsh criticism of new ideas, destructive internal competition, an avoidance of risk, and an overemphasis on the status quo. Questions on resources cover funding, materials, facilities, and information. Lastly, several questions ask about the amount of creativity that is called for at work and whether people believe they produce creative work.
The KEYS questionnaire is relatively short and all questions are in the same format, making the questionnaire reasonably easy to fill out. The different sections can operate independently and thus could be used separately, though this could require different scales. The questions obtain the opinions and perceptions of employees on their work environment, a broad range of motivational influences, and the quality of interactions between employees and management and within work teams. These add new dimension to the SELFIE and Innova surveys. However, the KEYS questionnaire does not cover many other influences on innovation or innovation activities. Lastly, given the focus on employees, the questionnaire is not designed to glean information from school leaders or students.
Education Week Research Center
The Education Week Research Center (EWRC) survey is a nationally representative online survey of nearly 500 K-12 teachers, principals, and school district leaders in the United States. The questionnaire obtains the perspectives of educators on innovation in education and provides results by professional roles and the socio-economic characteristics of schools and districts.
The questionnaire is considerably shorter than the others, at 17 items. The first set of questions cover the future priority of innovation, the extent of innovation in the past, and how often the school or district currently innovates. Other questions ask about the respondent’s risk-seeking behaviour regarding adopting innovations, the factors that drive efforts to innovate, different sources of pressure to innovate, challenges encountered in the innovation process, the role of school leaders, the necessary supports for successful innovation, innovation priorities, and general questions on the respondent’s experience and socio-economic characteristics of the district. The questionnaire focuses on the pressures and challenges experienced in the innovation process.
The questionnaire has a similar focus as Innova Organisation, yet is approximately 10 percent of the length of Innova, trading lower completion time costs of respondents against the depth of information gathered. The short length could be an advantage for local administrators who want to get a quick overview of innovation in their schools. The disadvantages are mostly due to its short length, which limits depth. The questionnaire therefore has limited capacity to support self-reflection.
Summary
Several key observations emerge from the review of the four key surveys.
First, the approaches to the questions differ. For example, SELFIE evaluates the perceptions of students, teachers, and school leaders, and is the only survey to cover all three, while also being adapted to different school levels. The questions in SELFIE are focused on the presence of various factors and not on their quality or necessity. Conversely, Innova has different versions for teachers and school leaders and is very comprehensive, requiring a very long questionnaire. The focus is on the factors that influence innovation processes.
Second, the various strengths of these four questionnaires complement each other, providing material for the design of a new questionnaire. For example, the KEYS questionnaire covers perceptions of the work environment, skills, and performance, the EWRC focuses on general information questions, and Innova provides many examples of questions on innovation processes. It is notable that the coverage of general information differs between all surveys, which emphasises the need to assess whether general information should be similar across different contexts or not.
Third, two of the four surveys (SELFIE and Innova) provide examples for how to orient questions to specific groups, including school leaders, teachers, and different levels of students.
Fourth, a single survey, due to length limitations, cannot cover all topics of interest. For example, none of these four questionnaires cover data management, which can be an important factor for supporting digital innovations and probably other types of innovations. Only one of the questionnaires covers skills and only one covers technology use. There are unavoidable trade-offs in selecting topics and questions, and the level of depth that questions can pursue. Decisions on what to cover depends on the intended users of the results and the targeted respondents.
The analysis and models provided by all these surveys provides a strong inspiration and basis to develop model questionnaires. The three model questionnaires developed in this report are presented in the subsequent chapters (chapters 3, 4, 5), after their scope, purpose, and reasons for including specific questions is introduced. As the surveys presented above, they showcase different possible methods to use questionnaires, trying to combine self-reflection as well as possible statistical uses, short and more lengthy questionnaires, as well as general and more specific targets (with questionnaires focusing on innovation climate and on innovation for equity).
Meeting new and existing challenges in education requires innovation to improve teaching and learning outcomes, administration processes, and the well-being of students and staff. An evaluation of whether educational institutions have the capabilities to innovate and if these capabilities lead to desired outcomes requires data on the factors that support innovation. These include a pro-innovation culture, sufficient resources, appropriate innovation management, and the necessary knowledge and skills.
Drawing on the literature on innovation in the private and public sector, we identified the factors that are important for innovation, and then reviewed existing surveys of innovation in the education sector. Both sources of information, plus advice and comments from external experts, were used to develop three model questionnaires for measuring innovation in the education sector that are presented in this part of the report: a general questionnaire that covers all innovation activities (chapter 3), a module of questions for inclusion in other surveys that collects data on the innovation culture of educational institutions (chapter 4), and a questionnaire on the use of innovation to improve equity in education (chapter 5). Each questionnaire is provided in two or more versions, suitable for school leaders, teachers, or students.
The general innovation questionnaire is designed to collect statistical information that can be used to build a profile of innovation capabilities within an education system (or an educational institution) or for comparison across several educational systems (or institutions). In addition, the general innovation questionnaire and the other two more subject-specific questionnaires are of value for self-reflection, where the questions inspire respondents to think deeply about how innovation occurs in their institution and what might be required to improve innovation capabilities or outcomes.
The questionnaires are designed for use by school leaders (principals, deans, rectors, etc.) or government departments at multiple levels (regional or district) that are interested in collecting representative data on innovation at a single educational institution or for all institutions in a defined region. Since some of these potential users may lack expertise in how to implement a survey, chapter 6 provides guidelines for a school principal or district leader on what is required to implement a questionnaire survey as well as links to other ‘how to’ resources for surveys.
References
[36] Agolla, J. and J. Van Lill (2016), “An empirical investigation into innovation drivers and barriers in public sector organisations”, International Journal of Innovation Science, Vol. 8/4, pp. 404-422, https://doi.org/10.1108/ijis-06-2016-0006.
[68] Alves, H. (2013), “Co-creation and innovation in public services”, The Service Industries Journal, Vol. 33/7-8, pp. 671-682, https://doi.org/10.1080/02642069.2013.740468.
[30] Amabile, T. (1997), “Motivating Creativity in Organizations: On Doing What You Love and Loving What You Do”, California Management Review, Vol. 40/1, pp. 39-58, https://doi.org/10.2307/41165921.
[92] Amabile, T., R. Burnside and S. Gryskiewicz (1995), User’s Guide for KEYS: Assessing the Climate for Creativity, Center for Creative Leadership.
[95] Amabile, T., R. Burnside and S. Gryskiewicz (1995), User’s manual for KEYS, assessing the climate for creativity : a survey from the Center for Creative Leadership, Center for Creative Leadership.
[91] Amabile, T., R. Conti, H. Coon, J. Lazenby and M. Herron (1996), “Assessing the work environment for creativity”, Academy of Management Journal, Vol. 39/5, pp. 1154-1184, https://doi.org/10.2307/256995.
[37] Andersen, S. and M. Jakobsen (2018), “Political Pressure, Conformity Pressure, and Performance Information as Drivers of Public Sector Innovation Adoption”, International Public Management Journal, Vol. 21/2, pp. 213-242, https://doi.org/10.1080/10967494.2018.1425227.
[5] APSC (2011), State of the Service Report 2010-11. State of the Service Series, Australian Public Service Commission (APSC).
[19] Arundel, A., C. Bloch and B. Ferguson (2019), “Advancing innovation in the public sector: Aligning innovation measurement with policy goals”, Research Policy, Vol. 48/3, pp. 789-798, https://doi.org/10.1016/j.respol.2018.12.001.
[90] Arundel, A., D. Bowen Butchart, S. Gatenby-Clark and L. Goedegebuure (2016), Management and service innovations in Australian and New Zealand universities: Preliminary report of descriptive results, LH Martin Institute, Melbourne.
[41] Arundel, A., L. Casali and H. Hollanders (2015), “How European public sector agencies innovate: The use of bottom-up, policy-dependent and knowledge-scanning innovation methods”, Research Policy, Vol. 44/7, pp. 1271-1282, https://doi.org/10.1016/j.respol.2015.04.007.
[17] Arundel, A. and D. Huber (2013), “From too little to too much innovation? Issues in measuring innovation in the public sector”, Structural Change and Economic Dynamics, Vol. 27, pp. 146-159, https://doi.org/10.1016/j.strueco.2013.06.009.
[87] Avvisati, F., S. Hennessyii, R. B. Kozma and S. Vincent-Lancrin (2013), “Review of the Italian Strategy for Digital Schools”, OECD Education Working Papers, No. 90, OECD Publishing, Paris, https://doi.org/10.1787/5k487ntdbr44-en.
[61] Bellefontaine, T. (2012), Innovation Labs: Bridging Think Tanks and Do Tanks.
[15] Bloch, C. and M. Bugge (2013), “Public sector innovation—From theory to measurement”, Structural Change and Economic Dynamics, Vol. 27, pp. 133-145, https://doi.org/10.1016/j.strueco.2013.06.008.
[23] Borins, S. (2014), The persistence of innovation in government, Brookings Institution Press, Washington, DC.
[35] Boxall, P. and J. Purcell (2011), Strategy and Human Resource Management, third edition, Palgrave Macmillan.
[60] Bugge, M., P. Mortensen and C. Bloch (2011), Measuring Public Innovation in Nordic Countries, Report on the Nordic Pilot Studies-Analyses of methodology and results.
[94] Carlsen, R. K. McFerrin, J. Price, R. Weber and D. Willis (eds.) (2007), LIKA - Digital literacy in Teacher Education, Association for the Advancement of Computing in Education (AACE), https://www.learntechlib.org/primary/p/24772/.
[62] Carstensen, H., C. Bason and H. Vibeke (2012), “Powering collaborative policy innovation: Can innovation labs help?”, The Innovation Journal, Vol. 17/2.
[46] Cels, S., J. de Jong and F. Nauta (2012), Agents of Change: Strategies and Tactics for Social Innovation, Brookings Institution Press, Washington, DC.
[56] Chesbrough, H. (2003), Open Innovation: The New Imperative for Creating and Profiting from Technology, Harvard Business School Press, Boston.
[28] Christiansen, J. and L. Bunt (2012), Innovation in Policy: Allowing for Creativity, Social Complexity and Uncertainty in Public Governance, Nesta and MindLab, http://www.nesta.org.uk/sites/default/files/innovation_in_policy.pdf.
[42] Cinar, E., P. Trott and C. Simms (2019), “A systematic review of barriers to public sector innovation process”, Public Management Review, Vol. 21/2, pp. 264-290, https://doi.org/10.1080/14719037.2018.1473477.
[27] Crosby, B., P. ‘t Hart and J. Torfing (2017), “Public value creation through collaborative innovation”, Public Management Review, Vol. 19/5, pp. 655-669, https://doi.org/10.1080/14719037.2016.1192165.
[57] Dahlander, L. and D. Gann (2010), “How open is open innovation?”, Research Policy, Vol. 39/6, pp. 699-709.
[43] de Vries, H., V. Bekkers and L. Tummers (2014), “Innovation in the Public Sector: A Systematic Review and Future Research Agenda”, SSRN Electronic Journal, https://doi.org/10.2139/ssrn.2638618.
[58] Demircioglu, M. and D. Audretsch (2020), “Conditions for complex innovations: evidence from public organizations”, The Journal of Technology Transfer, Vol. 45/3, pp. 820-843, https://doi.org/10.1007/s10961-018-9701-5.
[25] Eurofound (2012), Work Organisation and Innovation, http://eurofound.europa.eu/sites/default/files/ef_files/pubdocs/2012/72/en/1/EF1272EN.pdf.
[88] European Commission (2023), SELFIE resources, https://education.ec.europa.eu/selfie/resources.
[4] European Commission (2013), Powering European Public Sector Innovation: Towards A New Architecture, Report of the Expert Group on Public Sector Innovation, European Commission, European Commission, Brussels.
[33] Fernandez, S. and T. Moldogaziev (2012), “Using employee empowerment to encourage innovative behaviour in the public sector”, Journal of Public Administration Research and Theory, Vol. 23/1, pp. 155-187.
[96] Fink, A. (2003), The Survey Handbook, Sage Publications, London.
[83] Foray, D. and J. Raffo (2012), “Business-Driven Innovation: Is it Making a Difference in Education?: An Analysis of Educational Patents”, OECD Education Working Papers, No. 84, OECD Publishing, Paris, https://doi.org/10.1787/5k91dl7pc835-en.
[52] Freeman, C. (1987), Technology Policy and Economic Performance: Lessons from Japan, Pinter, London.
[32] Frey, B. and M. Osterloh (eds.) (2002), Successful Management by Motivation, Springer Berlin Heidelberg, Berlin, Heidelberg, https://doi.org/10.1007/978-3-662-10132-2.
[97] Fuller, C., M. Simmering, G. Atinc, Y. Atinc and B. Babin (2016), “Common methods variance detection in business research”, Journal of Business Research, Vol. 69/8, pp. 3192-3198, https://doi.org/10.1016/j.jbusres.2015.12.008.
[9] Gault, F. (2018), “Defining and measuring innovation in all sectors of the economy”, Research Policy, Vol. 47/3, pp. 617-622, https://doi.org/10.1016/j.respol.2018.01.007.
[16] Gault, F. (2012), “User innovation and the market”, Science and Public Policy, Vol. 39/1, pp. 118-128.
[86] González-Sancho, C. and S. Vincent-Lancrin (2016), “Transforming education by using a new generation of information systems”, Policy Futures in Education, Vol. 14/6, pp. 741-758, https://doi.org/10.1177/1478210316649287.
[13] Halász, G. (2018), “Measuring innovation in education: The outcomes of a national education sector innovation survey”, European Journal of Education, Vol. 53/4, https://doi.org/10.1111/ejed.12299.
[89] Halász, G. (2017), „INNOVA” Products, Downloadable Documents, http://halaszg.elte.hu/INNOVAENGLISHWEBSITE/Products.htm (accessed on 29 January 2023).
[14] Halász, G. and F. Ágnes (2021), “Who is innovating and how in the education sector? Combining subject and object approaches”, Journal of Adult Learning, Knowledge and Innovation, Vol. 5/1, pp. 22-35, https://doi.org/10.1556/2059.2021.00042.
[24] Hartley, J. (2006), “Innovation in governance and public services: Past and present”, Public Money & Management, Vol. 25/1, pp. 27-34.
[40] Helfat, C., S. Finkelstein, W. Mitchell, M. Peteraf, H. Singh, D. Teece, S. G. Winter (2007), Dynamic Capabilities: Understanding Strategic Change in Organizations, Blackwell Publishing, Oxford.
[39] Helfat, C. and J. Martin (2014), “Dynamic Managerial Capabilities”, Journal of Management, Vol. 41/5, pp. 1281-1312, https://doi.org/10.1177/0149206314561301.
[93] Junta Castilla y Léon (n.d.), Herramienta de autoevaluación de la competencia digital, https://autoevaluaciontic.educa.jcyl.es/ (accessed on 1 March 2023).
[81] Kärkkäinen, K. (2012), “Bringing About Curriculum Innovations: Implicit Approaches in the OECD Area”, OECD Education Working Papers, No. 82, OECD Publishing, Paris, https://doi.org/10.1787/5k95qw8xzl8s-en.
[79] Kärkkäinen, K. and S. Vincent-Lancrin (2013), “Sparking Innovation in STEM Education with Technology and Collaboration: A Case Study of the HP Catalyst Initiative”, OECD Education Working Papers, No. 91, OECD Publishing, Paris, https://doi.org/10.1787/5k480sj9k442-en.
[6] Kay, R. and C. Goldspink (2012), What Public Sector Leaders Mean When They Say They Want to Innovate, Incept Labs, Sydney.
[51] Kline, S. and N. Rosenberg (1986), “An overview of innovation”, in The Positive Sum Strategy: Harnessing Technology for Economic Growth, National Academies Press, Washington, DC.
[45] Kruiter, A. et al. (2008), De Rotonde van Hamed: Maatwerk voor Mensen met Meerdere Problemen, Nicis Institute, Den Haag.
[82] Lubienski, C. (2009), “Do Quasi-markets Foster Innovation in Education?: A Comparative Perspective”, OECD Education Working Papers, No. 25, OECD Publishing, Paris, https://doi.org/10.1787/221583463325.
[78] Lucas, B., G. Claxton and E. Spencer (2013), “Progression in Student Creativity in School”, OECD Education Working Papers, Organisation for Economic Co-Operation and Development (OECD), https://doi.org/10.1787/5k4dp59msdwk-en.
[53] Lundvall, B. (ed.) (1992), National Innovation Systems: Towards a Theory of Innovation and Interactive Learning, Pinter, London.
[44] Lunn, P. (2014), Regulatory Policy and Behavioural Economics, OECD, https://doi.org/10.1787/9789264207851-en.
[67] McGann, M., E. Blomkamp and J. Lewis (2018), “The rise of public sector innovation labs: experiments in design thinking for policy”, Policy Sciences, Vol. 51/3, pp. 249-267, https://doi.org/10.1007/s11077-018-9315-7.
[76] Mevarech, Z. and B. Kramarski (2014), Critical Maths for Innovative Societies: The Role of Metacognitive Pedagogies, Educational Research and Innovation, OECD Publishing, Paris, https://doi.org/10.1787/9789264223561-en.
[50] Mulgan, G. (2009), The Art of Public Strategy: Mobilizing Power and Knowledge for the Common Good, Oxford University Press, Oxford.
[1] Mulgan, G. and D. Albury (2003), “Innovation in the public sector”, Strategy Unit, Cabinet Office., Vol. 1/1, p. 40.
[31] Mumford, M. (2000), “Managing creative people: Strategies and tactics for innovation”, Human Resource Management Review, Vol. 10/3, pp. 313-351.
[54] Nelson, R. (ed.) (1993), National Innovation Systems. A Comparative Analysis, Oxford University Press, New York/Oxford.
[8] OECD (2021), OECD Digital Education Outlook 2021: Pushing the Frontiers with Artificial Intelligence, Blockchain and Robots, OECD Publishing, Paris, https://doi.org/10.1787/589b283f-en.
[11] OECD (2019), Measuring Innovation in Education 2019: What has changed in the classroom?, OECD, https://doi.org/10.1787/9789264311671-en.
[3] OECD (2017), Fostering Innovation in the Public Sector, OECD Publishing, Paris, https://doi.org/10.1787/9789264270879-en.
[84] OECD (2016), Innovating Education and Educating for Innovation: The Power of Digital Technologies and Skills, https://doi.org/10.1787/9789264265097-en.
[47] OECD (2015), Behavioural Insights and New Approaches to Policy Design: The Views from the Field, Summary of an international seminar 23 January, Paris, http://www.oecd.org/gov/behavioural-insights-summary-report-2015.pdf.
[18] OECD (2015), Frascati Manual 2015, OECD, https://doi.org/10.1787/9789264239012-en.
[22] OECD (2015), The Innovation Imperative in the Public Sector: Setting an Agenda for Action, OECD Publishing, Paris, https://doi.org/10.1787/9789264236561-en.
[10] OECD (2014), Measuring Innovation in Education: A New Perspective, Educational Research and Innovation, OECD Publishing, Paris, https://doi.org/10.1787/9789264215696-en.
[77] OECD (2014), PISA 2012 Results: Creative Problem Solving (Volume V), OECD, https://doi.org/10.1787/9789264208070-en.
[70] OECD (2013), Exploring data-driven innovation as a new source of growth: Mapping the policy issues raised by “big data”, Organisation for Economic Co-Operation and Development (OECD), https://doi.org/10.1787/9789264193307-12-en.
[73] OECD (2013), Innovative Learning Environments, Educational Research and Innovation, OECD Publishing, Paris, https://doi.org/10.1787/9789264203488-en.
[85] OECD (2013), Leadership for 21st Century Learning, Educational Research and Innovation, OECD Publishing, Paris, https://doi.org/10.1787/9789264205406-en.
[65] OECD (2011), Skills for Innovation and Research, OECD Publishing, Paris, https://doi.org/10.1787/9789264097490-en.
[55] OECD (1997), National Innovation Systems, https://www.oecd.org/science/inno/2101733.pdf.
[12] OECD/Eurostat (2018), Oslo Manual 2018: Guidelines for Collecting, Reporting and Using Data on Innovation, 4th Edition, The Measurement of Scientific, Technological and Innovation Activities, OECD Publishing, Paris/Eurostat, Luxembourg, https://doi.org/10.1787/9789264304604-en.
[21] Osborne, S. and L. Brown (eds.) (2013), Handbook of Innovation in Public Services, Edward Elgar, Cheltenham.
[48] Osborne, S. and L. Brown (2011), “Innovation in public services: engaging with risk”, Public Money & Management, Vol. 31/1, pp. 4-6, https://doi.org/10.1080/09540962.2011.545532.
[69] Osborne, S., M. Cucciniello, G. Nasi and K. Strokocsh (2021), “New development: Strategic user orientation in public services delivery—the missing link in the strategic trinity?”, Public Money & Management, Vol. 41/2, pp. 172-175, https://doi.org/10.1080/09540962.2020.1758401.
[29] Østergaard, C., B. Timmermans and K. Kristinsson (2011), “Does a different view create something new? The effect of employee diversity on innovation”, Research Policy, Vol. 40/3, pp. 500-509, https://doi.org/10.1016/j.respol.2010.11.004.
[2] Potts, J. and T. Kastelle (2010), “Public sector innovation research: What’s next?”, Innovation, Vol. 12/2, pp. 122-137, https://doi.org/10.5172/impp.12.2.122.
[63] Puttick, R., P. Baeck and P. Colligan (2014), I-teams: The Teams and Funds Making Innovation Happen in Governments Around the World, Nesta and Bloomberg Philanthropies, http://www.nesta.org.uk/sites/default/files/i-teams_june_2014.pdf.
[34] Shalley, C. and L. Gilson (2004), “What leaders need to know: A review of social and contextual factors that can foster or hinder creativity”, The Leadership Quarterly, Vol. 15/1, pp. 33-53, https://doi.org/10.1016/j.leaqua.2003.12.004.
[38] Simmons, R. and C. Brennan (2017), “User voice and complaints as drivers of innovation in public services”, Public Management Review, Vol. 19/8, pp. 1085-1104, https://doi.org/10.1080/14719037.2016.1257061.
[59] Sørensen, E. and J. Torfing (2011), “Enhancing Collaborative Innovation in the Public Sector”, Administration & Society, Vol. 43/8, pp. 842-868, https://doi.org/10.1177/0095399711418768.
[72] Speier, C., J. Valacich and I. Vessey (1999), “The Influence of Task Interruption on Individual Decision Making: An Information Overload Perspective”, Decision Sciences, Vol. 30/2, pp. 337-360, https://doi.org/10.1111/j.1540-5915.1999.tb01613.x.
[26] Torfing, J. (2019), “Collaborative innovation in the public sector: the argument”, Public Management Review, Vol. 21/1, pp. 1-11, https://doi.org/10.1080/14719037.2018.1430248.
[64] Torjman, L. (2012), Labs: Designing the future, MaRS Discovery District blog, http://www.marsdd.com/systems-change/mars-solutions-lab/news/labsdesigning-future/.
[7] Torugsa, N. and A. Arundel (2017), “Rethinking the effect of risk aversion on the benefits of service innovations in public administration agencies”, Research Policy, Vol. 46/5, pp. 900-910, https://doi.org/10.1016/j.respol.2017.03.009.
[49] Townsend, W. (2013), “Innovation and the perception of risk in the public sector”, International Journal of Organizational Innovation, Vol. 5/3, pp. 21-34.
[71] Ubaldi, B. (2013), “Open Government Data: Towards Empirical Analysis of Open Government Data Initiatives”, OECD Working Papers on Public Governance, No. 22, OECD Publishing, Paris, https://doi.org/10.1787/5k46bj4f03s7-en.
[75] Vincent-Lancrin, S. (2022), Educational innovation and digitalisation during the COVID-19 crisis: lessons for the future, OECD, https://doi.org/10.1787/bbeca162-en.
[80] Vincent-Lancrin, S. (2017), Understanding Innovation in Education: Where Do We Stand, Edward Elgar Publishing, https://doi.org/10.4337/9781785369070.00013.
[74] Vincent-Lancrin, S., C. Cobo Romaní and F. Reimers (eds.) (2022), How Learning Continued during the COVID-19 Pandemic: Global Lessons from Initiatives to Support Learners and Teachers, OECD, https://doi.org/10.1787/bbeca162-en.
[66] Vincent-Lancrin, S., C. González-Sancho, M. Bouckaert, F. de Luca, M. Fernández-Barrerra, G. Jacotin, J. Urgel and Q. Vidal (2019), Fostering Students’ Creativity and Critical Thinking: What it Means in School, Educational Research and Innovation, OECD Publishing, Paris, https://doi.org/10.1787/62212c37-en.
[98] Willis, G. (2004), Cognitive interviewing: A tool for improving questionnaire design, Sage publications.
[20] Windrum, P. and P. Koch (2008), Innovation in Public Sector Services, Edward Elgar Publishing, https://doi.org/10.4337/9781848441545.
Notes
← 1. These definitions have been simplified, excluding technical characteristics that are mostly applicable to the private sector.
← 2. An open question asks the respondents to provide a written answer, for instance a description of an innovation. A closed question provides a limited number of response options, such a ‘yes’ or ‘no’.
← 3. Common method bias occurs when question design creates artificial relationships between dependent and independent variables that is due to question design and not to factors of interest. Although concern over common method bias is overstated (Fuller et al., 2016[97]), using different question formats can reduce the likelihood of its occurrence.