6. Conclusions and outlook
An important aspect of the 2017 governance reform in Austria is to strengthen the use of evidence for decision making at all levels of governance to accompany the changes in responsibilities, specifically those of schools and school supervision (school quality managers). To this end, Austria partnered with the OECD Strategic Education Governance project to take stock of efforts promoting the systematic use of evidence in the Austrian education system. The following sections highlight overarching key findings and identify the relative strengths and weaknesses over five areas to promote use of evidence. Finally, the chapter provides an outlook on possible next steps.
This report sought to offer an indication of well-developed areas to promote the systematic use of evidence and areas with room for improvement. Carried out as an online survey to decision makers in the federal ministry, education directorates, school supervision and school leaders, it is meant as a “thermometer" gauging areas for further investigation and informing thinking about possible next practices. It is designed as a conversation starter rather than a definitive evaluation of practices. Following the spirit of a self-reflective exercise, the report is set to inform discussions at a workshop with stakeholders of all levels of governance in the Austrian education systems.
The report indicates that some provinces engage in efforts to promote the use of evidence highly systematically. In these provinces, school quality managers consistently report to exchange frequently with colleagues and with evidence providers to improve quality and preparation of evidence. Across schools, school leaders systematically report to exchange with school quality managers and to exchange with peers about methods and experiences to work with evidence. They report systematically that their exchanges are supported organisationally, such as through requisite time or staff resources. In other provinces, such efforts are emergent. For instance, some schools and school quality managers engage frequently in exchanges to improve quality and preparation of evidence; others do less so with reports of a lack of organisational support for exchanges being more frequent.
School quality managers play a pivotal role in fostering the use of evidence. They see room to be more involved in the provision and preparation of evidence. More than 90% of school quality managers engage in efforts to raise awareness of the merit and importance of using evidence in decision making. While there is a clear focus on working with school leaders to build awareness in schools, six in ten school quality managers also engage teaching staff in schools directly. While school quality managers are largely content with the evidence that is available, they express motivation to be more involved in preparing and providing evidence. They seek greater interaction with key providers of evidence.
Schools have some key organisational processes in place that can encourage the use of evidence in decision making. Approaches that promote transparency in decisions taken and how they were reached can motivate the use of evidence. School leader reports show an emphasis on internal exchange to increase clarity around school decisions and decision-making processes. Inviting a range of perspectives, experiences, and knowledge into different areas of decision-making can strengthen systematic use of evidence by motivating consideration of different sources of evidence. School leaders mainly report inviting diverse perspectives, for instance from teachers, parents, and students, to help develop classroom teaching. However, vocational and special-needs schools moreover underscore this approach also for staff development decisions.
Schools are important evidence producers. The vast majority of school leaders produce and use school-level evidence from internal evaluations (87%) and standardised student testing (82%). Many school leaders (84%) report that school staff prepares evidence themselves, which reflects the comparatively widespread use of teacher-developed tests in Austria as found by PISA (OECD, 2016[1]). Results have uncovered many concrete examples of evidence preparation at the school level. Many school leaders took the opportunity of the survey to highlight their school-level efforts of preparing evidence, which can provide a starting point to further dialogue and investigation. Examples include self-prepared statistical evidence within schools, such as school-developed competency catalogues, competence checks via the school’s own learning platform, annual student and teacher surveys and statistical information of registration and graduate numbers. Moreover, many examples referred to efforts to gather additional qualitative evidence. Examples included regular and intensive discussions with the teaching staff at the school, feedback from students and parents tailored to the school, and school leaders networking with peers. Some school leaders also provided examples of successful collaborations with universities.
Evidence provided to schools is not always adequately prepared for them and their work environment. Only half of all school quality managers who participated in the survey report that the preparation of evidence for schools is largely (42%) or very (8%) adequate. Shortcomings in the user-friendliness and practice-orientation of evidence with which schools are provided is a recurring topic among responses of school leaders. School leaders consider the BMBWF/ Education directorates as less concerned with preparing the evidence they provide to schools in a user-friendly way than other evidence providers. Among school leaders only a minority (41%) considers the BMBWF/ education directorates as largely or very interested in providing evidence in a user-friendly way; 43% of school leaders consider this to be the case little or not at all. Conversely, 56% of school leaders consider research institutions for the most part or very concerned with preparing evidence for schools in a user-friendly manner.
The online survey among executives at the BMBWF and education directorates, school quality managers, and schools leaders gathered multifaceted insights about the present opportunities and efforts carried out that research finds to promote the opportunity, capability and motivation to use evidence effectively and systematically. The survey covers five areas to promote the systematic use of evidence areas:
The skills to access and makes sense of evidence.
Making evidence conveniently available.
Organisational processes encouraging the use of evidence.
Collaboration with evidence producers and collegial exchange.
Building a common understanding of the importance of evidence, which evidence is useful and how its best used in specific situations.
Questions gathered information about the type of efforts and solicited additional information about the context and practices. A look at those questions that can be divided into positive and negative answers allows for an indication of relative strengths and weaknesses across the different areas.
The responses among school leaders and school quality mangers point towards relative strengths in terms of present opportunities and efforts in two areas (Figure 6.1). First, in the area of developing skills to access and makes sense of evidence, on average around eight in ten school leaders give positive responses. Within this area, school leaders answer particularly positive regarding opportunities to develop skills to access and make sense of evidence (nine out of ten school leaders). Also for school quality managers, opportunities to develop skills around using evidence are well established albeit on a lower level (85% positive responses among school leaders, 77% positive responses among school quality mangers). The main barrier to more a positive picture in this area is a perception among school quality managers of fewer opportunities to build skills to guide and instruct the use of evidence among schools (seven out of ten school quality managers answer positively) (Figure 6.2).
Note: see Figure 6.2 for positive responses to individual questions used to calculate averages in the areas of: the skills to access and makes sense of evidence (Skills), making evidence conveniently available (Availability), organisational processes encouraging the use of evidence (Organisational processes), collaboration with evidence producers and collegial exchange (Interaction), and building a common understanding of the importance of evidence, which evidence is useful and how its best used (Standards). Any response other than none/no/never is counted as positive. Questions with no negative answer options (no/none/never) are omitted.
A second area of strength among school leaders and school quality managers pertains to developing evidence-use as a principle of good decision-making and developing common understanding of how to use evidence (Figure 6.1). In this area, around eight in ten school leaders and school quality managers answer positively to questions in this area, albeit with different emphases. Close to nine in ten school leaders report efforts to develop common understanding in the school on how evidence should be used and to develop a common understanding which evidence is fit-for-purpose in specific decision-making challenges. However, only seven in ten school leaders report efforts to build awareness for use of evidence as principle of good decision-making. From the responses emerges a picture that school quality managers emphasise building awareness for use of evidence as a principle of good decision-making and efforts to develop a common understanding on how evidence should be used. Less frequently reported are efforts to develop a common understanding which evidence is fit-for-purpose in specific decision-making situations (Figure 6.2).
Out of the five areas to promote the systematic use of evidence, making evidence conveniently available is a relatively weak area with around six out of ten school leaders and school quality managers responding positively to questions in this area (Figure 6.1). The area covers the topics whether provided evidence is perceived as well-targeted and user-friendly, and whether the users feel adequately involved. Among school leaders, user-friendliness of evidence provided to schools by the federal ministry and education directorates particularly leaves room for improvement (four in ten school leaders answer positively). User-friendliness of evidence provided by research institutions is reported more favourably with overall around six in ten school leaders answering positively. While school leaders feel overall adequately involved in making evidence available (six in ten), only three in ten school quality managers regard their involvement in making evidence available favourably (Figure 6.2).
Note: Any response other than no/none/never is counted as positive. Questions with no negative answer options (no/none/never) are omitted. Ordered by decreasing share of positive answers within each area (average of school leaders and school quality managers).
For the areas of interaction and organisational processes, the responses show a more mixed picture. The area of interaction includes exchanges among colleagues and the exchange with evidence providers. Overall, six in ten school leaders report favourable opportunities within this area – however, while structured exchanges with colleagues are very widespread (eight in ten school quality managers and nine in ten school leaders report positively), only four in ten school leaders report exchanges with evidence providers. However, where school leaders do exchange with evidence providers, these exchanges are frequently aimed at improving provided evidence. Whereas school quality managers exchange substantially more frequently with evidence providers (six in ten) than school leaders, among both school leaders and school quality managers organisational support is reported favourably by only half of respondents. In the area of organisational processes, the analysis highlights very encouraging responses regarding school leaders’ efforts to clarify decision and how these decisions were reached, with around nine in ten school leaders reporting such practices overall. Among school quality managers, the adoption of such practice indicates some room for improvement. However, school quality managers’ responses show very widespread engagement in building awareness for the merits of evidence use for decision-making (nine in ten school quality managers engage in respective efforts). Inviting diverse perspectives in decision-making is overall a less employed practice both within schools and among school quality mangers (Figure 6.2).
Structuring collegial exchanges around explicit purposes taking into account already existing habits
Interactions with peers are an important channel for school leaders and school quality managers to exchange experiences and methods around using evidence. Empirical research emphasises the advantages of frequent interaction with a low threshold to interact, for instance in collegial exchange, over more ambitious, but (necessarily) less frequent interactions on the one hand and the benefits of structured exchange around an explicit purpose on the other (Shippee et al., 2013[2]; Langer, Tripney and Gough, 2016[3]; O’Mara-Eves et al., 2013[4]).
To maximise the potential of collegial exchanges to promote the use of evidence, exchanges should be structured around an explicit purpose to this end. Purposes include developing skills to gather, access and make sense of evidence, developing common understanding of what makes evidence fit for a specific purpose, and developing agreement on how evidence should be used in a specific situation.
Importantly, efforts to structure collegial exchanges should take current habits and processes as point of departure to avoid burdening decision makers and increase their adoption (Bunn and Sworn, 2011[5]; Langer, Tripney and Gough, 2016[3]). Where decision makers already engage systematically in relevant efforts, efforts to structure exchanges further can be counter-productive. Specifically, in some provinces school quality managers and school leaders already engage in efforts to promote the use of evidence highly systematically.
Increasing availability of trainings to develop skills to guide and instruct evidence use
For both school leaders and school quality managers, strengthening their evidence-related capabilities includes developing the skills to guide and instruct the use of evidence in schools. Yet, availability of respective trainings is reported comparatively infrequently. For both school leaders and school quality managers, exchange among peers is one of the main sources to strengthen evidence-related skills. Increasing the availability of trainings specifically aimed at fostering skills to guide and instruct the use of evidence can help promote use of evidence directly and insert important new knowledge into collegial exchanges to this end.
Supporting school leaders and school quality managers develop a common understanding around using evidence
The 2017 reform changed responsibilities of school quality managers and schools. Developing a common understanding around using evidence pertains to developing agreement around which evidence is fit-for-purpose for which tasks and how it is best used in concrete situations. This is particularly relevant in the transition to new responsibilities as specific decision-making situations and habits are still emerging.
With the reform, schools received greater autonomy, particular in terms of staff decisions and developing school quality. In this respect, developing a common understanding of using evidence includes determining which evidence is fit for purpose for specific responsibilities, such as staff development, and how and when is this most useful. School quality managers may take a role here in facilitating exchange across school types. For instance, responses show that special-needs schools (Sonderschule, ASO) engage more frequently in efforts to include diverse perspectives in staff decisions than primary schools, which are comparable in average number of classes. The BMBWF and education directorates can take a role in supporting schools and school quality managers in developing standards around using evidence. This includes facilitating exchanges between school quality managers and schools, as well as exchanges of school quality managers and schools with other evidence providers. Exchanges should be structured around explicit purposes; taking into account already existing habits and processes.
Collaboratively reflecting on which evidence is best gathered where
While schools are important evidence producers, in some circumstances others (e.g. school quality managers) will be in a better position to gather and prepare fit-for-purpose evidence. Similarly, some evidence may be best prepared and contextualised by research institutions, such as the IQS1. In the same vein, schools will be in a better position to gather evidence needed at other levels, for instance, by school quality managers to optimise the regional education offer. Different levels of governance – in particular, school leaders and school quality managers – should be involved in this reflection as decision-making situations and challenges are still emerging in light of changed responsibilities.
Improving tailoring of and user-friendly access to evidence through exchange with decision makers
Not all decision makers will be equally prepared to gather and prepare evidence as needed for their new responsibilities. Encouraging evidence providers to tailor evidence to the habits and capacities of decision makers – for instance to the needs of school leaders with less routine in using evidence – can help bring everyone along in the transition to new responsibilities. Tailoring evidence can support decision makers adjust to changed responsibilities and respective demands on using evidence. Making fit-for-purpose evidence available in a user-friendly way reduces burden for decision makers and increases take-up of evidence sources.
Tailoring of evidence and providing user-friendly access to evidence depends on information about decision makers’ work processes and habits. Evidence providers may gather respective information through exchanges with decision makers. Responses indicate that school quality managers are motivated to be directly involved in preparing evidence, while low-threshold information exchanges may be more suitable when soliciting schools’ input (Figure 6.2).
References
[5] Bunn, F. and K. Sworn (2011), “Strategies to promote the impact of systematic reviews on healthcare policy: A systematic review of the literature”, Evidence & Policy: A Journal of Research, Debate and Practice, Vol. 7/4, pp. 403-428, https://doi.org/10.1332/174426411X603434.
[3] Langer, L., J. Tripney and D. Gough (2016), The Science of Using Science - Researching the Use of Research Evidence in Decision-Making, EPPI-Centre, Social Science Research Unit, UCL Institute of Education, University College London, http://eppi.ioe.ac.uk/cms/Default. (accessed on 15 January 2018).
[4] O’Mara-Eves, A. et al. (2013), “Community engagement to reduce inequalities in health: A systematic review, meta-analysis and economic analysis”, Public Health Research, Vol. 1/4, pp. 1-526, https://doi.org/10.3310/phr01040.
[1] OECD (2016), PISA 2015 Results (Volume II): Policies and Practices for Successful Schools, PISA, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264267510-en.
[2] Shippee, N. et al. (2013), “Patient and service user engagement in research: A systematic review and synthesized framework”, Health Expectations, Vol. 18/5, pp. 1151-1166, https://doi.org/10.1111/hex.12090.
Note
Note
← 1. The Institute of the Federal Government for Quality Assurance in the Austrian School System (IQS) superseded the Federal Institute for Educational Research, Innovation and Development of the Austrian School System (BIFIE). The change took effect on 1 July 2020.
