9. A survey on educational R&D: a proposed international standard for questionnaire and methodology
Countries routinely collect information about their investment and operation in research and development, a key indicator of countries’ ability to innovate. Adapting the cross-sectorial approach promoted by the Frascati Manual, this chapter presents a survey instrument (questionnaire) and method to collect information about countries’ educational research and development. The questionnaire, presented in Annex, can be used (and adapted) by jurisdictions interested in knowing how much is spend on educational research and development, by whom, and on what broad topics.
This work has benefited from the inputs of many former OECD colleagues, notably Francisco Martinez as an intern, Richard Scott and Nicolas Jonas, as well as from country delegates.
Most governments collect some statistics about their investment in educational research and development (R&D) as part of their general statistics on R&D, which allow to compare their spending for educational research and for other socio-economic objectives. Education policy makers interested in driving their educational research agenda would need more granular and accurate information to better understand who performs and funds educational research within their system, and whether educational R&D focuses on their policy priorities.
Given limited budgets for educational R&D, governments need to invest them in a smart way and steer the system based on a better understanding of the educational R&D that is already performed or funded. For example, is enough knowledge generated to improve the quality of tertiary education and early childhood education compared to primary and secondary education? Is there enough educational research on teaching and learning to support practices in the classroom? Depending on what foundations and the higher education sector already fund or work on, what would it be more strategic or useful for governments to support? In the current state of affair, most education ministries and research agencies have very little information to take informed decisions. Therefore collecting specific information that can guide their decisions and inform them on how much their system in the generation and application of new knowledge is important.
This chapter proposes an exemplar questionnaire and methodology to collect information about the performance and funding of educational R&D within a country or education system – or internationally. Countries or regions could adapt it to their context or interest, or implement it as it is. Annex 9.A presents the model questionnaire. It follows the guidelines and definitions of the Frascati Manual (OECD, 2002[1]; OECD, 2015[2]), but mainly focuses on the public funding and performance of educational R&D and proposes questions that are customised and specific to the education sector.
The rest of the chapter is organised as follows. After presenting the structure of the questionnaire and explaining some choices made, a suggested method to define the universe of the respondents and collect the data is proposed. Finally, a couple of examples of implementation are briefly presented.
The questionnaire focuses on the funding or investment in educational research by collecting information about the performance and the funding of educational research and development, and proposes definitions of what should be counted as performance and as funding to avoid double counting.
After an introduction about the questionnaire, which can be customised by the administering organisation, and contact information about the respondents, section 2 tries to categorise the responding organisation and identify whether if is a funder and/or performer of educational R&D.
Compared to usual R&D questionnaires, the business sector has been left out. This is mainly for practical reasons: the possible units to survey would then be too numerous if those were included, but in principle the questionnaire could also work for for-profit companies performing or funding educational research. In education, most research comes from the higher education sector and the government. In some countries (for example the United States), the private non-profit sector (foundations) plays a large role too, either as a performer or funder of educational research. This is why this is an explicitly included in the scope of the survey. Because of the educational nature of the questionnaire and the fact that most of the answers will be of interest to departments of education, question 4 requires details about the nature of the responding organisation. All the questionnaire focuses on a specific fiscal year (to be specified depending on the date of administration).
The questionnaire is divided in two big sections: performance of educational R&D (section 3) and funding of educational R&D (section 4), with mirroring questions. Question 5 is both a summary and a “skip” question that collects the total amount that the responding organisation spends on performance and funding of educational R&D. The questionnaire is thus relevant for organisations that only perform educational research, only fund educational research or do both.
Section 3 requires details about the performance of educational R&D, using respondents to provide a (rough) percentage of the allocation of their performance between different research topics (question 6), levels of education (question 7), sources of funding (question 8).
The different areas of educational research propose high level categories that build on “special interest groups” as defined or identified by different research associations (e.g. American Educational Research Association (AERA), European Educational Research Association (EERA), European Association for Research on Learning and Instruction (EARLI), etc.). While educational research associations routinely have around 20-30 interest groups, our categories summarised the main themes in six broad umbrella topics (e.g. education “statistics”, “teacher education”, “learning, instruction and curriculum”, plus the possibility to add one’s own category). It is indeed unlikely that respondents can provide a very granular answer to this question so framing it according to what could be their main departments seemed more appropriate.
Question 6 asks about the level of education the educational research is about – noting that there might be differences across countries. (For an international survey, countries could use the International Standard Classification of Education (ISCED) classification and translate it into their national categories). Vocational education and training was omitted on the ground that it covers secondary education tertiary education and “adult learning”. Relevant categories could be added in a domestic context. When research is valid for all or several levels of education, a response category was provided.
The sources of funding defined in Question 8 take into account the federal nature of some countries (central/local government), the fact that for-profit companies (industry) sometimes fund research and development, the role of foundations and other non-profit organisations, the role of the organisations’ own funds, and finally, the role of international (or inter-governmental) organisations such as the European Commission, the World Bank, other banks of regional development, and others.
Questions 9 and 10 try to identify the amount of development in education, which is typically much lower for the target organisations than actual research. Question 9 requests a raw percentage on the divide between research (basic and applied) and development, while question 10 enumerates the typical development outputs an organisation involved in educational R&D could have.
Question 11 provides a shortened adaptation of a “barrier” question that is typical in R&D surveys, limiting the answer options to those most relevant to the education sector. Acknowledging that those answer proposals were mainly designed for the private sector, two open answers are suggested if need be.
Question 12 asks about the headcount and full time equivalent personnel devoted to educational R&D in the responding organisation. In order to make it easier to answer, but also to provide information about whether educational researchers work in “specialised” organisations or not, the question asks for the count for all fields of science, for educational R&D and for personnel that is not devoted to R&D (e.g. administrative or communication staff).
Section 5 asks for the same questions in the frame of the funding rather than performance of educational R&D, using the same sub-categories: sub-areas of R&D (question 13), target level of education (question 14), sector of recipient (rather than source) organisation (question 15), research or development (question 17) and type of development output (question 18).
Question 16 acknowledges that a funding institution of educational R&D could get its funding from different types of organisations. For example, a national research agency may get the funding it allocates to educational R&D from government appropriations or local appropriations, but also from other sources of funding. It is also common for foundations to pool their resources so that a funding organisation may allocate funds that were received from another non-profit organisation.
Section 6 allows respondents to enter comments to qualify and explain their answers.
Collecting information about educational research and development is challenging because only a small part of a country’s research and development ecosystem invest in or perform educational research and development. Surveying the entire ecosystem for this purpose would be a difficult and expensive exercise, often unrealistic for most countries or jurisdictions and leading to high rates of non-response. Therefore, the suggested methodology is comprised of three phases: the first is to conduct an initial mapping of educational R&D within the jurisdiction (or each country participating if the survey is conducted internationally). The second phase is to use this mapping to collect data on educational R&D from key actors. The third phase is to analyse the questionnaire responses and prepare a report.
Phase 1: Mapping of educational R&D
The main aim of the mapping exercise is to establish the universe for the survey on educational R&D. In the case of an international implementation, this phase also allows to verify that the approaches are consistent across countries and establishing which institutions should be involved in the survey in each country).
The target universe is institutions in the public, higher education and non-profit sectors financing or undertaking educational R&D. Relevant institutions in the mapping will typically include: government departments (not solely those in charge of education), government agencies, universities, public research centres/organisations, foundations, research councils and statistical agencies. The survey is intended to include both performers and funders of educational R&D (with separate questions to minimise any risk of double counting). The private sector and schools are not within the scope of the proposed survey instrument. In the higher education sector, the focus is not solely on education science departments –research on education topics carried out in a range of disciplines such as economics, sociology or medicine is within scope and should be included in the mapping process in principle.
The questionnaire takes into account the diverse activities within educational R&D. Curriculum development and educational evaluation or statistics should therefore be included in the mapping exercise if the results of these exercises are made public. Parts of the public educational inspectorate are also likely to meet the definition.
However, depending on the context, only the main actors with a significant activity in educational R&D should be included
The mapping exercise will typically include the following key actions for countries/jurisdictions:
Identify the universe of domestic educational research based on the definitions above (the focus of the project should be made explicit to avoid misinterpretation). The priority of the mapping is to identify large performers of educational R&D – an exhaustive census of relevant actors is not expected. Annex 9.B gives some example of the information that countries or jurisdictions could aim to collect about the educational research units for their internal discussions.
To conduct the mapping, jurisdictions should draw on expert knowledge to help identify where educational R&D takes place. In particular, national associations dealing with educational research or major educational research institutes should be consulted. Countries may benefit from establishing a review group of experts.
Countries may also wish to draw on bibliometrics to identify relevant institutions.
Countries should then review the mapping process and output, ideally with external experts not involved in the mapping group in order to introduce a layer of internal checks.
The mapping phase will allow countries to undertake cognitive testing of the draft questionnaire with national experts. If implemented locally, the questionnaires should be adjusted accordingly. If implemented internationally, feedback should be shared with the international coordinator at this early stage so that a wording that work for all participating countries/jurisdictions can be agreed upon.
Phase 2: Survey of educational R&D practitioners
The model of administration is similar to the one for an innovation survey (see chapter 2 of this report, (Van Lieshout, Arundel and Vincent-Lancrin, 2023[3])).
Depending on the size of the relevant R&D practitioners, they may opt for two strategies: a census of all large R&D performers and funders that they have identified; a stratified random sampling based on characteristics of importance within the concerned jurisdiction.
One of the key challenges in such a survey will be to ensure a reasonable response rate and to minimise selection bias. Establishing contact persons for each institution and conducting questionnaires in-person may assist this process. Another issue may come from how the possible use of the survey information will be perceived by the respondents, and how the confidentiality of the information will be managed. Countries could resort to their official statistical agency, if any, commission a third party that would provide them with the survey data in an aggregated form only to protect anonymity for the participating organisations. They may also consider guaranteeing that the responses will not be used for other purposes than statistical ones. Institutions may also be wary of responding to non-compulsory government surveys; partnering with educational research institutions may be one way to resolve this problem.
The data collection will typically include the following key actions for countries/jurisdictions:
Once finalised, countries will need to organise translation of the questionnaire into their national language (in the case of an international survey).
Identify contact persons in key institutions to whom to send the questionnaire (the expert group during the mapping exercise may help with this). For universities, targeting the relevant departments directly will be necessary, though in some countries the institutions would have to be formally notified.
Consider visiting institutions in-person in order to encourage participation. In many instances, responding to the questionnaire will require collaboration between researchers and administrative or budget officers (e.g. in defining how much expenditure is allocated to R&D).
Countries also have the option of using fieldwork teams to assist the data collection, or draw on the expertise of the community identified during the mapping. Working collaboratively with relevant educational associations or institutes is advised, as it may change the way the survey is perceived and encourage participation.
Reach at least a 60% response rate and analyse the non-response to ensure that it can be considered as random.
Undertake a check of the collected data, highlighting any issues in consistency and comparability.
Phase 3: Report, analysis and discussion
Once the data from the questionnaires are collected, countries should ensure some level of quality assurance of the data collection. The data analysis performed on the aggregated responses will allow for the preparation of a draft report providing comparative information on:
the amounts spent on educational R&D, presented in absolute terms and in relative terms to account for the size of the country/jurisdiction and its level of public expenditures on education;
the relative weight of different types of actors in the performance and funding of educational R&D (government, higher education, private non-profit);
the levels of education and topics of education the most commonly researched and funded in absolute terms and by different types of actors to see whether different actors are inclined to perform or fund different types of research, whether there is a good balance to improve different dimensions of the education process;
the share of experimental development in the total R&D and what are its typical outputs;
the level of overlap between funding and performance (are the sectors funding also the sectors that are performing, and do they do so through specialised institutions, within ministries or by other means);
Ideally, the repetition of the survey at regular intervals will allow to monitor the evolution of this funding and perhaps will allow for zeroing in on specific topics from one edition to the next.
A pilot implementation of the questionnaire was proposed to OECD countries in 2013-15. Three countries/education systems participated. One succeeded in doing the mapping and implemented the data collection, but failed to engage one of the larger educational research performers, a non-response that made the other collected results impossible to redress or interpret with enough accuracy. Two other countries successfully implemented the methodology and the data collection: the Netherlands (de Jonge, 2016[4]) and Norway (Gunnes and Rørstad, 2015[5]).
Norway has carried out surveys of educational R&D since 2007, so the mapping to identify the relevant units to survey for the data collection was straightforward. Adjustments consisted mainly in mapping Norwegian response categories to the ones provided by the OECD questionnaire (for example go from the “Institutes”, which includes a few private research institutes, to the “Government” category) and in adding some questions. Gunnes and Rørstad (2015[5]) present the implementation of the OECD survey. Since their 2013 survey, Norway has replicated the OECD questions in its subsequent surveys of educational R&D (Rørstad et al., 2021[6]; Rørstad et al., 2019[7]; Gunnes, Hovdhaugen and Olsen, 2017[8]) .
The Netherlands also successfully implemented the proposed approach in 2015, in their case with no prior survey infrastructure. De Jonge (2016[4]) summarises the main findings as follows:
“The most important finding is that investment in educational R&D is limited in terms of both budget appropriations and number of individuals working in this field. Approximately 150 institutions in the Netherlands are involved in educational R&D. They employ almost 1000 FTEs [full time equivalents] in research, with a further 175 FTEs providing research support. According to our calculations, the Netherlands invests a total of approximately EUR 120 million per annum in educational R&D. If we compare these figures to total employment in education or to total expenditure on education, we see that spending on educational R&D is limited. Expressed as a percentage, it comes to 0.28%. That is relatively little compared with the almost 2% spent on R&D nationally (across all sectors). Our conclusion is that education is a research-extensive sector in which investment in education itself has very little grounding in research and development work evaluating the effectiveness and efficiency of that investment.”
Both examples show that the methodology and standards could easily be used internationally once a political will emerges within OECD countries.
References
[4] de Jonge, J. (2016), Onderwijsonderzoek in Nederland - Inventarisatie voor de OECD, Rathenau Instituut.
[8] Gunnes, H., E. Hovdhaugen and B. Olsen (2017), Utdanningsforskning i Norge 2015: Ressurser og resultater, http://hdl.handle.net/11250/2442190.
[5] Gunnes, H. and K. Rørstad (2015), Educational R&D in Norway 2013: Resources and Results, http://hdl.handle.net/11250/2412424.
[2] OECD (2015), Frascati Manual 2015: Guidelines for Collecting and Reporting Data on Research and Experimental Development, The Measurement of Scientific, Technological and Innovation Activities, OECD Publishing, Paris, https://doi.org/10.1787/9789264239012-en.
[1] OECD (2002), Frascati Manual 2002: Proposed Standard Practice for Surveys on Research and Experimental Development, The Measurement of Scientific and Technological Activities, OECD Publishing, Paris, https://doi.org/10.1787/9789264199040-en.
[7] Rørstad, K., B. Sarpebakken, K. Wendt, H. Gunnes, B. Magne Olsen and M. Østby (2019), Ressursinnsatsen til FoU innenfor tematiske områder i 2017, http://hdl.handle.net/11250/2619010.
[6] Rørstad, K., B. Sarpebakken, K. Wendt, M. Nedberg Østby, H. Gunnes, E. Denisova and F. Sæbø Steine (2021), Ressursinnsatsen til FoU innenfor tema- og teknologiområder i 2019, https://hdl.handle.net/11250/2771665.
[3] Van Lieshout, K., A. Arundel and S. Vincent-Lancrin (2023), "Measuring Innovation through Surveys: Main Considerations and Applications to Education", in Vincent-Lancrin, S. (ed.), Measuring Innovation in Education 2023: Tools and Methods for Data-Driven Action and Improvement, OECD Publishing.
Innovation and improvement in education partly relies on educational research and experimental development. The [name of surveying organisation] seek to improve the quality of data on educational research and development by undertaking a new data collection following internationally comparable categories. The answers collected through this survey are intended to be used for drafting reports and papers produced by the [name of surveying organisation] and not be used for any other purposes. The questionnaire has been developed by the OECD Centre for Educational Research and Innovation (CERI), mainly for an international implementation allowing for comparative information. [If relevant: It has been adapted by <name of the surveying organisation>.]
Research and experimental development (R&D) is defined as “creative and systematic work undertaken in order to increase the stock of knowledge – including knowledge of humankind, culture and society – and to devise new applications of available knowledge”.
Research by students at the PhD level carried out at universities should be counted, whenever possible, as a part of R&D. However, all education and training of personnel in universities and special institutions of higher and post-secondary education should be excluded.
The questionnaire follows the methodological guidelines of the Frascati Manual. Most terms used in the questionnaire are clarified in the Frascati Manual.
This survey collects data on educational research and experimental development performed by organisations from the government, higher education and non-profit sectors in OECD countries.
The person responsible for completing this survey should be an individual or officer who leads or supervises R&D undertaken in the organisation. Complete responses are essential to support a comprehensive collection of key elements of educational research. Please answer all questions. Where exact data are not available, please provide an estimate and indicate it in section 5.
The questionnaire is structured as follows:
Section 1: Contact information
Section 2: Organisation information
Section 3: Performance of Educational R&D (for organisations performing R&D)
Section 4: Funding of Educational R&D (for organisations funding R&D)
Section 5: Additional information
For further assistance in interpreting the questions contained in the survey or for any related questions, please contact:
Section 1: Contact Information
Primary Contact (person who is responsible for the institution’s survey answers)
Full name __________________________________
Position __________________________________
E-mail address __________________________________
Telephone number __________________________________
Fax number __________________________________
Alternate Contact (person to contact if the primary contact is unavailable)
Full name __________________________________
Position __________________________________
E-mail address __________________________________
Telephone number __________________________________
Fax number __________________________________
Section 2: Organisation Information
☐ Higher Education ☐ Government ☐ Private non-profit
Please give additional details of your main activity (and only tick one box corresponding to the sector selected in your previous answer).
5. During fiscal year <20xx>, how much of your organisation’s expenditures were allocated to perform and fund educational R&D? (please write down the amounts in your national currency)
Educational R&D consists of a broad range of activities, including but not limited to research and experimental development on: educational policies; management, organisations and leadership; curriculum studies; learning; instructional methods; teaching and teacher education; assessment and evaluation; education statistics; and educational technology. Please consider also all other activities not listed here which you regard as educational R&D. All education and training of personnel in universities and special institutions of higher and post-secondary education should be excluded. However, research by students at the PhD level carried out at universities should be counted, whenever possible, as a part of R&D.
All your R&D expenditures should fall in either performance or funding of R&D. Please ensure you do not count any expenditure twice. Educational R&D can be performed with your own funds or other funds, and includes sub-contracted research. If your organisation has been sub-contracted to carry out educational R&D that belongs to another organisation, it should NOT be included.
Copyright is not the only criterion of belonging. If your organisation or its members have performed some research that will be published in a scientific journal or by a publisher that will own the copyright of your work, please report this research as a performer. If your organisation or its members perform research for another agency, for example a statistical agency, which will publish the work under its corporate name, please do NOT report it as performed by your organisation, even if you are acknowledged as author: the commissioning agency will report it in its performance of research (and consider your contribution as sub-contracted and thus part of its performance budget). If you produce by-products of commissioned work for scientific journals, commercial publishers, etc., they should be reported as performed educational R&D.
Funding only concerns research grants and awards, and not commissioned research as part of your own research. Research grants may be given through all sorts of mechanisms and can be given for specific areas of research, specific types of research, or with full freedom in topics and methods to researchers.
If your organisation did not allocate any expenditures to perform educational R&D, please skip to question 13 (Section 4).
Section 3. Performance of Educational R&D
The questions in this section refer to the educational R&D activities performed by your organisation in fiscal year <20xx>.
6. Specify the percentage of the total educational R&D performed in fiscal year <20xx> by sub-area of educational R&D.
If the R&D activities performed by your institution belong to several of the categories below, please allocate them proportionately to the amount spent on the different sub-areas in your research. Should this be too difficult to estimate, please allocate the percentage evenly between the sub-areas concerned.
Example: a research programme may produce indicators on the management of schools. If the research mainly emphasises the statistical dimension, you may allocate its expenditures as 80% in "education statistics" and 20% in "Management, Organisation, and Leadership". If you do not find a satisfactory allocation, please allocate 50% to each sub-area.
7. Specify the percentage of the total educational R&D performed in fiscal year <20xx> about specific levels of education.
If the R&D activities performed by your institution belong to several of the categories below, please allocate them proportionately to the amount spent on the different education levels in your research. Should this be too difficult to estimate, please allocate the percentage evenly between the sub-areas concerned.
8. Specify the percentage of the total educational R&D performed in fiscal year <20xx> by source of funding.
Funds received for R&D performed during earlier periods or for R&D not yet started should be excluded from the sources of funds reported for the specific period. If the funding passes through another organisation, you should report the initial source of funding.
If your organisation did not perform any development activities during fiscal year <20xx>, please skip to question 11 in this section.
10. During fiscal year <20xx>, did your organisation engage in the development of the following products/services, in whole or in part: (tick as many as applicable)
11. Please rank the two most significant barriers to your institution’s educational R&D activities? (place a “1” for the most significant barrier and a “2” for the second most important barrier.)
12. Specify the headcount and full time equivalent (FTE) personnel devoted to total R&D activities and educational R&D activities in fiscal year <20xx>, by occupation.
Headcount refers to the total number of persons who are mainly or partially employed in R&D. This includes both “full-time” and “part-time” employed staff.
Full-time equivalent (FTE) refers to the total effort devoted to R&D in terms of personnel. As R&D may be carried out by persons who work solely on R&D projects or by persons who devote only part-time to R&D, it is necessary to estimate the full-time equivalent of the persons working only part-time in R&D. Examples of calculation: if out of five researchers engaged in R&D work, one works solely on R&D projects and the remaining four devote only one quarter of their working time to R&D, then: FTE = 1 + ¼ + ¼ + ¼ + ¼ = 2 researchers; if out of 3 researchers, one works for 6 months, one for 3 months, and one for one year, then: FTE = ½ + ¼ + 1 = 1.75.
Researchers are professionals engaged in the conception or creation of new knowledge, products, processes, methods and systems and also in the management of the projects concerned. Managers and administrators engaged in the planning and management of the scientific and technical aspects of a researcher’s work also fall into this category. Their rank is usually equal or superior to that of persons directly employed as researchers and they are often former or part-time researchers. Postgraduate students at the PhD level engaged in R&D should be considered as researchers.
Technicians and equivalent staff are persons whose main tasks require technical knowledge and experience in one or more fields of engineering, physical and life sciences or social sciences and humanities. They participate in R&D by performing scientific and technical tasks involving the application of concepts and operational methods, normally under the supervision of researchers. Equivalent staff perform the corresponding R&D tasks under the supervision of researchers in the social sciences and humanities.
Administrative support staff includes skilled and unskilled craftsmen, secretarial and clerical staff participating in R&D projects or directly associated with such projects.
Section 4. Funding of Educational R&D
If your organisation did not allocate any expenditures to fund educational R&D, please skip to section 5.
R&D is considered funded by your organisation if its outputs (e.g., publications, patents, product, services) belong to the recipient of the funds. Sub-contracting research should not be considered as funding. Funds passing through your organisation to another organisation should not be reported as funding either, for example because you co-ordinate a project funded by another agency and allocate the funding to different partners.
13. Specify the percentage of the total educational R&D funded by your organisation during fiscal year <20xx> by sub-area of educational R&D.
If the R&D activities funded by your institution belong to several of the categories below, please allocate them proportionately to the amount spent on the different sub-areas in your research. Should this be too difficult to estimate, please allocate the percentage evenly between the sub-areas concerned.
14. Specify the percentage of the total educational R&D funded by your organisation during fiscal year <20xx> by level of education.
If the R&D activities funded by your institution belong to several of the categories below, please allocate them proportionately to the amount spent on the different education levels in the research you funded. Should this be too difficult to estimate, please allocate the percentage evenly between the sub-areas concerned.
15. Specify the percentage of the total educational R&D funded by your organisation during fiscal year <20xx> by the sector of recipient institution.
16. Specify the percentage of the total educational R&D funded in fiscal year <20xx> by source of funding.
Funds received for R&D funded during earlier periods or for R&D not yet started should be excluded from the sources of funds reported for the specific period.
17. Specify the percentage of the total educational R&D funded by your organisation during fiscal year <20xx> by type of R&D.
R&D is considered funded by your organisation if its outputs belong to the recipient of the funds. Please do not include any research sub-contracted to another organization.
18. During fiscal year <20xx>, did your organisation fund the development of the following products/services, in whole or in part: (tick as many as applicable).
Section 5. Additional information
Please provide comments below on any information you have supplied. Please specify the questions where you provided estimates rather than exact data.