Chapter 10. The object method for innovation measurement
This chapter provides guidance on collecting data on innovation from an object-based perspective. The object-based method collects data on a single, focal, most important innovation, facilitating information retrieval about enablers, features and outcomes of business innovations. Although the method can also be applied to unconventional data sources, this chapter describes how to implement the object approach within subject-based innovation surveys that cover the full range of innovation activities and innovations of the firm. Because focal innovations are not representative of the business as a whole, the main purpose of the object approach is to collect data for analytical and research purposes. The method can also be used to assess whether innovation is over- or under-reported by business respondents.
10.1. Introduction
10.1. The object approach to innovation measurement collects data on a single, “focal” innovation (the object of the study), in contrast to the subject approach, which focuses on the firm and collects data on all its innovation activities (the subject) (see Chapter 2). The main purpose of the object approach is not to produce aggregate innovation statistics but to collect data for analytical and research purposes. The method can also provide useful information for quality assurance purposes on how respondents interpret questions on innovation and whether they over-, under- or misreport innovation.
10.2. The object method can identify focal innovations through expert evaluations, or through announcements of innovations in trade publications (Kleinknecht and Reijnen, 1993; Santarelli and Piergiovanni, 1996; Townsend, 1981) or online sources (company websites, reports, investor announcements, etc.). An alternative method of using the object method is to incorporate the object approach within a subject-based innovation survey. In addition to questions on all of the firm’s innovation activities, a module of questions can focus on a single innovation. DeBresson and Murray (1984) were the first to use a version of this method as part of an innovation survey in Canada. More recently, this approach has been used in business enterprise surveys, for instance by Statistics Canada and the Japanese Statistical Office, academic researchers in Australia (O’Brien et al., 2015, 2014) and the United States (Arora, Cohen and Walsh, 2016), and in surveys of innovation in the Government sector (Arundel et al., 2016).
10.3. The inclusion of the object method within a subject-based innovation survey has several advantages over the use of experts or announcements to identify focal innovations. First, it can obtain information on a focal innovation for a representative sample of all innovative firms, whereas other methods will be prone to self-selection biases. Second, it can collect data on all types of innovations. Using experts or announcements to identify innovations will produce a bias towards successful product innovations. Third, it can collect information on innovations that are new to the firm only, or not sufficiently novel to be reported on line or in trade journals. It is therefore recommended, where cost-effective, to collect data on a focal innovation through representative surveys.
10.2. Including an “object module” in an innovation survey
10.4. In the survey context, there are several advantages of collecting data on a focal innovation in addition to data on all of the innovation activities of a firm. First, the inclusion of an object method module in an innovation survey can support the use of in-depth, quantitative and interval level questions that are too difficult for respondents to answer for all their innovations combined, for instance questions that require respondents to calculate the average importance of a variable across multiple innovations or innovation activities. Potentially difficult questions include expenditures on different innovation activities and the use of specific technical capabilities. Other difficult questions are those that require respondents to construct an “average” representation across the entire firm, such as questions on the importance of different knowledge sources, obstacles and outcomes.
10.5. Second, the use of questions on a single focal innovation ensures that the set of data collected refer to the same innovation. This is primarily an advantage for analyses on the relationships between innovation inputs, activities and outcomes, as in the research by Arora, Cohen and Walsh (2016) on the economic value of alternative knowledge sources for innovation. It can also assist other types of research, such as an evaluation of how respondents understand innovation survey questions (Arundel, O’Brien and Torugsa, 2013), and research into blended innovations that span both product and business processes (Bloch and Bugge, 2016), including changes to business models.
10.6. Nonetheless, it is not recommended to only include object-based questions in an innovation survey, or to allot a significant percentage of survey questions to an object module. Many research and policy questions cannot be addressed through questions on a focal innovation. These include questions that are relevant to the firm as a whole, such as questions on the firm’s internal capabilities and strategies (see Chapter 5) and external environment (see Chapter 7), as well as questions that are used to create aggregate indicators for all innovation activities, such as data on innovation expenditures (see Chapter 4) or the innovation sales share (see Chapter 8).
10.7. The object method is seldom useful for constructing simple statistics and indicators at the national or industry level because the answers do not fully reflect the overall innovation inputs, outputs and outcomes of an economy or industry. Furthermore, the focal innovation is unlikely to be representative of all of the responding firm’s own innovations or innovation activities. Data for a firm’s most important innovation should therefore not be used to produce indicators that require data for all of a firm’s innovations, such as total expenditures for specific innovation activities, the importance of different types of knowledge sources for innovation, or the frequency of collaboration with different types of partners.
10.8. Many of the guidelines in this manual for collecting data on innovation at the subject level can be directly applied to collecting data at the object level. There are no additional methodological limitations to including an object-based module in a subject-based innovation survey.
10.2.1. Identifying a focal innovation within surveys
10.9. An object module must include an initial prompt that asks respondents from innovative firms to think of a single innovation and limit all subsequent questions in the module to this innovation. Respondents from firms that are innovation-active, but with no innovations in the observation period, can be also asked to think about a single innovation project. As a device to ensure that the responses are focused on the innovation, it is helpful to ask the respondent, in an open question, to provide a short description of the innovation.
10.10. It is recommended to ask respondents to select a focal innovation that was introduced or implemented during the observation period. This ensures that other data from an innovation survey on the general capabilities or strategies of the firm are relevant to the focal innovation and that data on the focal innovation can be linked to outcome data from other surveys with a known time lag interval. It also reduces recall biases for innovations that occurred before the observation period (see Chapter 9). However, respondents should be permitted in their responses to include activities, where relevant, that occurred before the start of the observation period, such as collaboration with specific types of partners or the receipt of government subsidies for the innovation.
10.11. The questionnaire should also provide guidance for the choice of a focal innovation (or innovation project) to improve comparability between respondents. Possible options include:
-
the most important innovation with respect to its actual or expected contributions to the firm’s economic performance
-
the innovation with the highest share of total innovation expenditures invested in its development
-
the product innovation with the greatest actual or expected contribution to sales
-
the business process innovation with the greatest actual or expected contribution to reducing costs
-
the most recent innovation.
10.12. The first option has several advantages. The question is usually well understood by respondents and the innovation is memorable, which ensures that respondents can answer questions about it. In addition, the most important innovation is relevant to many areas of research, such as on the factors that lead to success. Leaving the first option open to all types of innovations can collect useful data on the types of innovations that firms find important. It can also identify innovation inputs that are likely to be of high value to a firm. For instance, a respondent could give a moderate importance ranking to universities as a source of knowledge for all innovation activities, but the use of this source for its most important innovation would indicate that the value of knowledge from universities could vary by the type of innovation.
10.13. The second option requires respondents to have a good knowledge of the development cost for different innovations. The third and fourth options are a variant of the first option, but limited to either product or business process innovations and therefore will not be relevant to firms that did not introduce an innovation of that type. The fifth option is useful for research that requires a random selection of all types of innovations.
10.14. Unless there are good research reasons for using a different option, the first option is recommended because it is better understood by respondents and is relevant to all firms. Furthermore, the first option is useful for research into the types of innovations with the largest expected economic benefits to the firm. These results can be used to construct aggregate indicators by industry, firm size, or other firm characteristic on the types of innovations (i.e. product or business process innovations) that respondents find of greatest economic value to their firm.
10.15. Cognitive testing shows that respondents are able to identify their most important innovation as defined by its actual or expected contribution to the firm’s economic performance. For small and medium-sized enterprises (SMEs), there is usually one innovation that stands out from all others. Respondents from firms with many different innovations (often, but not always large firms) can find it difficult to identify a single innovation that stands out in comparison with the rest, but this does not affect their ability to select a single innovation and answer subsequent questions about it. Respondents from firms with many innovations are still likely to find it easier to answer questions on a focal innovation than to summarise results for multiple innovations.
10.16. If resources permit, written information in an open-ended description of the most important innovation can be coded and analysed to assess how respondents interpret questions on the types of innovation and the novelty of the innovation (Arundel, O’Brien and Torugsa, 2013; Cirera and Muzi, 2016; EBRD, 2014). This requires written information to be coded by experts, but text mining software tools can significantly reduce coding costs. Textual data on novelty can also be used to estimate if respondents understood the questionnaire definition of an innovation (Bloch and Bugge, 2016).
10.2.2. Non-innovative firms
10.17. Firms with no innovations or innovation activities cannot be asked about a focal innovation or a focal innovation project. However, it can be useful to ask respondents from non-innovative firms to describe their most important change to products or business processes during the observation period. This information can be analysed to determine whether respondents correctly report innovations and can distinguish them from changes that are not innovations (Arundel O’ Brien and Torugsa, 2013). Combined with information on the novelty of reported innovations, the object approach can help identify potential biases towards under- or over-reporting innovations of different types by firm characteristics such as size or industry.
10.3. Questions on a focal innovation
10.18. Subject-based innovation surveys that include an object-based module should place such module after all other innovation questions in order to ensure that respondents do not confuse questions about all innovation activities with questions limited to a focal innovation.
10.3.1. Characteristics of the focal, most important innovation
10.19. It is recommended to include a list of innovation types (two types of product innovations and six types of business process innovations) and ask respondents to identify all innovation types that are part of their focal innovation (see Chapter 3). This can provide data on the prevalence of “bundled” innovations that have the characteristics of more than one innovation type (for instance both a service innovation and a business process innovation for product delivery) and which types of innovations are most important to firms.
10.20. It is recommended to collect information on the comparative importance for the responding firm of the focal innovation. Useful measures include the share of total innovation costs spent on the focal innovation and the contribution of the focal innovation to a firm’s performance outcomes (e.g. sales or profits) (see subsection 10.3.2 below). Outcome questions will not be relevant to respondents reporting on an innovation project.
10.21. Respondents can be asked several questions on the novelty of their focal innovation, including if it is new to their market or only new to their firm, if it is part of a new business model, or if it is a radical or disruptive innovation (see subsection 3.3.2). However, data collection on radical, disruptive and related types of innovations will require experimentation to determine if these concepts can be properly measured in an innovation survey.
10.3.2. Innovation activities contributing to the focal innovation
10.22. Cognitive testing shows that respondents find it easier to provide interval level expenditure data (either in currency units or in person-months) for a single innovation than for all innovations combined (see Chapter 4). Consequently, it may be possible to obtain expenditure data for the entire period that the focal innovation was under development, instead of only for the reference year.
10.23. A question on expenditures for a single innovation can be particularly appropriate for SMEs or service sector firms that do not organise their innovation activities into clearly defined projects with a separate accounting budget.
10.24. It may be possible to obtain the following data for the focal innovation:
-
the total time, in calendar months, from the initial idea for the focal innovation, to its introduction or implementation
-
the year of introduction for a product innovation or the year of implementation for a business process innovation
-
total expenditures in currency units or person-months on the focal innovation
-
total external expenditures by type of activity on the focal innovation (research and experimental development, training, design, engineering, and other creative work activities, etc.)
-
the use of and expenditures on follow-on activities after the introduction of a product innovation onto the market. This can include marketing, training, and after-sales services (see subsection 4.5.3).
10.25. Some of these questions could ask for data on activities before the observation period, such as the question on calendar months or total expenditures, but this is only likely to be relevant for major innovations.
10.3.3. Business capabilities contributing to the focal innovation
10.26. Business capabilities related to management or workforce skills are a characteristic of the firm (see Chapter 5) and generally not limited to a focal innovation. However, intellectual property (IP) strategies and technological capabilities can vary significantly among different types of innovations.
10.27. Depending on research interests, it can be worthwhile to ask about the use of different IP protection methods for the focal innovation, for instance whether a patent, design, trademark, or other IP right application was made for the focal innovation or if it is covered by copyright or trade secrecy. In addition, respondents can be asked if they licensed-in technology for their focal innovation or if the focal innovation was licensed-out (Arora, Cohen and Walsh, 2016).
10.28. Questions on technical capabilities are appropriate for an object module that can link capabilities to specific types of innovations. Relevant capabilities include design capabilities (engineering design, product design, and design thinking), digital capabilities, and digital platforms (see section 5.5).
10.3.4. Knowledge flows contributing to and generated by the focal innovation
10.29. The types of internal and external knowledge sources of value to innovation activities can differ between those used to identify an idea for an innovation, to develop and test an idea, including problem-solving; and to implement business process innovations or introduce a product innovation onto the market (see section 6.1). Differences in the use or importance of knowledge sources at different stages of the innovation process can be too complex for a respondent to track for all innovations, but it may be possible to include questions on such topics for a single focal innovation. An option is to ask for the knowledge sources of the original idea for the innovation, and the knowledge sources used to develop the innovation. These questions can list both internal and external sources (see Table 6.6).
10.30. It is also of interest to collect data on the contribution of external actors to the development of the focal innovation, such as whether the innovation replicates products or business processes already available on the market, was developed as part of a collaborative agreement with other organisations, or was mainly developed by the firm on its own (see Table 6.2). Further information on collaboration with different types of partners for the focal innovation can also be of value.
10.3.5. External factors influencing the focal innovation
10.31. The effect of some external factors can vary by the type of innovation (see Chapter 7). External factors of interest include the type of customer and customer engagement in a focal product innovation, the use of government support policies and other external drivers for the focal innovation.
10.32. Questions on innovation obstacles can be applied to the most important innovation or to a focal ongoing or abandoned innovation project or an innovation that did not meet expectations. This information can be used to differentiate between the factors that impede the implementation of an innovation, result in unsatisfactory outcomes, or result in an innovation project being cancelled or put on hold.
10.3.6. Objectives and outcomes of the focal innovation
10.33. Innovation objectives and outcomes can vary substantially by the type of innovation and therefore it can be useful to collect this information for a focal innovation. Table 8.1 provides a list of common innovation objectives and outcomes, such as increasing customer satisfaction or reducing environmental impacts that can be measured on a nominal or ordinal scale. Data collection for quantitative outcomes is particularly suitable for a focal innovation because respondents should find it easier, compared to all innovations combined, to provide data on the innovation sales share in the reference year, the market share or profit margin for a focal product innovation, or the cost savings for a focal business process innovation.
10.34. Data on all types of outcomes can also be collected by asking respondents if a specific outcome of the focal innovation was above, at the same level, or below the outcome level typically obtained by the firm for other innovations of the same type. For instance, respondents can be asked about the relative outcome of a focal product innovation on sales compared to the firm’s other product innovations.
10.35. The factors that influence outcomes can be investigated if data on inputs and innovation activities are also collected for the focal innovation.
10.4. Summary of recommendations
10.36. The decision to include an object-based module in an innovation survey depends on the needs of users, particularly policy analysts and researchers, and if there are sufficient available resources to conduct analyses of the object data, for instance on the effect of inputs and strategies on outcomes. An object module is not recommended if use of the relevant data is limited to constructing aggregate indicators. Recommended questions for an object-based module are given below. Other types of data covered in this chapter are suitable for specialised data collection exercises.
10.37. Key items for data collection using an object-based module include:
-
define the focal innovation as the most important innovation with respect to its expected contribution to the firm’s economic performance (subsection 10.2.1); or the most important change for non-innovative firms (subsection 10.2.2), providing an open-ended description if possible
-
the type of innovation (subsection 10.3.1)
-
a measure of the novelty of the innovation (subsection 10.3.1) and the sources of knowledge contributing to the innovation
-
the year in which the innovation was introduced on the market or implemented in the firm’s business processes (subsection 10.3.2). This will be implicit if the observation period is one year
-
the time span between the beginning of the relevant innovation project or activities and implementation (subsection 10.3.2)
-
a measure of the efforts made towards the innovation by the firm, such as the total expenditure (in currency units or person-months) on the focal innovation (subsection 10.3.2)
-
the contribution of internal and external actors to the development of the focal innovation, in order to identify potential success factors (subsection 10.3.4)
-
an outcome measure such as the innovation sales share for a focal product innovation or cost savings from a focal business process innovation (subsection 10.3.6).
10.38. Supplementary topics for data collection using an object-based module include:
References
Arora, A., W.M. Cohen and J.P. Walsh (2016), “The acquisition and commercialization of invention in American manufacturing: Incidence and impact”, Research Policy, Vol. 45/6, pp. 1113-1128.
Arundel, A. et al. (2016), “Management and service innovations in Australian and New Zealand universities: Preliminary report of descriptive results”, Australian Innovation Research Centre (University of Tasmania) and LH Martin Institute (University of Melbourne).
Arundel, A., K. O’Brien and A. Torugsa (2013), “How firm managers understand innovation: Implications for the design of innovation surveys” in Handbook of Innovation Indicators and Measurement, Edward Elgar, Cheltenham, pp. 88-108.
Bloch, C. and M. Bugge (2016), “Between bricolage and breakthroughs – Framing the many faces of public sector innovation”, Public Money & Management, Vol. 36/4, pp. 281-288.
Cirera, X. and S. Muzi (2016), “Measuring firm-level innovation using short questionnaires: Evidence from an experiment”, Policy Research Working Papers, No. 7696, World Bank Group.
DeBresson, C. and B. Murray (1984), “Innovation in Canada – A retrospective survey: 1945-1978”, Cooperative Research Unit on Science and Technology (CRUST), New Westminster.
EBRD (2014), Transition Report 2014: Innovation in Transition, European Bank for Reconstruction and Development, London.
Kleinknecht, A. and J.O.N. Reijnen (1993), “Towards literature-based innovation output indicators”, Structural Change and Economic Dynamics, Vol. 4/1, pp. 199-207.
O’Brien, K. et al. (2015), “New evidence on the frequency, impacts and costs of activities to develop innovations in Australian businesses: Results from a 2015 pilot study”, report to the Commonwealth, Department of Industry, Innovation and Science, Australian Innovation Research Centre (University of Tasmania), Hobart, www.utas.edu.au/__data/assets/pdf_file/0009/772857/AIRC-Pilot-survey-report-for-DIS_Dec_2015.pdf.
O’Brien K, et al. (2014), “Lessons from high capability innovators: Results from the 2013 Tasmanian Innovation Census”, Australian Innovation Research Centre (University of Tasmania), Hobart.
Santarelli, E. and R. Piergiovanni (1996), “Analyzing literature-based innovation output indicators: The Italian experience”, Research Policy, Vol. 25/5, pp. 689-711.
Townsend, J. (1981), “Science innovation in Britain since 1945”, SPRU Occasional Paper Series, No. 16, Science Policy Research Unit (SPRU), University of Sussex, Brighton.