4. Making room for change: Recommendations for Slovenia
This chapter explores key actions that Slovenia could pursue to reform their procurement system. Co-creation sessions with Slovenian officials and stakeholders gave rise to three priority actions: creating communities of practise, fostering collaboration between procurement officials and policy specialists, and creating room for innovation and experimentation. The OECD further recommends three additional actions that are grounded in global best practice to help address challenges that exist in Slovenia.
As part of the work for this report, officials from the OECD Observatory of Public Sector Innovation (OPSI) and the public procurement team, conducted over 50 interviews with a varied and diverse set of stakeholders. Among the interviewees were officials from many different ministries and offices of the central government, including an independent review commission, auditing bodies and anti-corruption officials; sub-national and local government officials and procurement authorities; representatives from the private sector and civil society; journalists; and a member of parliament.
The observations from these interviews were combined with OECD research and a body of work produced by the European Commission, to develop a series of initial findings. These findings were discussed, evaluated and validated in a series of workshops with the participation of diverse representatives from across the public procurement ecosystem. These workshops confirmed some of the key challenges in the procurement system, as discussed earlier in this report, and also identified core areas of opportunity for engaging in reforms. In the workshops, the OECD worked together with participants to co-create solutions that uniquely balanced a number of factors. In particular, the co-created solutions sought to:
achieve change where ambition and support currently exist or could be rallied
address areas where Slovenia has the most immediate opportunity to capitalise on elements of what participants agreed were the “purposes of public procurement” (Chapter 2)
Focus on high-priority areas where innovation is most needed, as determined by the participants.
In addition, when co-created solutions identified by participants as “high priority” were considered against the identified purposes of innovation, discussed earlier in this report, the OECD noted the existence of a few gaps (e.g. they did not generally focus on the purpose of “driving the economy”). In addition, gaps emerged when comparisons were made to the tensions and dilemmas Slovenia faces in the context of global practice and European procurement strategy (e.g. the need to enhance professionalisation of the procurement workforce, which could help support all identified purposes).1 Thus, in addition to the three priority action items identified through the co-creation process, this report provides additional action items that Slovenia could explore as objectives of future reform of its procurement system.
The OECD thus recommends six actions that could help ease Slovenia’s foundational challenges and the systems dilemmas discussed in this report. The first three are priority actions, the last three are additional actions.
Develop procurement communities of practice in order to facilitate connections and the exchange of knowledge.
-
Foster understanding and collaboration among technical experts, policy specialists and procurement officials, in order to move beyond Slovenia’s “hand-off mentality” where procurements are conducted in a siloed and linear manner without a strong focus on outcomes or user needs.
Manage risk and create safe spaces for experimentation to introduce new governance arrangements in relevant procurement process which would help procurement officials engage in experiments without fear.
Enhance the professionalism of the procurement workforce to remedy challenges associated with the current disparity of knowledge and experience across government, and to ensure all procurement officials are equipped with the capacities and opportunities to succeed.
Establish pathways to facilitate the flow of innovation and new ideas across government and bring innovation and new ideas into government from the outside.
Take deliberate steps to drive the economy and enhance competition, in order to more strategically leverage the 10% of GDP that Slovenia spends on public procurements, with a view to strengthening local companies and achieving a healthy market and improved outcomes.
The following section provides real-world examples of how other governments and stakeholders have sought to implement these actions and are working to achieve similar objectives. While the OECD is not necessarily recommending that Slovenia replicate these examples, these cases may provide inspiration or elements that could Slovenian officials could adapt to their own unique context.
The highest priority item identified in the co-creation workshops was the need for procurement communities of practices and other networks, both formal and informal, to facilitate the exchange of knowledge. Indeed, throughout the course of the OECD’s engagement with Slovenia for this project, the OECD team noted the lack of mechanisms to bring together officials to learn from one other. The participants themselves remarked that the workshops provided a rare opportunity for them to meet stakeholders from other parts of government, an experience they found valuable and would like to see repeated on a regular basis. While addressing this priority item could have cross-cutting benefits, it would clearly help address challenges associated with “fragmentation versus connectedness”. The ability to share experiences and learning can also assist with the “legal versus behavioural” dilemma by increasing the visibility around potential actions.
Around the world, countries are increasingly setting up effective yet often relatively simple networks and communities of practice to help civil servants transcend bureaucratic silos and fragmented government structures. Such communities or networks help advance implementation in a consistent, unified manner. In fact, a recent OECD (2016[1]) survey found that that 63% of OECD countries (22 out of 35) have innovation networks in place across the Civil Service. Such a network in Slovenia could focus on innovative procurement, or all types of procurement, to help people connect and learn from one other.
These communities of practice and networks can take a number of different forms; for instance:
They can be led from the Centre of Government (CoG), such as through the Ministry of Public Administration, or be more ground-up and employee-driven
They can be formally structured with governance structures and set processes, or more informal, such as meetup groups
They can be government-only or open to external parties from civil society and the private sector
They can be fully virtual, in-person or a combination of the two.
Many such cases exist around the world that could help inspire Slovenia and provide examples of different approaches. Communities can be started up with little overhead costs; for example, the UK government has built a series of communities for civil servants hosted on Google Groups and through Slack Channels on a wide variety of topics, some of which touch on procurement.2 In Canada, GCpedia and GCconnex provide connection points for individuals working in government (see Box 4.1), with different digital discussion groups focusing on a variety of subjects. Portugal’s Common Knowledge Network provides more open collaboration opportunities by inviting non-governmental participants to join the community (Box 4.2); and in the Netherlands, a one-stop-shop for procurement officials takes the same approach for targeted areas of procurement (Box 4.3).
GCpedia
The Government of Canada has developed GCpedia, an open source government-wide wiki for collaboration and knowledge sharing. It allows federal employees to share files and post, comment and edit articles placed on GCpedia by their peers, helping to break down walls between departments that are traditionally siloed. While access is available only to those with a government e-mail address, limiting the possibility for third-party collaboration, the tens of thousands of active users within government are a testament to the collaborative power of the platform.
GCconnex
The Government of Canada also created GCconnex, an open source government-wide internal social media network, designed to help public servants build connections and collaborate. Users are able to connect with other public servants with similar interests or with skills that can help them become more productive in their work. The systems aims to foster a public sector culture of collaboration and to promote the creation of information that is streamlined, relevant, user-driven and integrated.
Sources: Government of Canada (2016), “GCTools: Re-imagined for you”, www.canada.ca/en/treasury-board-secretariat/corporate/news/gctools-reimagined.html; Janelle (2009), “GCPedia a success, says Government of Canada CIO”, https://techvibes.com/2009/10/06/gcpedia-a-success-says-government-of-canada-cio; GCConnex on GitHub: https://github.com/tbs-sct/gcconnex.
The Common Knowledge Network is a collaborative network built by the Portuguese government to promote the sharing of best practice and information about modernisation, innovation and the simplification of public administration. Membership of the network is open to public bodies, central and local administrations, private entities and any citizen who wishes to participate. Participation involves presenting and describing a best practice and its results. The network aims to become a central reference point for the dissemination of good practices and lessons learned. It currently hosts over 500 examples of best practice documented from all levels of government.
The network also serves as a place to conduct debate on public policies and their implementation at local, regional and national levels, as well as for participatory decision making with interest groups or communities of practice. It works to strengthen relationships between the various stakeholders and co-ordinate information sharing.
Lastly, the network helps participating government organisations obtain a common perspective on the activities of public administration, with a view to standardising services and identifying similar quality standards in different services.
Source: www.rcc.gov.pt/Paginas/Home.aspx.
The Public Procurement Expertise Centre (PIANOo) of the Ministry of Economic Affairs operates a one-stop shop called the Dutch Public Procurement Expertise Centre, which centralises a large amount of information related to procurement and tendering. In fact, PIANOo is recognised as the main website for procurement and tendering in the public sector, gathering reliable, accruable and up-to-date information. Public procurement practitioners use the website to share information and good practices, although PIANOo also offers other information services such as a bi-weekly procurement e-newsletter, a weekly legal review and summary of case law, model templates and online tools. On average, each month the one-stop shop has 50 000 visits and around 13 500 downloads.
In addition, the platform provides direct access to the PIANOo forum, a community of 1 600 procurement professionals, and to TenderNed, the Dutch government’s online tendering system. Available online material includes an innovation procurement toolbox, factsheets on various procurement subjects and sector-specific guidance, as well as a dedicated Q&A section.
Sources: https://ec.europa.eu/docsroom/documents/32184, www.pianoo.nl/en/public-procurement-netherlands.
In another example, the United States Federal Enterprise Data Resources portal (Box 4.4) serves as a hybrid community of practice and open, living policy guidelines to ensure that all stakeholders have access to the same uniform guidance. While this example focuses on open data, such a community for procurement could exist in Slovenia and help to ensure alignment across government and sectors. It could also provide all involved in procurement with the possibility to improve policy implementation, and the community with an opportunity to share ideas and build on each other’s work.
The Federal Enterprise Data Resources portal is a one-stop shop for resources related to federal data management. It serves as an online repository of policies, schema standards, tools, best practices and case studies to help government organisations implement national data strategy and policies. The portal replaced the Project Open Data site, which had similar functionality.
The portal is accessed via the code repository and social media platform GitHub, through which the White House co-ordinates and collaborates with government officials and the public to continually innovate around the implementation of US data strategy.
As a community resource, the portal facilitates adoption of federal data resources. Through GitHub, anyone – government employees, contractors, developers or the general public – can view, contribute and communicate through threaded discussions. Resources and staff are dedicated to collaborating and communicating with users, reviewing feedback and revising policy guidance based on feedback, as needed.
Source: https://resources.data.gov.
Beyond simply serving as a community, such an approach is unique in that it serves as a source of living policy guidance able to adapt and be iterated upon as the government and open data ecosystem grows, learns and evolve. The traditional approach to developing policy guidance often involves a central team of skilled civil servants drafting guidance based on their knowledge, experience and research. At the end of the process the guidance is published, typically in PDF form, for government agencies to follow. However, sometimes the policy and guidance has flaws or gaps that become apparent, and there is little or no ability to make adjustments as implementation evolves and the context changes. The traditional approach also limits iterative learning based on experiences that accumulate during implementation.
The contexts in which governments operate today are complex and change in rapid and potentially unforeseeable ways. The traditional approach of issuing inflexible guidance increasingly is no longer suitable. The Federal Enterprise Data Resources portal is an example of an attempt to leverage a community to produce guidance for agencies that is iteratively refined and enhanced based on lessons learned. Models such as this and the other examples presented could allow Slovenian procurement officials from experienced teams to share their processes and templates, and help procurement officers from smaller authorities (e.g. schools) to access and re-use them. They also enable procurement officials to communicate and work from the same baseline of knowledge.
Governments are increasingly developing or otherwise supporting public sector and cross-sector communities of practices and collaboration networks within their countries. Momentum is also increasing for international collaboration communities. One such example is the informal and growing OneTeamGov Global community (Box 4.5), which Slovenian procurement officials could freely join. By participating in international dialogue at a future OneTeamGov Global event, or even forming a chapter or its own, the Slovenian Government could promote informal interactions and learn from others internationally.
The OneTeamGov community “is made up of people who are passionate about public sector reform” with an emphasis on improving the services offered to citizens and the way in which they work.
Originating in the United Kingdom, One Team Gov has since expanded with individual chapters in multiple countries (e.g. Canada, Finland, Norway and Sweden). In addition to country-specific networks, the group has built the OneTeamGov Global platform to grow as an international community and bring together public sector reformers from around the world.
The main highlight and opportunity for participants to interact and collaborate is an annual OneTeamGov Global “unconference” (participant-driven events without an agenda). This international event brings together hundreds of people from numerous countries to engage on a wider variety of topics, including procurement.
Sources: www.oneteamgov.uk/mission, www.oneteamgov.uk/global, https://oecd-opsi.org/one-team-gov-global-event, https://twitter.com/hashtag/OneTeamGovGlobal.
Communities and networks can also be very informal. Box 4.6 presents #ProcurementHour, an example of a highly informal procurement community of practice in the United Kingdom. The Government of Slovenia could engage in a similar open communications community, or could encourage ground-up communities, by making civil servants aware that such open dialogue is acceptable and even encouraged. By leveraging Twitter instead of a proprietary or closed solution, #ProcurementHour operates in a space with an existing broad base of users. This reduces the hurdles associated with enticing users to use a system with which they are not familiar. Because the dialogue is open, non-governmental actors can also participate in the discussion, enabling questions to be addressed and resolved, and government, industry and civil society to operate on the same page.
The concept of #ProcurementHour was created by agile procurement expert David Kershaw and senior local government procurement official Mark Culley. It is designed to be a public and open space where procurement officials and stakeholders from all sectors can come together to exchange thoughts and answer questions on any topic related to procurement. At a specific date and time, people converge on Twitter, using the hashtag #ProcurementHour, to discuss specific procurement themes.
The founders explained their motivation in an interview with industry magazine Supply Management. Culley stated that, “The biggest problem I see in procurement at the moment is everyone’s a boss of somebody and we’re trying to move that out of the way and reduce the hierarchy so people feel free to ask questions.” Co-originator Kershaw added, “Especially in the public sector we talk about open, fair and transparent procurement but if I say to my colleagues, ‘Do you have a Twitter account or a blog talking about your procurement’, the answer is, ‘Oh no we don’t do that because it’s locked in a cupboard’. Well how is that “open”? Surely you should use the Internet to be more open.”
#ProcurementHour is held every two weeks, with a different theme for each session. Potential themes envisioned include relationship management, market engagement, data analysis and legal aspects.
Sources: www.cips.org/en/supply-management/news/2019/july/time-has-come-for-procurementhour, https://twitter.com/hashtag/ProcurementHour.
A second key theme throughout the OECD interviews was the disconnect between technical and programme staff, who have specific expertise in their policy and managing domains, and procurement officials, who are responsible for procuring goods and services for these domains. This is a core aspect of the “fragmentation versus connectedness” systems dilemma (see Chapter 3), with interviewees and workshop participants commonly referring to a “hand-off mentality”, where procurements are conducted in a siloed and linear manner. Programme staff spend time developing their requirements, and then give the file to procurement officials to execute the procurement process. Interviewees and workshop participants reported little communication between these groups, and noted that each operated with a different set of objectives and a different vision of what constitutes “success”. Challenges in this area also contributed significantly to the “strategic versus tactical” dilemma (Chapter 3), as staff involved in procurement often move from one procurement to the next without broader strategic considerations.
Developing procurement communities of practices, as discussed above, would likely assist in relieving some of this tension, as long as the community is designed through engagement with both technical and procurement officials and guided by higher-level strategic principles. However, participants in the co-creation workshop stated that additional, targeted actions were needed to bridge the gap between these groups and to help change the dynamics of the “hand-off” culture.
Instead of structuring procurement processes around separate, disconnected linear processes, with different types of people associated with each step, Slovenia could move towards a model where multi-disciplinary teams work together from the beginning of the procurement process through to the end, with each having shared goals. Such goals could be communicated at team, organisational and broader strategic levels. This approach would mean including both programme officials and procurement officials from the outset, as well as other potentially relevant stakeholders.
OPSI (2019[2]) research has shown that multi-disciplinarity is one of the most critical factors for the success of projects, especially those where innovative or different approaches are used, or projects involving technology. As Slovenia seeks to leverage procurement approaches that move beyond traditional procurement processes, this will become even more important. OPSI has recommended that, at the outset of any project, governments should convene a group consisting of the skilled individuals necessary to make the process a success. Such individuals could include policy analysts and advisors, field experts, user-experience designers, software developers, attorneys and, of course, procurement officials. While their level of engagement may vary throughout the lifecycle of a project, they should have the ability to become involved and provide feedback throughout the process, from design and initial requirement gathering through to implementation and evaluation.
The OECD recently published an ICT commissioning playbook that emphasises this point. It states that in order “to design user-driven procurement processes and contracts, public sector organisations should form multidisciplinary teams with a set of different capabilities. Teams should ideally include procurement and commercial capabilities as well as user-centred service design and agile delivery capabilities, from the beginning of the commissioning process.”3 The playbook also calls for government teams to “set the context”, in order to understand how things fit together in a more strategic way. Although this playbook was drafted with ICT in mind, these particular approaches are applicable for all procurement activities.
Shifting to more strategic procurement practices further emphasises the need for multi-disciplinary collaboration. As procurement becomes more of a strategic tool, the focal point of action across the lifecycle of the approach shifts. As a result of this change, effort is greater at the pre-tendering and contract execution phase, including planning, research and analysis, than at the contract management phase (Figure 4.1). This means that procurement officers will need to work together with content matter experts and policy experts beforehand, to ensure that any research and preparation respects the rules of procurement, but also allows for exploration and thorough canvasing of possibilities.
Governments are increasingly recognising the importance of multi-disciplinarity to achieving success in mission outcomes and a more strategic approach. The example of the Service Innovation Lab in New Zealand shows how the government has developed mechanisms for multi-disciplinary experimentation (Box 4.7). Meanwhile, the city of Ghent, Belgium views collaboration with multi-disciplinary actors and experts as a core component of its integrated procurement strategies (Box 4.8). Finally, Data61 (Box 4.9) provides an intensive example of how Australia is promoting multi-disciplinarity around the topic of Artificial Intelligence. However, significant improvement can be achieved through much simpler mechanisms, for example, by encouraging programme staff and procurement officials to work together from the early design stages through to implementation on the basis of broad strategic aims.
Sources:(OECD, 2017[3]; OECD, 2019[4]).
The Service Innovation Lab is an all-of-government neutral space that enables public sector organisations to collaborate on innovations in order to facilitate public access to government services. It serves as a design and development lab to experiment, drive and enable systemic change in government for the benefit of society, focused on the needs of the user. The Lab also works to direct public funding towards systemic improvements, horizontal efforts around shared goals, high-value reusable components and actionable innovation for all participating public sector organisations.
The Service Innovation Lab collaborates with agencies and partners across New Zealand to promote greater innovation throughout the public service. Importantly, it reinforces the importance of multi-disciplinarity and ensures that there is a skilled and multi-disciplinary core team at the centre of every project. The Lab provides an example of cross-agency, multi-disciplinary teams working to experiment, address systemic barriers to innovation and prototype new approaches to integrated service delivery designed around user needs. It therefore offers an example of how governments can bring together diverse actors in order to adopt an agile and adaptive approach to systemic innovation.
Sources: https://oecd-opsi.org/innovations/the-service-innovation-lab, https://trends.oecd-opsi.org.
The city of Ghent has developed different strategies that are combined and translated into real-life procurement, thus enabling municipal authorities to use its purchasing power to make changes simultaneously on several fronts. Together with internal and external experts, stakeholders and citizens, the city's leading political coalition developed an ambitious mission statement, which was converted into different strategies and plans, including a procurement strategy.
The integrated strategies implemented by the municipality rely on support at the political level and require a long-term vision and awareness of the bargaining power of public procurement practitioners, together with the necessary room for trial and error.
Another important element has been collaboration with experts belonging to different functions and professions within the municipal government, as well as partners outside the city government in academia and professional organisations, as well as other local authorities across the European Union.
Implementation of these integrated strategies also demands co-operation with economic operators and other stakeholders through market consultation, in order to better translate the goals embedded in the strategy into practical matters.
Finally, the integrated strategies include building up expertise on particular subject matters (i.e. food procurement strategy for local schools) or grouping similar topics so that practitioners can acquire specialised knowledge and skills.
Source: https://stad.gent/smartcity-en.
Data61 is a “data innovation network” with “porous boundaries” designed to allow multi-disciplinary members of the network from various backgrounds to lend their expertise to different projects and programmes, including those concerning Artificial Intelligence. In an interview with The Mandarin, Data61 CEO Adrian Turner stated that the model enabled the organisation “to tackle larger-scale, multi-disciplinary work in a way that we couldn’t if it was just us and our employees.”
Through collaboration agreements, Data61 has grown to become a combined network of 1 100 individuals, including experts from 32 universities, as well as civil servants. Turner emphasises “that raw technical capabilities must be combined with domain expertise in whatever sector they are applied, such as government, health, or agriculture”. The expertise convened includes those who understand and can execute on procurement.
Using a multi-disciplinary approach could help Slovenia move away from a hand-off culture and ease tensions associated with the country’s “strategic versus tactical” systems dilemma. However, additional measured may be needed to align goals and measures of success among programme staff and procurement officials. In interviews and workshops, procurement officials stated that success, for them, was defined simply in terms of completing the procedure and/or obtaining the lowest price. Obtaining the optimal solution that best suits the mission and strategic goals was generally not a consideration. Building multi-disciplinary teams therefore also requires aligning the goals and incentives for all team members.
Australia has sought to align goals and incentives by introducing a Procurement Awards for Excellence programme at the national level (Box 4.10). While focusing on procurement, the awards also emphasise positive outcomes delivered through procurement, and not simply outputs related to completing procedures and obtaining a low price. Awards and other forms of recognition could help Slovenia to raise the profile of public procurement officers and also highlight good and innovative practices in the field.
The Government of Australia established the inaugural Commonwealth Procurement Awards for Excellence in 2019 to recognise leadership and commitment to excellence in procurement. The awards are a new initiative, and provide an opportunity to recognise the positive outcomes delivered to citizens, businesses and the Australian Government through procurement. The nominations period lasted about five weeks and covered four categories:
Building entity capability. This category recognises leadership or excellence in building procurement capability to ensure that all procurements deliver value for money.
Delivering innovation through procurement. This category recognises innovation in delivering an innovative outcome through procurement.
Engaging with risk. This category recognises excellence in engaging with and managing risk in procurement, with a focus on identifying, managing and appropriately apportioning risk to deliver value for money.
Engaging with small and medium enterprises. This category recognises strategies or approaches that maximise the potential for SMEs to engage and participate in procurement.
A panel of judges from inside and outside the public sector was convened to evaluate nominations. The judges used a set of category-specific criteria to evaluate the nominations. Shortlisted nominees were given an opportunity to give a 10-minute presentation to the panel, which then selected the winners.
A government information pack stated that, “The Awards program will help promote the important role procurement plays in delivering services and outcomes for government, citizens and the business community; develop case-studies to showcase the commitment to excellence and achieving value for money outcomes demonstrated by entities; and build a community of highly engaged procurement professionals.” It also outlined the following benefits of participating in the awards programme:
contributing to activities aimed at showcasing better practice, effective private sector engagement, innovation and excellence in procurement.
The government announced the award winners in November 2019.
Source: www.finance.gov.au/procurement/cope-awards-nomination.
Finally, in addition to encouraging understanding and collaboration between programmatic and procurement roles, a number of governments have developed innovative human resources programmes to give individuals real-world experience of serving in different roles. Temporary movement of staff from one agency to another or from one position to another can help staff gain new experiences, provide access to new skills, build horizontal relationships and help build a broader understanding of their work and the work environment. It can also expose staff to different ways of approaching innovation. Offering procurement officials the chance to work in a programmatic role in Slovenia (or vice-versa), would give them the opportunity to learn more about and understand the values, strategic aims, principles and practices of other roles. Such an approach could take place within one organisation, or could be structured to allow people to experience work in other organisations in the public sector.
Slovenia already has some experience in engaging in innovative human resource programmes. The country’s “Partnership for Change”4 initiative represents an innovative practice, based on building a strong partnership between the business sector and public administration. The main objectives of the programme are to overcome the gap between these two worlds, enhance understanding about the different goals and views, establish knowledge transfer between organisations and build a strong partnership for addressing common challenges. Slovenia could perhaps develop a programme to better connect the technical and programmatic worlds with the world of procurement.
A number of government initiatives and models exist around the world that Slovenia could consider as it explores such a programme. Canada’s Free Agents programme (Box 4.11) is an example of a large centralised effort. Slovenia could also pursue smaller and/or decentralised initiatives where employees are seconded to other roles temporarily.
The Canada Free Agents (CFA) programme is a model for internal talent mobility that offers federal public servants the autonomy to select work that matches their skills and interests, and allows them to make contributions that they find meaningful across the Public Service. It also supports managers looking to rapidly and easily acquire top talent in emerging and core skills, in order to support short-term project needs.
Dozens of Free Agents exist today, mainly located in central government in Ottawa as well as regions across the country. If applicants are accepted into the CFA selection process, they find their first assignment and are then deployed to one of the three home departments that manage administration of the programme. Free Agents work on assignments across the Federal Public Service that vary in length (generally between 6-12 months) and organisation (100+ federal organisations) with all details outlined in an agreement. They have access to far-reaching networks with broad skillsets and opportunities for learning and development. They are supported by a Talent Manager who provides career advice and assists them in identifying assignments. After each assignment, Free Agents are eligible for promotions based on their experience on assignment. Not only does the programme offer flexible workforce mobilisation, it also serves as a way to share practices and expose the Public Service to new perspectives and knowledge within the organisation.
Source: OECD (2018), The Innovation System of the Public Service of Canada, OECD Publishing, Paris, https://doi.org/10.1787/9789264307735-en.
One of the biggest challenges uncovered through the work for this report in Slovenia is a significant aversion to risk. Procurement officials – while often seeking to ensure “legitimacy and trust in the process” – tend to use traditional procurement processes that are well understood and minimise personal risk, even when these processes may be less likely to yield optimal results and other processes (e.g. pro-commercial procurement, innovation partnerships) are permitted. Risk avoidance is also the main reason why many procurement officials use price as the only real criterion when selecting a winning bid (see the section on the “strategic versus tactical” systems dilemma in Chapter 3). This behaviour reflects the tension between perception versus reality, with procurement officials habitually conducting procedures that seem safer and less likely to be perceived as corrupt, even though the end results may be less likely to succeed. Procurement officials, and Slovenia in general, are not alone here. Risk aversion is perhaps the most classic and commonly discussed challenge for public sector innovation.
As is common in risk-averse cultures, the OECD observed some inconsistency in what interviewees believed was permitted under procurement rules. What some believed to be allowed (e.g. challenge-based procurements), others believed to be forbidden or even illegal. Even those that believed innovative approaches to procurement were possible stated that few officials, if any, were conducting these types of procurement (see the section on the “legal versus behavioural” systems dilemma in Chapter 3 for a discussion of these types of perspectives). These challenges with regard to perceptions versus reality are well documented in research (Pykett et al., 2016[3]). However, the view that public servants are facing insurmountable barriers to policy change and innovation may not be entirely substantiated in reality. Research has found that a perceived lack of flexibility may be a more powerful barrier to taking initiatives than actual barriers in law or the lack of financial resources (de Jong, 2014[4]).
As a result of the challenges brought about by fear and risk aversion, the co-creation workshops identified a pressing need to create mechanisms to better understand and manage risk, including the creation of safe spaces to allow officials to experiment and try new approaches. Rather than changing procurement procedures themselves, which generally already include provisions for flexible and innovative practices, these mechanisms focus on communications, education, training, governance arrangements and creating exploratory spaces.
A number of governments have sought to manage problems associated with risk aversion by providing uniform guidance for all relevant stakeholders to help ensure the same baseline knowledge of what is and is not allowed, as well as tools and resources to undertake permitted agile and innovative processes. For example, the US government has issued the TechFAR Handbook to document existing flexibilities for agile procurement and show how they can be achieved (Box 4.12). In another example, the Australian government has developed ClauseBank to provide a repository of pre-vetted contract clauses that procurement officials can cut and paste (Box 4.13). Slovenia could develop such guidance and a repository for innovative and agile contract language, templates and tips on how to use them for use by communities of practice. The mechanism should also stimulate administrations testing new approaches to promote them to others. This could help fulfil procurement needs and crowd-in resources in more efficient and effective ways, as well as help to overcome challenges associated with several systems dilemmas by providing easy-to-access baseline knowledge.
In government, digital service projects too often fail to meet user expectations or contain unused or unusable features. Several factors contribute to these outcomes, including the use of outdated practices and, in some cases, overly narrow interpretations of what is allowed by procurement and acquisitions regulations.
The TechFAR Handbook highlights flexibilities in the US Federal Acquisition Regulation (FAR), a 2 000-page document, and can help agencies enact procurement practices for goods and services in in an agile way that meets the human-centred principles laid out in the US Digital Services Playbook. TechFAR discusses relevant FAR authorities and includes practice tips, sample language and a compilation of FAR provisions relevant to agile software development. The handbook focuses particularly on how to use contractors to support an iterative, customer-driven software development process, as is routinely done in the private sector.
TechFAR acknowledges that successful procurement relies on the involvement of multiple stakeholders, including programme officials, IT specialists, procurement officials and agency legal counsel. It is designed to facilitate a common understanding among these stakeholders of the best ways to use acquisition authorities to make investments and to set common expectations and maximise the likelihood for success.
The US government also launched the TechFAR Hub to connect individuals conducting procurement and acquisition with those wishing to do so. The Hub forms part of a greater collaborative effort to create a cross-government community on this topic, and provides access to field guides, case studies and methods for collaboration.
While this particular example focuses on procuring goods and services related to technology, the underlying concept – providing centralised guidance to clarify what flexibilities exist in procurement rules – could be used for all types of procurements in Slovenia.
Sources: https://techfarhub.cio.gov, https://playbook.cio.gov, https://techfarhub.cio.gov.
To help make procurement and contracting easier, more consistent and efficient, and less complex, the Government of Australia has developed ClauseBank, an evolving bank of standard terms and conditions established for use by procurement authorities in government contracts. ClauseBank includes pre-drafted contract terms that can be used within existing contract templates or in bespoke contracts by government entities. Because the clauses have already been vetted, no additional legal review is necessary if they are used without substantive changes.
At present, ClauseBank includes contract language for about 30 subject areas. It also provides additional guidance on how to use the language, relevant notes to help users understand the context and rationale for the clauses, and points of contact at the Department of Finance for questions.
In another example, Chile is taking a co-creation approach to developing government procurement guidance that identifies potential opportunities for innovation based on the experience of procurement officials (Box 4.14).
In Chile, the national procurement regulator, ChileCompra, is working with the Laboratorio de Gobierno to engage with procurement professionals on the development of new guidelines defining the space available for innovative procurement in existing regulations. The Laboratorio is co-ordinating human-centred design (a development method that helps to ensure a service will be easy to use for end users) workshops in three different government agencies to connect procurement practitioners with the central regulator. The workshops aim to gather insights into approaches to implementing regulations and to co-design new guidelines that make space for innovation.
Source: https://oe.cd/eig.
In addition to providing a common understanding and centralised guidance for achieving innovative procurement, governments can also create safe spaces for public officials to test innovative procurement approaches. Such small-scale experiments are generally already permitted within existing procurement procedures, and enable governments to mitigate potential risks associated with inadvertently breaking rules or even outright failure.
These types of safe spaces are most commonly manifested in the form of small groups of public servants dedicated to innovation. Common examples include innovation labs, incubators and digital service teams. There has been significant growth in these types of groups in recent years, partly in response to the increased complexity of public policy issues, which require new approaches and ways of working. They provide an alternative to governments investing time, money and social capital in large-scale policies and programmes that may fail to achieve the expected results. Innovation labs, for instance, are dedicated spaces for investigating and experimenting through trial and error to understand better what works in public service design and delivery. They often give birth to innovative projects or take the first step in scaling successful ones. As procurement is the lifeblood of government programmes and services, such groups may have a strong focus on procurement, or even be entirely dedicated to public procurement efforts. The United States has made several efforts in this area (Box 4.15). Such spaces help procurement officials test different approaches in a controlled setting without the fear that any mistake made could diminish legitimacy and trust in the process. The knowledge and experience gained can help them more efficiently and effectively meet outcomes, which can enhance public and business trust in the system.
Acquisition Innovation Labs
In 2016, the White House issued a government-wide policy instructing each federal government Chief Financial Officer to ensure it had in place an acquisition innovation lab to help agencies achieve better procurement results for taxpayers. The policy also encourages agencies to participate in pilot programmes to develop digital procurement abilities through hands-on coaching of multi-disciplinary teams.
Procurement Innovation Lab
The Procurement Innovation Lab (PIL) experiments with innovative acquisition techniques across the entire Department of Homeland Security (DHS). The Lab provides a safe space to test new ideas, share lessons learned and promote best practices. It fosters cultural changes that promote innovation and managed risk-taking through a continuous feedback cycle. It answers directly to the Chief Procurement Officer of the DHS.
Through its efforts, the Lab seeks to:
lower the barriers to entry for small innovative, non-traditional contractors seeking to compete for DHS business opportunities
increase the likelihood of successful outcomes by focusing on evaluation techniques that help obtain the most qualified vendors.
To ensure a systems-wide perspective, the Lab includes an innovation advocate from each division of the DHS.
The Lab also maintains a website with case studies, webinars, handbooks and an innovation resource library to help spread innovation throughout the DHS, as well as a dashboard presenting current PIL innovation projects.
Sources: www.dhs.gov/sites/default/files/publications/March%202016%20Memo.pdf, www.dhs.gov/pil.
In addition to innovation groups such as those discussed above, some governments have created special “sandboxes” that enable innovations to be tested in environments where certain rules are relaxed. For the most part, these have been regulatory sandboxes,5 which allow businesses to test innovative products, services, business models and delivery mechanisms in a live environment. The sandbox framework also allows businesses to ask the government to waive or modify rules that have become unduly burdensome or are failing to achieve their objectives, sometimes on a limited or temporary basis. Governments are just starting to explore the use of sandboxes internally to create safe spaces where public sector organisations can try new things on a limited scale without the fear of breaking the rules. The results of such experiments enable governments to identify new ways of working within the rules, or to identify pain points and propose changes to rules in areas where the burden of compliance outweighs the potential benefit. Although there are few examples of public sector sandboxes, the concept does appear to be advancing, especially for tech projects including AI (Box 4.16). With approval from parliament, other policy makers and potentially the European Commission, Slovenia could pilot similar sandboxes for experimentation. These could serve specifically to promote the use of innovative procurement processes, or focus on core programmatic mission goals, with procurement as a core factor for success.6
Estonia
In July 2019, Estonia adopted the Estonian National AI Strategy (“Kratts” strategy). One aspect of this strategy is the development of pilot projects benefiting from more flexible public funding and the creation of sandboxes to test and develop public sector AI solutions and accelerate their uptake. These sandboxes would provide regulatory flexibility and temporary access to testing infrastructure resources (e.g. high-performance data processing). The strategy also emphasises the need for technological as well as regulatory sandboxes.
Lithuania
In April 2019, the Government of Lithuania published a national strategy to modernise and expand the current AI ecosystem in Lithuania and prepare the nation for a future with AI. A key recommendation is the development of a regulatory sandbox to allow the use and testing of AI systems in the public sector for a limited timeframe. This would allow developers to test products in a live environment and allow the public sector to determine which solutions can be fully integrated.
Finland
Finland’s AuroraAI strategy calls for a regulatory sandbox to experiment with citizen-authorised data in a controlled way, and to explore whether any legislative changes are needed to achieve the full potential of the AuroraAI national public sector AI strategy.
Sources: https://e-estonia.com/estonia-accelerates-artificial-intelligence, www.riigikantselei.ee/sites/default/files/riigikantselei/strateegiaburoo/eesti_tehisintellekti_kasutuselevotu_eksperdiruhma_aruanne.pdf, http://kurklt.lt/wp-content/uploads/2018/09/StrategyIndesignpdf.pdf, https://vm.fi/documents/10623/1464506/AuroraAI+development+and+implementation+plan+2019%E2%80%932023.pdf.
Professionalising the procurement workforce across Europe is one of the European Commission’s top strategic procurement priorities. It involves clear and concrete action to transform public procurement into a powerful instrument in each EU country’s economic policy toolbox, leading to substantial benefits in procurement outcomes (European Commission, 2017[5]). The priority is seen as so critical that in October 2017, the European Commission adopted a Recommendation on the Professionalisation of Public Procurement7 to encourage EU countries to take steps to increase the professionalism of contracting authorities (see Box 4.17).
The EC Recommendation encourages the development and implementation of professionalisation policies in the Member States, by offering a reference framework for consideration. It focuses on three key areas, each with a number of underlying action items. These key areas and a selection of action items for Member States are given here. The complete Recommendation can be found at the link.
1. Defining the policy for the professionalism of public procurement
Develop and implement long-term professionalisation strategies for public procurement, tailored to their needs, resources and administrative structure.
Encourage and support contracting authorities in implementing the national professionalisation strategies, developing initiatives, and securing institutional architecture and co-operation.
2. Human Resource – improving training and career management
Identify the baseline skills and competences any public procurement practitioner should be trained in and possess, taking into account the multidisciplinary nature of procurement.
Develop appropriate training programmes – initial and lifelong – based on data and needs assessment, as well as on competence frameworks where available.
Develop and support the uptake by contracting authorities of sound human resources management, career planning and motivational schemes, in order to attract and retain qualified staff and encourage them to deliver quality procurements.
3. Systems – providing tools and methodologies
Encourage and support the development and uptake of accessible IT tools.
Support and promote integrity, at individual and institutional level.
Provide guidance to give legal certainty on EU and national law or requirements, and to facilitate strategic thinking, commercial judgment and intelligent/informed decision making.
Source: http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32017H1805.
Likewise, the OECD Recommendation on Public Procurement (OECD, 2015[6]), which all OECD member countries have adhered to, states that governments need to “develop a procurement workforce with the capacity to continually deliver value for money efficiently and effectively”.
Despite its strategic importance, governments have struggled in this area. The most prominent weaknesses in public procurement systems are lack of capability in the workforce (defined as the skills-based ability for an individual, group or organisation to meet its obligations and objectives) and lack of capacity (defined as the ability to meet obligations and objectives based on existing administrative financial, human or infrastructure resources). Challenges facing public procurement practitioners include the transition from an ordering function to a more strategic one, increasingly complex rules, the multidisciplinary nature of the profession and the lack of professionalisation (OECD, 2017[7]).
A public procurement workforce with adequate capacity and capability is crucial for achieving the strategic objectives of government organisations. Professionals who possess a wide range of skills and competencies, including negotiation, project management and risk management skills, are necessary for the successful delivery of strategic procurement initiatives (OECD, 2017[8]). Moreover, the skillset required of procurement professionals needs to be flexible, as the contexts and priorities involved in their everyday work are constantly changing. Many procurement professionals work in roles that demand high-level strategic, tactical and operational skills (OECD, 2013[9]).
The OECD has developed a checklist (OECD, 2016[10]) for the overall Recommendation, which also outlines steps that can be taken to build capacity (see Box 4.18). In addition, the European Commission has recently finalised a professional competency framework for civil servants, with a major sub-component dedicated to procurement professionalisation (see Box 4.19). These resources could assist Slovenia and other countries in strengthening procurement professionalization, which can be seen as a systemic, underlying factor hat significantly affect all other purposes of procurement identified by Slovenian officials.
The checklist provides three key steps that countries can take to strengthen their procurement capacity, each with sub-principles and a checklist of actions. The following text touches on the key steps, sub-principles and a selection of action items. The full checklist can be found at the link.
High professional standards for knowledge, practical implementation and integrity
Develop a public procurement capacity strategy and action plan:
Develop a competency framework, job profiles and a certification system tailored to public procurement:
Have a unit or team that covers the capacity-development needs of the public procurement workforce, able to:
Develop and implement public procurement training on a regular basis:
Provide central advisory services or help desks to answer questions:
Ensure that public procurement entities meet high integrity standards from the procurement workforce to handle ethical dilemmas:
Attractive, competitive and merit-based career options for procurement officials
Provide attractive career options for procurement officials:
Provide competitive and merit-based career options for procurement officials:
Collaborative approaches with knowledge centres:
Improve the capacity of procurement entities through collaborative approaches with knowledge centres (e.g. universities, think tanks and policy centres).
Source: www.oecd.org/governance/procurement/toolbox/search/checklist-implementation-oecd-recommendation.pdf.
The EC has developed a set of instruments that support administrations in their efforts to improve their administrative capacity for management of funds, by helping them to identify and address potential competency gaps.
The subject-specific Competency Framework for Public Procurement Officials aims to valorise the procurement profession as a strategic function and to make it fit for future challenges. This voluntary tool is provided by the EC to help contracting authorities, public procurement authorities and training organisations identify and address competencies that require strengthening. It can help self-assess individuals’ skills and strengths, identify gaps and training needs, design and plan a personal development and career path, and improve performance. At the organisational level, the Competency Framework can be used to assess and enhance organisational and personnel performance, and to ensure a highly capable procurement function able to respond to an organisation’s policy priorities.
The framework’s Competency Matrix, which outlines competencies and skills that public procurement professionals should possess depending on their role (e.g. public procurement specialist, category specialist), includes 30 competencies grouped into two categories:
Procurement-specific competencies are necessary at each stage of the public procurement lifecycle.
-
Professional competencies are the soft skills that public procurement professionals should have in order to master procurement-specific competencies.
Within each category, competencies are further grouped into competency clusters, as shown below. These include different competencies (e.g. planning, innovation procurement, negotiations, market analysis, tender evaluation, ethics and compliance, collaboration, project management, etc.).
The Competency Matrix is structured around competency descriptions, which provide a detailed explanation of each competency, and proficiency-level descriptions, which set out the level of competency an individual should be able to demonstrate. The proficiency levels are: 1) basic, 2) intermediate, 3) advanced, and 4) expert. For each proficiency level, the Matrix describes the competencies an employee should possess.
The Competency Framework for Public Procurement Officials also includes:
Implementation Guidance offering examples and cases for how the tools can be used
a Self-Assessment Tool that public procurement professionals and organisations can use to assess their levels of proficiency and organisational maturity
a Self-Assessment User Guide which explains the technical steps necessary to perform a self-assessment at individual or organisation level
a Generic Training Curriculum which shows how public administrations can upskill their procurement professionals.
Source: https://ec.europa.eu/regional_policy/en/policy/how/improving-investment/competency, European Commission officials" https://ec.europa.eu/regional_policy/en/policy/how/improving-investment/competency, European Commission officials.
A number of governments have put in place projects, initiatives or resources to enhance the capabilities and capacities of their procurement staff. The European Commission has aggregated many of these and published a library of 90 examples of good practices and tools (European Commission, 2018[11]).8 In addition, Box 4.20 provides an example of an innovative approach to professionalising the procurement of digital services from outside the European Union.
In an effort to become a smarter buyer of technology, the US government decided to train procurement specialists to understand the digital and IT marketplace, agile software development methodology, cloud hosting and user-centred design, among others, having determined that potentially all of the government’s 6 500 acquisitions employees needed training on these subjects.
To achieve this objective, in 2015, the White House Office of Federal Procurement Policy (OFPP) and the US Digital Service (USDS) hosted a USD 360 000 challenge on Challenge.gov, a platform run by the federal government where government agencies can post challenges and offer prizes for solutions. Challenge.gov provides a Toolkit that includes information and resources to guide government employees working on challenges. It includes guidance on each challenge phase from preparation to execution, information on different types of challenges, case studies and a list of resources for more detailed guidance and support.
In this case, the challenge was for a vendor to develop a comprehensive training and development programme for procurement and acquisitions officials, to enable them to understand and apply strategic thinking, industry best practices, market place conditions and acquisition strategies to the procurement of digital services. Small amounts of money we provided to multiple vendors to design detailed concept programme proposals. On proposal was selected and the vendor was awarded a larger sum of USD 250 000 in milestone payments to fully develop and pilot their proposed programme with actual procurement officials.
As part of the pilot, 54 acquisitions and procurement employees have now been trained on new techniques. Graduates from the initial cohort, armed with new skills and capacities, are also helping to spread knowledge about best practices they learned from the course to other parts of their agencies.
While the training programme began as a pilot, it is now being scaled up and rolled out. Findings from the initial training cohorts also fed into the development of a new specialisation and certification programme for federal procurement officials. The White House has issued a deadline of 2022, by which date all digital service purchases over USD 7 million will require the expertise of a procurement officer trained through the programme.
Sources: www.usds.gov/report-to-congress/2017/fall/procurement, www.challenge.gov/toolkit, https://federalnewsnetwork.com/acquisition-policy/2018/05/ofpp-sets-2022-deadline-to-train-acquisition-workers-to-buy-digital-services, www.actiac.org/groups/proposed-digital-service-contracting-professional-training-and-development-program-challenge (mirror of original challenge).
Finally, some governments have also sought to professionalise external providers, such as SMEs, to equip them with the knowledge and resources necessary to better navigate procurement processes. Box 4.21 provides an example of such an initiative from Italy that was disseminated through the European Commission’s good practice library. Such professionalisation outside government coupled with procurement professionals inside government could assist Slovenia in optimising the results of procurements and foster positive relationships across the procurement ecosystem.
In 2003, Italy launched a national initiative for the rationalisation of public administrations aiming at streamlining public procurement. As a result, the central purchasing body, Consip, partnered with a nationwide association of enterprises-suppliers to develop a network of dedicated training desks called “Sportelli in Rete”. The overall purposes of the initiative are to:
instruct suppliers in more optimal use of available eProcurement tools, to foster innovation and efficiency
promote use of the eMarketplace tool MEPA among suppliers, especially SMEs
increase the participation of SMEs in Consip’s e-procurement activities, especially at the local and regional level.
The replication model of CONSIP consists of providing training to local associations of suppliers and subsequently allowing them to train local SMEs free of charge. In this context, geographical barriers no longer exist. Training helps to expand business opportunities for both contracting authorities and suppliers, thus contributing to the creation of a multiplication effect.
Italy has set up over 200 SME public procurement training bureaus within 11 supplier associations across the country. These bureaus play fundamental role as reference institutions recognised by local enterprises. The initiative has been popular and training attendance is high. More than 2 250 SMEs have been trained to date. The initiative has also transformed perceptions of Consip, which is now recognised as a body fostering business opportunities in a competitive and transparent environment.
Sources: https://ec.europa.eu/docsroom/documents/32184, www.oecd.org/governance/procurement/toolbox/search/supplier-training-desks-std-str-italy.pdf.
OECD workshop participants co-generated ideas about sharing innovation and new ideas across government, and bringing innovation and new ideas into government from the outside. These ideas generally revolved around supporting experimentation, testing new ideas as part of the procurement process (e.g. prototyping and agile contracts), finding ways to bring attractive innovative companies into the process, and exploring alternatives to traditional procurement to bring good ideas into government. The participants agreed that such ideas were good, but stated that they were not prioritised at the top level. However, based on experience and its alignment with achieving the identified purposes of public procurement, the OECD believes that such ideas hold great promise and can represent new ways of thinking in the Slovenian public service. Doing so can help to ease tensions associated with several of the systems dilemmas discussed earlier in this report. Importantly, finding new ways to surface ideas and identify innovative companies could help improve the foundational economic challenges that Slovenia faces. For instance, building larger blanket contracts for procurement authorities to use, or holding competitive challenges, could increase competition – an issue with which the country has struggled.
One key approach to this challenge is to focus centre-of-government efforts and expertise on building rosters of, or conduits to, proven innovative companies that others in government can easily leverage to bring new approaches into the public sector. The centre of government may be particularly well placed to perform the work needed to clear a path for innovative companies. Box 4.22 provides an example of such an approach.
The Government of Canada has created an AI Source List with 73 pre-approved suppliers “to provide Canada with responsible and effective AI services, solutions and products”. This framework allows government agencies to expedite procurement from firms that have demonstrated a capability to provide quality AI goods and services. Approval of vendors occurred at the central level, and other government organisations are now able to access and rapidly hire pre-screened vendors.
The framework requires suppliers to demonstrate competence in AI ethics, as well as implementation and access to talent. Firms that responded to the “Invitation to Qualify” had to prove to an inter-disciplinary panel that they satisfied these requirements. The framework also has three bands with escalating requirements. The lowest band has less stringent requirements, making it easier for small start-ups to qualify, thereby driving innovation and creating a deeper market.
The framework supports mission-driven and iterative innovation by allowing agencies to commission multiple firms to develop early-stage services to address a problem. This enables effective information sharing and an agile approach, thereby mitigating the uncertainty of potentially disruptive approaches.
The process of establishing and maintaining this list of AI service providers also enables the government to establish longer-term relationships with private companies. Such dialogue facilitates the development of shared expectations and a mutual understanding of the potential challenges facing public sector organisations.
Sources: https://buyandsell.gc.ca/procurement-data/tender-notice/PW-EE-017-34526, https://buyandsell.gc.ca/cds/public/2018/09/21/5e886991ecc74498b76e3c59a6777cb6/ABES.PROD.PW__EE.B017.E33817.EBSU001.PDF.
In addition to bringing in companies with fresh ideas, Slovenia could benefit greatly from innovative methods of procurement. Throughout interviews and workshops, different officials emphasised that all requirements for procurements need to be made explicit up front before the tendering process. However, strict adherence to this belief can lock out interesting and innovative ideas, and for some products such as software, significantly increase the risk of failure. Interviewees and workshop participants also told the OECD that agile procurements rarely occurred. In the United States, agile contracting has been introduced recently (Box 4.23), but has not always been easy or without problems (Mergel, 2016[12]).
The introduction of agile contract formats has enabled the US government to create simple, effective contracts that take advantage of post-award agile methods to procurement. This work, led by the US Technology Transformation Services’ Office of Acquisition, is focused on increasing the adoption of an agile contract format that changes the commissioning process approach. The new approach explains what suppliers are expected to do in regard to one specific project, instead of prescribing how they should do it.
The US government’s 18F group has also experimented with building a pool of agile vendors. 18F created a Request for Quotation (RFQ) where vendors were evaluated based on a working prototype built by each vendor, instead of a narrative document. They ultimately awarded a contract to 17 vendors, both large and small, who were integrated into a “blanket purchasing agreement” (BPA) that allows other government agencies to hire these vendors more quickly and easily. While other agencies could leverage this BPA, 18F documented the development process in order to help other agencies understand how agile contracts can work and pursue similar efforts in their own context.
More information about this approach can be found in the Agile Software Development Solicitation Guide: https://18f.gsa.gov/2019/08/20/an-agile-software-development-solicitation-guide.
Sources: UK GDS (2018), The ICT Commissioning Playbook, Government Digital Service, London, https://playbook-ict-procurement.herokuapp.com (accessed on 25 September 2019); https://18f.gsa.gov/2018/07/26/what-we-learned-from-building-a-pool-of-agile-vendors.
Many governments have also completely flipped the way in which they obtain certain goods and services. In particular, many countries around the world have pursued challenges and competitions to help remedy challenges associated with a lack of fresh ideas. In these processes, instead of specific requirements being stated up-front, government officials articulate an end goal or outcome that they would like to achieve. External parties are then invited to propose their ideas for solutions that can achieve the goal. Canada’s recent policy change (Box 4.24) shows how they took proactive action at a policy level to enable such processes. A number of examples of government challenges and competitions (both programmes and specific projects) are discussed in Box 4.25, Box 4.26 and Box 4.27.
The Public Service of Canada has expressed an openness to doing things in new ways. A key example from a procedural viewpoint is a new policy on transfer payments, which opens up the ability for agencies to use incentive-based funding over a five-year pilot period.
The Treasury Board Secretariat is giving departments new options for distributing government grants and establishing contribution programmes that aim to resolve existing problems. The new TBS approach, entitled Generic Terms and Conditions, applies to all departments and agencies covered by the Treasury Board Policy on Transfer Payments. This policy enables agencies to use incentive-based funding, prizes/challenges and micro-funding over a five-year pilot. These tools will help the Government of Canada make the transition from funding based on tasks and activities to funding based on the achievement of concrete goals.
Source: Government of Canada (2018c), Enabling the innovative use of Transfer Payments, Ottawa, www.canada.ca/en/treasury-board-secretariat/services/innovation/enabling-innovative-use-transferpayments.html.
The United States Air Force launched an Air Force Pitch Day programme in 2019. Modelled after commercial investment pitch competitions (e.g. television’s Shark Tank), the aim is to deliver a faster, smarter approach to competing for ideas in the accelerating technology ecosystem. The process represents a major departure from the lengthy procurement and contracting processes typically expected of the government. It focuses on rapidly awarding small business innovation research (SBIR) contracts to companies based on a simpler streamlined evaluation of proposals papers and in-person presentations. Such grants have special procurement rules to promote innovation under the 1982 Small Business Innovation Development Act,
The first Pitch Day was held in March 2019. In the weeks leading up to the event, Air Force contracting officials reviewed 417 submissions received during the 30-day application period and then invited 59 businesses to pitch their proposals in person. The Air Force subsequently awarded 51 contracts to different companies for a total value of almost USD 9 million. Each of the initial contracts was paid immediately using a government purchase card, with winning contractors being paid in an average of 15 minutes. More than 500 attendees from government, industry, academia, venture capital and investment communities participated in the Pitch Day, which was an open event. About 12 pitch days were held around the United States throughout 2019. Each focuses on a different topic, such as communications systems, space, aviation technology and Artificial Intelligence.
This concept is not limited to defence. The US General Services Administration (GSA) has created the Assisted Acquisition Services Express Programme to provide services to non-military agencies, in order to help them obtain ideas and proposals from the private sector for innovative solutions that may be a good fit for their missions. Special contracting authorities used by this programme can help provide a faster and more efficient route to obtaining innovative products and services in government. GSA helps to pay for the service by charging a fee to participating agencies.
Sources: www.afwerx.af.mil/stories/space-pitch-day.html, www.af.mil/News/Article-Display/Article/1779609/inaugural-air-force-pitch-day-new-contracts-new-partners, www.c4isrnet.com/battlefield-tech/space/2019/06/24/watch-this-space-air-force-announcing-pitch-days , www.govexec.com/management/2018/05/all-agencies-can-now-buy-innovative-tech-shark-tank-speed/148456.
CivTech is the Scottish Government’s challenge programme for innovation. Based on leading technology incubators, CivTech offers entrepreneurs the chance to compete in open challenges designed to identify new companies and increase innovation. By bringing together public sector expertise and private sector creativity, the programme pioneers a smarter, faster approach to public procurement. The objective is to harness entrepreneurial tech innovation and citizen engagement, improve public service delivery, create economic development opportunities and foster an entrepreneurial mindset within government.
CivTech sponsors define specific challenges and then invite the private sector to propose solutions. A handful of proposing teams are selected and given a small amount of funding (GBP 3 000) for three-week exploration sessions. One teams is then selected for a 15-week accelerator, which involves GBP 20 000 in funding and interaction with innovation centres and citizens. After the accelerator phase, minimum viable products (MVPs) are presented at a demo day, and the resulting product(s) are prepared to scale. Because CivTech is oriented around solving challenges, the process allows for the creation of solutions that the government was not aware existed.
With CivTech, Scotland aims to shift from closed prescriptive tendering to open challenge-based questions. In so doing, the government seeks to move away from traditional methods of procurement that favour larger supplier companies able to afford the time and money to navigate their way through the complex public procurement process, and open itself up to small, talented tech companies who are rarely engaged by the public sector.
The government has also launched the CivTech Academy, a structured, comprehensive programme designed to pass on the overall methodology and ethos of the CivTech Programme to partner organisations. The methodology includes the infrastructure, systems and operating principles that need to be put in place, and the knowledge and knowhow required to successfully operate them.
Sources: https://oecd-opsi.org/innovations/civtech-scotland, https://civtech.atlassian.net/wiki/spaces/C4/overview.
The Government of Poland has endeavoured to ensure that public policies are prepared in co-operation with stakeholders, in order to provide new invaluable inputs, innovative ideas and evidence about the problems as well as the solutions. It recognises, however, that not all stakeholders have the capacity, knowledge and resources to provide structured responses and suggestions to policy makers.
GovTech Poland is designed to respond to this issue. This challenge-based procurement model allows any business or citizen to pitch ideas to government, and encourages small businesses to bid on government projects, which may be inspired by the ideas received from citizens. Companies that pitch or can carry out the best ideas win a full implementation contract without the need for an additional tender. The model is designed to open up procurement to all creative individuals, and covers the entire process from identification to implementation.
Ideas are evaluated anonymously by a panel to mitigate potential biases. Participants then progress through competitive rounds designed to select the best projects. In the first round, they must develop minimum viable products (MVPs), then those who make it to the second level create functional prototypes. A small handful of winners is selected and awarded contracts for implementation.
The GovTech programme has already had a significant impact, by helping to address the common challenge of too few vendors bidding on a tender – an issue of particular relevance for the Slovenian context. In an interview with industry publication GovInsider, Justyna Orłowska, Director of GovTech Poland, noted that “usually there are just two or three [bids]”, but the new process has increased this number to “about 50 on average, with a peak of 96” bids, giving the government more options and increasing the potential for positive outcomes. In addition, the majority of winners to date have come from companies with fewer than 250 employees, an outcome that is helping small and medium-size enterprises to break into the public procurement ecosystem.
Besides bringing innovative ideas into government from external sources, it is also critical to enable ideas and innovative practices to flow and diffuse within the public sector. This can be greatly enabled through communities of practice and networks, as discussed as a priority item above. Formalised procedures focused on innovation diffusion can assist, with communities of practice perhaps serving as a vector for these mechanisms. See for example the intentional activities in Denmark around spreading learning and innovation (Box 4.28).
In order to break down bureaucratic silos and help governments create new ways of working, the Centre for Offentlig Innovation (COI) created the Spreading Innovation guide. Its aim is to support institutions and individuals in the process of replicating innovation in the public sector, and thereby reduce the risks and costs associated with innovation. COI designed the guide in three phases over the course of a year. The process involved over 100 people and enjoyed the support of an Advisory Board of handpicked practitioners and researchers.
In the first phase, a team of researchers carried out a literature review of processes for the diffusion of innovation in government.
In the second phase, the COI conducted field studies in 11 Danish municipalities involved in innovation in the area of welfare technology. Using observation techniques and approaches linked to behavioural economics and “nudge”, the COI endeavoured to answer the following question: What happens when institutions spread innovation?
In the third phase, the main findings were tested in other contexts (e.g. business growth immigration services and child care), across all levels of government (central government, local authorities, hospitals, etc.) and even in non-governmental organisations working on welfare innovation.
The result of the process was Spreading Innovation, a guide deeply anchored in concrete experience and research. As a tool, it supports dialogue between sharers (teams who want to share an innovation) and re-users (teams that want to replicate an innovation). Although any interested sharer/re-user can download and use the guide independently, the COI firmly believes that personal relationships are essential for innovation and organises networking events to facilitate matching between potential sharers and re-users.
The guide gives an overview of an otherwise complex process. It is structured around six key steps to help government officials share and reuse an innovation (as shown in Appendix IV). For each of these steps, there is a checklist of proposed actions to take and questions in the form of dialogue tools to guide discussions between teams. This approach enables the Spreading Innovation guide to support the spreading of innovations in government, while respecting institutional conditions that may require reinvention or re-contextualisation of some aspects of innovation processes – for example, taking into account the complexities that can exist between municipal, regional and state workplaces.
Sources: www.oecd.org/gov/innovative-government/embracing-innovation-in-government.pdf, www.coi.dk/en/what-we-do/spreading-innovation.
A number of ideas discussed during the interviews and workshops, although not prioritised like three of the items in this chapter, focused on driving the economy and enhancing competition. These generally involved encouraging companies to compete for tenders, reducing the burden and barriers to entry for SMEs, competing in international markets and attracting cross-border bids. Such actions can help drive the economy and also help ease one of Slovenia’s foundational challenges discussed earlier in this report: economic challenges stemming from the size of its market. Taking action to bring in different and diverse types of companies is important for public sector innovation broadly. Because of its complexity, public procurement can favour incumbents with track records over “lean start-ups” that might actually have the relevant innovation know-how. How can the values of public procurement be aligned with the principles of agile and iterative processes? Initiatives are needed that help put the small and agile on a level playing field with the large and the incumbent, and help them participate in public procurement processes.
A number of countries around the world have developed innovative solutions to try to tackle similar challenges. For example, the government of New South Wales, Australia has piloted ways to help small companies break into the government market (Box 4.29).
The New South Wales (NSW) government has launched the Local Procurement Pilot to make it easier for small companies to break into the government procurement ecosystem. It provides tools to cut red tape and simplify the tendering and procurement process, thus streamlining the process to give small businesses a better chance at winning government contracts.
As part of the pilot, the NSW Government published Doing Business with your Local Council: A Guide for Small Business Owners. The guide discusses how to maximise chances for winning government contracts, including information on how to find out about opportunities, how to demonstrate capability and experience, and how to be environmentally sustainable.
The Local Procurement Pilot also includes a road map and other resources for councils to make it easier for small businesses to win contracts.
Sources: https://www.smallbusiness.nsw.gov.au/what-we-do/our-work/improving-local-procurement-opportunities, www.smallbusiness.nsw.gov.au/sites/default/files/2019-10/14860%20OSBC%20Local%20Govt%20Tender%20Guide%20web.pdf .
The United Kingdom has developed a digital marketplace to make it easier for businesses to engage with government and to help centralise common products and services, in order to take advantage of economies of scale. While many countries have developed digital marketplaces, the United Kingdom is unique in that theirs is “global” and open to use by other countries.
Slovenia could potentially benefit from developing a nationwide digital marketplace of its own for goods and services. Such goods and services would not necessarily need to be related to technology needs, such as in the UK example. To drive its economy, Slovenia could also consider allowing other countries, such as those in the region, to procure items from Slovenian vendors on the marketplace. To enhance competition and thereby help address challenges associated with the size of its market, Slovenia could invite cross-border listings from other countries or even explore the potential of participating as a buyer in the UK’s Global Digital Marketplace (Box 4.30).
The UK Government Digital Service (GDS) launched the Digital Marketplace in 2014 as an online service designed to help public sector organisations find people and technology for digital projects. The aim was not only to help make it simpler, clearer, faster and more cost efficient for government to buy technology, but to also redefine the UK Government’s relationship with the technology market. Suppliers must apply to sell services, and any public sector organisation can buy services or products using the Digital Marketplace. The Digital Marketplace offers an opportunity to support the growth of the UK’s digital sectors, particularly for start-ups and scale-ups. Indeed, a contract from government can transform a small business, giving it credibility, income and the crucial first customer.
The Digital Marketplace has led to clear results. In 2009, fewer than 20 companies retained 80% of the UK’s annual technology spending. As of 1 October 2018, almost 5 100 suppliers are available to the UK public sector through the Digital Marketplace, over 92% of which are SMEs.
Given the success of the Digital Marketplace, the United Kingdom is seeking to use its expertise to support other governments. The Global Digital Marketplace aims to help international governments make their procurement more transparent, thereby helping to prevent corruption and boost their digital, data and technology sectors. Its delivery model includes working in partnership with international and domestic technology service providers and educational institutions in host countries. Global Digital Marketplace project interventions include:
Ensuring plans are made before money is spent. This involves planning, business case development and spending controls, including associated codes of practice.
Designing procurements and contracts. This includes Digital Marketplace commercial routes to market, and associated procurement and contracting reforms, including reviews of domestic regulatory contexts at national and subnational government levels.
Ensuring service delivery. This involves contract awards and managing service delivery and supplier relationships through service assessments and associated standards.
Embedding the Open Contracting Data Standard. This includes supporting the progressive public disclosure of information relating to procurements, and generating detailed information in human-readable and machine-readable formats.
Building capability and capacity. This involves developing new professions within government related to digital, data and technology and associated capabilities frameworks; and building institutional capacities in the Civil Service and private sector through targeted learning and development modules covering the above areas, with training delivered through an academy model focused on integrity, procurement reform, digital service delivery and government transformation.
Sources: https://oecd-opsi.org/innovations/global-digital-marketplace; www.digitalmarketplace.service.gov.uk.
Finally, driving the economy can also mean using the government’s purchasing power to shape the products offered by vendors and promote the strategic goals of government. Promoting social goals is considered to be one of the most important purposes of the procurement system in Slovenia, as discussed earlier in this report. Slovenia could use its procurement function to effect change and signal the government’s desire to drive social change. Victoria, Australia’s Social Procurement Framework, provides a strong example of this approach, as discussed earlier in Box 2.7.
References
[43] Amann, M. and M. Essig (2015), “Public procurement of innovation: empirical evidence from EU public authorities on barriers for the promotion of innovation”, Innovation: The European Journal of Social Science Research, Vol. 28/3, pp. 282-292.
[27] Baldi, S. et al. (2016), “To bid or not to bid: That is the question: Public procurement, project complexity and corruption”, European Journal of Political Economy, Vol. 43, pp. 89-106.
[34] Barrad, S., R. Valverde and S. Gagnon (2018), The application of system dynamics for a sustainable procurement operation., Springer, Cham.
[54] Book, M., V. Gruhn and R. Striemer (2012), “adVANTAGE: A fair pricing model for agile software development contracting”, Agile Processes in Software Engineering and Extreme Programming, Vol. 111, pp. 193-200.
[21] Brugnach, M. and H. Ingram (2012), “Ambiguity: the challenge of knowing and deciding together”, Environmental science & policy, Vol. 15/1, pp. 60-71.
[17] Burns, T. and F. Köster (2016), Education Governance in Action: Lessons from Case Studies, OECD Publishing, Paris, https://doi.org/10.1787/9789264262829-en.
[16] Burns, T. and F. Köster (2016), Governing Education in a Complex World, OECD Publishing, Paris, https://doi.org/10.1787/9789264255364-en.
[56] Chapman, J. (2002), System Failure: Why Governments Must Learn to Think Differently (2nd edn)., Demos, London.
[36] Cingolani, L. and M. Fazekas (2017), Methods Paper on Administrative Quality.
[4] de Jong, J. (2014), Dealing with Dysfunction: Innovative Problem Solving in the Public Sector, Washington D.C.: Brookings Institution Press.
[67] Director General for Communications (2019), Flash Eurobarometer 482: Businesses’ attitudes towards corruption in the EU.
[48] Djankov, S., F. Saliola and A. Islam (2016), Is public procurement a rich country’s policy? World Bank Blogs., Washington, DC: Peterson Institute for International Economics.
[42] Edler, J. and L. Georghiou (2007), “Public Procurement and Innovation—Resurrecting the Demand Side”, Research Policy, Vol. 36/7, pp. 949–963.
[24] Edler, J. et al. (2005), Innovation and Public Procurement – Review of Issues at Stake. Study for the European Commission..
[37] Eurobarometer (2017), Businesses’ attitudes towards corruption in the EU. Slovenia, Country factsheet..
[65] European Commission (2020), Slovenia. Key Facts and Figures..
[64] European Commission (2019), Single Market Scoreboard. Procurement., https://ec.europa.eu/internal_market/scoreboard/performance_per_policy_area/public_procurement/index_en.htm.
[11] European Commission (2018), Building an architecture for the professionalisation of public procurement.
[40] European Commission (2018), Commission notice Guidance on Innovation Procurement. C(2018) 3051 final..
[66] European Commission (2018), Public Procurement Scoreboard.
[47] European Commission (2017), COM(2017) 572 final. OMMUNICATION FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS Making Public Procurement work in and for Europe..
[5] European Commission (2017), COUNCIL RECOMMENDATION on the 2017 National Reform Programme of Slovenia and delivering a Council opinion on the 2017 Stability Programme of Slovenia, https://ec.europa.eu/info/sites/info/files/2017-european-semester-country-specific-recommendations-commission-recommendations_-_slovenia.pdf.
[68] European Commission (2017), Measurement of impact of cross-border penetration in public procurement.
[38] European Commission (2017), Special Eurobarometer 470. Corruption..
[62] Government of Slovenia (2018), KOALICIJSKI SPORAZUM O SODELOVANJU V VLADI REPUBLIKE SLOVENIJE ZA MANDATNO OBDOBJE 2018–2022.
[61] Government of Slovenia (2014), KOALICIJSKI SPORAZUM O SODELOVANJU V VLADI REPUBLIKE SLOVENIJE ZA MANDATNO OBDOBJE 2014–2018.
[23] Guenther, E. et al. (2013), “The ´why not’ perspective of green purchasing: a multilevel case study analysis”, Journal of Change Management, Vol. 13/4, pp. 407-423.
[26] Hommen, L. and M. Rolfstam (2009), “Public Procurement and Innovation: Towards a Taxonomy.”, Journal of Public Procurement, Vol. 9/1, pp. 17–56.
[35] Ison, R. and M. Shelley (2016), “Governing in the Anthropocene: contributions from systems thinking in practice?. Systems Research and Behavioral Science”, ystems Research and Behavioral Science,, Vol. 33/5, pp. 589-594.
[41] Lember, V., T. Kalvet and R. Kattel (2011), “Urban Competitiveness and Public Procurement for Innovation”, Urban Studies, Vol. 48/7, pp. 1373-1395, https://doi.org/doi:10.1177/0042098010374512.
[19] Loosemore, M. and E. Cheung (2015), “Implementing systems thinking to manage risk in public private partnership projects”, International Journal of Project Management, Vol. 33/6, pp. 1325-1334.
[57] Mazzucato, M. (2011), The entrepreneurial state: Debunking public vs. private sector myths, Anthem Press.
[45] McKinsey & Company (2017), Government Productivity Unlocking the $3.5 Trillion Opportunity,.
[12] Mergel, I. (2016), “MAgile innovation management in government: A research agenda.”, Government Information Quarterly, Vol. 33/3, pp. 516-523.
[58] Nannicini, T. et al. (2013), “Social capital and political accountability.”, American Economic Journal: Economic Policy, Vol. 5/2, pp. 222-250.
[52] OECD (2019), Productivity in Public Procurement: A Case Study of Finland: Measuring the Efficiency and Effectiveness of Public Procurement, OECD Publishing, Paris.
[14] OECD (2019), Enhancing the Use of Competitive Tendering in Costa Rica’s Public Procurement System STREAMLINING THE EXCEPTIONS AND REDESIGNING THE THRESHOLD SYSTEM., OECD Publishing, OECD.
[71] OECD (2019), Government at a Glance 2019, OECD Publishing, Paris, https://dx.doi.org/10.1787/8ccf5c38-en.
[31] OECD (2019), Improving Lithuania’s Public Procurement System, COMPONENT 1 – IMPLEMENTATION OF PROFESSIONALISATION AND CERTIFICATION FRAMEWORKS, OECD Publishin, Paris.
[18] OECD (2019), OECD (2019), Reforming Public Procurement: Progress in Implementing the 2015 OECD Recommendation, OECD Publishing, Paris,, https://doi.org/10.1787/1de41738-en.
[72] OECD (2019), Public Procurement in Germany: Strategic Dimensions for Well-being and Growth, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/1db30826-en.
[33] OECD (2019), Public Procurement in Malta - Re-engineering the Department of Contracts, OECD Publishing, Paris.
[30] OECD (2019), Public Procurement Toolbox.
[29] OECD (2019), Reforming Public Procurement: Progress in Implementing the 2015 OECD Recommendation, OECD Publishing, Paris,, https://doi.org/10.1787/1de41738-en.
[59] OECD (2018), Data from the 2018 OECD Survey on the Implementation of the 2015 OECD..
[28] OECD (2017), Public Procurement in Chile: Policy Options for Efficient and Inclusive Framework Agreements, OECD Public Governance Reviews, OECD Publishing, Paris,, https://doi.org/10.1787/9789264275188-en.
[13] OECD (2017), Public Procurement in Chile: Policy Options for Efficient and Inclusive Framework. OECD Public Governance Reviews., OECD Publishing, Paris., https://dx.doi.org/10.1787/9789264275188-en.
[32] OECD (2017), Development and implementation of a national e-Procurement strategy for the Slovak Republic., OECD Publishin, Paris.
[73] OECD (2017), Government at a Glance 2017, OECD Publishing, Paris, https://dx.doi.org/10.1787/gov_glance-2017-en.
[74] OECD (2017), OECD Guidelines on Measuring Trust, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264278219-en.
[44] OECD (2017), Public Procurement for Innovation: Good Practices and Strategies, OECD Public Governance Reviews,, OECD Publishing, Paris, https://doi.org/10.1787/9789264265820-en.
[7] OECD (2017), Public Procurement in Peru: Reinforcing Capacity and Co-ordination, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264278905-en.
[8] OECD (2017), Public Procurement Review of Mexico’s PEMEX: Adapting to Change in the Oil Industry, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264268555-en.
[70] OECD (2017), Systems Approaches to Public Sector Challenges: Working with Change, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264279865-en.
[55] OECD (2017), What’s possible? Finding and filtering innovative ideas, OECD Observatory of Public Sector Innovation..
[46] OECD (2016), Improving ISSSTE’s Public Procurement for Better Results, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/9789264249899-en.
[1] OECD (2016), Survey on Strategic Human Resources Management in Central/Federal Governments of OECD Countries, OECD Publishing, Paris.
[10] OECD (2016), Checklist for Supporting the Implementation of the OECD Recommendation of the Council on Public Procurement.
[60] OECD (2016), OECD (2016), Data from the 2016 OECD Survey on public procurement (partially published)..
[6] OECD (2015), OECD Recommendation of the Council on Pulic procurement.
[15] OECD (2015), Systems Innovation: Synthesis Report, OECD, Paris, http://www.innovationpolicyplatform.org/system-innovation-oecd-project.
[9] OECD (2013), Public Procurement Review of the Mexican Institute of Social Security: Enhancing Efficiency and Integrity for Better Health Care, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264197480-en.
[53] OECD Publishing, P. (ed.) (2017), Embracing Innovation in Government: Global Trends, http://www.oecd.org/gov/innovative-government/embracing-innovation-in-government.pdf.
[2] OPSI (2019), Embracing Innovation in Government. Global Trends 2019..
[20] Peters, G. (2005), “The problem of policy problems. Journal of Comparative Policy Analysis”, Journal of Comparative Policy Analysi, Vol. 7/4, pp. 349-370.
[51] Preuss, L. (2011), “On the contribution of public procurement to entrepreneurship and small business policy”, Entrepreneurship & Regional Development, Vol. 23/9-10, pp. 787-814.
[3] Pykett, J. et al. (2016), Governing mindfully: Shaping policy makers’ emotional engagements with behaviour change., Routledge.
[69] Rolfstam, M. (2012), “An institutional approach to research on public procurement of innovation”, Innovation – The European Journal of Social Science Research, Vol. 25/3, pp. 303–321.
[39] Rolfstam, M. (2012.), “An Institutional Approach to Research on Public Procurement of Innovation.”, Innovation – The European Journal of Social Science Research, Vol. 25/3, pp. 303–321.
[22] Sönnichsen, S. and J. Clement (2019), “Review of green and sustainable public procurement: Towards circular public procurement. Journal of Cleaner Production.”, Journal of Cleaner Production.
[63] Sta (2019), Revizijska komisija razveljavila del razpisne dokumentacije za prenovo Hotela Brdo.
[49] Vogel, L. (2009), “Macroeconomic effects of cost savings in public procurement”, Economic Papers, Vol. 389, https://doi.org/10.2765/29810.
[25] Yeow, J. and E. J. (2012), “Innovation procurement as projects.” Journal of Public Procurement.”, Journal of Public Procurement, Vol. 12/4, pp. 472–504.
[50] Zeng, S., X. Xie and C. Tam (2010), “Relationship between cooperation networks and innovation performance of SMEs.”, Technovation, Vol. 30/3, pp. 181-194.
Notes
← 1. https://ec.europa.eu/docsroom/documents/25612.
← 2. www.gov.uk/service-manual/communities.
← 3. https://playbook-ict-procurement.herokuapp.com.
← 4. www.amcham.si/en/partnership-for-change.html.
← 5. See, for example, www.fca.org.uk/firms/regulatory-sandbox.
← 6. See also an interesting discussion on this topic at www.howtocrackanut.com/blog/2017/5/22/procurement-sandboxes-mock-procurements-and-some-other-thoughts-on-trying-to-create-space-for-practical-real-world-experimentation.
← 7. http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32017H1805.