3. System analysis of the public procurement system
This section outlines the procurement system as it is understood today by the civil servants in Slovenia. It includes a discussion on legal reforms that have set the formal agenda, but it mainly focuses on the behaviours of public actors and is based on their lived experiences and realities. The section also touches on issues that have created systemic challenges that need to be tackled.
Systems analysis examines the functioning of a system and the various elements that affect the behaviours of the actors inside. Rather than analysing the laws, regulations and policies that drive the system, the OECD takes a more holistic perspective to determine how the various purposes, stakeholders, rules and intervening factors relate to one other.
The OECD’s previous work on systems has given rise to the observation that while policy and laws are critical factors, perceptions, relationships and an absence of perspective often have an equal or greater effect. Two dominant factors that influenced the procurement system were the media (which drove the narrative with politicians and citizens) and procurement reforms. However, as this chapter explores, the laws provide opportunities for innovation within the system that are often ignored, and confusion persists about what is allowed. Additionally, the purpose of procurement, explored in Chapter 2, is not reflected strongly in the various reforms or media stories. Instead, there is a focus on eliminating corruption, streamlining the process and transparency. Accordingly, the procurement system pulls actors in different directions. So what does the system actually look like today?
This section seeks to provide insights into how the system functions today. Through missions and interviews in Slovenia, the OECD developed an understanding of current system operations, including: the purpose of public procurement as defined by system actors, the key goals of effective public procurement, the main systems dilemmas and trade-offs connected to the afore-mentioned purpose, the key stakeholders, and intervening factors within the system that facilitate outcomes.
This chapter unpacks some of the key connections that help actualise and explain the various relationships and behaviours of the system. According to the OECD’s system analysis, these relationships centre around the legal framework, dilemmas within the system, foundational challenges, and the engagement and interactions between actors within and outside government.
This view was then validated through multiple co-design workshops. The goal of conducting this activity is not to create value judgements regarding the system as it operates today; rather, the aim of this exercise is to create a shared understanding of the system and how it operates today. This will allow the actors concerned to reflect on the system and determine if it is operating as desired. It will also provide a more holistic approach to allow these actors to identify areas and levers for innovation.
Major procurement reforms in Slovenia and their relationship with politics and the media
The Slovenian procurement system is rooted in law; therefore, it is important to understand the main principles and objectives of the laws themselves. Through workshops and interviews, the OECD observed that the main lever for improving the procurement system in Slovenia was legal reform. Much of the early discussion of the project centred around what is legally permitted and where the law grants explicit permissions to conduct certain activities. An understanding of the law, the major reforms and how they both interact with the system is therefore critical to the analysis. This section focuses on the evolution of the law and major reforms. The ways in which they interact with the system, primarily through oversight organisations, such as the National Review Commission and auditors, are explored later in this chapter.
The foundation of the Slovenian procurement system was the first public procurement law, issued in 1997. This law was replaced in 2000 and then amended extensively through legislation in 2004 to align with EU procurement standards. This section provides a brief history of the various major procurement reforms and also focuses on major news stories shaping the system today. In total, six major procurement reforms have taken place between Slovenia’s accession to the European Union and the writing of this report. As the OECD discovered during interviews, reform is viewed as the only legitimate path towards resolving problems within the system – a mindset that has led to quite frequent reforms.
In August 2006, Slovenia underwent its first major reform after joining the European Union. The reform focused on aligning and harmonising national public procurement rules with the procurement directives of the European Commission. The resulting law introduced better payment discipline between contractors and subcontracts. Additionally, it attempted to reduce some administrative barriers, such as:
the introduction of a simple system for the publication of contracts above and below the European threshold through the public procurement portal
a uniform simplified procurement procedure below European thresholds, with the possibility to negotiate
the introduction of the ability to complete missing elements in the tender offer that are not crucial for consideration.
The reform opened up new avenues to procure goods and services. Additionally, the flexibility to allow for sustainable development as part of a procurement strategy enabled procurement to serve as a market signal for social priorities. While the 2006 reform focused on alignment with the European Union, there remained a need to address corruption, cartels and unfair business practices, all of which were highlighted in the media (see Box 3.1).
According to stakeholders within the Slovenian procurement system, the media has been a major driver of reform in Slovenia, promoting a narrative regarding the ethics and efficiency of procurement through various news stories. These stories have increased political pressure to reform the system and centred attention on reducing corruption and increasing transparency.
One such example is the “floppy disk affair” of 2001. The state Highway Company (DARS) selected a construction contractor for the Podmilj tunnel, and the company provided a yellow floppy disk containing its bid and pricing information. However, the floppy disk presented at the subsequent public opening of tenders was pink, indicating clearly that the contractor had modified the information in the offer to secure an advantage during the selection process. At the request of the State Audit Committee, the bidding company lost the contract.
The floppy disk affair represented a turning point in the construction of the state highway network, and shed a light on controversial practices in the construction business. It exposed how the three largest construction companies acted as a cartel enabling them to win 90% of all public works contracts. The cartel negotiated with each other to increase prices and prevent other contractors from applying for tenders. Bribery also helped to secure contracts. One company leader was even quoted as saying that despite offering the lowest price, they could be paid as much as they wanted through annexes to the contract.
This story formed part of wider media exposure designed to reduce corruption and cartels, and informed the 2006 procurement reform.
Sources: OECD interviews and https://siol.net/siol-plus/kolumne/alenka-bratusek-ima-tezavo-z-gradbenimi-baroni-2-0-komentar-503474.
A subsequent 2007 reform sought to correct some of the systemic issues and restore legitimacy to the system by increasing transparency and reducing corruption. The reform attempted to strike a balance between increasing transparency to reduce corruption and reducing burdens that internal and external stakeholders faced during the procurement process. The changes in the procurement system focused almost exclusively on creating trust in the procurement process itself and providing guidance to increase clarity around practices that affected legitimacy.
The 2009 reform focused more on issues that directly affected businesses, crowding-in resources and getting needs fulfilled. It established the foundation for the price vs. value debate in the procurement system, which is still ongoing. Specifically, this time period saw extensive debate about the role of public procurement in driving social behaviour. This involved a discussion about social values, such as green procurement, as well as ethical and desired behaviours from industry itself, such as addressing issues with subcontracting that existed within the system.
Overall, the creation of this reform helped to establish greater financial discipline across the system, and opened up new opportunities across the multiple facets of procurement. However, the 2009 reform, as with its predecessors, ultimately increased the burden on both government and industry during the procurement process.
This pattern continued with the next reform in 2011. While some innovations within the procurement process sought to provide new opportunities and flexibility, most focused on increasing the burden to reduce corruption and unwanted behaviours in the system. Motivated by contractors attempting to avoid rules, especially regarding payments to subcontractors, the 2011 reform sought to close certain loopholes, and empower contracting authorities with more flexibility and more responsibility.
Throughout the OECD’s time in Slovenia, it became apparent that political decisions and reforms are driven frequently by construction issues involving schools, hospitals, tunnels and others. Box 3.2 highlights the influence of the construction sector in particular on the reforms in Slovenia.
Stakeholders in Slovenia noted that some of the largest, most complex and expensive procurements were found in the construction sector. Errors within these procurements have been amplified by the media, have political implications and can lead directly to reforms, such as legal changes affecting subcontractors, cartels and transparency.
The construction of the Markovec tunnel provides one example of these issues. In 2004, a request was made for over EUR 23 million to cover unforeseen work, delaying the project by over three years. Prime Minister Miro Cerar highlighted the issues at the opening in June 2015: “We have been waiting for this tunnel for a long time. The proceedings were delayed by problems with the legislation as well as the bankruptcies of the companies. The government is aware of the legislative problem, and will make changes accordingly, so that the next construction of a 5-km long section does not take ten years.”
While legislation was indeed an issue, the bankruptcy of an electrical equipment company resulted in a lengthy revision of the tender. The total cost of all the combined challenges during construction was higher than that of a Slovenian competitor.
Another construction issue relates to the Karavanke tunnel and has been ongoing since 2015. The original tunnel opened to traffic on 1 June 1991 and stretches over 7.9 km, 4.4 km of which lies in Austrian territory and 3.4 km in Slovenian. The tunnel is managed jointly by the Austrian motorway operator ASFiNAG and the Slovenian company DARS, and is used on average by 10,859 vehicles per day. Preparatory works for the construction of the second pipe of the tunnel began in 2015, with drilling work foreseen to start in 2022 and a target date of early 2024 for carrying vehicles. The existing pipe will be renovated and the smooth passage of traffic in both pipes is planned for the beginning of 2026.
Construction work on the Austrian side started in September 2019 and is proceeding as planned. After initial preparatory work, ASFiNAG issued a public tender for the work at the end of 2017 – the same time as DARS, the Slovenian motorway operator. However, there have been delays in construction on the Slovenian side. At the end of July 2019, Slovenian DARS selected a Turkish company for the construction of the second tube of the Karavanke tunnel. Their bid of EUR 89.3 million (excluding VAT) was significantly cheaper than the other eight bids. Following the selection of the Turkish contractor, however, the process was halted by three non-selected contractors who filed audit claims.
As of November 2019, the State Audit Committee has rejected the fourth appeal from non-selected companies, but work has still not begun on the Slovenian side of the tunnel.
Sources: OECD interviews, https://siol.net/avtomoto/reportaze/predor-markovec-koncno-odprt-olajsanje-odgovornih-dvomi-domacinov-98500, https://siol.net/posel-danes/novice/avstrijci-zaceli-z-gradnjo-druge-cevi-predora-karavanke-478173.
One important thing to note about the 2011 reform is that, due to austerity measures, this reform actually eliminated the exam and qualifications required to be a procurement officer in the government. Instead, these requirements were replaced by a call for training. This decision was made in response to the shortage of available trained officials to conduct the necessary procurements. However, this de-professionalisation of procurement officials also had negative affects throughout the system in terms of capabilities, empowerment and trust within the system.
The macro and micro-economic environment was also an important motivational factor in many of the changes. An increased focus on competition created more flexibility to ensure local industry remained able to access and compete, and obtain fair pay for work.
In 2014, the 13th Government of Slovenia released its coalition co-operation agreement (Government of Slovenia, 2014[1]). The system was aligned for the first time with EU standards, and procurement was cited as a priority in four different sections:
Information society. Help optimise procurement through greater centralisation, transparency and information systems.
Public finance: Simplify the procurement system, train strong buying professionals and introduce performance criteria that avoid price seeking as the sole criteria for procurement. Additionally, advocate for greater joint procurement in the areas of healthcare, education and IT.
Environment and space: Create new standards for environmental protection and green procurement requirements.
Health: Establish transparent and uniform joint procurement processes to promote greater inclusion and higher standardisation in the health sector.
Alongside these priorities, the coalition agreement also listed numerous projects that would help achieve these goals. Unlike previous laws that ended up increasing burdens on both industry and contracting officials, the 2014 law sought to increase procurement efficiency. Introducing mandatory joint public procurements, increasing the value for simplified procurements and creating exemptions to the standstill period in certain situations, enhanced the capability to getting needs fulfilled quickly, drive more economic activity and reduce burden.
The subsequent 2015 law built on the 2014 EU Directives and focused on the social aspects of the coalition agreement. It simplified procedures, increased flexibility and improved economic efficiency, while introducing sustainability, environmental factors, and social and innovative procurement. The law also considered the social aspects and costs of procurement over the entire lifespan. The integration of these aspects and the principle of sustainability into public procurement is expected to increase the shares of green, social and innovative public procurement and promote greater economic efficiency.
In addition, the 2015 law provided an information-based foundation for the implementation of electronic public procurement in all phases, as called for by the European Directives. This innovation was designed to help reduce the burden for both contractors and providers.
While the procurement system, as defined by law, seemed to create a flexible and trusted system that could improve outcomes, signal social changes and drive the economy, the external view of the system was focused on a handful of expensive and political issues, such as the Markovec tunnel.
In 2018, the 14th Government of Slovenia released its coalition agreement (Government of Slovenia, 2018[2]). However, despite all the alterations to the law, there was an absence of behavioural change within the system. The agreement re-emphasised many of the previous coalition values within the procurement system, namely: the need for a high level of transparency, the use of criteria other than the lowest price, a focus on protecting the environment, and protection and respect for labour rights.
Lastly, on 1 January 2018, Slovenia renewed the Green Public Procurement Regulation. This regulation was created to set forth the conditions for green procurement. Examples of environmental requirements and criteria for green procurement, which are in many aspects similar to the environmental requirements of the previous regulation, were created or updated, and became more flexible and non-binding, giving contracting authorities greater choice as to how to achieve the required target for a particular subject.
The 2014 and 2015 reforms helped to build much of the present public procurement system in Slovenia. However, despite the necessary flexibilities and clarity in the system, innovative procurement remained nascent. Meanwhile, capacity building and best practice documents created by the Ministry of Public Administration did little to increase new procurement practices.
System stewardship and the legal framework
System stewardship refers to the individual or group responsible for steering a system towards certain outcomes. In the case of the Slovenian procurement system, the Directorate for Public Procurement within the Public Administration has the role of system steward. The responsible team drafts reforms and laws related to procurement, conducts procurement for the Public Administration as well as centralised procurements (e.g. utilities) across government, collaborates with industry associations, staffs a help desk for procurement officials and conducts training across the procurement system.
Interviews conducted for this report revealed that this small team often has to balance multiple priorities and emergencies. This has created a situation where diffusion and learning within the system occur through individual relationships rather than systemically across the system. As law and precedence exert a strong influence on the behaviours of actors in the system, the team also attempts to create space for testing new tools and methods. The team hopes that by acting as a proving ground for innovation, even without a mandate, others will see the benefits and adopt new and innovative solutions to better deliver public value through the procurement system.
Aside from conducting procurement, the Directorate is also initiating reforms. However, reforms are a time-intensive and lengthy process, and, as the previous chapter showed, there have been an average of almost one reform every two years since Slovenia joined the European Union in 2004.
As system stewards, the team has worked to build better procurement practices across the system. The team attempts to organise yearly roundtables with other procurement officials, although there is a perception that these take place only during reform efforts. Additionally, the team has partnered with industry associations related to construction and buying to create good practice guidance, although industry representatives had found limited evidence of the use of these documents.
The complexity of procurement rules is not just limited to Slovenia. While Slovenia continues to evolve its own procurement system, there has also been a substantial increase in EU Directives that need to be understood and interpreted within a country’s procurement system.
While system actors viewed the Slovenian procurement system as overly complex, there was very little interest, demand or stated need for further major reforms, even if those reforms helped further simplify the process. While complexity was an issue, the OECD observed that no individuals interviewed for this project cited the law as an impediment to innovative, flexible or effective procurement strategies and methods. There was an understanding that procurement reform will always be needed as new challenges arise, but that the law provides the necessary framework for meeting the purposes of procurement and allowing space to do things differently. Individuals reflected that the process could be simpler, but overall, stakeholders did not view the law as a key contributor to current challenges within the system.
Intergovernmental Influences
In the case of a complex, large bureaucracy such as the public sector, engagement on the part of actors is not limited to the laws and policies within a specific system (i.e., procurement-specific laws and policies). Public procurement officials interviewed for this report discussed the influence that budget and finance processes and laws have on procurement priorities, timelines, and approaches. For instance, since the approved budget covers only two years at the national level and one year for local government, all procurement actions must be planned and executed within that time period, in spite of the fact that many procurement processes take significantly longer. Interviewees also noted the influence of non-procurement related governmental processes and tasks in creating an environment where only limited time is available for strategic planning. Because procurement is seen as a time-intensive and lengthy process, procurement officials working in such environments are forced to rapidly create procurement strategies that prioritise award speed above all else. This may also explain why innovation procurement and pre-commercial procurement are uncommon, as these procedures usually take more than two years to complete.
Oversight organisations that serve to enforce the rules and ensure procurement is fair and unbiased also play a large role in the system. According to OECD missions, research and interviews, oversight organisations such as the Court of Audits, the Revision Commission, the Competition Commission and the Corruption Commission have the greatest effect on the system. These organisations are responsible for issuing findings that set new standards, some of which may appear contradictory. Indeed, many stakeholders cited a disconnect between procurement law and the interpretations of oversight organisations. However, as oversight organisations are responsible for final rulings during disputes, their decisions become a form of unwritten law driving behaviour across the system.
Outside government pressures
One factor relatively not yet touched on in this report is the role of actors in the business sector. Competition is key to better outcomes, better value and a mature procurement system. However, for a small country such as Slovenia, lack of competition can be a threat to effective procurement.
In a smaller economy, fluctuations within business engagement can have a more dramatic effect on public procurement competition. As the macro-economy shrinks, businesses seek more steady and stable government contracts; however, as the macro-conditions improve, businesses look for new (and perceived easier) means to obtain revenue. Interviews also highlighted a countrywide trend towards smaller companies with a perceived lack of incentive to grow. As a result, these small businesses often do not bid outside of their own municipality.
Additionally, there is a perception among Slovenian businesses that the government is concerned more with pricing and less with value. Those engaged in the procurement process perceive a lack of expertise among government buyers, while those less engaged view the government market as difficult to penetrate and more favourable towards those with existing strong relationships. This results in a business environment with few new entrants, a “race to the bottom” culture regarding price without consideration for outcomes, and a lack of formal or informal engagements with industry outside of the procurement process.
Finally, external factors are strongly related to corruption. As per the previous section, many procurement reforms, aside from those necessary to transpose EU Directives, were driven by media stories depicting some form of corruption. While this has contributed to a less efficient system, it also greatly affects behaviours within the system. Media stories that involve seemingly topic-specific areas such as medicine or construction can have ripple effects on all aspects of procurement.
Behaviour and culture
Despite the stated goals and purpose of procurement, intergovernmental influences, outside pressures and oversight organisations have shaped a system focused on following a specific process – one that defines success in terms of awarding contracts quickly on the basis of the lowest possible price, and avoiding any innovative activity that could delay the process. Overall, the OECD observed that procurement seems to be treated less as a strategic asset to achieve better results, and more as a means to an end where speed, the absence of risk and low prices are the primary concerns.
Even though innovative approaches are not required for every single procurement process, but rather target specific product categories (e.g. ICT, health, infrastructure, etc.), innovative solutions are still perceived as risky and slower than regular procurement. Furthermore, there are no incentives to conduct procurement processes with increased risk. Doing new things implies necessary learning and mistakes, however the time constraints and demands made on procurement officials undermine the willingness to use any innovative approach that may slow down a process, even if it means better results. This applies not only to the use of methods besides open procurement, but also to testing out different methodologies for assessing value beyond price or procurement of innovation.
As mentioned earlier in the report, the Slovenian procurement system is based heavily on the respect of law. The written law is seen to grant permission to conduct certain activities; without a law these activities would be considered impermissible. In such a system, doing new things may be perceived not only as difficult, but illegal. The challenge is that in addition to slowing down procurement, innovation may also undermine the entire process, potentially resulting in the need to restart the entire activity.
In the end, challenges and rulings that cause delays are considered failures within the procurement community. When the OECD asked procurement and programme officials what success looked like, the most common answers were: 1) a fast awarding process, 2) a lack of appeals or legal challenges, and 3) dismissal of any appeal or legal challenge, if lodged.
In one case, the procurement office defined success in terms of achieving its procurement plan on time and spending its budget.
These issues, combined with other pressures within the system, create a risk-averse environment that focuses on speed over value. Doing new things is not actively encouraged or discouraged, but instead is rarely even considered. Instead, procurement officers seek paths to simplified procurement below thresholds to increase speed and limit complexity.
There is also little thought about the specific outcomes that procurement is supposed to achieve. Moreover, the prevailing “hand-off” culture between programme staff (e.g., requirements developer) and procurement officials involves limited feedback loops and learning. Even within the procurement community itself, there was little evidence of learning or engaging with each other to share knowledge.
Any system has certain fundamental challenges that cannot be solved over the short term, and are unlikely to be resolved in the long term. However, such macro-challenges should be defined explicitly in order to better understand how they alter specific challenges within the system. Many of these fundamental challenges are driven by macro-economics, politics, competition and sustainability issues.
Economic challenges
In 2017, Slovenia had the 6th smallest share of GDP in the European Union (Figure 3.1). As a small country, the challenges that its procurement system faces differ from those of larger countries.
Source: Eurostat.
Slovenian officials told the OECD that the size of the market significantly limits competition. This is supported by European Commission work that indicates a likelihood of lower competition for contracts from domestically located firms in small countries (European Commission, 2017[3]). The Slovenian market alone is unlikely to be able to provide the broad range of products, services and innovative solutions necessary to meet the societal challenges the government is trying to address. Additionally, the small size of the market serves to discourage large, multi-national companies from setting up operations that could bring innovative solutions to the public sector. For example, the OECD’s missions found that many of the major innovative technology providers have little or no presence in Slovenia.
Even with this dynamic, public procurement still has a significant influence on the market, accounting for 10.01% of GDP in 2018 (Ministry of Public Administration of Slovenia, 2019). At present, the Slovenian government is trying to determine how to properly allocate such a large percentage of GDP. A balance needs to be found between supporting the Slovenian market, upholding the overarching principles of non-discrimination and equal treatment embedded in EU treaties, and creating a competitive environment that will allow the public sector to procure the greatest value while meeting its needs.
This does not mean that such tensions are a constant presence or that solutions provided within the Slovenian market represent the best value. However, as Slovenia continues to seek innovative solutions, it may need to look outside its borders for major innovation investments. Because the public sector has the ability to drive investment, serve as a market signal and support Slovenian economic growth, it is important to keep a watchful eye on this balance and tension.
Disparity of knowledge and experience
In many governments there seems to exist a strong desire to have a strong cohort of professional experts across each core function of government. As procurement is a core function vital to the success of government, public sectors are continually seeking a strong community able to recruit, train, retain and reward procurement talent. This is a never-ending challenge within the system.
According to the OECD interview with Slovenia’s Chamber of Purchasers, developing strong buyers in the private sector requires knowledge of a specific subject matter, time, training, relationships and experience, and the process often takes over two years. However, government faces many challenges that do not affect the private sector:
Existing relationships with certain companies present a risk of appearing corrupt.
The diversity of products and services required provides little time for the government procurement specialist to build up the necessary expertise.
Little time or budget is allocated to learning due to high workloads and fiscal austerity.
The procurement expert is not required or expected to have specific knowledge of the products or services they are procuring.
There are also challenges around retention and career paths within procurement in the Slovenian system. Procurement is perceived as a job characterised by political pressure, stress and limited flexibility or demand for creativity. Procurement officials in Slovenia do not feel empowered and often find themselves in no-win situations trying to balance the needs of the government with a very strict interpretation of the procurement process. During workshops with the OECD, participants report that this dynamic created a disengaged cohort of procurement specialists that often eventually left their job for a new challenge.
Even when individuals do stay in procurement positions, there appears little upward trajectory with clear career paths. Expertise in procurement is valued but does not lead directly to career advancement. Instead, many of the best procurement officials leave the procurement field to seek new opportunities, with few new experts, either through recruitment or training, available to fill the void.
As discussed earlier in relation to system stewards, capacity building and dissemination of best practices within the procurement system is minimal. The Public Administration has created best practice documents, but these have enjoyed limited application – although it is unclear if this is due to lack of awareness or no real appetite for recommendations that differ from current procurement methods.
Procurement officials are not alone in this challenge. The OECD has observed these same challenges in many specific and specialised professions in the public sector. A profession needs a strong community that is able to recruit, build and retain the necessary experts. But even if expert development systems are mature, there will always be a tension between building the future and executing today. Systems must attempt to balance both of these functions and seek creative ways to make the system more resilient.
Centralisation versus decentralisation
The debate around whether to centralise or decentralise functions spans government functions including procurement, ICT, budgeting and others. In Slovenia, discussion focused on how to strike the right balance between centralising procurement programmes, practices and activities at the centre of government (CoG) and respecting the autonomy and contextual awareness of non-CoG ministries and sub-national procurement authorities. The idea of centralised procurement was discussed as a means to drive down costs, reduce repetitive activities, and become smarter buyers in medicine, technology, schools and utilities.
Slovenia has not created a “perfect formula” to determine when to centralise procurement. The idea of centralising procurement was discussed primarily as a method to use economies of scale to drive down costs, while saving time on conducting multiple unnecessary procurements. However, as the Competition Commission in Slovenia noted, this approach can also have unintentional adverse effects.
As an example, if there are four major providers of a product needed by every school in Slovenia, because each school procures this product individually, the market will be sufficiently decentralised to allow all four providers to succeed. If purchasing becomes centralised, the cost per product will likely decrease over the short term due to the size of the procurement through a single provider. However, this situation could drive the other providers out of business, resulting in higher prices the next time the product is needed, due to lack of competition. This hypothetical scenario simplifies many factors, but it helps to illustrate the potential trade-offs that a small market like Slovenia must consider.
This issue has already been realised in the construction and education sectors, and is likely to continue. Because of the size of the public procurement market, centralising procurement can dramatically reduce immediate costs, but could have adverse long-term effects on the system.
One approach discussed in OECD workshops for dealing with this challenge was to focus CoG efforts and expertise on building repositories and other resources that others can leverage to make procurement easier and more consistent across the public sector.
The traditional approach to developing government policy guidance, such as guidance on conducting public procurements, involves a central team of skilled civil servants drafting the policy and guidance based on their knowledge, experience and research. It also generally involves getting input from other stakeholders in government, and maybe conducting a public consultation. At the end of the process, the policy and guidance are published, typically in PDF form, for government agencies to follow. However, sometimes the policy and guidance has flaws or gaps that become apparent, and there is little or no ability to make adjustments as implementation evolves and the context changes. The traditional approach also limits iterative learning based on experiences that accumulate during implementation.
Openness of the procurement process
The hierarchy of procurement procedures represents a trade-off between the amount of competition versus a more targeted approach. Procurement procedures in general, thus, canvas open procedures, restricted tenders and direct awards. The more open the process, the more the process enables new entrants such as start-ups and SMEs to engage, thus encouraging wide participation. Open procedures can include processes such as competitive dialogue and phases. However, they also require significant investment of time and resources on the part of procurement officials. As procurement becomes more targeted, more specific strategic exceptions can be made. This may also mean that (based on thorough market research) specific partners for the procurement process can be selected that match the needs the procurement processes are trying to meet. The ability to do this well depends on the ability of procurement authorities to perform market research and identify qualified and capable suppliers up front. However, it may provide a disincentive for suppliers to submit competitively priced bids. At the same time, it can speed up procurement processes in the case of specialised goods and services, in limited supply market conditions, emergencies and so on.
Public scrutiny and accountability versus risk avoidance
Creating a transparent process is critical to ensuring fairness within the system and building public trust. Public scrutiny is a pillar of democracy and should therefore be actively encouraged. However, public scrutiny can also dramatically shape the perceptions of the system.
As stated in the purposes of procurement, being stewards of taxpayer funds is a core tenant of a strong and effective process. While the Slovenian government performs thousands of procurements a year, the public’s perceptions are generally shaped by a select few that are generally political, complex or both, and therefore have a high likelihood of creating negative perceptions, such as corruption or lack of competence (see Box 3.3). This perception not only influences public opinion, but also encourages procurement officials and authorities to avoid mistakes. This is, of course, a positive outcome, but an associated side-effect is that it discourages officials from taking any risks and creates disincentives to explore innovative working methods.
The renovation of the Brdo Protocol Hotel is considered a critical project for Slovenia in the light of its forthcoming Presidency of the European Union in July 2021. The project is valued at EUR 26.2 million with work projected to take 20 months. Accordingly, the renovation will need to begin soon to ensure completion on time. However, the State Audit Committee partially granted the audit request of the construction company and cancelled part of the tender.
Bids were supposed to be open in June 2019, but were delayed due to the requested review of the tender documentation that claimed the conditions were unlawful and disproportionate. The National Audit Commission also found that some of the requirements violated the Public Procurement Act. These challenges would have caused unnecessary burden during the reference process and also limited the ability for companies to subcontract.
With over 150 questions from bidders and the changing of the tender documentation three times, there was a prevailing perception that the government was unsure what it needed, was ill prepared to undertake the project, and lacked the necessary skills to tender appropriately for the contract.
Sources: OECD interviews; (Sta, 2019[4]).
In Slovenia, the depiction of the procurement system in the media has created an antagonistic relationship with government in a context where communication is already limited and information flows are scarce. While there will always be tension between public oversight and the government, it is the responsibility of both parties to create trusted relationships. These relationships may change and shift over time, but the goal should be to ensure that the right information is available and the proper training for public oversight groups is available in order to correctly interpret the information.
While the above challenges are unlikely to be fully resolved, there are unique dilemmas within every system that prevent it from operating optimally. These dilemmas are due to system pressures and challenges that drive behaviours which may be suboptimal. In most cases, these dilemmas are also implicit. They may be hidden in phrases such as “this is how we have always done things.” Because these challenges are not explicit and behavioural, they also are often the hardest to resolve, especially in a legalistic system that seeks explicit permission from the law.
Through interviews, observations, workshops, co-design and co-creation, the OECD has defined seven core trade-offs and dilemmas within the system. While these dilemmas may be interconnected, they each lead to different behaviours and actions within the system. The remainder of this chapter explores these trade-offs and dilemmas.
Strategic versus tactical
As procurement is rarely viewed as a strategic activity in Slovenia, it is not surprising that most of the time, effort and priorities within the procurement system are tactically focused. With limited capacity, procurement officials restrict themselves to awarding a contract and then moving onto the next one. There is little time to consider or test innovative methods; instead, the key question is which of the two readily acceptable procurement paths are available: an open procurement or simplified procedure. Procurement organisations only display evidence of a more strategic approach during the preparation of yearly plans, but even then they have limited time to create the plan or space for testing or new thinking.
Instead, strategic thinking is the responsibility of the Directorate of Public Procurement. This body has served as a model for new procurement techniques, centralised procurement and reforming the system. Much of the strategic thinking and effort within the Directorate concerns reforming the law; however, existing laws appears to already grant much of the space and innovative thinking that the system desires, but is not currently undertaking.
The Directorate has also attempted to drive new and better procurement by partnering with industry on guidelines and standards. But there is little evidence that these standards are being used across the system. They have also partnered with the Association of Purchasers to learn how to strengthen their ability to purchase, but participation has waned as other priorities have intervened.
Taking all of these factors together, the system is not actively testing or trying new things, and strategic thinking is not occurring at the levels necessary across the system. Instead, procurements are focused on cost rather than value. The OECD found evidence of cost accounting for over 80% of weighting in some cases and interviews could not provide examples of when the lowest cost (besides extreme outliers) was not selected due to other factors.
This problem is not limited to the purview of procurement specialists. Even during the requirement stage, there is limited time to understand the market and develop requirements that can determine value beyond price. In such contexts, price becomes the default measure.
Yet, there is a strong desire for procurement to improve, to have the space to try new things, and to reduce the burden inside and outside government. The OECD’s efforts to bring together system actors were rewarded by a high level of engagement and no lack of new ideas for improving the system. The question is therefore not how to come up with new ideas, but rather how to make time to think and act more strategically.
Fragmentation versus connectedness
The Slovenian procurement system currently operates in a fragmented manner, both in terms of how the actors within the procurement system operate and learn, as well as how the process is conducted. This fragmentation leads to tactical thinking, less focus on the mission and outcomes, limited learning and spreading of good practices, and reduced understanding of how the system works.
For this report, the OECD conducted multiple workshops with actors across the system to help co-design challenges and solutions. During these workshops, members of oversight organisations and procurement specialists raised differences of opinions, asked questions and even exchanged contact information with a promise to continue the conversation. While interviewees were keen to correct negative perceptions held by those outside the system, system actors also discussed their own incorrect perceptions of other actors within the system. This became most clear during an activity where OECD asked the system actors to map out the entire lifecycle of a procurement process. This task required a collective effort and involved misconceptions and misunderstanding throughout the process.
With such a small community, this came as a surprise to the OECD. If the experts within key system organisations are not connected, learning and strategic thinking can be difficult to achieve. In such a context with no systemic approach to learning and development, sharing and learning are confined to an individual’s office and network. While the Directorate of Public Procurement has made some effort in this regard, learning cannot only happen in a top-down manner; it must be continuous and occur across the system horizontally and from the bottom up. Other procurement systems also function in a similar manner and have integrated new processes to minimise risks (Box 3.4).
The public procurement system in Germany is highly decentralised with 30 000 Contracting Authorities at all governmental levels, performing about 2.4 million procurement procedures annually. Out of all procurement activities, 58% happen at the municipal level, 30% at the level of the federal states and only 12% at the federal level.
Even though the German system is highly decentralised, specific solutions are developed at the central level:
Prevention of corruption through mandatory training sessions and workshops, in addition to functional solutions such as designating a Corruption Prevention Officer and rotation of employees.
Promotion of sustainability through a dedicated strategy and a Competence Centre for Sustainable Public Procurement which proposes good practices, technical assistance, and training sessions. This helps lead to horizontal integration of knowledge and skills at the federal level and vertical integration at the state level, and promotes networking among administrations and with NGOS and science.
Promotion of innovation with the KOINNO centre of excellence, an innovation award and participation in the Initiative for a European Competence Network of Innovative Procurement.
Centralisation of IT procurement in a single point of contact at federal level through a Central office with a project to develop a central e-Procurement portal.
A decentralised procurement system requires a differentiated approach to aggregation at different levels consisting of central purchasing bodies (CPBs) at central level along with a tendency to perform joint procurement at the local level. Together with the Coordination Unit - Government Purchasing Authority of the Ministry of Internal Affairs (Kaufhaus des Bundes), there are four central purchasing bodies at federal level, which, in terms of professionalisation, provide the following benefits for buyers: 1) specialised know-how and training, 2) standard technical solutions for the federal administration, 3) e-procurement, and 4) price reductions due to higher purchase quantities and favourable contract conditions.
Finally, practical advice is provided to contracting authorities and companies by the Joint Service Bodies of the Chambers of Commerce and Crafts (Auftragsberatungsstellen) at the federal state level.
Sources: www.bmwi.de/Navigation/EN/Home/home.html; www.koinno-bmwi.de.
Fragmentation is also prevalent during specific procurement processes. In both the open and simplified procurement process, there is a clear split in roles and responsibilities that leads to a hand-off mentality. During interviews, procurement officials discussed the clear separation of roles, with project managers responsible for the requirements and market understanding, and procurement officials responsible for executing the procurement process once requirements were done.
This hand-off mentality showed evidence of ignoring key reflections during the process. Rather than a collective effort that served to educate the procurement officials in the product or service being purchased, as well as the programme, in order to better understand the process, there was a lack of insight or empathy for the other side of the process. When the results of the process (time, outcomes, process or selection) were sub-optimal, there was evidence of finger pointing. Procurement officials and industry cited lack of good requirements, but programme officials cited a lack of understanding from the procurement officials of what the programme was attempting to procure.
Legal versus behavioural
Procurement systems generally take two forms: legal-based or principle-based. Legal-based procurement systems rely on the law to grant permission for specific activities. Anything not declared as permissive within the law is, therefore, not permissible. Principle-based systems focus on core principles that outline the broad behaviours, ethics and principles that should guide officials.
Slovenia has a strong legal-based system with little tolerance for mistakes. The OECD’s analysis of the law found clear signals for new thinking, new techniques and creativity, yet these are still missing. This could be due in part to the tactical dilemma outlined above, but there is also a perception that some things are not allowed due to confusion around the appeals process.
This perception stems from the perceived inconsistency of rulings in the Review Commission. This was an issue specifically identified by the interviewed stakeholders and workshop participants. As procurement officials view the Review Commission as relatively unpredictable and there is little tolerance for when things go wrong, there is a lack of ambition for new behaviours and activities. It is widely believed that the Review Commission and the Audit Commission take an overly legalistic view which squashes new ways of working, especially if they do not work the first time.
This issue became particularly evident during a conversation about sandboxes at an OECD workshop. What would a procurement sandbox look like? Would it be legal? If it was challenged or audited, how would those issues be resolved? While there are no clear answers to these questions as of this report, there was a strong demand for a safe space to try new things not explicitly permitted in law. This signalled an interest to test principle-based systems where certain failures would be reframed as learning opportunities for improvement, rather than evidence that something does not work and should not be attempted again.
User-centred versus process-driven
The OECD has identified six core skills for innovation, one of which is user-centricity (Figure 3.2). By understanding the end-users and designing a system that aims to achieve certain outcomes, systems become more effective, more responsive and more user-centred. However, the Slovenian procurement system has rarely adopted a user-centred process.
Source: OPSI, 2019
As a legal-based system, the majority of efforts have traditionally added barriers and burden for users rather than reducing them. One example that emerged during research was the various procurement systems with which users need to engage to effectively conduct e-procurements. As of this report, both procurement officials and providers need to interact with two separate systems for different parts of the process. In some cases, the same information needs to be input into each system, as there is no connectivity between the systems. These issues should soon be resolved by the introduction of a new system which will merge the two other systems. Two examples of the inclusion of user-centred processes in public procurement are outlined in Box 3.5.
Overall, there is a lack of feedback loops within organisations and ministries and across and outside government. In the absence of these loops, it is difficult to create an iterative improvement process that focuses on achieving the necessary legal requirements and reducing unnecessary burdens within the system. When the focus of a system becomes the proper execution of a repeatable process, there is less time to reflect and consider whether the process is achieving the strategic goals and expected outcomes.
The Mystery Shopper Service (United Kingdom)
The Mystery Shopper Service was launched in 2011 as part of a range of measures aimed at increasing and facilitating the participation of SMEs in public procurement. Its purpose is to provide public procurement suppliers with a structured channel at the central level that economic operators can use to report on the quality of the contracting authorities’ practices and escalate issues they may have experienced when participating in a procurement procedure.
In particular, the Mystery Shopper Service helps to detect practices that are not friendly to SMEs, provides recommendations to contracting authorities/entities and monitors their implementation.
In addition to educating contracting authorities through the provision of recommendations, the service conducts on-the-spot checks to verify whether a contracting authority previously investigated is properly applying the recommended measures. The Mystery Shopper Service also proactively verifies that contracting authorities use practices that are compliant with the various procurement policy notes prepared by the Crown Commercial Service and aimed at facilitating SME participation in procurement.
To make use of the Mystery Shopper Service, an economic operator needs to send a request that is first analysed by the Mystery Shopper team. If the request is deemed eligible, it becomes a case for investigation. Once the investigation has started, and depending on the seriousness of the case and whether it is located at central or local level, the service will provide its help within two or three months.
Results of investigated cases are evaluated and published on the Mystery Shopper service webpage. Since its inception, the service received the highest number of cases during 2013 and 2014. Since then, the service has experienced a reduction in the number of cases. This could indicate that contracting authorities are changing their current practices toward economic operators and are using more SME-friendly practices.
Tallinn’s smart port (Estonia)
In Estonia, Tallinn’s smart port is used to tackle the challenge of managing increasing traffic to the Tallinn Port Authority in Estonia. The port authority wanted to purchase a new electronic check-in system for both passenger and cargo vehicles. The public buyer identified the port’s needs by conducting 40 interviews with passengers, 6 interviews with drivers, 4 interviews with representatives of ferry operators, 2 interviews with stevedore service providers and 4 interviews with employees of the port. An assessment of these user needs allowed the buyer to procure an innovative solution that addresses the whole travel process for cars and lorries, from online pre-registration to check-in, as well as a fully automated traffic management process that directs vehicles onto the ship.
Sources: www.gov.uk/government/publications/mystery-shopper-scope-and-remit, www.portoftallinn.com/smart-port.
Perception versus reality
As discussed above, perceptions among businesses and the general public of corruption levels in Slovenia are among the highest in the European Union. Thus, it is entirely understandable that reducing corruption was the most-cited reason for reform, with corruption also cited as the cause of complicated processes and additional burden during the procurement process. During interviews, officials from media outlets stated that investigating potential corruption issues was one of their most important government-related stories to seek out, and that these stories influence the perceptions of audience members and readers. Meanwhile, industry believes that relationships are the most crucial factor in winning bids during the procurement process. Such stories and issues, which often result from the largest and most complex procurements, create issues for the rest of the system. All of these perceptions exist within the Slovenian procurement system.
Yet, there is little evidence that actual rates of corruption live up to the very high perceptions of corruption. For example, a 2016 study from the European Parliament “The cost of non-Europe in the area of corruption” estimated that corruption in Slovenia accounted for EUR 40 million. While that may seem like a large number, it accounts for just 1% of the monies spent through public procurement in that year. Slovenian officials argue that the procurement processes in the country are simply too transparent to allow for significant corruption. The European Commission also reported that, “Slovenia has made important steps in fighting corruption and increasing transparency in recent years. Overall, it has sufficiently strong institutions to detect and highlight corruption… Slovenia’s efforts in enhancing transparency have been remarkable and have been noticed by the international community (European Commission, 2020[5]).” There will likely always be some form of corruption in the system, but framing reform around the perception of corruption generally increases the burden for actors both inside and outside the system.
There is also little evidence that the system is fighting to correct these perceptions, with the exception of layering on more anti-corruption reforms. The government seems to be doing little to combat these perceptions and build trust among influential actors within the system. Instead, there are anecdotal examples of the opposite, with tightening controls increasing the complexity of the procurement process. For example, top-level leadership is required to approve every step of the procurement process in the form of a signature for every non-simple procurement.
Allowing these perceptions to continue unchecked reduces the space to try new things and innovate, and creates a risk-averse culture. The government appears to have the necessary data to combat much of these perceptions, but may lack the requisite skills and abilities in terms of analysis and storytelling (another OPSI core innovation skill).
Isolated incentives versus aligned actions
In a fragmented system, procurement officials view success in terms of passing through the procedure with few delays, no appeals or no successful appeals. Incentives are related to implementing procurement plans and the ability to execute the procurement process. The OECD’s analysis of the system found little evidence of shared incentives within the system focused on shared outcomes. Additionally, programme goals may implicitly be affected by procurement if the programme cannot procure the things it needs to succeed; however, such success measures rarely incorporate the procurement process.
Within the procurement process this dilemma manifests itself as a lack of ambition to try things outside the normal process. Because of the lack of incentives around the outcomes of the procurement, the discussion around price versus value remains relatively immature. The lowest price is seen as the safest and quickest route to award and, therefore, the lowest price that meets the minimum requirements of the tender is almost always selected.
These issues, combined with budget pressures, creates a tactical, output-focused culture that seeks the path of least resistance and lowest price, even if it means suboptimal outcomes. Aligning outcomes within the system and creating a partnership culture can be a challenge due to the time involved, but it can also create better outcomes and produce more public value related to the core purposes of procurement.
Scarcity versus need for resources
Procurement officials in Slovenia feel overworked and lack the support, expertise and time necessary to meet all of their challenges. This is not an abnormal feeling within public procurement systems or in government in general. Since the most recent recession, governments have sought austerity measures that have reduced capacity, limited training and asked the public sector to do more with less.
Because of this reality, Slovenia is lacking a strong cohort of experts, a pipeline of future procurement talent, and time to learn and develop. However, this challenge of time and money is common across all countries with which OPSI works. System actors need to start thinking more creatively about how to create space within the system to allow for new and different forms of thinking, to prioritise learning and to test new things.
This is not easy; it requires strong leadership and a will to seek opportunities for greater efficiencies within the system.
References
[35] Amann, M. and M. Essig (2015), “Public procurement of innovation: empirical evidence from EU public authorities on barriers for the promotion of innovation”, Innovation: The European Journal of Social Science Research, Vol. 28/3, pp. 282-292.
[19] Baldi, S. et al. (2016), “To bid or not to bid: That is the question: Public procurement, project complexity and corruption”, European Journal of Political Economy, Vol. 43, pp. 89-106.
[26] Barrad, S., R. Valverde and S. Gagnon (2018), The application of system dynamics for a sustainable procurement operation., Springer, Cham.
[47] Book, M., V. Gruhn and R. Striemer (2012), “adVANTAGE: A fair pricing model for agile software development contracting”, Agile Processes in Software Engineering and Extreme Programming, Vol. 111, pp. 193-200.
[13] Brugnach, M. and H. Ingram (2012), “Ambiguity: the challenge of knowing and deciding together”, Environmental science & policy, Vol. 15/1, pp. 60-71.
[9] Burns, T. and F. Köster (2016), Education Governance in Action: Lessons from Case Studies, OECD Publishing, Paris, https://doi.org/10.1787/9789264262829-en.
[7] Burns, T. and F. Köster (2016), Governing Education in a Complex World, OECD Publishing, Paris, https://doi.org/10.1787/9789264255364-en.
[49] Chapman, J. (2002), System Failure: Why Governments Must Learn to Think Differently (2nd edn)., Demos, London.
[28] Cingolani, L. and M. Fazekas (2017), Methods Paper on Administrative Quality.
[54] de Jong, J. (2014), Dealing with Dysfunction: Innovative Problem Solving in the Public Sector, Washington D.C.: Brookings Institution Press.
[65] Director General for Communications (2019), Flash Eurobarometer 482: Businesses’ attitudes towards corruption in the EU.
[40] Djankov, S., F. Saliola and A. Islam (2016), Is public procurement a rich country’s policy? World Bank Blogs., Washington, DC: Peterson Institute for International Economics.
[34] Edler, J. and L. Georghiou (2007), “Public Procurement and Innovation—Resurrecting the Demand Side”, Research Policy, Vol. 36/7, pp. 949–963.
[16] Edler, J. et al. (2005), Innovation and Public Procurement – Review of Issues at Stake. Study for the European Commission..
[29] Eurobarometer (2017), Businesses’ attitudes towards corruption in the EU. Slovenia, Country factsheet..
[5] European Commission (2020), Slovenia. Key Facts and Figures..
[63] European Commission (2019), Single Market Scoreboard. Procurement., https://ec.europa.eu/internal_market/scoreboard/performance_per_policy_area/public_procurement/index_en.htm.
[58] European Commission (2018), Building an architecture for the professionalisation of public procurement.
[32] European Commission (2018), Commission notice Guidance on Innovation Procurement. C(2018) 3051 final..
[64] European Commission (2018), Public Procurement Scoreboard.
[39] European Commission (2017), COM(2017) 572 final. OMMUNICATION FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS Making Public Procurement work in and for Europe..
[8] European Commission (2017), COUNCIL RECOMMENDATION on the 2017 National Reform Programme of Slovenia and delivering a Council opinion on the 2017 Stability Programme of Slovenia, https://ec.europa.eu/info/sites/info/files/2017-european-semester-country-specific-recommendations-commission-recommendations_-_slovenia.pdf.
[3] European Commission (2017), Measurement of impact of cross-border penetration in public procurement.
[30] European Commission (2017), Special Eurobarometer 470. Corruption..
[2] Government of Slovenia (2018), KOALICIJSKI SPORAZUM O SODELOVANJU V VLADI REPUBLIKE SLOVENIJE ZA MANDATNO OBDOBJE 2018–2022.
[1] Government of Slovenia (2014), KOALICIJSKI SPORAZUM O SODELOVANJU V VLADI REPUBLIKE SLOVENIJE ZA MANDATNO OBDOBJE 2014–2018.
[15] Guenther, E. et al. (2013), “The ´why not’ perspective of green purchasing: a multilevel case study analysis”, Journal of Change Management, Vol. 13/4, pp. 407-423.
[18] Hommen, L. and M. Rolfstam (2009), “Public Procurement and Innovation: Towards a Taxonomy.”, Journal of Public Procurement, Vol. 9/1, pp. 17–56.
[27] Ison, R. and M. Shelley (2016), “Governing in the Anthropocene: contributions from systems thinking in practice?. Systems Research and Behavioral Science”, ystems Research and Behavioral Science,, Vol. 33/5, pp. 589-594.
[33] Lember, V., T. Kalvet and R. Kattel (2011), “Urban Competitiveness and Public Procurement for Innovation”, Urban Studies, Vol. 48/7, pp. 1373-1395, https://doi.org/doi:10.1177/0042098010374512.
[11] Loosemore, M. and E. Cheung (2015), “Implementing systems thinking to manage risk in public private partnership projects”, International Journal of Project Management, Vol. 33/6, pp. 1325-1334.
[50] Mazzucato, M. (2011), The entrepreneurial state: Debunking public vs. private sector myths, Anthem Press.
[37] McKinsey & Company (2017), Government Productivity Unlocking the $3.5 Trillion Opportunity,.
[59] Mergel, I. (2016), “MAgile innovation management in government: A research agenda.”, Government Information Quarterly, Vol. 33/3, pp. 516-523.
[60] Nannicini, T. et al. (2013), “Social capital and political accountability.”, American Economic Journal: Economic Policy, Vol. 5/2, pp. 222-250.
[44] OECD (2019), Productivity in Public Procurement: A Case Study of Finland: Measuring the Efficiency and Effectiveness of Public Procurement, OECD Publishing, Paris.
[56] OECD (2019), Enhancing the Use of Competitive Tendering in Costa Rica’s Public Procurement System STREAMLINING THE EXCEPTIONS AND REDESIGNING THE THRESHOLD SYSTEM., OECD Publishing, OECD.
[68] OECD (2019), Government at a Glance 2019, OECD Publishing, Paris, https://dx.doi.org/10.1787/8ccf5c38-en.
[23] OECD (2019), Improving Lithuania’s Public Procurement System, COMPONENT 1 – IMPLEMENTATION OF PROFESSIONALISATION AND CERTIFICATION FRAMEWORKS, OECD Publishin, Paris.
[10] OECD (2019), OECD (2019), Reforming Public Procurement: Progress in Implementing the 2015 OECD Recommendation, OECD Publishing, Paris,, https://doi.org/10.1787/1de41738-en.
[69] OECD (2019), Public Procurement in Germany: Strategic Dimensions for Well-being and Growth, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/1db30826-en.
[25] OECD (2019), Public Procurement in Malta - Re-engineering the Department of Contracts, OECD Publishing, Paris.
[22] OECD (2019), Public Procurement Toolbox.
[21] OECD (2019), Reforming Public Procurement: Progress in Implementing the 2015 OECD Recommendation, OECD Publishing, Paris,, https://doi.org/10.1787/1de41738-en.
[61] OECD (2018), Data from the 2018 OECD Survey on the Implementation of the 2015 OECD..
[20] OECD (2017), Public Procurement in Chile: Policy Options for Efficient and Inclusive Framework Agreements, OECD Public Governance Reviews, OECD Publishing, Paris,, https://doi.org/10.1787/9789264275188-en.
[55] OECD (2017), Public Procurement in Chile: Policy Options for Efficient and Inclusive Framework. OECD Public Governance Reviews., OECD Publishing, Paris., https://dx.doi.org/10.1787/9789264275188-en.
[24] OECD (2017), Development and implementation of a national e-Procurement strategy for the Slovak Republic., OECD Publishin, Paris.
[71] OECD (2017), Government at a Glance 2017, OECD Publishing, Paris, https://dx.doi.org/10.1787/gov_glance-2017-en.
[74] OECD (2017), OECD Guidelines on Measuring Trust, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264278219-en.
[36] OECD (2017), Public Procurement for Innovation: Good Practices and Strategies, OECD Public Governance Reviews,, OECD Publishing, Paris, https://doi.org/10.1787/9789264265820-en.
[70] OECD (2017), Public Procurement in Peru: Reinforcing Capacity and Co-ordination, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264278905-en.
[72] OECD (2017), Public Procurement Review of Mexico’s PEMEX: Adapting to Change in the Oil Industry, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264268555-en.
[67] OECD (2017), Systems Approaches to Public Sector Challenges: Working with Change, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264279865-en.
[48] OECD (2017), What’s possible? Finding and filtering innovative ideas, OECD Observatory of Public Sector Innovation..
[38] OECD (2016), Improving ISSSTE’s Public Procurement for Better Results, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/9789264249899-en.
[51] OECD (2016), Survey on Strategic Human Resources Management in Central/Federal Governments of OECD Countries, OECD Publishing, Paris.
[57] OECD (2016), Checklist for Supporting the Implementation of the OECD Recommendation of the Council on Public Procurement.
[62] OECD (2016), OECD (2016), Data from the 2016 OECD Survey on public procurement (partially published)..
[45] OECD (2015), OECD Recommendation of the Council on Pulic procurement.
[6] OECD (2015), Systems Innovation: Synthesis Report, OECD, Paris, http://www.innovationpolicyplatform.org/system-innovation-oecd-project.
[73] OECD (2013), Public Procurement Review of the Mexican Institute of Social Security: Enhancing Efficiency and Integrity for Better Health Care, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264197480-en.
[46] OECD Publishing, P. (ed.) (2017), Embracing Innovation in Government: Global Trends, http://www.oecd.org/gov/innovative-government/embracing-innovation-in-government.pdf.
[52] OPSI (2019), Embracing Innovation in Government. Global Trends 2019..
[12] Peters, G. (2005), “The problem of policy problems. Journal of Comparative Policy Analysis”, Journal of Comparative Policy Analysi, Vol. 7/4, pp. 349-370.
[43] Preuss, L. (2011), “On the contribution of public procurement to entrepreneurship and small business policy”, Entrepreneurship & Regional Development, Vol. 23/9-10, pp. 787-814.
[53] Pykett, J. et al. (2016), Governing mindfully: Shaping policy makers’ emotional engagements with behaviour change., Routledge.
[66] Rolfstam, M. (2012), “An institutional approach to research on public procurement of innovation”, Innovation – The European Journal of Social Science Research, Vol. 25/3, pp. 303–321.
[31] Rolfstam, M. (2012.), “An Institutional Approach to Research on Public Procurement of Innovation.”, Innovation – The European Journal of Social Science Research, Vol. 25/3, pp. 303–321.
[14] Sönnichsen, S. and J. Clement (2019), “Review of green and sustainable public procurement: Towards circular public procurement. Journal of Cleaner Production.”, Journal of Cleaner Production.
[4] Sta (2019), Revizijska komisija razveljavila del razpisne dokumentacije za prenovo Hotela Brdo.
[41] Vogel, L. (2009), “Macroeconomic effects of cost savings in public procurement”, Economic Papers, Vol. 389, https://doi.org/10.2765/29810.
[17] Yeow, J. and E. J. (2012), “Innovation procurement as projects.” Journal of Public Procurement.”, Journal of Public Procurement, Vol. 12/4, pp. 472–504.
[42] Zeng, S., X. Xie and C. Tam (2010), “Relationship between cooperation networks and innovation performance of SMEs.”, Technovation, Vol. 30/3, pp. 181-194.