4. Challenges in the development and procurement of ICT/digital projects
This chapter presents the challenges identified in developing and procuring digital and ICT projects in the Greek Public sector. It highlights key areas of improvement to achieve greater competency and maturity in the development of ICT and digital projects. The challenges presented include the governance perspective, the planning and funding of digital initiatives, the procurement practices, monitoring and evaluation mechanisms and public sector capabilities.
While Greek authorities have been gathering considerable experience in implementing ICT/digital projects over the past years, challenges often persist, leading to ineffective project implementation that hinders benefits realisation. In particular, large digital projects are at greater risk of failure, as highlighted already by Greek authorities in the National Digital Strategy 2016-2021 (Ministry of Digital Policy Telecommunications and Information, 2016[1]) as well as during the interviews and data gathering process for this review.
Namely, digital projects generally take a long time from the moment of conception and design until they are effectively implemented, are often not flexible enough to respond to changing needs, face challenges to interoperate and adopt cross-governmental standards, do not incorporate the latest technologies, and lead to complex and closed systems with the risk of vendor lock-in and legacy issues that increase maintenance costs, among other challenges. Digital transformation’s expected benefits may not be achieved if ICT/digital projects do not adhere to system-wide standards, are not tailored and driven to users’ needs, take too long to be implemented, or are complex to manage during and after the implementation and procurement processes (Ministry of Digital Policy Telecommunications and Information, 2016[1]).
This section outlines key areas where the Greek government can achieve greater competency and maturity in order to secure ICT/digital projects’ benefits realisation. It comprises reflections and suggestions throughout the project life cycle as well as other key components of the digital government ecosystem in Greece.
As highlighted in the previous chapter, the creation of the Ministry of Digital Governance represents a step forward in terms of the governance of digital government in Greece. Created in 2019, the Ministry is empowered to control all relevant functions for digital government, including policy design, strategic planning, funding management and public procurement for most of the critical and relevant projects. Within this governance model, Information Society S.A. is also under the mandate of MDG and acts as its implementation arm. Information Society plays a critical role to support public sector organisations with limited institutional capacities to implement and deliver, acting as an executive arm supporting the implementation of ICT projects in the public sector.
The reforms undertaken by the Greek administration are in line with the OECD Recommendations for Digital Government Strategies, which calls governments to secure leadership, political commitment and institutional capacities to co-ordinate digital transformation projects and achieve the digital government imperative (OECD, 2014[2]). When looking at the organisational structure and operational model of MDG, evidence collected from various departments and units within and outside the Ministry shows that it operates under a fragmented approach as a consequence of the successive institutional reforms undergone in the past decade. This siloed functioning becomes particularly clear in the ways departments and units involved in the life cycle of ICT/digital projects do not collaborate and interact throughout this process. This also includes the limited awareness of roles and mandates of each department/unit involved in the implementation and procurement of ICT/digital projects within and outside MDG, which constrains a smooth and co-ordinated approach in project assessment, prioritisation, funding and procurement. Further capacity building activities to bring together different areas within MDG are needed to create opportunities for policy dialogue, knowledge transfer and co-ordination beyond the formal procedures for project approval, funding and implementation. Institutional culture and appropriate incentives play an important role in this context, too. Namely, a culture that promotes co-operation between teams and institutions with concrete incentives (e.g. career rewards for co-operative behaviours) can reduce fragmentation and silo-based thinking.
In addition, the Digital Transformation Bible (DTB) includes specific provisions on the governance framework supporting the implementation of the strategy. This comprises setting an inclusive governance model to involve relevant public sector stakeholders in this process, including the Digital Transformation Steering Committee (relevant ministers and state secretaries) and the Digital Transformation Executive Network (institutional CIOs and heads of public sector bodies and organisations that have a key role in the implementation of the DTB).
Accordingly, both co-ordination bodies reflect well-intended efforts to secure effective governance mechanisms for the implementation of the strategic roadmap at top political and execution level. However, overcoming institutional siloes requires clarifying the responsibilities of each body, including the advisory and decision-making roles of both the Committee and the Network to effectively support the implementation of the Bible. Similarly, clear communication of their roles and mandate are needed within and outside the public sector to set expectations, ownership and awareness of the inclusive governance model for digital government in the country.
An effective digital transformation requires adopting user-driven approaches in the formulation and development of ICT/digital projects. Establishing concrete mechanisms for understanding and meeting user needs increases the pertinence, expected impact and successful implementation of ICT/digital projects throughout the development process, capturing feedback and acting accordingly to shape them around and driven by user needs. In this sense, adopting agile methodology approaches to frame the design and implementation of digital transformation process can help encompass increased engagement with users to collect feedback with the importance of addressing the uncertainties and risks of large and complex projects by using proof of concepts and minimum viable products (MVPs) instead of detailed upfront requirements (Agile Alliance, 2001[3]; Meyer, 2014[4]).
The Greek public sector requires a cultural shift in the development of ICT/digital projects to better understand and place user needs at the core of the development processes. Beyond anecdotal cases, MDG and public sector organisations in Greece do not have institutionalised methods to conduct user research processes that help better understand and define the requirements when planning an ICT/digital project nor as well as concrete mechanisms to channel user feedback at early development stages.
Additionally, evidence collected during the interviews suggests that user research and service design skills are not common abilities among Greek public workforce. Within the OECD, several countries are progressing to setting standards and a culture of user research, such as the case of the US and the development of the Digital Services Playbook (see Box 4.1). The Playbook guides on the driving principles of digital services, including a specific checklist and recommendations on the role of users in the design and delivery of services.
Promoting user research and a culture around user needs implies establishing formal engagement mechanisms with users, where they can channel their needs into the design phase so that solutions provided fully meet their demands and expectations. In addition, the government needs to adapt the process of formulating digital initiatives to ensure that users’ input effectively impacts the design of solutions. Successful engagement with users should be continuous and iterative throughout the project development cycle, promoting ongoing exchanges and constant user feedback.
In 2014, The US Digital Services published the US Digital Service Playbook, summarising the driving principles for digital services in thirteen practical steps. The playbook supports the development of digital services nationwide by leveraging successful practices from both the private sector and government to pursue effective digital services. Each of the thirteen plays includes a checklist and a set of critical questions to guide digital services design and delivery. The plays of the playbook are:
Source: (US Digital Service, n.d.[5]).
Similarly, the limited adoption of agile project management principles and standards is impeding a culture of experimentation, testing and iteration. Agile development is critical when addressing the risks of implementing large and complex digital transformation projects. Similarly, there is a limited culture and room for experimentation practices and the use of proof-of-concept in the design and delivery of digital solutions, in line with existing gaps on institutional capacities for delivery (see next section). Embracing a user-driven approach calls for flexibility in product development, promoting scalability, and encouraging continuous learning and improvement. The use of agile methodologies can help Greece foster a culture driven around user needs, such as it has been observed in New Zealand and the UK with dedicated guidelines and practices to support adoption of agile methodologies in designing and delivering services (see Box 4.2).
Different OECD member countries are developing guidelines and standards to strengthen the use of agile methodologies in service delivery and design. There are different agile management methodologies, such as Scrum or dynamic systems development methods. However, standardisation efforts seek to leverage the guiding principles promoting an incremental approach through a systematic discovery of user needs, experimentation, and iteration.
The Government of New Zealand issued in 2019 the Assurance guidance for Agile delivery to support government organisations in applying good practice assurance to agile delivery in government. This guidance extends the All-of-Government Portfolio, Programme, and Project Assurance Framework, which outlines the principles of good assurance in service design and delivery. The Assurance guidance for Agile delivery describes the different roles in agile management, propose advice on a coherent between assurance thinking and agile delivery
In 2017, the Infrastructure and Project Authority (IPA) and the Government Digital Service (GDS) in the United Kingdom issued the Assurance and approvals for agile delivery of digital services guidance to support units within government embracing agile methodologies. The direction calls for the use of agile methodologies in the delivery of digital services. It describes the different phases of the agile development lifecycle and requirements to assure digital government standards.
Source: : Own elaboration, adapted from (New Zealand Government, 2019[6]) and (UK Government, 2017[7]).
MDG is currently responsible for the approval, funding allocation and management of most ICT/digital initiatives, along with the support of Information Society S.A. in certain projects. Currently, the development process is fragmented and limitedly co-ordinated within the Ministry. The relevance attestation (project approval mechanism) acts as a method to join-up departments/units within MDG but it does not act as a policy lever to effectively support the implementation of the DTB. Instead, interviewees indicated that the process often creates delays in each phase as well as limited internal and external monitoringregarding the progress and status of each project as well as accountability of relevant stakeholders across the project life-cycle.
This process results in the limited information managed by MDG concerning the approval, funding, procurement, management and monitoring/evaluation of projects above EUR 60 000 set as budget threshold to be approved by MDG. Despite collecting general information about goals, technical specifications and expected timelines, MDG does not have a comprehensive knowledge management approach to leverage this information to effectively steer the implementation of the DTB, prioritise efforts and create synergies between relevant authorities to avoid project duplication and fragmentation. Interviewees identified that existing budget thresholds and approval mechanisms are inducing project fragmentation to avoid lengthy procurement processes with MDG and/or Information Society. Such an approach impedes synergies and alignment, producing a silo-based digitalisation of the public sector.
Additionally, the Ministry does not have clear visibility and overview of relevant projects below the budget threshold. The limited information held by MDG regarding these projects impedes the Ministry to take actions to secure efficiency, coherence and co-ordination among relevant line ministries (for example to centralise the procurement of ICT/digital commodities or to co-ordinate public sector institutions with similar needs). This approach may be also fostering project fragmentation and shadow IT in the Greek public sector, with the eventual impact on legacy issues in the medium and long-term.
All these issues have an ultimate impact on the role of MDG within the digital government ecosystem in Greece and its capacity to focus on the key priorities set in the DTB. Considering the comprehensive set of initiatives defined in the strategy, MDG could prioritise and give larger institutional attention to the projects with a systemic impact for the whole-of-government digital transformation in Greece. This includes critical digital transformation enablers, such as interoperability systems, digital identity, or payment platforms.
Instead, the Ministry currently devotes significant time and effort in dealing with the long-tail of small projects which could eventually be implemented by line ministries. While this requires further capacity building at institutional level to de-centralise the implementation of the strategy, MDG would benefit by concentrating efforts and resources in ensuring the right implementation of the objectives set for the public sector in the DTB. For this, the Ministry could make a strategic use of management tools to streamline the ICT/digital project development life cycle in order to support project prioritisation and the level of involvement (strategic, implementation or monitoring) of MDG and line ministries in this process.
The Agency for Digitalisation develops and maintains the cross-governmental ICT project model and portfolio management of ICT projects and systems. In this role, the Agency’s Division for Central Government ICT Management is responsible for standard contracts and advising public authorities to implement large ICT projects in the public sector.
This approach includes the development of a portfolio management system to strengthen governmental ICT projects' planning, management, and implementation. The model supports a responsible and secure management of government ICT systems, promoting national standards alignment while decentralising decision-making. By supporting governmental institutions in assessing their capacities and needs, the model allows for informed decision-making on direction and priorities when developing ICT solutions.
The model analyses six dimensions: Business support, Technical state, Documentation and knowledge, Finances, Contracts and sourcing, and Security. Altogether, the model serves as basis for the formulation of an ICT Action Plan. This plan provides a strategic prioritisation of operation and maintenance activities for all ICT systems. The plan indicates strategic goals, limitations, expected impacts, details on implementation and an overview of all initiatives. This plan is reviewed by the Danish National ICT Council every three years, securing national alignment and coherence at a national level.
Source: Own elaboration, adapted from (Agency for Digitalisation, n.d.[8]).
Several countries have addressed similar issues by implementing a comprehensive ICT portfolio management approach that supports the implementation of digital government strategies. An ICT portfolio management approach can serve as a policy lever to address the development cycle of ICT/digital projects (from approval to monitoring/evaluation) recording relevant information to oversee the extent to which public sector organisations are meeting the objectives and goals set for each project. Additionally, such an approach can provide more granular information at project level in order to set concrete key performance indicators (KPIs) to assess and monitor progress (see Box 4.3 above).
MDG has different funding sources to support the implementation of ICT/digital initiatives, including European and national funds. According to the evidence collected, there are different procedures and internal responsibilities within MDG depending on the funding source, following different approval workflows and legal formalities. Currently, the limited alignment in the way projects are approved and funded is challenging MDG in terms of administrative and policy coherence.
For example, in the case of ESIF funds, ICT/digital projects are handled directly by MDG, which is not the case for the Public Investment Programme or line ministries budgets. These different procedures risk causing inefficient resource allocation by affecting investment decisions based on different criteria and parallel paperwork needed. Stakeholders within MDG acknowledge the need to develop coherent management frameworks that replicate the experience of the ESIF in other funding sources, in particular given the experience with the implementation of the Public Investment Programme or upcoming Resilience and Recovery Funds (RRF).
This coherence can be supported by revisiting and aligning the relevance attestation and funding mechanisms, unifying the criteria and procedures to have a coherent approach to approving and prioritising ICT/digital projects in the Greek government. In line with good practices on coherent capital budgeting frameworks, investment assessments and decisions should be independent of specific funding mechanisms (OECD, 2015[9]).
Additionally, and in line with the OECD Recommendation of the Council on Digital Government Strategies, the approval mechanism should include a multi-dimensional value proposition (analysis of the costs and benefits of ICT/digital projects) in the short, medium and long-term. This approach supports the identification of financial, social, environmental or administrative benefits and costs to support planning and prioritisation of initiatives. As outlined previously, the revision of the approval mechanism can be supported by the development of ICT portfolio management in order to have a comprehensive overview of the different stages.
MDG holds the critical role of approving and funding most of ICT/digital projects in the public sector. However, to date the Ministry is not leveraging this process to secure that digital investments adhere to the priorities and standards for digital government in Greece prior to allocating funding.
Currently, the DTB sets several priority areas to digitally transform the Greek government, economy and society. Concerning the public sector, the implementation of interoperability frameworks, the national central service delivery portal gov.gr and the development of digital skills require solid policy levers for MDG to secure a coherent and aligned approach across public sector organisations. This includes making effective use of budget thresholds and approval processes to enforce the implementation of digital government objectives as well as the needed standards and tools for its implementation. MDG could strategically use this process for public sector organisations to adopt existing and future digital tools and standards that secure a coherent implementation across the public sector. The evidence collected shows that line ministries acquire ICT goods such as software licenses or basic hardware using this waiver. These kinds of items are often highly standardisable. Therefore the government could benefit from co-ordinated procurement practices to secure the best value for money.
This is the case of Chile, where the Digital Government Division is co-ordinated with the Budget Office to assess all ICT/digital projects in the public sector in order to assess their adherence to strategic goals and adoption of digital government standards and tools (e.g. cloud first, agile management, digital identity, interoperability). This information is leveraged by the Budget Office to assess all digital initiatives included in the annual budget (See Box 4.4). This process also involves ChileCompra, Chile’s public procurement authority, which identifies common ICT/digital needs that can be centralised through collaborative or co-ordinated purchase mechanisms, as currently done with IT hardware, mobile phone plans, software licences and data infrastructure, among others.
In Chile, the Digital Government Division (DGD) and the Budget Office (DIPRES) developed a whole-of-government approach to assess and align all central government ICT/digital projects, both in-house and outsourced, as part of the annual budgeting process. The procedure requires all initiatives to adhere to relevant digital government priorities and standards, such as cloud first, digital identity (ClaveUnica), agile project management, among others.
Line ministries and agencies submit their ICT/digital project proposals through the EVALTIC platform prior to the annual budget discussions, co-ordinating efforts between financial managers, digital experts, and CIOs within each institution.
On an annual basis, digital, technology, and data projects are peer-reviewed by institutional CIOs, providing a binding technical recommendation before the budget allocation decision-making. When needed, the procurement authority (ChileCompra) demands a validation and approval code from the EVALTIC platform for opening new purchase orders or tenders of ICT goods or services.
The objectives of EVALTIC process are:
Increase efficiency in public expenditure by leveraging economies of scale and network effects of digital tools.
Align public expenditure in ICT with the strategic goals of the Government digital transformation act.
Increase the quality of ICT projects through standardisation focusing on public value creation and efficiency gains.
Provide information for evidence-based decision-making on digital transformation.
Strengthen control mechanism over ICT projects to secure efficiency, enabling timely detection of mismanagement.
Source: Own elaboration, adapted from (División Gobierno Digital, 2021[10]).
A critical success factor for ICT/digital projects is the speed of implementation. In Greece, however, several elements contribute to extended implementation timeframes, including the project approval (relevance attestation), communication between relevant ministries, and management of the procurement process. Evidence indicates that some ICT/digital projects may require several years between the project proposal and contract award activities. Such delays pose challenges to attract suppliers, particularly SMEs that may not have the organisational and financial capacities to engage in overly long processes. At the same time, delayed implementation of procurement processes, particularly if not implemented in a flexible and agile way, risks delivering outdated solutions i.e. the technology may be already outdated, or the needs of end-users may have evolved. This means that solutions may no longer be fit-for-purpose or relevant (e.g. institutional mergers or reorganisations, regulatory changes, technology upgrades to different systems or platforms etc.).
Need to speed up project preparation, approval and selection financing scheme
As a starting point, there is a need to speed up the early preparatory stages of a digital project, which entail early project preparation, the validation on the basis of the DTB, as well as the selection of the appropriate financing schemes. While these steps are broadly understood by stakeholders within MDG and line Ministries, their implementation is often not as streamlined as it could be.
For instance, the approval of digital projects based on the DTB is in essence a simple and fast process, but institutional communication channels between the ministries involved often create delays. Greek authorities could think how to set-up direct communication and validation channels to ensure that the process is not slowed down unnecessarily.
Furthermore, the project preparation may entail a lengthy decision-making and negotiation process to define the responsibilities of the procurement process itself (i.e. whether a line ministry or Information Society S.A. will take over the procurement process) as well as the scope of the procurement project, in case it is executed by Information Society S.A. Limited capacity by the line ministries is linked to the fact that the project preparation is still at early stage when the support of Information Society S.A. is requested, requiring important preparation work as well as lengthy negotiations to establish a Memorandum of Understanding that frames the relationship for the development of ICT/digital projects, including its procurement.
Finally, as discussed above, the selection of the financing scheme is a process that also requires significant time, in particular when ESIF funding is involved, as additional requirements may be connected with EU-funded projects.
Extreme delays in each phase of the procurement process
In addition to the slow implementation of the preparatory phase of digital projects, the procurement process itself is often burdened by extreme delays throughout each of the procurement phases. The project data analysed by the OECD team Figure 4.2 and Figure 4.3) reveals bottlenecks identified in each stage of the procurement process. In addition to delays, many ESIF financed projects do not reach sub-sequent implementation stages after their approval (Figure 4.2). In fact, while all the ICT projects approved under the ESIF programme period 2014-2020 obtained the approval of starting public procurement procedures, only 62% of them reached the stage of tender publication. Out of the projects that publish a tender, only 59% issued a contract award. Ultimately, only 22% of the initially approved projects reached the stage of contract implementation.
Beyond these bottlenecks in implementation, the overall execution of digital projects is severely slowed down. On average, the project procurement process takes 686 days, from the project approval to the start of contract implementation. This data excludes the time needed for actual contract implementation.
The tender preparation alone is takes significant time, taking on average 476 days from the project approval to the approval of starting procurement procedures. Then, it takes 24 days from the approval of starting procurement procedures to publishing tender, and 82 days from tender publication to issuing contract award. For instance, the tender preparation is lengthy due to the lack of an effective co-ordination processes with the project beneficiary. Lack of personnel as well as limited technical understanding of the subject matter of the contract may also contribute to slow implementation. Reportedly, delays are also linked to the evaluation stage, as evaluation committee’s members are not allotted dedicated time for this task, but it is an additional responsibility to their day-to-day work. This limits the incentive for speedy conclusion of the evaluation phase.
A key challenge in the procurement process of digital projects lies in frequent legal challenges. While the Review Body AEPP is relatively speedy in providing a decision, each phase of the procurement may be challenged without deadlines, thus taking up to a year for the process to conclude. If the decision of AEPP is further challenged in court, the process may take several years to conclude. Namely, it took 262 days on average from issuing contract award to starting contract, which could be mainly attributable to the legal challenges on the contract award decision.
Addressing the slow execution times of the remedies system goes well beyond the scope of the Ministry of Digital Governance. Nevertheless, procurement bodies within the Ministry could explore in greater detail whether recurring aspects of the procurement procedure are challenged in court, and identify measures to address these. Greater clarity of the tender documentation and of award criteria could help in limiting legal challenges. Similarly, enhancing the capacity of suppliers in understanding the procurement process may also reduce ambiguity and the related need to make use of the remedies system.
Analysing the duration of each phase of the procurement process gives insight into, it appears that specific bottlenecks and challenges that may emerge during the procurement process. Namely, the data suggest that challenges emerge in defining the detailed concept of ICT items to be procured, as this process takes a long time (476 days) (see Figure 4.3). Beyond the length of this phase, only 62% of the approved projects reach the stage of tender publication, suggesting additional challenges in preparing tender documentation, or other difficulties at this preparatory stage. Specific data or insights into the challenges that approved project face at this stage are not available, e.g. cancellations or other causes.
Note: 74 ICT projects were approved under the ESIF programme period 2014-2020. Calculation of dates between each steps is based on data availability.
Source: Prepared by the author based on the statistics provided by the Government of Greece.
In contrast, the process to publish a tender once the approval of starting public procurement procedure has been received is relatively speedy (24 days). In parallel, the vast majority projects that published a tender also issued a contract award: 59% of approved projects issued contract award, compared to 62% of approved projects that published a tender. In addition, the tender process took a comparatively shorter amount of time: 82 days from publishing tender to issuing contract award. As such, there seem the tender process from tender issuance to contract award appears comparatively less challenging in terms of execution and speed. The stage from contract award to start of contract execution also appears fraught with difficulties, as a significant number of projects do not advance to this stage and this stage is very delayed. Specifically only 22% of all the approved projects started its contract implementation, and these 22% projects required 262 days from issuing contract award to starting a contract. This data could evidence the difficulties encountered through the legal challenges, as one of the potential causes for this low progress rate and the long time spent in this stage.
Overall, challenges around lengthy and burdensome procurement processes affect many OECD countries, and some have taken concrete steps to streamline and speed up these processes, such as the UK, as discussed in Box 4.5 below.
LEAN is a management methodology designed to reduce waste at each step of the process, and increase value from the perspective of the client. While it was originally applied in manufacturing, the principles of LEAN have been applied to streamline the procurement process, notably in the UK. In 2012, the UK Cabinet Office mandated LEAN sourcing principles across the central government with the goal to reduce the time of the procurement process, and reduce administrative burden linked to delays.
This process led to the publication of so-called ‘Standard Operating Procedures’, which aim at optimising each process of the procurement procedure. Staff dealing with public procurement was encouraged to receive training on how to implement the new Standard Operating Procedure.
Source: (European Commission, n.d.[11]).
Beyond the speed of procurement processes, the effective implementation of ICT/digital projects requires advanced procurement practices. In the context of the Ministry of Digital Governance, a mixed picture has been observed with the level of advancement of procurement practices suited for digital projects. For instance, Information Society S.A., as a longstanding actor in the field, is more advanced in engaging with the market on a regular basis. In contrast, the Department of Procurement and Logistics covers the full spectrum of products and services, and relies on in-house expertise for the procurement of digital projects.
Limited focus on market engagement in the pre-tender stage
It is important to give enough focus and allocate enough time and resources to the preparatory phase for the success of the tender process. The more time spent in the preparatory phase of the procurement process, the less time will be needed to fix misunderstandings, discrepancies and mistakes in the procurement process and when managing contracts. The preparation entails gathering a sound understanding of what the market can deliver by conducting early market engagement. However, as discussed above, the practices for market engagement vary within MDG, but overall do not exploit the full possibilities permitted by the law, and the related benefits for the procurement of digital transformation projects.
In some instances, contracting authorities lack full understanding of the benefits of market engagement. Often, however, the barrier to conducting market engagement lies in contracting authorities’ lack of practical knowledge about how to conduct such market engagement, as well as knowledge of alternative modalities. In particular, contracting authorities fear legal repercussions for interacting with market providers. In contrast, market engagement is foreseen by the EU legal framework, provided the principles of transparency and equal treatment are respected. Thus, it is important for the process to be planned and managed carefully.
Beyond the respect of fundamental principles, the EU Directive leave room of manoeuvre to contracting authorities by providing the following definition of “preliminary market consultations”: “Before commencing a procurement procedure, contracting authorities may conduct market consultations with a view to preparing the procurement and informing economic operators of their procurement plans and procurements” (Article 40, Directive 2014/24/EU).The use of preliminary market consultations has been transposed into the Greek public procurement framework with Article 47 and 48 of L. 4412/2016. Furthermore, HSPPA has produced a dedicated technical instruction to provide contracting authorities with greater clarity on how to apply preliminary market consultations in accordance with the law (Technical Instruction 5/2019: Issuance of a Technical Instruction on: "Preliminary market consultations in innovation public procurement" Έκδοση Τεχνικής Οδηγίας με θέμα: «Προκαταρκτικές διαβουλεύσεις της αγοράς στις δημόσιες συμβάσεις καινοτομίας»).1
Market engagement can take several forms, and can also be used throughout the procurement cycle to continuously inform the contracting authority of market developments. Table 4.1 below summarises various forms of market engagement throughout the procurement cycle.
In the context of digital transformation projects, it is important to regularly exchange with the industry and keep abreast of technology developments. Several OECD countries are using digital platforms to facilitate market engagement between buyers and suppliers, i.e. so-called digital marketplaces. These platforms allow suppliers to present their products and facilitate the interaction with government buyers. Box 4.6 describes the UK’s initiative in this field.
In 2014, the UK’s Government Digital Service (GDS) launched the Digital Marketplace as an online service to facilitate the government’s ability to find and procure technology for the public sector. The goals of the Digital Marketplace were twofold: first, introduce a simpler, faster and cost effective method for the government to buy to technology; second, re-define the government’s relationship with technology providers.
The Digital Marketplace was also introduced in response to the concentration of the government’s IT spending: according to the UK’s National Audit Office, in 2009 less than 20 companies accounted for 80% of the UK’s GBP 16 billion of annual IT spend.
In contrast, the launch of the Digital Marketplace has transformed the UK’s technology market into a highly competitive and diversified market with 5 100 suppliers (92% SMEs) registered on the platform as of 2018. The platform allows a simpler exchange, as suppliers must apply to sell services, and public sector organisations are able to buy products or services. As such, the Digital Marketplace offers an opportunity to support the growth of digital companies.
Furthermore, the Digital Marketplace GOV.UK blog can also be used as a tool to support market engagement and a multidisciplinary approach at the pre-procurement stage. Namely, GDS and the Crown Commercial Service (CCS) published a series of communications over a 6-month period ahead of the 9th iteration of the ‘G-Cloud’ cloud services commercial framework. This entailed the publication of procurement plans and timetables, draft service categories, service questions, and terms and conditions, in advance of and during the build up to the formal procurements to deliver framework agreements.
Source: (OECD, n.d.[12]) (OECD, 2022[13]).
Regular interaction with the market also provides the opportunity of identifying upcoming and innovation trends, as perceived by market players. Through the Emerging Market Initiative, National Association of State Procurement Officials (NASPO) in the U.S. surveys market players on what they consider to be innovative trends and upcoming needs for the public sector (Box 4.7).
The National Association of State Procurement Officials (NASPO) is composed of the U.S. State Chief Procurement Officials. NASPO ValuePoint (NPV) represents the largest procurement cooperative in the US, as it establishes framework agreements that are open for use by state governments and local public entities (“cooperative master agreements”). Traditionally, NPV surveyed the needs of its members and analysed past purchases to identify relevant goods and services for its framework agreements.
In 2020, NASPO turned to the market to gather insights on what future needs of state and local governments could be served through its master agreements. In fact, the private sector invests heavily in researching trends, as well as developing innovative products and technologies. NASPO intended to harness the knowledge and insights of industry, as it often has a broader view of market developments compared to the public sector, and is therefore in a strong position to anticipate market trends.
NASPO used a structured market engagement process to gather ideas from the market and assess their feasibility. The market engagement process consisted of 6 main phases:
Phase 1 – Submission of ideas by suppliers based on three pieces of information: 1) overview of the concept; 2) whitepaper describing the concept in fuller detail; 3) relevant market information including references to existing contracts that address this concept
Phase 2 – Collection of submission through cloud-based interface
Phase 3 – Presentation by selected suppliers on: 1) current market demand; 2) market growth potential; anticipated resources needed to develop the idea into a cooperative master agreement
Phase 5 – Issuance of Requests for Information (RFI) to gather a broader perspective of the good/service from industry
Phase 6 – Approval for the development and release of a cooperative procurement
As a result of this initiative, several project ideas have turned into master agreements operated by NASPO. These include Citizen Engagement Platforms, Enterprise Content Management, Online Marketplaces.
Source: (OECD-OPSI, 2021[14]).
Understanding beneficiary needs
Some measures are in place to understand beneficiary needs and engage them throughout the project cycle, but these appear not to be sufficiently developed to deliver outcomes in a consistent and satisfactory way. For instance, Information Society S.A. relies on setting up a multi-disciplinary team for each of its projects. Beneficiaries are represented in this team, and are key in contributing to all phases of the project development, including procurement the preparation of technical specifications. Despite this mechanism, the process appears to be running slowly and there can be challenges in creating a shared understanding between the technical/procurement team and the subject matter experts (i.e. the beneficiary).
In contrast, the Department of Procurement and Logistics does not appear to have a formalised process in place for understanding and engaging beneficiaries; instead, it relies on informal co-operation with the departments. While such a mechanisms may prove effective on an ad-hoc basis and solve short-term issues, they do not substitute for dedicated mechanisms to understand user needs and involve them effectively.
Additionally and as previously outlined, there is room for ICT/digital projects to adopt agile project management approaches that create spaces for increased dialogue and interaction between MDG, Information Society S.A., beneficiaries and end-users. Limiting the definition and development of ICT/digital projects only within the public sector fosters top-down approaches that do not reflect the final needs and expectations of end-users, affecting the pertinence and relevance of final solutions.
No focus on quality award and overly prescriptive technical specifications
Another key aspect of success in the procurement of digital projects is the focus on quality in the award of a contract. In many instances, however, contracting authorities rely on the lowest price instead of awarding contracts to the best price-quality ratio. A common misconception is that market participants are not ready or willing to compete on the basis of quality. Fear of audit is also a frequent cause for relying on the lowest price as evaluation criterion. Not least, the lack of technical skills by procurement practitioners may also limit the use of quality criteria. As such, contracting authorities are not able to award contracts to high-quality providers that offer value-for-money solutions. Instead, choosing the lowest price brings along the risk of poor execution and less advanced technology. Despite a high prevalence of lowest price, the Greek procurement legal framework fully supports contract award based on best-price quality ratio, i.e. taking into account quality dimensions that are related to the subject matter of the contract, as per Article 86 of L. 4412/2016.
In a similar vein, overly prescriptive technical specifications are potential causes of sub-optimal execution of digital projects. In addition, overly prescriptive technical specifications are often the consequences of the almost exclusive use of the lowest price criteria. Importantly, prescriptive specifications do not leave any opportunity for the market to propose innovative solutions. Furthermore, such specifications may turn out overly rigid and inflexible. In the digital environment, however, it is especially important to allow for agility, modifications and iterations. Thus, a product or service procured on the basis of an overly defined specifications may prove not to be fit-for-purpose. Over-specifying details hide further risks and disadvantages. Customised solutions are generally more expensive than standard ‘off-the-shelf’ options. In addition, they are more difficult to be reused. Subsequently, suppliers who develop and manage custom-made systems can retain all the information about the system and make it very difficult to migrate to another supplier or to maintain or upgrade the system in the future. Excessive customisations might also lead to supplier dependence (vendor lock-in as it was already mentioned earlier). Contracting authorities should define the problem to be solved (the expected outcome of the purchase) rather than designing the solution.
Instead, functional specifications offer greater flexibility and room for innovation. When using functional specifications (or functional requirements), the contracting authority only specifies a required performance or outcomes, without providing the technical detail for reaching such a result. While much welcomed by market participants, as reported by private sector stakeholders, the procurement bodies appeared to lack awareness about such types of specifications and do not make use of them in practice. Nonetheless, the use of functional specifications is fully supported by the Greek procurement framework. Namely, Article 54 of L. 4412/2016 specifies that technical specifications shall be formulated, among other methods, “in terms of performance or functional requirements, including environmental characteristics, provided that the parameters are sufficiently precise to allow tenderers to determine the subject-matter of the contract and to allow contracting authorities to award the contract”.
Limited use of advanced procurement practices
Public procurement has been evolving with emerging advanced tools to drive efficiency throughout the public procurement cycle in satisfying the needs of the government and its citizens. Indeed, the OECD Recommendations of the Council on Public Procurement calls upon countries to develop and use tools to improve procurement procedures, reduce duplication and achieve greater value for money, including centralised purchasing, framework agreements, e-catalogues, dynamic purchasing, e-auctions, joint procurements and contracts with options. It also recommends countries to use public procurement as policy lever to pursue broad policy objectives such as stimulating innovation (OECD, 2015[15]).
Currently, Greece uses advanced procurement practices (such as dynamic purchasing systems and public procurement for innovation, including innovation partnerships) in a very limited way in the area of ICT public procurement, regardless of the fact that the Greek Public Procurement Law foresees these advanced procurement tools and schemes since it transposed the EU Directives on public procurement.
Dynamic purchasing system (DPS) is a procurement tool to support agile ICT procurement through a completely electronic system. Namely, under a dynamic purchasing system, new economic operators may apply for participating at any time throughout the life of the dynamic purchasing system, unlike framework agreements in which the participation of new economic operators is not allowed after the framework is set up (OECD SIGMA, 2017[16]). This dynamic feature allows contracting authorities to change suppliers more easily, and economic operators to continuously incorporate advances in technology. The DPS is thus very suitable for repeated purchases of standardised items, where technology developments occur fast. For instance, the European Commission adopted the use of DPS for procuring its cloud services. In addition to staying abreast of technological evolution, the tool significantly shortened lead times by 80% compared to the previously used framework agreements. In fact, even for complex tenders, the time between tender opening and final evaluation could be reduced to 2-3 weeks (Mercell, n.d.[17]).
From the evidence gathered during the fact-finding missions, stakeholders within MDG are still hesitant about the use of DPS, given the lack of experience with using this tool, and an overall risk-averse attitude towards new approaches in procurement, as procurement procedures are typically highly scrutinised at audit. This is despite the fact that DPS is regulated by Article 33 in L. 4412/2016 and the Greek e-procurement system fully supports the implementation of DPS from a technical perspective. To further support the uptake of DPS, HSPPA has recently translated a guide on DPS (“Dynamic Purchasing Systems: Use Guidelines” – “Δυναμικά συστήματα αγορών Κατευθυντήριες γραμμές χρήσης”) prepared by the European Commission (European Commission, 2021[18]).
Piloting the use of DPS with a limited scope, could be an initial starting point to test the technology, learn from a first iteration and potentially apply the DPS more widely going forward. Procurement officials would need to be supported by sufficient capacity-building to undertake such a pilot with confidence.
DPS in practice
When considering setting up a DPS, it is important to take stock of experiences from countries and organisations that have already undertaken this effort. Namely, in some countries, there is a relatively widespread use of this instrument, which allows drawing lessons learnt and gathering insights on practical aspects related to organisational and management arrangements for running a DPS, benefits accrued to the organisations and suppliers, as well as key success factors. The example from the Danish central purchasing body SKI (Statens og Kommunernes Indkøbsservice) provides a detailed example of DPS implementation for standard software (see Box 4.8).
Furthermore, Central Purchasing Bodies across Europe have also implemented DPS in recent years. Their experiences are discussed in detail in Annex A.
The Danish central purchasing body SKI (Statens og Kommunernes Indkøbsservice) is a limited company owned by the Danish state and Local Government Denmark (KL). While it mainly serves Danish local authorities, its services can be accessed by any public entity in Denmark. It currently manages 45 framework agreements covering a wide range of purchasing categories, such as ICT, travel, vehicles, professional services, food etc. Its procurement activities represent 2.5% of the Danish procurement spend.
SKI also makes use of five DPS for several purchasing categories, and recently introduced a DPS for the purchase of standard software. As reported by the organisation, three main characteristics need to be taken into DPS when thinking about implementing a DPS. For starters, a fast-developing market makes a DPS an attractive instrument. Second, it should be considered whether the market is characterised by few large suppliers or a multitude of small suppliers. And finally, the level of maturity of customers is also relevant in this process, too.
Standard software is defined as software that is prefabricated and commercially available, otherwise known as COTS (Commercial Off The Shelf). The DPS for standard software is divided in two categories: the first category based on price-only award and the second category taking into account best-price quality ratio. For the second category, contracting authorities using the DPS may chose quality criteria based on which to award their specific procurements based on pre-defined options. The contracting authority may decide which of the criteria to use and how to weigh them.
To support the use of the DPS, SKI defined standard terms and documentation. The documentation related to the DPS is available online, including partially in English.1 It includes the following documents:
In addition to the tender documents listed above, SKI prepared a guide for applicants on how to apply for admission to the DPS, as well a supplier guide on how to submit a tender through the DPS.
The DPS for standard software has so far gathered as many as 235 suppliers (as of April 2022). Despite initial concern that contracting authorities (SKI’s clients) would receive too many offers for each of their call-offs, this did not materialise. Instead, contracting authorities were able to benefit from a health rate of competition and a high participation rate of SMEs in the DPS. In fact, requirements for acceptance in the DPS are not overly extensive and thus promote SME participation. Furthermore, client satisfaction with the DPS is high.
The initial set-up and administration of the DPS necessitates investment from a technical and legal perspective. In fact, SKI invited a group of suppliers to participate in the development process. The investment, however, it not dissimilar for the set-up of a framework agreement, which equally requires careful preparation. A learning curve was necessary to deal with the first-time registration of suppliers into the DPS, which included training and informing both suppliers and customers about the use of a DPS.
In terms of organisational set-up to put in place DPS, SKI does not have specialised unit responsible for DPS but considers it a shared responsibility between the procurement division, the contract management division and the business development division. Specialised skills and personnel dedicated to the DPS also work horizontally in the organisation. Going forward, SKI would like to further improve and digitalise the process of using DPS for its customers, i.e. making the process online as much as possible.
← 1. https://www.ski.dk/aftaler/se-aftale/?id=02060021.
Source: Information provided by SKI.
Public procurement for innovation
The scheme of public procurement for innovation is another emerging public procurement practice, allowing governments to boost innovation at both the national and local levels. The use of public procurement for innovation is essential in the context of improving digital governance that always requires high-level innovative solutions which might not be available in the market. Strategic use of public procurement for innovation (PPI) is defined as “any kind of public procurement practice (pre-commercial or commercial) that is intended to stimulate innovation through research and development and the market uptake of innovative products and services” (OECD, 2017[19]). It includes schemes such as pre-commercial procurement (PCP), procurement of innovative solutions and Innovation Partnership.
Public procurement for innovation has been promoted by the European Commission since 2007 with the Commission Communication entitled ‘Pre-commercial Procurement: Driving innovation to ensure sustainable high quality public services in Europe’ (European Commission, 2007[20]). The 2014 Directives have further anchored the concept of public procurement of innovation in the legal framework by introducing the Innovation Partnership as a specific procedure to facilitate innovation procurement in the EU. Other procedures, such as the competitive dialogue or the negotiated procedure, are also suitable to conduct innovation procurement. In this sense, public procurement of innovation is fully aligned with the European legal framework. This is also reflected by the fact that in 2021, the European Commission published a Guidance on Innovation Procurement in order to offer guidance for policymakers and practitioners throughout the European Union on how innovation procurement can improve the economic recovery of the European Union after the COVID-19 crisis with better public investment (European Commission, 2021[21]) (Box 4.9). The relevant procurement methods apply in Greece, too, as defined in L. 4412/2016. Specifically, including competitive dialogue (Article 30, 267), innovation partnership (Article 31, 268), and negotiated procedure with prior call for competition (Article 266). To support the implementation of innovation procurement HSPPA developed dedicated Technical Instructions on Innovation Contracts (Τεχνική Οδηγία 2/2018, Συμβάσεις καινοτομίας).
The guidance clarifies the concept of innovation procurement and gives concrete examples and practical tools for public buyers to implement procurement strategies that are open to innovators and to help them acquire the new solutions the market can provide.
The guidance covers topics relevant for public buyers in terms of innovation procuremeent, amongst others:
Engaging with the market (e.g. preliminary market consultation)
Using standards, open data, open interfaces and open source software
The guidance aims to help public buyers throughout the EU navigate the provisions of the 2014 EU directives on public procurement, as it illustrates how to open up public procurement to innovators, including start-ups and innovative small and medium-sized enterprises (SMEs). The guidance also includes technical advice, especially with regard to the management of intellectual property rights (IPR).
The guidance furthermore describes how public procurement procedures can help modernise public services with innovative solutions, and how they can generate economic growth and create jobs. The guidance in this regard is also addressed to policymakers. It covers topics that are necessary for an enabling policy framework for innovation procurement such as,
Source: (European Commission, 2021[21]).
Across the OECD, governments are using innovation procurement to address challenges related to digitalising government. This can be done by using a so-called “challenge-driven” approach, whereby the contracting authority formulates its problem as a challenge and invites economic operators to come up with solutions. An example of this is used by Scotland, as described below (Box 4.10).
In 2016, Scotland launched its CivTech approach to solve the problems of the public administration by using a challenge-driven approach to drive innovative solutions from the market. In the first stage, interested participants can submit their application. In the next stages, called ‘exploratory’ and “accelerating” respectively, selected teams further work on their solution in co-operation with the challenge owner (the challenge “sponsor”).
The exploratory phase has a duration of three weeks and leads to a refined solution. Participation to this phase is compensated with GBP 3 000 for each team. Only selected teams proceed to the next the accelerating phase, in which they are tasked to develop a prototype of the solution. This phase is compensated with GBP 20 000. Finally, the winning solution can receive a government contract to implement it. Over the course of the implementation of the programme, 26 public sector challenges have been solved. For instance, NHS Scotland has been able to create a flexible online registration system, and Transport Scotland improved its ability to detect and repair roads wholes through a dedicated app.
Source: (Polish Economic Institute, 2019[22]).
Greece is still behind in the use of this procurement scheme for innovation in the ICT procurement, and overall investment and adoption of ICT-based innovative solutions. Namely, its share of procurement dedicated to innovative ICT amounts to 2.9% of out of total procurement expenditure, compared to the European average of 3.5%. Furthermore, Greece ranked 13th position (low performer) out of 30 European countries in the adoption of ICT-based public procurement for innovative solutions, according to a 2018 benchmarking study carried out by the European Commission on national innovation procurement policy frameworks (European Commission, 2021[23]).
The role of these advanced procurement practices has not been fully recognised yet by the stakeholders in the field of digital procurement in Greece due to the lack of awareness and specific skills. Greece ranked at the 15th position (modest performer) out of the 30 European countries (27 EU Member States plus the UK, Switzerland and Norway) on national innovation procurement policy frameworks, according to the benchmarking study carried out by the European Commission (European Commission, 2021[23]). The study points out the weakness in the absence of various elements to promote the use of public procurement for innovation such as the action plan, spending target, monitoring system, capacity-building and assistance measures.
However, Greece has been taking initiatives at a country level. The General Directorate for Public Procurements (GDPP) of the Ministry of Development and Investments, as National Central Purchasing Body for goods and general services in Greece, is mandated with setting up within its structural organisation a National Competence Centre on Innovation Procurement in Greece, under the Procure2Innovate project funded by the European Union Horizon 2020 programme. The centre aims at promoting mainstreaming innovation procurement throughout Greek contracting authorities (See Box 4.11) The Ministry of Digital Governance could benefit from getting involved in the process of establishing this competence centre.
The main goals to be achieved by the Competence Centre for Innovation Procurement in Greece are:
Promoting and mainstreaming of innovation procurement throughout Greek Public authorities.
Increasing familiarisation of Greek contracting authorities and entities with innovation procurement procedures facilitating also networking with relevant contracting authorities from other EU countries. In this manner, they will increase knowledge for their sector innovation needs and as a result their participation in EU PCP/PPI procedures.
Enhancing capacity building of procurers in the field of innovation procurement by designing educational courses in PCP/PPI procedures as well as in IPRs, in co-operation with the National Training School for Public Administration.
Enabling procurers in Greece to identify public needs that could be tackled through innovative solutions
Increasing the percentage of contracting authorities implementing PCP/PPI procedures.
Boosting the participation of economic operators in PCP/PPI procedures and as a result raising their economic activities in procurement markets.
Using innovation procurement as a springboard for achieving strategic goals in a national or regional economy level by setting horizontal policies in the areas of entrepreneurship, economic, financial and competition policy.
Facilitating the rapprochement between the demand and the supply (of innovative solutions) sides in Greece.
Connecting innovation procurement with the Green Public Procurement network (circular economy, green economy) and thus enhancing the achievement of environmental targets.
Promotion of sectored policies in the area of health, smart cities, defence sector etc.
Preparing the next programming period for securing financing from European structural funds in order to promote innovation procurement in Greece.
Connection with universities in order to include courses regarding innovation procurement.
Opening opportunities to create new calls from EU projects (for innovative goods and services) through demand of contracting authorities due to the development of sectoral clusters of procurers specialised in production of innovative goods or services.
Providing technical assistance on the implementation through GDPP website and helpdesk.
Source: (Procure2Innovate, n.d.[24]).
In addition, international experiences identified in Poland and Finland show some options to promote the use of advanced practices such as starting by a pilot project, and supporting implementation with a competence centre. Poland has proven experiences in using PPI schemes such as pre-commercial procurement (PCP) and innovation partnership, starting from pilot projects. In particular, Poland has been using PCP to procure innovative solutions to advance the digital agenda. Namely, the National Centre for Research and Development (NCRD) of Poland, the agency in charge of ESIF R&I projects, initiated an e-Pioneer project financed under the ESIF Operational Programme "Digital Poland 2014-2020”. This initiative contributed to encouraging a larger pool of interested contractors to participate in the ESIF R&I projects by using PCP (see Box 4.12). Greece could benefit from considering the pilot initiative of this good practice in order to promote and support the use of advanced procurement practices in digital projects.
In Poland, the new Public Procurement Law (PPL) of 11 September 2019, which entered into force on 1 January 2021, directly links the procurement of innovation with a broader perspective of national policy.
National Centre for Research and Development (NCRD) have experiences in using the scheme of public procurement for innovation in ESIF-funded R&I projects. NCRD is in charge of carrying out tasks in the area of the scientific, technical and innovation policy including financing innovative solutions through R&D programmes. NCRD decided to launch pilot research programmes based on both procurement models: pre-commercial procurement (PCP) and innovation partnership, both of which are adapted to procuring innovative solutions that cannot be specified at the beginning of the procedure, i.e. those that require research and development works.
The first experience of NCRD related to pre-commercial procurement dates back to 2013. It was further developed in 2016 owning to the implementation of the e-Pioneer project, which aims to support talented programmers in solving identified social or economic problems with communication technologies (ICT). This is a joint project of NCRD and the Accelerators selected by NCRD, which uses the PCP model. PCP is not used directly by NCRD, but serves as an obligatory project implementation tool for the Accelerators selected by NCRD. The e-Pioneer project is financed under the ESIF Operational Programme “Digital Poland 2014-2020”.
For example, Akcelerator Accelpoint Sp. z o. o., one of the selected accelerators in the e-Pioneer project, developed a Cleanbox to fight against Covid-19 by using the PCP model. Cleanbox is an easy-to-use, modern device with antibacterial and antifungal properties, which eliminates 99.99% of microbes and bacteria, ensuring an additional antibacterial effect for up to 30 days. With its functionalities, the project helps to prevent common infections and answers the growing need for surface disinfection, contributing to the fight against the spread of coronavirus. The MVP (a Minimum Viable Product, or, to put it simply, a demo version of the target product) was created and tests were successfully carried out using the infrastructure of the public partner (Municipal Sport and Recreation Centre in Lublin).
NCRD promoted themselves as an agency which not only finances innovation but also carries out innovative activities itself. It presented new forms of research programme implementation to the market and the potential contractors to encourage a larger pool of interested contractors to participate in the ESIF R&I projects.
With the experience gained so far and the competences built within the organisation, NCRD has been working since 2020 to launch further initiatives based on pre-commercial procurement. By doing so, NCRD aims at ensuring that this instrument is permanently included in its portfolio of support instruments.
Source: (Public Procurement Office of Poland, 2020[25]).
Finland set up a target of public procurement for innovation and established a Competence Centre for Sustainable and Innovative Procurement (KEINO), in order to support the use of innovative public procurement procedures (see Box 4.13).
The Finnish government set a target of public procurement for innovation at 10 % of total procurement, in order to increase competitiveness of the country. In March 2018, the Finnish Competence Centre for Sustainable and Innovative Procurement (KEINO) was established as a joint initiative of eight founding members from both the public and private sector, including the CPBs Hansel and Kuntahankinnat. Additional members of the consortium are Motiva Ltd, the Association of Finnish Local and Regional Authorities (FLRA), VTT Technical Research Centre of Finland Ltd, the Finnish Funding Agency for Innovation Business Finland and the Finnish Environment Institute SYKE. KEINO serves as a network-based competence centre to support innovation procurement in Finland.
KEINO’s three main objectives are:
Increasing the number of innovative public procurement procedures in Finland
Promoting public procurement as a strategic tool to reach wider societal objectives
Fostering the dissemination of information and peer learning across contracting authorities
In order to reach these goals, KEINO supports the development of procurement competence in Finland through advisory services, events and networking meetings. It promotes peer learning among procurement professionals and it fosters closer collaboration and international networks on sustainable and innovative procurement topics. In addition, KEINO offers specific capacity building measures to public procurement professionals in the area of innovative procurement, such as issuing guidelines, disseminating best practises and case studies, and providing templates and tools.
Source: (European Commission, 2020[26]).
Contract implementation
The contract implementation stage should not be overlooked, as it represents the stage in which the solution is actually delivered by a supplier. Examples for poor performance are cost overruns, contract modifications, additional works and vendor lock-in.
The feedback gathered from conversations with stakeholders pointed out that there is room for improvement in the contract delivery. Reportedly, suppliers tend to deliver the minimum on required goods and services. While it is not clear to what extent minimum performance by suppliers undermines the overall contract implementation, and whether it affects all contracts equally, it should be noted that contracting authorities have limited awareness about the links between proper tender preparation and supplier performance. Namely, the pre-tender stages such as early market engagement and user involvement allow to have very solid expectations about what the market can deliver and at what price, and therefore limit the risk of poorly drafted tender documentation (from selection criteria, to award criteria, technical specification and even contract clauses). Conversely, well drafted tender documents and contract (contract clauses) mitigate risks for contract underperformance.
Overall, contracting authorities in Greece, however, place limited emphasis on the planning stages of a procurement. As outlined in this report, this phenomenon also applies in the context of procurement of digital projects.
From stakeholder conversation, it emerged that final beneficiaries often lack the skills and capacity to take full ownership of the solutions that has been procured. As such, they may not fully benefit from the gains of digitalisation. Furthermore, there are no comprehensive monitoring processes and reporting mechanisms that would allow beneficiaries to provide feedback, such as a satisfaction survey. Without this kind of feedback loop, it is difficult for procurement bodies to continuously improve their processes and interactions with beneficiaries.
Increasingly, contracting authorities across the OECD have been turning to agile approaches of project development, including procurement functions, and particularly when dealing with digital technology and ICT. At its core, agility refers to the ability to create and respond to change. The concept derives from the Agile Manifesto2 (Manifesto for Agile Software Development) published in 2001 (Agile Alliance, 2001[3]). The “Agile” approach takes its name because this word represents adaptability and response to change.
In the context of public procurement, applying agile principles refers to using collaboration, working flexibly and allowing for adaptation, proceeding in iterations and reviewing on a continuous basis. As such, the software end product is the result from a collaborative effort between developers and end-users (Deloitte, 2017[27]). In contrast, the common way of procuring ICT goods and services has been the waterfall method, which consists of a structured step-by-step approach through the various phases of conception, initiation, analysis, design, testing, and implementation. This approach implies involving users at an early development stage to have fully-defined and accepted set of requirements and needs that are not supposed to change in the course of implementation; and at the end of the process when end-users are involved and have the opportunity to provide feedback. This approach leaves no room for interaction and engagement with user on an ongoing basis, and puts pressure on ‘getting it right’ at the first step because the approach does not foresee the return to an earlier stage. As such, the waterfall model is considered less flexible and effective in software development (OECD, 2022[13]).
Despite a number of benefits of using the waterfall approach for software development, criticism revolves around the fact that the uncertainty around end-user’s needs, and the accuracy required of cost estimates at early stages of preparation are often not reflective of realities. Instead, the approach can be useful in the context of well-known technologies and stable user preferences. Data from Poland give insights into the success rates of agile projects versus waterfall projects. Namely, the success rate for agile projects lies at 49% compared to 14% when using waterfall. More importantly, the failure rate for waterfall is much higher (29%) compared to agile (9%), underscoring the strong risk mitigating aspects of an agile approach (Polish Economic Institute, 2019[22]).
Currently, Information Society S.A. makes use of the waterfall approach in the procurement of digital projects. As discussed, this model provides a clear framework for operations, but increases the risks of Information Society S.A. working for several months (or years) on services that no longer meet end-user needs once finalised. Furthermore, contracts for digital and ICT projects tend to be very large, thereby posing a further challenge to a more nimble and agile procurement process. As reported by private sector stakeholders, the size of the Greek public sector ICT market is relatively small compared to its private sector equivalent. Partly due to fiscal constraints of the past decades, the market has been characterised by a small number of large contracts, which are highly attractive to the pool of available suppliers. Such large contracts, however, also increase the incentives for litigation, as suppliers are aware that only few large opportunities are present on the market.
Increasing agility in the procurement process also foresees the breakdown of complex and large projects into smaller and iterative projects. The agile approach thus requires the use of modular contracting as a procurement strategy, i.e. procuring a series of tightly-scoped procurements to implement technology systems in successive and interoperable increments. The benefits of such a strategy include risk mitigation, reduction of vendor lock-in, as well as increased speed of delivery (Mark Headd, 2018[28]). Indeed, with shorter contracts, the risk of failure of the entire project is much reduced, and instead is contained to a small piece of the system. Similarly, planning a successful contract for a timeframe between six and twelve months is an easier undertaking compared to planning for a six-year horizon (Laura Gerhardt, 2019[29]).
Furthermore, these smaller and modular contracts can be accessible to a larger pool of suppliers, including innovative SMEs and start-ups. At the same time, procuring multiple contracts increases the need to procure much faster and requires re-thinking the procurement process through this lens. Shortening the procurement time and breaking up large contracts has been the approach taken in the State of California (Box 4.14).
In 2015, California’s Health and Human Services Agency sought to replace its outdated Child Welfare Services case management system. The agency intended to change its previous modus operandi, i.e. instead of spending years on procurement and development, it would break up the large project into smaller pieces and deliver value more iteratively. However, such an approach required the rethinking the procurement process, including taking into account systems integration if multiple vendors were to be involved.
The Agency and with its Office of Systems Integration (OSI) reached out for support from 18F, a federal office within the U.S. General Services Administration, and Code for America, a non-profit that enhances local governments’ efforts on technological innovation. Previously, OSI had a longstanding history of relying on a single vendor to be the systems integrator. For this project, it decided to play the role of systems integrator itself.
The first question mark for the project team was related to the compatibility of an agile procurement practice with the legal framework. Importantly, the major shift to iterative and agile procurements did not require any legal changes.
Once that potential hurdle was cleared, California identified the need to procure much faster to ensure the success of a modular and iterative implementation of the procurement process. To do so, it adopted a model used by 18F, which consults of creating and pre-approving a pool of suppliers capable of doing agile work and responding to the small procurements. In a short amount of time, several types of vendors qualified for the procurement process, including smaller suppliers. Gradually, the Office has expanding this new thinking around procurement away from specific products, but towards defining a way of collaborating with suppliers by sharing and co-owning projects.
Source: (Rath, 2017[30]); (OECD, 2022[13]).
Since agile development relies heavily on the interaction and collaboration between end-user, buyer and supplier, the process of contract management becomes a key success factor for the implementation of digital projects. This requires regular engagement throughout the process and a partnership approach to contract delivery. Trust in the relationship with suppliers is also an essential ingredient to the success of agile delivery, given that contractual non-performance cannot be defined as easily in a context where the contract scope is flexible. Often this represents a cultural barrier for public buyers, where typically civil servants rely on heavily detailed contract specifications. To overcome such barriers, the United States Digital Service developed the TechFAR Hub guidelines, which promote a more lenient interpretation of contract rules (Eggers and O’Leary, 2017[31]).
TechFAR Hub
The TechFAR Hub is an online platform created by the U.S. Digital Service aimed at bringing together procurement practitioners dedicated to the procurement of digital services. Among other resources, the Hub hosts the TechFAR Handbook. It also includes interactive tools and learning resources dedicated to the execution of digital strategies through contracts.
TechFAR Handbook
The TechFAR Handbook is a comprehensive guide for public sector agencies on to conduct how agile procurement processes within the flexibilities provided for by the U.S. Federal Acquisition Regulation (FAR). The TechFAR Handbook provides practical tips, sample language and provisions relevant for agile software development. It was launched in 2014 to support the implementation of the Digital Service Playbook. The Playbook describes 13 “plays” that support the development of effective government digital services.
As part the TechFAR Handbook, the goals for modular contracting and agile software development are outlined:
Shared Goals of Modular Contracting and Agile Software Development
1. Improvement in investment manageability and budgetary feasibility
3. Frequent delivery of usable capabilities that provide value to customers more rapidly
Source: (The U.S. Digital Service, n.d.[32]).
While moving from a waterfall approach to agile and modular contracting certainly requires a learning curve, it would be beneficial for Greek contracting authorities to gradually move towards an agile implementation methodology, especially for those projects where the technology and user needs are bound to evolve over time. As a starting point, Greek authorities could consider piloting such an approach, with a view of potentially scaling it up over time. In parallel, dedicated guidance and support material should be made available. Overall, a key goal of MDG should be to substantially reduce the duration of the procurement process. Applying modular contracting requires designing the development and procurement process in such a way to ensure its speedy execution. Finally, reducing the contract size in favour of multiple small projects would provide key benefits in terms of responsiveness to user needs, as well as limiting the risk of failure of large and expensive digital transformation projects.
Finally, despite potential hesitation from public buyers, using agile approaches in public procurement is not in contradiction with the regular procurement process, and does not require an adaptation of the regulatory framework. Similarly to other procurement practices discussed throughout this report, an agile procurement process also heavily relies on a solid preparation of the tender through needs assessment and early market engagement. In terms of procurement procedure, the regular process applies. All types of procurement procedures can be applied for agile implementation, though complex projects may be more suited for procedures that allow greater flexibility between the contracting authority and the supplier (e.g. competitive dialogue), in particular if the subject matter of the contract is not fully known from the onset. Specific agile elements may be included in the tender specifications and the contract conditions, such as modularity or multistage delivery. Selection criteria can ensure that suppliers have experience in agile working methods, while award criteria can give more weight to relevant aspects such as quality of UX design, or project management capacity. Importantly, contract conditions can be designed to introduce a greater degree of flexibility compared to more standard approaches.
To succeed in the implementation of transformational digital projects, the Greek government needs to rely on a pool of innovative and capable suppliers that are able to deliver quality solutions to the public administration. To this end, the Greek government needs to be considered an attractive client, in particularly given that the share of the ICT market by the public sector has been relatively small compared to the private sector (although the influx of funds from the Recovery and Resilience Facility (RRF) will likely change this dynamic). It needs to ensure a strong track record to increase competition and attract capable and innovative suppliers, especially in the context of an open and dynamic regional market such as the European Union. Speedy procurement procedures and payment processes are part of what constitutes an attractive client. At the same time, several barriers persist to making the public procurement market accessible to suppliers, particularly small, innovative start-ups and SMEs, notably the long duration of procurement cycle and competition that does not reward the highest quality offer, or innovative solutions. These aspects may deter small and innovative companies from participating in public sector bids, given that a significant amount of liquidity and capital is required to financially sustain a procurement process spanning over several years. Furthermore, as noted by procurement stakeholders, vested interests in the ICT market may also pose a barrier to participation of smaller companies. Namely, large ICT players often have more resources to lobby and engage with Government ICT leaders and politicians. This can contribute to the risk-aversion of smaller businesses who do not have the same financial capacity and access to lobbying resources.
From the buyer’s perspective, access to cutting-edge, innovative, start-up supplier eco-system including international companies remains difficult in their operations. The Greek ICT market overall is quite localised and characterised by small and medium-sized enterprises, as most companies do not surpass the 250 employees. In some instances, these SMEs represent Greek subsidiaries of international corporations.
Important work is ongoing to promote the involvement of new and innovative companies in the tendering arena, and is considered one of the priorities of Information Society S.A. Indeed, it has taken part into several activities that support the creation of a vibrant ecosystem of innovative suppliers. Information Society S.A. is also looking into how to harness international experience and expertise, if it is not yet available on the Greek market. Going forward, Information Society S.A. also plans to build an ecosystem for technology matching, further supporting the capacity and maturity of a local ICT ecosystem.
Similarly, the limited institutional capacities to plan, develop, procure and maintain ICT/digital projects increase the strategic relevance of the private sector as critical partners for the implementation of the DTB (Ministry of Digital Governance, 2021[33]). The strategy underlines the role of the GovTech ecosystem to support its implementation, and requires further actions to foster and strengthen this ecosystem to take this expected role.
While digital projects are often custom-made, procuring ICT and digital goods and services also consists purchasing standardised, highly-demanded and off-the-shelf solutions. For such purchases, centralising demand is an effective strategy to generate economies of scale, ensure efficient procurement operations and increased coherence and interoperability of solutions. Indeed, centralisation of procurement operations has proven to lead to significant benefits, such as better prices, lower transaction costs, as well as increased capacity and expertise since such procurements are typically carried out by specialised procurement professionals. In the context of technological change, specialisation and centralisation are necessary to accelerate delivery. Namely, they facilitate alignment on cultural aspects as well as coordination across departments and line ministries that may otherwise hamper the implementation of digital transformation. Over time, knowledge and capability become more pervasive as they spread across government entities. At the same time, centralisation entails risks, if not managed adequately (OECD, 2022[13]). When dealing with multiple clients, there is a risk of repeatedly encountering similar challenges as well as knowledge loss, which slows delivery and stifles innovation. Hence, it is important to ensure a learning effect within the CPB’s clients.
Recognising the potential for centralisation of ICT, several OECD countries have already introduced specialised ICT Central Purchasing Bodies (CPBs). For instance, Germany introduced the Central Office for IT Procurement within the Federal Procurement Office of the Federal Ministry of the Interior (Zentralstelle für IT-Beschaffung) in 2017. This entity is the central point of contact for ICT procurement at the federal level. In its initial phase, ZIB was tasked with an advisory function, i.e. supporting users throughout the entire procurement process, from the first notification of a need, through planning and commissioning, including contract management throughout the length of the agreement. Since its creation, it has moved to an implementation phase, whereby it also carries out tenders for on behalf of contracting authorities (OECD, 2019[34]).
Similarly, Ireland introduced centralised ICT procurement in order to deliver on its Public Service ICT Strategy. This effort is led by the Office of Government Procurement, i.e. Ireland’s CPB. Ireland’s approach focuses on whole-of-government IT applications that are crosscutting and impact multiple departments, as opposed to technologies that are specific to one ministry, and streamlining applications such as payroll or messaging, while leaving devolved decision-making for agency-specific technologies with the relevant agency.
In the context of MDG, no such centralisation or advisory function of digital technology and ICT needs has been identified. It is unclear to what extent common needs and repetitive needs within MDG are bundled together and procured via dedicated framework agreements by the Department of Procurement and Logistics. To create or strengthen such a function, MDG could also consider co-operating with the central-level CPB in Greece, namely the General Directorate of Public Procurements of the General Secretariat of Commerce and Consumer Protection of the Ministry of Economy, Development and Tourism.
This function can be supported by leveraging existing project approval processes or future efforts to adopt a whole-of-government ICT portfolio management to identify common needs of highly standardised and demanded ICT/digital products or goods. This information, along with other data sources that provide further information on purchasing behaviour, can serve as basis to analyse and conduct centralised procurement exercises.
One of the most pressing challenges in the Greek public sector to effectively implement the DTB is the limited institutional capacities to plan, manage, procure and implement ICT/digital projects. The restricted managerial and technical capacities in line ministries to carry out digital transformation projects has created a significant overload in MDG and its dependent units, including Information Society S.A., as they concentrate most of the digital expertise in the public sector and take complete control over the development lifecycle of these projects. As a consequence, line ministries take often a passive role and have limited ownership over the process, preventing a coherent and more decentralised implementation of the DTB.
The DTB includes several provisions on digital skills development to underpin the national digital transformation strategy, including the development of a national digital competence framework. Accordingly, OECD governments as the UK have developed professional capability frameworks to support institutions identifying skills and professional profiles needed to drive transformation (see Box 4.16). However, such an approach may be insufficient to address the existing digital talent and skill challenges. Concrete initiatives are needed to equip line ministries and their workforces with the skills needed to implement the goals set in the DTB. This includes assessing existing competences and developing training programmes to effectively accompany the implementation of the DTB, including the management and procurement of ICT/digital projects.
Interviewees indicated the limited conditions to attract and retain digital talent in the public sector as one of the reasons for constrained institutional capacities to manage ICT/digital projects. As outlined previously, the Greek public sector observes a limited culture for horizontal collaboration and co-operation, which constrains the development of an agile culture to co-ordinate digital transformation projects. Similarly, organisational conditions to foster digital talent and skills are limited to date, including promotion and training schemes as well as attractive employment packages to keep the public workforce abreast of new practices in the ICT/digital domain. Management culture can further exacerbate existing issues of talent attraction and retention by not properly valuing staff, providing opportunities for career advancement as well as promoting a culture of co-operation necessary for bringing about the transformational changes sought by MDG.
Source: (OECD, 2021[35]).
Conversely, institutional and management culture may strengthen positive developments of collaboration, continuous improvement and talent attraction by setting the tone at the top. In this sense, Greece could benefit from promoting multidisciplinary teams in ICT/digital project development, bringing together digital professionals with non-digital backgrounds in designing and delivering digital investments. Merging digital and non-digital expertise can be helpful to create a culture for collaboration and a better understanding of the multiple policy, legal, and financial factors that may determine the scope and impact of digital transformation projects Figure 4.4).
Finally, a comprehensive approach towards equipping public sector organisations with the tools to implement digital transformation projects includes setting guidelines and standards that fosters coherence and alignment of efforts. Guidelines and standards are an effective way to enable system-wide transformation while promoting coherence and alignment in the development of ICT/digital projects both within and outside MDG. As outlined previously, this may include service design and delivery, agile management, and ICT/digital procurement standards and guidelines.
When looking at the availability of standards and common practices for ICT/digital projects, MDG has not set a concrete and actionable set of common tools that help empower service teams to have a more leading role in the development of these projects. This fosters incentives for line ministries to rely on MDG and Information Society S.A. to implement these projects on their behalf and take a less active role as beneficiaries rather than project owners. Further developing a comprehensive set of standards and guidelines for digital transformation in the Greek government can help build capacities across the public sector and reinforce ownership over digital initiatives.
In March 2017 the UK Government published the digital, data, and technology profession capability framework describing the roles and skills needed in government to support digital transformation efforts. The framework provides a detailed description of the digital, data, and technology roles, listing the skills required for each role seniority. Job roles are grouped in job families (data, IT operations, product and delivery, quality assurance testing (QAT), technical and user-centered), providing specific details on expected skills for each profile.
The framework has helped the UK government develop a shared understanding of the required skills for DDaT roles, facilitating cross-government communities of DDaT professionals, mapping career paths, identifying skills gaps and developing new training programs in Government. The DDaT capability framework supports institutions to:
understand what skills are needed by professionals in particular jobs
identify skills that need development to help career progression
Source: Own elaboration, adapted from (Central Digital and Data Office, 2019[36]) and (OECD, 2021[35]).
The implementation of digital projects requires a specific set of procurement and digital competencies. As reported by stakeholders during the fact-finding missions, there has been limited investment and recruitment of personnel with specific ICT skills over the past several years. As a result, procurement entities are short in specialised personnel that can bridge the divide between expertise in digital technology and procurement competence. Such professional profiles would be needed to effectively translate ICT needs into clear procurement documents (i.e. technical specifications, selection criteria, award criteria). As a result, stakeholders consider understaffing and the lack of appropriately skilled resources as one of their main challenges in the implementation of procurement of digital projects.
Importantly, stakeholders within MDG see the potential for training and upskilling existing staff as a means to overcome some of the skills gaps. In fact, the required capabilities are often available in-house throughout the Greek government, but are not organised to ensure knowledge transfer and capacity-building. In parallel, better planning of the project pipeline, and effective prioritisation of projects are also useful tools to manage a growing workload within existing constraints. Initial efforts in prioritisation have proven effective in delivering more effectively on multiple projects.
Across the OECD, countries are increasingly recognising the need for specific skills related to digital and ICT procurement, covering the full procurement cycle as well as agile methodologies. The US for instance, invested heavily in developing dedicated training (Box 4.17).
Recognising that the digitalisation of public services often requires IT skills that are difficult to master for public procurement professionals, the United States Digital Service joined forces with the Federal Acquisition Institute (FAI) to set up the Digital Acquisition Professional Training, i.e. a specialised programme that entails training and a certification.
The programme teaches federal procurement professionals how to design procurement processes for IT and digital services that are flexible and innovative. The aim is for participants to become ambassadors for change. Students that complete the programme acquire learning credits and receive a certificate from FAI in FAC-C Core Plus Specialization in Digital Services. After a pilot phase, the training is offered to all professional levels across several agencies.
Source: (European Commission, 2020[37]); (U.S. Digital Service, n.d.[38]).
Evidence indicates that MDG does not have a comprehensive monitoring and assessment policy for digital transformation initiatives in the public sector. Despite the DTB indicates the future development of a monitoring system for the activities comprised in the strategy, it does not provide further details about what and how the system will measure digital government progress and performance. Similarly, evidence collected indicates that systematic use of monitoring tools in MDG is limited and responds to efforts scattered across the different phases of ICT/digital project development and the respective departments/units involved in the process. Considering the fragmented approach towards ICT/digital project management within MDG, efforts to improve monitoring should be part of a broader strategic approach for benefits realisation within the Ministry, involving all relevant stakeholders.
Evidence indicates that currently there are no clear indicators to track progress and/or performance of ICT/digital projects. Additionally, KPIs can help create incentives within the public sector if managed in a transparent and open way, fostering accountability of all related stakeholders involved in the process. For example, in Australia some regional governments are using online dashboards to track progress and monitor the implementation of their digital transformation strategies, including progress of ICT/digital projects through data visualisation and open data (see Box 4.18). In line with the progress and existing culture for open government data in the country, MDG can leverage this information and make it available to relevant stakeholders to foster transparency, reinforce accountability and align incentives to pursue the strategic goals set in the DTB.
Additionally, performance data can be an effective way to feed development cycles and take concrete actions in the redefinition of strategic priorities. Considering the steering role of MDG in setting goals and priorities for digital government, having a sound approach to assess the performance of ICT/digital projects as well as to proactively use this information to redefine priorities and set new goals is critical to secure the realisation of intended outcomes.
Similarly, a user-driven approach for digital government calls for improved and standardised ways to assess the experience of end-users with ICT/digital projects. This implies establishing coherent feedback mechanisms for end-users (citizens and businesses) to communicate their experience with a certain service. However, in line with the limited initiatives to understand and meet the needs of end-users, MDG does not have a standardised approach to measure user satisfaction in digital government. Given the nature of some ICT/digital projects (service design and omni-channel delivery) and the priority set in the DTB for the implementation of the centralised service platform gov.gr, MDG would benefit by setting a comprehensive user satisfaction measurement policy.
Currently, the DTB includes a dedicated provision to collect and process user feedback (the Integrated Citizen Relationship Management System). In order to improve the assessment of ICT/digital projects, MDG should accelerate its implementation, setting a common methodology for the Greek public sector organisations to assess user satisfaction in line with the limited institutional capacities observed to support the development of ICT/digital projects.
In Australia, the Governments of Queensland and Victoria are using online dashboards to promote transparency and reinforce oversight on the deployment of ICT/digital projects in the public sector. These centralised platforms provide relevant KPIs to support the monitoring of their respective digital strategies, e.g., on departments leading ICT spending, identifying the number of projects and their status, expenditure, and timeframe, flagging cost overruns, and development delays.
Queensland Digital Projects Dashboard
The Digital Projects Dashboard displays presents ICT/digital project data published by each department on the Queensland Government Open Data Portal. The platform provides timely information on the progress of all initiatives carried out by the State Government and the alignment of each project with the digital priorities set by the government. Users can explore the status of digital projects by department, approved expenditure, and digital priority.
The online platform also presents information on ICT tenders, including planned, open and closed tenders. The platforms reuse the data of the procurement platform QTenders, integrating different data sources to foster transparency and openness. The platform has been also used to communicate to small providers about potential opportunities in ICT procurement at the government.
The Victorian Government ICT dashboard
The Victorian Government ICT dashboard covers digital projects over A$1M, providing details on cost, timeframes, phases of development, performance status, and beneficiaries. The dashboard is updated quarterly and is part of the Information Technology Strategy 2016-2020. The initiative was implemented as part of the State Auditor General recommendations in 2015, reflecting the need for monitoring mechanisms to support the implementation of the digital strategy. Through the Digital Strategy and Transformation branch, the Department of Premier and Cabinet (DPC) is responsible for the dashboard based on the information reported by departments and agencies in the State Government.
Source: Own elaboration, adapted from (Queensland Government, 2021[39]) and (Victorian Government, 2021[40]).
Colombia’s MinTIC developed the Digital Government Index as a measurement tool to support the implementation of the digital government strategy. The measurement provides disaggregated data on the performance of national and territorial entities regarding digital government policy. The Index allows the MinTIC to assess good implementation practices and identify opportunities and gaps. The MinTIC publishes the index results at a disaggregated level using an interactive platform, and the data is available on the open data platform of the Government of Colombia. The data is collected on an annual basis, using a survey based on the three enablers and five objectives of the digital government strategy:
The Ministry of Interior and Safety in Korea measures the compliance of the performance, operation, and cost efficiency management and impact on users every three years. The Korean Government uses mobile app uptake to assess part of the impact by measuring the number of downloads, the satisfaction levels, latest updates. At an external level, the government used the Digital Government Usage Survey and the National Survey on Digital Divide, to gather information about service usage rate and internet accessibility.
The system considers specific activities in the four different steps on the development of information system projects:
Planning & budgeting: Establishing impact and performance assessment plan for a project.
System development: Pre-development project consultation and adjusting and finalising assessment plan.
Operation and Management: collecting data, measuring indicators, and assessing projects.
Feedback: Publishing an annual report, evaluating performance management level of government entities, and providing recognition.
After the assessment, systems can be categorised into five different results: to be maintained, to be re-developed, to be improved, to be merged, or to be terminated.
Source: Own elaboration, adapted from (Ministry of Interior and Safety, 2021[42]).
References
[8] Agency for Digitalisation (n.d.), ICT portfolio management, https://en.digst.dk/ict-portfolio-management/ (accessed on 27 September 2021).
[3] Agile Alliance (2001), Manifesto for Agile Software Development, http://agilemanifesto.org/.
[36] Central Digital and Data Office (2019), Making the Digital, Data and Technology Capability Framework more user friendly, https://digitalpeople.blog.gov.uk/2019/10/24/making-the-digital-data-and-technology-capability-framework-more-user-friendly/ (accessed on 27 September 2021).
[27] Deloitte (2017), Going Agile: The new mind-set for procurement officials, https://www2.deloitte.com/us/en/insights/industry/public-sector/agile-in-government-procurement-mindset.html.
[10] División Gobierno Digital (2021), Instrucciones para la formulación de proyectos TIC - EVALTIC 2022, https://digital.gob.cl/transformacion-digital/estandares-y-guias/instrucciones-para-la-formulacion-de-proyectos-tic-evaltic-2022/ (accessed on 27 September 2021).
[31] Eggers, W. and J. O’Leary (2017), “Agile in Government: A playbook from the Deloitte Center for Government Insights”, https://www2.deloitte.com/content/dam/insights/us/articles/3897_Agile-in-government/DUP_Agile-in-Government-series.pdf (accessed on 27 September 2021).
[21] European Commission (2021), COMMISSION NOTICE - Guidance on Innovation Procurement - C(2021) 4320 final, https://ec.europa.eu/docsroom/documents/45975.
[23] European Commission (2021), The strategic use of public procurement for innovation in the digital economy, https://op.europa.eu/en/publication-detail/-/publication/7f5a67ae-8b8e-11eb-b85c-01aa75ed71a1/language-en (accessed on 5 October 2021).
[18] European Commission (2021), “Δυναμικά συστήματα αγορών Κατευθυντήριες γραμμές χρήσης”, https://www.eaadhsy.gr/images/docs/ET0121340ELN.el.pdf (accessed on 22 February 2022).
[37] European Commission (2020), “ProcurCompEU: Study on professionalisation of public procurement in the EU and selected third countries”, https://doi.org/10.2873/1198.
[26] European Commission (2020), “Study on professionalisation of public procurement in the EU and selected third countries”, https://doi.org/10.2873/1198.
[20] European Commission (2007), “Pre-commercial Procurement: Driving innovation to ensure sustainable high quality public services in Europe”, https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52007DC0799&from=EN (accessed on 7 February 2022).
[11] European Commission (n.d.), “Streamline public procurement procedures”, https://ec.europa.eu/regional_policy/sources/good_practices/GP_fiche_19.pdf (accessed on 27 October 2021).
[29] Laura Gerhardt, M. (2019), Why we love modular contracting, https://18f.gsa.gov/2019/04/09/why-we-love-modular-contracting/.
[28] Mark Headd, E. (2018), Modular contracting and working in the open, https://18f.gsa.gov/2018/10/25/modular-contracting-and-working-in-the-open/.
[17] Mercell (n.d.), 80% decrease in tender lead times with the implementation of a Dynamic Purchasing System supported by Mercell, https://www.negometrix.com/en/cases/european-commission/.
[4] Meyer, B. (2014), Agile! The Good, the Hype and the Ugly, Springer International Publishing: Switzerland.
[33] Ministry of Digital Governance (2021), “Bible of Digital transformation 2021-2025”, https://digitalstrategy.gov.gr/ (accessed on 17 August 2021).
[1] Ministry of Digital Policy Telecommunications and Information (2016), National Digital Strategy, 2016-2021 (ΕΘΝΙΚΗ ΨΗΦΙΑΚΗ ΣΤΡΑΤΗΓΙΚΗ).
[42] Ministry of Interior and Safety (2021), Assessing Impacts of Digital Government in Korea.
[41] MinTIC (2021), Índice de Gobierno Digital, https://colombiatic.mintic.gov.co/679/w3-propertyvalue-36675.html (accessed on 2 November 2021).
[6] New Zealand Government (2019), Assurance guidance for Agile delivery, https://www.digital.govt.nz/dmsdocument/115-assurance-guidance-for-agile-delivery-full/html (accessed on 27 September 2021).
[13] OECD (2022), Towards Agile ICT Procurement in the Slovak Republic: Good Practices and Recommendations, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/b0a5d50f-en.
[35] OECD (2021), “The OECD Framework for digital talent and skills in the public sector”, OECD Working Papers on Public Governance, No. 45, OECD Publishing, Paris, https://doi.org/10.1787/4e7c3f58-en.
[34] OECD (2019), Public Procurement in Germany: Strategic Dimensions for Well-being and Growth, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/1db30826-en.
[19] OECD (2017), Public Procurement for Innovation: Good Practices and Strategies, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/9789264265820-en.
[15] OECD (2015), OECD Recommendation on the Council on Public Procurement, http://www.oecd.org/gov/public-procurement/recommendation/OECD-Recommendation-on-Public-Procurement.pdf (accessed on 28 May 2018).
[9] OECD (2015), “Recommendation of the Council on Budgetary Governance”, https://www.oecd.org/gov/budgeting/Recommendation-of-the-Council-on-Budgetary-Governance.pdf (accessed on 22 September 2021).
[2] OECD (2014), “Recommendation of the Council on Digital Government Strategies”, Public Governance and Territorial Development Directorate, Vol. July.
[12] OECD (n.d.), OECD Public Governance Policy Papers, OECD Publishing, Paris, https://doi.org/10.1787/14e1c5e8-en-fr.
[16] OECD SIGMA (2017), Procurement Tools, https://www.oecd-ilibrary.org/docserver/5js4wzv3mtvj-en.pdf?expires=1629455540&id=id&accname=ocid84004878&checksum=492A5A5C9F0D36277DA73F7F1DA88CC8 (accessed on 20 August 2021).
[14] OECD-OPSI (2021), Cooperative Procurement Emerging Markets Initiative, https://oecd-opsi.org/innovations/cooperative-procurement-emerging-markets-initiative/.
[22] Polish Economic Institute (2019), “GovTech - New technology in the public sector”.
[24] Procure2Innovate (n.d.), National Competence Centre on Innovation Procurement in Greece, http://www.procure2innovate.eu (accessed on 25 August 2021).
[25] Public Procurement Office of Poland (2020), Public Procurement of Innovation, https://www.uzp.gov.pl/__data/assets/pdf_file/0017/45503/Zamowienia_publiczne_na_innowacje_EN_WCAG-2021-01-12.pdf (accessed on 16 April 2021).
[39] Queensland Government (2021), Digital Projects Dashboard, https://www.qld.gov.au/digitalprojectsdashboard (accessed on 2 November 2021).
[30] Rath, D. (2017), Agile Acquisitions: Rethinking Public-Sector Purchasing, Government Technology, https://www.govtech.com/budget-finance/GT-September-2017-Agile-Acquisitions-Rethinking-Public-Sector-Purchasing.html (accessed on 6 December 2022).
[32] The U.S. Digital Service (n.d.), TechFAR Hub, https://techfarhub.cio.gov/handbook/general-considerations/.
[38] U.S. Digital Service (n.d.), Digital IT Acquisition Professional Training (DITAP), https://techfarhub.cio.gov/initiatives/ditap/.
[7] UK Government (2017), Assurance for agile delivery of digital services, https://www.gov.uk/government/publications/assurance-for-agile-delivery-of-digital-services (accessed on 27 September 2021).
[5] US Digital Service (n.d.), Digital Service Playbook, https://playbook.cio.gov/ (accessed on 27 September 2021).
[40] Victorian Government (2021), Victorian Government IT project dashboard, https://www.vic.gov.au/IT-project-dashboard (accessed on 2 November 2021).