6. Developing a comprehensive measurement framework in Malta
This chapter discusses the development of a comprehensive public procurement performance measurement framework in Malta. To support Malta in setting such a framework, the chapter highlights key elements to consider in setting up a performance measurement framework, including i) the category of indicators (efficiency/effectiveness, compliance, strategic ones), ii) the procurement stage, and iii) the relevant stakeholders. The chapter also reviews the availability of data to assesses the procurement processes.
Public procurement is a key tool for providing public services to citizens. From an economic perspective, it should contribute to an efficient and effective management of public resources. (OECD, 2015[1]) How countries spend taxpayers’ money and how they deliver services makes procurement an increasingly important tool to go beyond the economic aspects of “value for money.” (OECD, 2021[2]). As seen in previous chapters, public procurement has been widely used as a strategic tool for achieving different policy objectives and supporting governments in advancing their strategic agenda.
To achieve these different objectives, countries implemented specific strategies and used specific tools and mechanisms throughout the procurement cycle. In this context, measurement frameworks are needed to i) assess progress and achievements and ii) to identify potential gaps. This will enable governments and contracting authorities to take relevant actions and/or tailoring specific strategies. The OECD Recommendation on Public Procurement highlights the need to drive performance improvements through evaluation of the effectiveness of the public procurement system, from individual procurements to the overall system, at all levels of government where feasible and appropriate (OECD, 2015[1]) (see Box 6.1).
Public procurement systems should collect consistent, up-to-date and reliable information and use data on prior procurements, particularly regarding price and overall costs, in structuring new needs assessments, as they provide a valuable source of insight and could guide future procurement decisions.
ii. Develop indicators to measure performance, effectiveness and savings of the public procurement system for benchmarking and to support strategic policy making on public procurement
Source: (OECD, 2015[1])
This evaluation is usually conducted by defining key performance indicators (KPIs) that are monitored over time. The performance of public procurement systems can be assessed at three levels: tender/ contract level (micro-level), contracting authority level (macro level) and national level (meta level) (see Figure 6.1. . There are clear links between the 3 levels as each level is feeding the upper one (OECD, 2018[3]).
Source: (OECD, 2018[3])
Only 45% of countries responding to the 2018 survey on the Implementation of the 2015 OECD Recommendation of the Council on Public Procurement1 mentioned the establishment of a formal performance management system, including KPIs. (OECD, 2019[4])
Currently in Malta, no KPIs related to public procurement have been set. Aware of the benefits of establishing such indicators, however, the DoC is fully committed to advancing the performance measurement agenda. Indeed, monitoring procurement activities with the aim of ensuring adherence to the Maltese regulations and to ensure that the results of these monitoring activities are made available to the public through appropriate means of information is part of the duties of the DoC (DoC Malta, 2016[5]). The Procurement Policy and Quality Assurance Directorate oversees the monitoring of public procurement activities. In addition, it is worth mentioning that as an EU member state, Malta is committed to sending its “Procurement Monitoring Report” to the European Commission on a regular basis in view of the Member States' reporting process under the Directives 2014/23/EU, 2014/24/EU and 2014/25/EU. The first report was published in 2017 and the second one in 2021 (European Commission, 2021[6]). This report includes key information on the procurement system of each member state and includes relevant quantitative indicators that could be relevant to consider (see Box 6.2).
Number of procedures by procurement category (supplies, services, works)
Number of notices/ contracts above and below European Union thresholds
Share of contracts awarded to companies with concealed ownership structure
Number of above-threshold contracts awarded to SMEs and total value of contracts above threshold won by SME
Share of public procurement procedures incorporating green criteria out of the whole volume of procurement at national level
Source: (European Commission, 2017[7])
In terms of reporting obligations, contracting authorities must submit the following documents/ information:
Publication every 6 months in the Gazette of a full list of contracts awarded with a value exceeding EUR 5 000, including direct awards, and a list of all cases involving variations which exceed the original contract values by more than 5%.
GPP reporting to the Ministry for the Environment, Energy and Enterprise (MEEE) (see section 4.1.4).
The only reporting obligation that is tied with indicators is the one related to GPP reporting. This calls upon Malta to develop a comprehensive measurement framework for public procurement, with relevant KPIs, which could, on its turn, allow the country to measure progress and their efforts to improve the public procurement system against a given baseline in different domains.
The OECD Recommendation on Public Procurement calls for developing indicators to measure performance, effectiveness and savings of the public procurement system for benchmarking and to support strategic policy making on public procurement (OECD, 2015[1]). Each country has its own institutional settings, policy objectives and legislative framework (OECD, 2019[8]), but all of them need to assess progress and achievements and to identify potential gaps. In this context, different elements need to be considered: i) the category of indicators (efficiency/effectiveness, compliance, strategic ones), ii) the procurement stage, and iii) the relevant stakeholders (procurement authorities vs contracting authority or Central Purchasing Body - CPB, etc.).
When establishing and implementing measurement frameworks, countries can adopt different strategies. For instance, a country can decide to start with a specific procurement stage, or to start with specific categories of procurement indicators.
6.2.1. Different categories of procurement performance indicators
Public procurement refers to the process of identifying what is needed; determining who the best person or organisation is to supply this need; and ensuring that what is needed is delivered to the right place, at the right time, for the best price and that all this is done in a fair and open manner. It is increasingly considered as a crucial pillar of services delivery for governments. Because of the sheer volume of spending it represents, well governed public procurement can and must play a major role in fostering public sector efficiency and establishing citizens’ trust and advancing the government agenda. This involves considering indicators in three main categories: compliance, efficiency and strategic objectives (OECD, 2015[1]).
Compliance KPIs aim at assessing whether procurement processes and outcomes are in line with the national or any applicable legislation. In this context, KPIs could cover issues related for instance, the use of appropriate procedures, integrity breaches, transparency requirements, challenges of procurement decisions or litigation during the contract execution phase.
Efficiency KPIs aim at assessing whether the procurement processes enable to achieve the best procurement outcomes and effectiveness and the best “value for money”. This involves assessing inputs and outputs, the use of the different procurement methods, and the implementation of different efficiency tools such as FAs, DPS, etc. Efficiency can be commonly defined as a ratio between outputs and inputs while effectiveness is the ratio of defined outcomes to defined inputs and is conditional on the quality-of-service provision (OECD, 2019[8]). For instance, KPIs could cover savings (in monetary value and time), level of market participation in specific procedures, duration of procurement processes (including vetting), etc.
Strategic KPIs aim at assessing how public procurement processes and outcomes contribute to achieving pressing policy goals such as environmental protection, innovation, job creation and the development of small and medium enterprises (OECD, 2015[1]). In this context, KPIs could include the share of sustainable goods and services, the share of procurement awarded to SMEs (in number and volume), the share of procurement involving innovation features. In addition, in mature systems, KPIs could be related to effective impacts such as the reduction in CO2 emissions or energy consumption
6.2.2. Indicators covering the whole procurement cycle
Performance indicators should be related to different stages of the procurement cycle from tender preparation to the completion of the contract. Figure 6.4. provides an example of some performance indicators throughout the procurement cycle.
6.2.3. Indicators considering the national institutional framework
When developing and establishing KPIs, it is pivotal to consider the national institutional setting and the potential existence of specific entities in charge of implementing public procurement procedures. In Malta there are 3 categories of contracting authorities; Each of them must follow specific processes depending on specific thresholds for the vetting of procurement procedures that could be undertaken by the DoC, namely the SPD directorate (if the contracting authority is under schedule 16) or the Directorate for Operations (if the contracting authority is under schedule 2) (see section 1.1.2). In this context, specific indicators might be relevant for each of these categories but also for the directorates of the DoC involved in the procurement process. In addition to the SPD and the Directorate for Operations, the Procurement Policy and Quality Assurance Directorate could also be considered as it is in charge of vetting BPQRs. In addition, other entities could be considered, such as those in charge of the remedies system (The Public Contracts Review Board in Malta).
In addition, data from the 2016 Public Procurement Survey show that an overwhelming majority of respondents have at least one Central Purchasing Body (CPB) to conduct central purchasing (OECD, 2019[4]). CPBs usually have specific processes and tailored indicators to assess the performance of their activity. For example, indicators related to Framework Agreements used by CPBs can include calculating the number of staff and time to develop framework agreements centrally. Table 6.1. provides a description of examples of indicators and metrics for measuring CPB performance related to framework agreements. The public procurement regulation mentions that DoC and Malta Information Technology Agency (MITA) are considered as CPBs. In addition, aware of the benefits of centralisation, the DoC is currently considering the possibility of expanding its CPB function. (see section 2.3.2). Therefore, Malta should consider integrating indicators related to the aggregation of needs.
6.3.1. General indicators for CAs
The level of formal performance monitoring with KPIs varies among OECD countries and the systematic evaluation of procurement outcomes remains a challenge (OECD, 2019[4]). In addition to monitoring frameworks that may exist at the national level, contracting authorities can also develop relevant KPIs that will help them assess the performance of their procurement and their contribution to achieve their mandate.
Some surveyed contracting authorities (14%) use reporting mechanisms to monitor project outputs with limited and very basic public procurement KPIs such as the number of direct orders/negotiated procedures or the number of contracts signed. The majority of surveyed contracting authority (86%) do not have specific KPIs. The main reasons mentioned by contracting authorities for not setting KPIs are: i) lack of capacity of procurement officials, ii) lack of digitalisation, and iii) the perception that performance indicators were not a value added in the procurement system.
In this context, Malta should consider raising awareness of the different contracting authorities on the benefits of setting public procurement measurement frameworks and ensuring that the framework to be developed at the national level does not involve extensive workload from contracting authorities.
6.3.2. Specific indicators for DoC
Despite the lack of formal performance management system with relevant KPIs in Malta, the DoC is committed to advancing the performance measurement agenda. In addition to the indicators on the performance of the public procurement processes, it is key to monitor the effectiveness of the organisational structure and related processes of key stakeholders in Malta. Given the involvement of different DoC directorates in public procurement processes and their impact on time and outcomes, it is key for the DoC to develop tailored internal KPIs to measure its efficiency and effectiveness. In particular these indicators are relevant for, the SPD, the Directorate for Operations (OD), and the Directorate Procurement Policy and Quality Assurance (DPPQA).
Both the SPD and the OD are in charge of administering procurement procedures and vetting tender documents depending on the schedule to which the entity belongs to, the threshold and the type of procedure. However, the administration of procurement processes and in particular the vetting process requires time that may lead to some delays for contracting authorities (See section 2.2.1)
However, as mentioned in section 2.2.1, the SPD and the OD also mentioned the lack of responsiveness of some contracting authorities that may delay some procurement processes. As a mitigation measure, for instance, the SPD and OD are setting deadlines to cancel procedures when no feedback is received from contracting authorities.
Therefore, it is key to set up the indicators to measure the efficiency and effectiveness of the organisational structure and related processes of DoC directorates. To establish such indicators, it is key to first detail the processes, indicate an estimated timeline or a maximum timeline, and indicate who is responsible for each of them. These indicators could be developed per procurement procedure, procurement categories or DoC directorate or unit. Table 6.2. provides examples of relevant indictors of internal processes for DoC to consider.
In addition to the duration of each process, it could be relevant to assess the performance of DoC by assessing for instance, the share of procedure that received clarification requests. In addition, given the intermediate role of DoC, it could be relevant to assess in a regular basis the satisfaction of the users of its services: contracting authorities
There are many challenges in measuring the performance of public procurement systems including the scarcity of available data (OECD, 2019[4]). As highlighted in the previous work of the OECD in Malta, access to data is essential to measure the efficiency and effectiveness of a procurement system. The data need to be able to be collected for each step of the procurement process (OECD, 2019[9]).
To measure the performance of the Maltese public procurement system throughout the public procurement cycle, data could be provided from the e-procurement system: ePPs and internal system of contracting authorities and DoC, as the department in charge of specific aspects of procurement processes (See Figure 6.6. ).
6.4.1. Improving data availability from the e-procurement system and governmental systems
Improving data availability in the e-procurement system of Malta
The use of e-procurement systems is a prerequisite for effective measurement. Indeed, data availability is improved by widespread and coherent use of these systems across levels of government, for all procedures and throughout the procurement cycle (OECD, 2019[4]).
In Malta, ePPS was established in 2011 and since 2016 the vast majority of procurement procedures with a value of EUR 5 000 (excluding VAT) or above are published electronically through the system. (OECD, 2019[9]). In 2021, the ePPS introduced some new functionalities and fields that enable for instance to collect information on the category of economic operators and on the GPP criteria of the tender. Box 6.3 provides the list of data available in ePPS as of January 2021. In addition to general information, data covers the pre-tendering and the tendering stage and the contract management stage (for certain cases). The analysis of the available data shows three categories of gaps: i) data is not available, ii) data cannot be extracted automatically, or iii) contracting authorises or economic operators are not filling or filling partially some information (poor data quality).
Regarding the pre-tendering and tendering phase, data is not available or cannot be extracted automatically regarding the social criteria, innovation features, the duration of efficiency tools such as DPS and framework agreements. In addition, data related to the number of participating suppliers in DPS and framework agreements and the financial volume for each call cannot be extracted from ePPS. It is extracted manually from the original contract notice. Data on the number of qualified bids is not available, neither. Regarding appeals, ePPS includes a functionality to challenge public procurement decisions. However, despite the existence of this functionality, economic operators still challenge procurement decisions by sending paper letters to the public contracts review board. In addition, data about the challenges outcomes and the duration of the process are not recorded in ePPS.
Significant data gaps have also been identified in the contract management phase. Indeed, ePPS did not have the function to record key information regarding this phase such as contract amendments and contract payments until 2020 when it recently introduced a module on contract management that is optional and can include data on key milestones, timeframes and payments. However, discussions with contracting authorities highlighted that the system is not widely used as i) it duplicates with other tools available within contracting authorities and ii) it is not accessible to all relevant people involved in contract management (as they would need to be registered in ePPS). In this context, the government of Malta should work towards improving data availability and quality in the e-procurement system to ensure a comprehensive evaluation of the performance of the procurement function. It should also consider gradually mandating contracting authorities and economic operators to use some modules and functionalities such as the contract management one and the functionality to challenge decisions. Regarding the contract management module, the DoC should explore the possibility to integrate it with other systems used by contracting authorities.
In addition, in specific cases, after the DoC approval, some direct award procedures are taking place outside the ePPS. Discussions with the DoC and contracting authorities highlighted that those cases refer mainly to urgency situations or exceptional cases when the market is not capable of using the e-procurement system. However, although stakeholders mentioned that those cases represent a minor share of the national public procurement volume, data related to these procedures is not recorded in ePPS, even after signing of contracts. This impacts the data availability and quality on these procedures. For those specific cases, despite the fact that economic operators might be granted some exceptions to be exempt of using ePPS, it is recommended that contracting authorities record all the relevant data in the system to ensure the availability of comprehensive data on these categories of procedures.
Advancing the integration of the e-procurement system with other platforms
Countries have been expanding functionalities of e-procurement systems to achieve better outcomes and deliver services more effectively and efficiently (OECD, 2021[10]), including Malta with the implementation of ePPs and its improvement through new features since 2016. Following these technological advances, vertical and horizontal integration of e-procurement systems with other governmental platforms are the next steps to achieve a fully integrated procurement system to provide government with full visibility on the use of public funds across different government departments (OECD, 2018[11]) and to achieve various efficiency gains for both the public and the private sector (OECD, 2018[11]). In this context, there are two categories of integrations to consider: horizontal integration and vertical integration (see Figure 6.7. ). Vertical integration refers to the integration of the national e-procurement system with the internal digital procurement and management tools of contracting authorities (see section 6.4.2). Horizontal integration involves integrating the e-procurement system with other governmental systems such as the national tax system, the national budgeting/accounting system, or the national security system. In Korea for example, the central procurement agency introduced a fully integrated, end-to-end procurement system called KONEPS which is integrated with about 140 external systems (OECD, 2018[11]).
Source: Adapted from (OECD, 2018[11])
In Malta, the “Once Only Principle” system has been applied in the ePPS, with the integration of the European Single Procurement Document (ESPD). Therefore, economic operators only have to submit information related to their eligibility once as the information is pre-filled the next time (Office of the Prime Minister, n.d.[12]). However, after the award, the supplier needs to get different documents.
Regarding contract management, its module is not integrated with the Corporate Financial Management Solution (CFMS), the accounting system used by public entities in Malta. This integration could provide further efficiency gains and enhance the visibility on public procurement spending. Malta could consider the possibility of integrating the ePPS contract management module with the CFMS.
Finally, when it comes to external platforms, it is key to consider the integration with the banking system, in particular for processing performance guarantees which are required for the majority of procurement operations when the procurement value exceeds EUR 5 000. Currently, the performance guarantee is not provided in a digital format which impacts the efficiency of the system and increases the administrative burden for tenders. In this context, Malta could consider advancing the integration of ePPS with the banking system.
6.4.2. Improving internal information systems of contracting authorities and DoC
In addition to e-procurement platforms, the digitalisation of the procurement process relies also on the digitalisation of internal systems supporting whole-of-procurement activities, including tender preparation and contract management till the completion of the contract. In addition to better spend visibility, better stock management, faster validation of processes, a strong IT system also enables improved data collection (OECD, 2018[3]). The COVID 19 outbreak highlighted the need for interoperable IT infrastructures and digital services to avoid the disruption of public services and contributed to accelerate digitalisation of public procurement systems (European Commission, 2021[13]), including in Malta.
Progressively digitalising the internal procurement system of contracting authorities
Some KPIs rely on data available at the contracting authority level. In Malta, they include, but are not limited to, the data on procurement procedures below EUR 5 000 (excluding VAT) and the use of social criteria. To enable the regular monitoring of such indicators, it is key to digitise internal procurement systems of contracting authorities.
With the COVID-19 pandemic, some contracting authorities enhanced their digital tools and introduced digital signature. Overall, in Malta, contracting authorities use either manual or digital processes to manage and support their public procurement activities. However, for the majority of contracting authorities, digital processes include basic IT tools such as emails, Teams, Excel, Word, and scanned copies of documents. These basic digital tools have limited features, poor interoperability with governmental systems and might not be sustainable in the future due to the poor storage capacity and the fact that there are not fit data sharing, processing and safety. Only a few contracting authorities (21%) mentioned the use of more sophisticated digital tools such as Enterprise Resource Planning (ERP) for some procurement stages, task management tools or dedicated procurement internal system. However, those systems do not cover the whole procurement stages and are not integrated with governmental systems. For example, approvals with DoC and SPD are performed by email. In this context, contracting authorities could consider progressively digitalising their internal procurement system.
Progressively, the challenge for OECD countries and other countries such as Malta goes beyond introducing digital technologies into public administrations, it is also about integrating their internal information system into the public sector modernisation efforts (vertical integration) (OECD, 2018[11]). Indeed, for further efficiency gains, contracting authorities’ internal information systems should be integrated with the e-procurement system and other governmental platforms. As already mentioned, public procurement internal systems need to be linked to the public sector digital environment to enhance workflows, reinforce digital policy coherence and provide valuable data across all levels of government on public spending.
Digitalising the internal information system of DoC
Good management and communication tools enhance efficiency and effectiveness of public procurement systems. They are also necessary to ensure data quality and relevant KPIs. In Malta, whilst it is acknowledged that the public procurement system has been digitised through the use of ePPS, as mentioned in section 2.2 some indicators such as those developed in Table 6.2. require data from DoC. However, DoC is using various systems and tools such as Teams, emails and Excel to communicate with contracting authorities and to perform its mandate and role. Indeed, the DoC doesn’t have a formal digital management system, which is essential for the efficiency of processes and to track their progress.
Currently, all different stakeholders can contact DoC via a generic email address. A generic number is provided for each request and sent to the relevant department, without any shared tracking mechanism. For instance, OD has an Excel file to track the different requests. This file needs to be filled manually by all staff and on a regular basis. The director reviews the file on a regular basis to identify potential issues such as contracting authorities that are still waiting for feedback from DoC, or contracting authorities that have not been responsive following DoC’s feedback. The implementation of a digital management system could enable DoC to track the different requests, to send alerts when needed, and to monitor the performance of its processes in an efficient manner, reducing the risks of delays in the response, mistakes, errors or duplication. In addition, DoC could consider integrating this system with ePPS, if possible.
References
[5] DoC Malta (2016), Manual of Procedures: Public Procurement Regulations, 2016.
[6] European Commission (2021), Malta - Procurement Monitoring Report, https://ec.europa.eu/docsroom/documents/47779?locale=en.
[13] European Commission (2021), Report on Public Administrations’ Digital Response to COVID-19 in the EU, https://op.europa.eu/en/publication-detail/-/publication/8b1a7024-9816-11eb-b85c-01aa75ed71a1.
[7] European Commission (2017), Procurement Monitoring Report Template in view of the Member States’ reporting process under the Directives 2014/23/EU, 2014/24/EU and 2014/25/EU, https://ec.europa.eu/docsroom/documents/47775/attachments/1/translations/en/renditions/native.
[10] OECD (2021), Public Procurement in the State of Mexico: Enhancing Efficiency and Competition, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/cc1da607-en.
[2] OECD (2021), Unlocking the Strategic Use of Public Procurement in Bratislava, Slovak Republic, OECD Publishing, Paris, https://doi.org/10.1787/d616e4d9-en.
[8] OECD (2019), Productivity in Public Procurement, A case study of Finland: measuring the efficiency and effectiveness of public procurement, https://www.oecd.org/gov/public-procurement/publications/productivity-public-procurement.pdf.
[9] OECD (2019), Public procurement in Malta Re-engineering the Department of contracts, https://www.oecd.org/governance/public-procurement/publications/public-procurement-in-malta.pdf.
[4] OECD (2019), Reforming Public Procurement: Progress in Implementing the 2015 OECD Recommendation, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/1de41738-en.
[11] OECD (2018), OECD Public Governance Reviews Mexico’s e-Procurement System Redesigning CompraNet through Stakeholder Engagement, https://www.oecd-ilibrary.org/governance/mexico-s-e-procurement-system_9789264287426-en;jsessionid=EFolGsLCJMaNVXAoU9NrNOtB.ip-10-240-5-150.
[3] OECD (2018), Second Public Procurement Review of the Mexican Institute of Social Security (IMSS): Reshaping Strategies for Better Healthcare, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/9789264190191-en.
[1] OECD (2015), OECD Recommendation of the Council on Public Procurement, https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0411.
[12] Office of the Prime Minister (n.d.), A national platform for eProcurement, https://publicservice.gov.mt/en/Pages/News/2020/20200219_EPPS.aspx (accessed on 23 March 2022).
Note
← 1. Data gathered from 33 respondents (30 OECD countries plus Morocco, Costa Rica and Peru).