Annex B. Data-driven integrity
Applying the concepts of a data-driven public sector to integrity actors
Data are a driver of many initiatives to prevent fraud or corruption, such as the use of data to assess risks, automate control activities, or manage asset declarations and conflicts of interests. This case study explores the application of a data-driven public sector (DDPS) framework to integrity actors in government. It considers how and to what extent the different elements of a DDPS – data governance, data for creating value in public service delivery and data for promoting citizen’s trust – are applicable to the policies, practices and tools of governmental integrity actors. It emphasises both opportunities and limitations for applying the DDPS framework, taking into account the mandates of integrity actors vis-à-vis those responsible for defining and executing digital government strategies.
Conversely, the case study highlights ways that the DDPS framework can benefit from the expertise and data issues facing integrity actors. Integrity actors have subject matter expertise that can be a useful input for key elements of the DDPS, such as data privacy and ethics. In addition, integrity actors are also key users of data from across government and sectors. Their use of data for preventing and detecting fraud affords them a unique perspective to support refinements to the formulation and implementation of policies, frameworks and guidance for a DDPS.
The term “data-driven” is inherent in the DDPS framework, and therefore is used throughout the case study. Nonetheless, the idea that integrity actors can also benefit from simply being “data-informed” is worth noting upfront. Implicit in this idea is the notion that data are often one of many critical inputs for the work of integrity actors, which also relies on human elements, such as sound judgement, professional scepticism and expertise.
This case study refers generally to integrity actors as the entities or individuals responsible for designing, implementing or overseeing policies and practices to promote integrity and prevent fraud or corruption in the public sector. The definition is broad to accommodate a range of institutions or individuals that could benefit from using data for the said purpose. For instance, integrity actors in government can include ethics offices, anti-corruption bodies, supreme audit institutions, ombudsmen, and internal audit or control functions within line ministries. They can also be law enforcement agencies, prosecutor’s offices, the courts or other institutions with judicial or punitive mandates, all of which can advance and support the use of data to prevent and detect corruption. In addition, integrity actors can be individuals, such as managers responsible for controls and risk assessments as part of service delivery and operational activities.
This case study does not attempt to capture the experiences of all the institutions or individuals that could be considered integrity actors. This inclusive, broad definition of integrity actors works for the purposes of this case study because the primary focus is assessing the application of the principles and practices of the DDPS framework, and use of data more specifically, for preventing and detecting fraud, corruption, waste and abuse. As such, this broad definition provides some flexibility for exploring the practices and examples that could be transferable across institutions and sectors.
Effective data governance is a critical precondition for data-driven approaches to prevent and detect fraud or corruption. For instance, Korea’s Bid Rigging Indicator Analysis System (BRIAS) facilitates quantitative assessments of collusion in public procurement (OECD, 2017[1]). The ability of BRIAS to facilitate electronic transfers of bidding information from hundreds of government institutions, and to convert those data into corruption indicators, relies on a robust data architecture and infrastructure. In addition to these elements of data governance, how governments manage the data value cycle – collecting, generating, storing, securing and processing of data – has direct implications for the ability of integrity actors to prevent and detect fraud and corruption. For example, data interoperability across government is essential for auditors or anti-corruption bodies to carry out data matching using databases maintained by different government entities in order to identify fraud, waste and abuse. Figure B.1 summarises the various elements of data governance, many of which are preconditions for integrity actors to create value from data.
Figure B.1 shows the key elements for which centre-of-government (CoG) ministries are responsible in order to strengthen data governance across government and enhance a DDPS. Nonetheless, integrity actors are prosumers of data – both creators and users – and can offer unique experiences and insights to help the CoG shape broader digital government strategies. This has been the case in many countries for advancing the open data movement. For instance, in Indonesia, the Corruption and Eradication Commission played a critical role in facilitating the communication and co-ordination among stakeholders not only to implement anti-corruption policy reforms, but also to work with the Office of the President to advance the open data agenda. In Argentina, the Anti-Corruption Office contributed to the work of the Chief of Cabinet Office and the then Ministry of Modernisation to develop a decree on open data, which mandated the publication of key datasets and set timelines for ministries to develop open data plans (OECD, 2019[2]).
Source: OECD (2019[2]), Digital Government Review of Argentina: Accelerating the Digitalisation of the Public Sector, https://doi.org/10.1787/354732cc-en.
Integrity actors can be useful contributors to a DDPS beyond the open data context. They are often at the frontlines of inter-ministerial or inter-sectoral discussions about the technical and regulatory challenges for using data while balancing competing policy priorities, such as privacy issues and use of personally identifiable information. For example, the supreme audit institution of the United States, the Government Accountability Office, has organised several multi-stakeholder forums and communities of practices with government, civil society and the private sector to explore ways the government can maximise the benefits of data and limit possible drawbacks. One forum brought together experts from across sectors to discuss the use of data analytics to address fraud and improper payments, including identifying ways to improve collaboration, such as overcoming legal and data-sharing barriers. Supreme audit institutions, like other integrity actors, have a whole-of-government view based on their work and use of government-wide data sources that can help the CoG shape and refine digital government strategies and policies.
By contributing to the national dialogue on digital government strategies, integrity actors can also help to advance their own agendas and address their unique challenges. For example, data interoperability and data-sharing issues can be especially pronounced in the integrity space, as illustrated in Box B.1, since integrity actors often rely on multiple data registries and databases across government and sectors to identify risks and conduct investigations. Ultimately, by improving data governance and promoting consistency and coherence across institutions, governments can help integrity actors spend more time and resources (re)using data, and less time and fewer resources managing data governance issues.
In May 2016, the European Commission set up a high-level expert group on information systems and interoperability. In its final report in May 2017, the high-level expert group highlighted, among others, the potential added value of interoperability between the customs and Justice and Home Affairs (JHA) systems. Improving interoperability between the systems of the two authorities could enable sharing of real-time information in a systematic and automated way. The high-level expert group focused on the reality that customs and JHA information systems were not interoperable, leading to blind spots for both JHA and custom authorities. For the custom authorities, the systems are a critical tool for managing risk-based controls at the external borders, which includes detecting and preventing the trafficking of goods posing security or safety risks.
To address this challenge, the commission established an expert group of practitioners with operational knowledge of border management, customs and security. The group analysed the data that could be compatible by mapping the different systems of customs authorities and the JHA in the area of security, border and migration management. The mapping informed a feasibility study on the specific interoperability efforts that could enhance the management of security risks.
The work also supported the group’s efforts to inform policy makers on their specific opportunities and challenges related to interoperability. For example, to strengthen investigative capacity, the experts determined that law enforcement authorities should have direct access to a centralised Advance Cargo Information System (ICS2). From 2021, this system will replace the existing system (ICS) for collecting electronic information on goods coming into or through the European Union customs territory.
Source: European Commission (2018[3]), Interoperability of Security and Border Management Systems with Customs Systems: Assessment Report of the Practitioners, www.statewatch.org/news/2019/mar/eu-council-interoperability-customs-5574-19.pdf.
The DDPS framework highlights three main areas of opportunity for a data-driven public sector: 1) anticipatory governance; 2) design and delivery of services; and 3) performance management. This section explores the relevance of these areas for integrity actors and the use of data to prevent and detect fraud and corruption. Overall, these three areas provide a useful framework for understanding how data can support decision making and the activities of integrity actors, although some elements, such as performance management, pose challenges for integrity actors given the hidden nature of fraud and corruption. In this context, prevention refers to ex ante approaches for avoiding fraud and corruption, or lessening their impact and likelihood. Detection is an ex post activity for identifying fraud or corruption that has already occurred. This distinction helps to understand the value of data-driven approaches, as well as their limitations.
Data for anticipating governance
Anticipatory governance in the DDPS framework generally refers to the use of data as part of a systematic effort to take into account a possible future to take decisions in the present. The framework focuses on two aspects of anticipatory governance: forecasting and foresight. According to the DDPS framework, data-driven “foresight” activities take into account possible future outcomes, without necessarily predicting specific cases of fraud and corruption based on past experiences. The DDPS framework defines forecasting as the use of data to “predict” the most likely developments and outcomes (van Ooijen, Ubaldi and Welby, 2019[4]). The distinction between the two in the integrity context is subtle. In general, activities to prevent fraud or corruption reflect the DDPS framework’s definition of foresight, while detection activities have a conceptual link to forecasting. Nonetheless, some activities of integrity actors, like risk assessments, can serve both purposes.
Data-driven risk assessments can be one example of using data for both foresight and forecasting. Data can support quantitative assessments of the likelihood, impact and velocity of risks, and inform the identification of high-risk organisations or individuals that are susceptible to fraud or corruption (OECD, 2019[5]). Risk assessments can be carried out manually (i.e. with experts) or automated, such as the case of the Arachne tool for the European Union (see Box B.2). Insights gleaned from the data can then be used to pre-emptively prevent fraud or corruption by adapting controls before expenditures are made. In this sense, data facilitates the foresight that ultimately shines a light on the various vulnerabilities in the control environment, without necessarily targeting specific cases or individuals. This is meant to avoid the classic “pay and chase” model, whereby a government entity makes an improper payment and then spends additional taxpayer resources to identify, investigate and possibly recoup those payments.
The European Commission encourages the use of data analytics to enrich the risk assessment process, and in particular, the use of Arachne, a web-based tool with data on contractors, contracts, beneficiaries and projects (European Commission, 20014[6]). Arachne became operational in 2013 as a tool to support authorities of EU member states to identify and prioritise fraud risks, conflicts of interest and irregularities in European Structural and Investment Funds. Data sources for Arachne include data from managing authorities as well as external databases (e.g. ORBIS and World Compliance). In December 2018, according to the European Commission, 21 member states used Arachne for 165 operational programmes, which accounted for 54% of all EU cohesion funding for 2014 20 (excluding the European Territorial Co-operation objective of the European Regional Development Fund) (European Court of Auditors, 2019[7]).
Arachne helps managing authorities to identify high-risk projects, contracts, contractors and beneficiaries with the help of over 100 risk indicators linked to 7 risk categories. Those risk categories include procurement, contract management, eligibility, performance, concentration, reasonability and reputational, and fraud alerts. The specific types of fraud risks and red flags that Arachne supports include:
financial: overall financial performance of beneficiaries, contractors/suppliers and subcontractors, based on financial reporting data
relationship: the existence of relationships between beneficiaries and contractors/suppliers or sub-contractors and their respective personnel
reputation: involvement in activities (such as bankruptcies) that could possibly result in reputational damages
sanctions: identification of beneficiaries, contractors/suppliers, subcontractors or their respective personnel, blacklisted by appearing on any type of sanctions list
change: any type of changes to the company structure
procurement: the lead time between the publication of the tender notice and contract signature
contract management: contract addenda cost (total) for the project and actual project cost
eligibility: project costs outside the eligibility period, such as before the start date or after the end date
performance: number of people trained or to be trained
concentration: beneficiaries involved in multiple projects
other checks: European Commission financial assistance and total project cost.
The OECD supported the government of the Slovak Republic to improve its fraud and corruption risk management in European Structural and Investment Funds, which Included an analysis of its use of Arachne. Managing authorities in the Slovak Republic supplement Arachne with several other databases, including the Information Technology Management System, the Irregularity Management System, the Early Detection and Exclusion System, company registers, and procurement databases. Managing authorities said that the most commonly applied data analytics techniques they use, with the support of these systems, are traditional rules-based detection and descriptive tests, such as data matching and data mining. The success of Arachne relies in large part on the ability of these systems to communicate with each other, and on the input of data by managing authorities for calculating risk indicators.
Source: OECD (2019), Tackling Fraud and Corruption Risks in the Slovak Republic: A Strategy with Key Actions for the European Structural and Investment Funds, OECD Public Governance Reviews, Source: OECD (2019), Tackling Fraud and Corruption Risks in the Slovak Republic: A Strategy with Key Actions for the European Structural and Investment Funds, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/6b8da11a-en.
Fraud and corruption are deliberate and non-random, therefore traditional audit methods involving the use of statistical sampling can be ineffective for identifying corruption or fraud (Dilla and Raschke, 2015[6]). Data can support methodologies for forecasting, or “predicting”, where fraud or corruption could have occurred, or is ongoing, in a particular programme or transactions. For instance, auditors and investigators routinely make use of data analytic techniques, such as data mining, to uncover fraudulent activities. Forecasting in this sense, unlike foresight, is not necessarily aimed at taking preventive actions (e.g. adapting controls) based on possible outcomes, but rather to target audit or investigative resources towards specific instances of possible fraud or corruption.
Examples of data mining are common for detecting collusion in public procurement or conflicts of interest; however, there are other applications. For example, in the People’s Republic of China, the supreme audit institution used data-mining methods to analyse data and geographic information from across government ministries to assess compliance with environmental regulations. The analysis showed that smaller factories had more compliance problems than larger ones in certain regions, and that specific factories continued production at night or surreptitiously to avoid emissions controls. This finding led to more targeted oversight and reforms (OECD, 2019[7]).
In addition to data mining, data-driven predictive audits can serve a similar purpose to support programme objectives. For example, such audits can help to judge the probability of default of government loans or the likelihood that certain transactions on government credit cards or applications for government assistance are indicative of fraud. A DDPS – with robust data governance, in particular – promotes and facilitates such efforts to forecast specific problem areas, and help integrity actors to take advantage of administrative data, as well as other types of data.
As discussed below, some integrity actors (e.g. supreme audit institutions or sector-specific integrity units) attempt to conduct government-wide analysis about the extent of fraud and the financial losses incurred by taxpayers and the government. This analysis can help policy makers and line ministries to take decisions about legal and policy reforms, resource allocation, and co-ordination strategies, for instance. Moreover, data (e.g. “big data”) can provide insights into the future needs of society, which in turn can inform the potential strategies and foresight activities of integrity actors. For example, data on climate change can inform anti-fraud strategies and targeting of possible hotspots for delivering disaster assistance, and data on the changing demographics of populations can help shape fraud and corruption controls when distributing social benefits, pensions and healthcare subsidies.
While integrity actors can support anticipatory governance, by and large, using data for this purpose is aspirational and evidence of “what works” is either anecdotal or raises questions for further research. For instance, in its flagship report, the Association of Certified Fraud Examiners (ACFE) reported that the use of “proactive data monitoring and analysis” and surprise audits were associated with a more than 50% reduction in fraud losses, based on a median loss calculation before and after controls (ACFE, 2018[8]). While interesting to note, the ACFE’s survey does not distinguish between private sector and government sectors, and the methodology for the calculation is unclear. Perhaps as a telling sign for the quality of data in or provided by government, the 2018 “Government Edition” of the ACFE’s report does not provide the same indicator for proactive data monitoring and analysis (ACFE, 2018[9]). In reality, data for anticipatory governance in the integrity context is a maturing area. As discussed, data governance at a government-wide level is often a hindrance, which has implications for practical issues such as skills, co-ordination challenges for sharing data, time and resources.
Data for enhancing integrity activities and engaging stakeholders
The DDPS highlights the ability of data to improve public service delivery and engage civil society to promote trust and participation in government (van Ooijen, Ubaldi and Welby, 2019[4]). This concept is uniquely reflected in the work of integrity actors. The service integrity actors provide links directly with broader principles of governance, such as ensuring the accountability of managers. Their work facilitates citizens’ trust in government, and as interpreters of data, integrity actors can shine a light on governance problems for policy makers, line ministries and the public at large. In addition to data for anticipatory governance (i.e. foresight and forecasting), Table B.1 illustrates some of the everyday key questions that a DDPS and data can help integrity actors address.
To answer these questions, integrity actors rely on a variety of data sources and formats (e.g. data that are open, big, administrative, structured, unstructured, etc.). Some examples of data sources that can support the work of integrity actors to answer the questions above, particularly to identify and analyse integrity risks, including the following:
employee, household or business surveys;
other survey data, such as user surveys, or polls from local research institutions;
data from public registries (e.g. law enforcement, audit institutions, national statistics office);
published research documents from national or international organisations or academia (e.g. articles, reports, working papers, political economy analysis) ;
commissioned research;
indicators from international organisations or research institutions;
interviews or focus groups with relevant stakeholders; and
risk assessments conducted by ministries or other government entities for their own programmes (OECD, forthcoming[11]).
The Office of the Auditor General of Western Australia (OAG-WA) illustrates the use of transactional data and the value of using data analytics to provide both hindsight and foresight to improve public financial management. The OAG-WA analysed payroll data to identify possible fraud, errors and omissions. Data matching and data interrogation techniques allowed auditors to analyse 4 million transactions from 2014 to 2015 totalling over EUR 7.5 billion from 12 different government agencies. Auditors did not find evidence of fraud, but they found overpayments and a need for improved controls in half of the agencies tested (OAG-WA, 2016[12]). The data governance in the local government where the OAG-WA operates, along with its own capacity and skills, allowed the OAG-WA to use data analytics in this way and recommend solutions to pressing governance and control issues. Other countries have similar successful experiences of using data to prevent and detect improper payments, and more generally, to promote accountability (see Box B.3 for an example from Brazil).
Better strategies, capacity, skills and tools for using data has allowed supreme audit institutions to perform increasingly innovative oversight activities over the last decade. The following examples, selected by Brazil’s supreme audit institution, illustrate how data can be used to detect fraud and foster a more efficient and accountable public administration.
A data-mining system to detect fraud in the Brazilian public healthcare system
The large size of the healthcare sector and the amount of money it represents make it an attractive target for fraud. Using several databases, Brazil’s supreme audit institution, the Tribunal de Contas da União, uses the InfoSAS system to spot statistical anomalies in the services delivered by SUS, Brazil’s publicly funded healthcare system.
While individually analysing each of the 5 000 medical targets and approximately 6 000 providers to check for anomalies would require extracting billions of factsheets through the system, InfoSAS uses various algorithms to detect discrepancies, producing scores that can sort and prioritise factsheets. It detects sudden variations in a provider’s production and assigns a discrepancy score to each institution, which draws the analyst’s attention to carry out more in-depth analyses.
While statistical discrepancies should be considered cautiously due to the existence of various explanations, their use represents a step forward in modernising selection processes for audit and control items, allowing the Tribunal de Contas da União to detect fraud more efficiently.
Geo-technologies and the monitoring of Sustainable Development Goals by supreme audit institutions
Recent resolutions of the UN General Assembly emphasise the key role of supreme audit institutions and of the International Organization of Supreme Audit Institutions (INTOSAI) in meeting Sustainable Development Goals: supreme audit institutions perform an important role in promoting the efficiency, accountability, effectiveness and transparency of the public administration, fostering national development around the Sustainable Development Goals. The use of geospatial data being one of the initiatives proposed by the UN, the International Standards of Supreme Audit Institutions (ISSAIs) has described the possible applications of geo-technologies to several audit phases and issued guidance on the use of geographic information systems.
In particular, spatial data sources may be exceptionally useful to supreme audit institutions when checking environmental issues with clear geographic aspects, such as environmental protection areas or polluted areas. Geospatial data are also useful to select samples from different sites, find high-risk areas and standards in data, which would not be possible without this spatial component. Supreme audit institutions may also use spatial data to present audit results, making them more tangible.
Source: TCU (2016[13]), “InfoSAS: A data mining system for production control of SUS”, https://revista.tcu.gov.br/ojs/index.php/RTCU/issue/view/68/102; TCU (2016[14]), “Geotechnologies and monitoring of Sustainable Development Goals by supreme audit institutions”,https://revista.tcu.gov.br/ojs/index.php/RTCU/issue/view/68/102.
The work of some integrity actors is directly aligned with an institution. For instance, internal audit functions are meant to support organisational goals through assurance and control of expenditures, thereby ensuring that taxpayer funds are used for their intended purposes. Ethics offices have a similar arrangement, as they are embedded within an institution’s organisational structure. Data can help these actors to mainstream their activities into the internal governance system. This can include reducing uncertainties in managerial decision making through data-driven risk assessments, or using data analytics to pull information from across an organisation to combat silos and strengthen entity-wide knowledge and information. The preconditions for effective data governance at this level are similar to those across government. Institutional leadership, capacity, skills and a willingness to experiment at an institutional level are all critical factors for keeping pace with a data-driven environment (OECD, 2019[7]).
The OECD’s work to support public sector integrity in a number of areas, from managing fraud risk in European Structural Investment Funds to preventing fraud and corruption in government in Latin America, Asia, and the Middle East and North Africa, demonstrates a wide-ranging need for improving “the basics” for data-driven approaches. In the Middle East and North Africa, for instance, the OECD surveyed seven economies (Egypt, Jordan, Lebanon, Morocco, Oman, the Palestinian Authority and Tunisia) and identified common government-wide and institutional challenges for using data to support integrity and anti-corruption work. This included typical challenges of poor data strategies and data infrastructure across government, as well as specific issues like the need for tailoring interventions to build data skills based on job roles (OECD, 2017[15]).
In addition, responses from a non-generalisable survey of the OECD’s Auditors Alliance suggested a heavy reliance on Microsoft Excel and the need to improve simple data extraction tools to support data management and pre-analytical processing of data. Respondents also reported a limited use of automated data extraction tools, which has the potential to become a more significant issue in the future as financial data and digitalisation of “evidence” across government accelerates (e.g. financial transactions, purchase orders, signatures, invoices, etc.). This underscores the importance of a DDPS to not only advance the delivery and strategic layers of data governance, but also the tactical layer, in particular, developing capacities for coherent implementation. The challenges are not just technical. In the same survey, nearly a quarter of respondents highlighted budget limitations as the top challenge for making better use of data and computer-assisted auditing techniques (OECD, 2019[7]).
In the integrity context, the benefits of using data often focus on identifying areas of high risks, yet data also offer value for identifying and adapting controls related to low-risk areas. Traditional risk treatments generally fall into four main categories, depending on the level of risk. Terms vary for the treatments, but they typically reflect the ideas of risk mitigation, avoidance, transfer and acceptance. Acceptance is particularly relevant for risks that are perceived as low likelihood and low impact. Fraud and corruption prevention strategies are calibrated to minimise controls of low risks. This can have practical effects on individuals and their experiences with government services. For instance, for social benefit programmes, data can reveal which claims and recipients pose a greater fraud risk, as well as claims and recipients who are minimal risks.
Through data-informed risk management, a DDPS approach can reduce checks on recipients who have a good record of compliance and a low risk of engaging in fraud. In Denmark, the government entity that manages social security and pension payments (Udbetaling Danmark) noted that the increased use of data for preventing fraud and error comes with other benefits for legitimate claimants, such as streamlining the administration of social benefits across branches of the entity (European Commission, 2015[15]). In this sense, data-driven fraud control is not about targeting bad actors, but serving the good ones well – a notion reflected in the fraud control strategy of the Irish Department of Social Protection, which has the stated goal of ensuring “the right person is paid the right amount of money at the right time” (European Commission, 2015[15]).
So far, the discussion has focused on the pre-analytical and analytical processes, tools, benefits and challenges in the integrity context. However, public value of data does not just come from analysing the data, but from conveying the findings in a way that leads to actionable insights and solutions. Methods such as interactive data visualisation help make conceptual issues real in ways that can be presented for different audiences and that provide specific entry points to engage stakeholders in contributing potential solutions (van Ooijen, Ubaldi and Welby, 2019[4]). Data visualisations and data dashboards are also critical tools for interpreting risks, and creating actionable insights for internal and external stakeholders, as illustrated in Box B.4. Data visualisation is the “visual representation of statistical and other types of numeric and non-numeric data through the use of static or interactive pictures and graphics” (Gatto, 2015[16]). In essence, in the context of integrity actors and applications of anticipatory governance, data visualisation can be a communication tool for sharing results, such as the outcomes of risk assessments. In this example, the results of the analysis should facilitate a common understanding of risks, while complementing perceptions with evidence and countering biases inherent in qualitative approaches. Data visualisations can help promote this value proposition.
The Crime and Corruption Commission, in its mission to combat and reduce the incidence of major crime and corruption in the public sector in Queensland (Australia), receives complaints about corrupt conduct from members of the public and public sector agencies.
In the interests of transparency and to assist public sector agencies to better understand corruption risk, the Crime and Corruption Commission created a data dashboard through which ordinary citizens can learn about the number and types of allegations as well as the institutions and activities related to the alleged cases of corruption between 1 July 2015 and 31 March 2019.
By providing a user-friendly data-visualisation tool, downloadable raw data, help resources and a tutorial video, the commission empowers citizens: using this anonymised “barometer” of corrupt conduct as a first step, citizens can request specific information on a case-by-case basis and hold the Crime and Corruption Commission accountable if the allegations do not seem to be followed by actions. This transparency tool also allows citizens to better understand the data used by this integrity body without disclosing identifiable information.
Source: Crime and Corruption Commission (2019[17]), Corruption Allegations Data Dashboard,. www.ccc.qld.gov.au/corruption-prevention/corruption-allegations-data-dashboard/corruption-allegations-data-dashboard-about
Data for performance management
The DDPS framework identifies how data can support performance management, including more efficient use of resources, evaluation and continuous improvement. For integrity actors, this particular aspect of the DDPS framework poses challenges because of difficulties in measuring the impact of prevention measures. This issue is not simply a matter of improving data governance, such as data access, quality or interoperability; the issue is a measurement challenge, which in the best of cases relies on statistical models to determine fraud rates and create baselines. Without such baselines, it is difficult for institutions to measure the efficacy of control activities. As a result, assessment of the effectiveness of prevention and detection activities are often anecdotal, or captured during the course of control planning processes, such as risk assessments.
While a baseline for the extent of fraud and corruption is difficult to establish, countries are exploring ways to make use of the data they have to paint a “good enough” picture that can still be useful for policy making. Examples are often sector-specific, particularly in health or social benefit programmes, focus on detected fraud or corruption, or capture bigger concepts like improper payments. For instance, in 2014, the French government detected fraud related to social benefits equivalent to an estimated EUR 425 million (Comité national de lutte contre la fraude, 2015[18]). In addition, the US Government Accountability Office estimates that improper payments – any payment that should not have been made or was made in an incorrect amount, including underpayments – amounted to nearly USD 141 billion for fiscal year 2017 (US Government Accountability Office, 2019[19]).
While imperfect, such measurements offer baselines for government to assess the effectiveness and efficiency of policies, governance, management and internal controls. Data can also support similar measurements at a micro-level, for instance, within a subset of procurement contracts or among specific beneficiaries of a social benefits programme. These baselines can provide a snapshot for managers to take decisions about controls. In theory, a baseline will allow for monitoring changes in the rates of fraud, corruption or improper payments based on changes in the control environment, which is a critical feedback loop for managerial decision making. For instance, baselines for fraud prevention in social benefit programmes (or other areas) could be established by:
Examining historical data for fraud to establish a fraud rate, preferably time-series data over a number of years, i.e. what percentage of claims are fraudulent; what percentage of recipients submitted a fraudulent claim.
Comprehensive large-scale audits or risk assessments can help establish the rate of fraud based on identifying cases of suspected fraud. If fraud is confirmed, results can then be extrapolated to a likely fraud rate based on programme size (e.g. number of recipients; value of detected frauds).
Random sampling of cases where there is specific focus on finding suspicious cases. Given proper methodological design, identified results can be generalised to entire programmes.
Baselines can also inform performance management systems. For instance, government institutions in Canada (Ontario) and New Zealand have developed fraud reduction-related targets for managers responsible for social benefit programmes. As an example, the following data points could support the development and regular updating of such targets related to social benefit programmes as well as overall performance management over the internal control system:
the number, percentage and value of fraudulent claims
the number and percentage of recipients engaging in fraud
the number and percentage of private providers engaging in fraud
the number, percentage and value of fraudulent transactions involving different goods and services.
In practice, for many government institutions such activities require skills, capacity and time that is in short supply. Moreover, such approaches to performance measurement provide insights about the effectiveness of controls, but not necessarily their efficiency. Value-for-money is its own measurement challenge, and cost-benefit assessment relies on even richer datasets and information to be carried out.
Looking beyond the walls of their institution, integrity actors can also contribute to government-wide performance evaluation and improvement, using data to substantiate and inform analyses. One way integrity actors do this is by helping policy makers and line ministries to interpret the outputs and outcomes of their own decision making. A range of integrity actors serve this function. For instance, ethics offices and anti-corruption bodies provide feedback on the policies and practices to manage complaints mechanisms, based on their own interpretation of data collected from hotlines, whistleblowing mechanisms and data from asset declarations. Internal audit bodies contribute to decisions about controls and risk treatments based on their audits and risk assessments, which can influence the ability of an organisation to achieve broader objectives. The more the data are of high quality and timely to support this type of work, the more valuable this input can be for real-time decision making.
A citizen-focused DDPS has the normative frameworks, policies and safeguards in place to ensure the ethical and accountable use of data, protect citizens’ privacy and promote transparency (van Ooijen, Ubaldi and Welby, 2019[4]). As discussed, integrity actors rely on a variety of databases and registries to perform their basic duties. These data can include personally identifiable information of citizens. For many applications, anonymised data is insufficient. For instance, forensic auditors, investigators and regulators rely on the ability to use and reuse data with personal information to identify fraudulent patterns and assess criminal behaviour of specific individuals. Nonetheless, to preserve citizen’s trust in government institutions, integrity actors can do their part to ensure that their activities strike a balance between oversight, privacy and transparency. Unique identifiers, for example, can be useful for anonymising early stages of analysis, before there is a need to know more about individual cases then simply patterns of behaviour. In addition, integrity actors are well-positioned to lend expertise and insights to the government-wide dialogue about the ethical use of data, privacy and transparency. This section unpacks these issues further.
Integrity actors leading by example to use data ethically and protect privacy
In general, the work of integrity bodies, whether ethics offices, anti-corruption bodies, audit institutions or others, is meant to promote trust in government by ensuring effective and efficient governance. Accountability, transparency and integrity are key principles that underpin their mandates. Integrity actors can further promote these principles by taking practical steps to ensure that ethics, privacy and transparency are considered in their use of data. Failing to consider these issues in their strategy and operations could undermine citizens’ trust and integrity actors’ own arguments for data access and reuse.
A major challenge for governments is to create the necessary institutional conditions to realise the potential of a DDPS (van Ooijen, Ubaldi and Welby, 2019[4]). Institutional conditions can mean many things, including the various elements of data governance described above, as well as the factors that reflect the unique policy, personnel and technical environment of integrity actors. Some entities have institutionalised the policies around data privacy by dedicating responsible individuals. For instance, the French High Authority for Transparency in Public Life (Haute autorité pour la transparence de la vie publique), which works to prevent conflicts of interest and manages asset disclosures of public officials, took the concrete step of assigning a “data protection delegate.” This individual is in charge of dealing with “any issues susceptible of having an impact in terms of data protection and privacy”. According to the law, this delegate “should have the necessary resources at their disposal to accomplish this mission and request any training that they deem useful to maintain their knowledge of the subject” (Haute autorité pour la transparence de la vie publique, 2018[20]).
Organisational codes of conduct are another institutional mechanism that integrity actors can use to promote ethical handling of personal data. In general, ethics laws or codes of conduct serve as the backbone to ensuring integrity in the public service. They act as reference point for public servants regulating ethical norms and principles and conflict of interest (OECD, 2019[21]). In addition to national level codes of conduct, organisational codes of conduct can be targeted at individual contexts and issues. For instance, in Argentina, several entities and state-owned enterprises have adopted their own codes of ethics, tailored to their specific functions and risk profiles (OECD, 2019[21]). Addressing the ethical use of data in institutional codes of conduct can serve as a mechanism to communicate expectations and priorities in the context of the organisation’s day-to-day activities. This would help to make the issue of the “ethical use of data” relatable and actionable for employees. Institutional codes of conduct also can be a vehicle for creating consensus, ownership and guidance related to key risk areas across the data value cycle.
In each stage of the data value cycle, from data collection to sharing and reuse, there is a potential for integrity actors to lead by example with regards to the ethical use of data and protecting data privacy. Safeguards can begin early in the cycle, at data collection and access. The United Kingdom’s Information Commissioner’s Office, an executive non-departmental public body for information rights, offers insights in its 2011 Data Sharing Code of Practice (to be revised in 2019). The code provides frameworks, considerations and good practices for sharing data. It highlights the importance of understanding objectives first, and considering the benefits and risks to individuals or society for sharing data. It also emphasises the need to determine in advance the specific data to be shared, security and controls for use, anonymisation and defining who has access to data (UK Information Commissioner's Office, 2011[22]).
Integrity actors can benefit from taking this systematic approach to data sharing. Data-driven activities to prevent and detect fraud can focus on identification of known suspected criminals or “bad actors,” such as the matching of national databases with debarment, sanctions or terrorist watch lists. These techniques are meant to identify whether nefarious entities have infiltrated government services. In general, the public would expect governments to use data for this purpose, given the implications for national and personal security, as well as the integrity of government. Individuals do not have the ability to provide consent for use of these data, as one might consent to use of Internet browsing or health data. However, many data analytics techniques, both manual and automated, cast a wider net in order to identify anomalies and patterns of unknown individuals. For instance, the matching and mining of health or welfare data for fraudulent patterns will take into account entire segments of the population. These techniques can go beyond identifying broad risk areas, such as the identification of risky types of contracts (e.g. single-bid contracts) or individuals (e.g. males under the age of 30), and target specific individuals. Indeed, data of well-meaning citizens can act as a baseline for outliers that could represent fraud or corruption, and therefore innocent citizens by default are a central part of the analysis.
For this reason, it is critical for integrity actors to take steps to maintain citizen’s trust by following data-informed codes of conduct, and ensuring proportionality between the value generated from using data and the risks to citizens’ privacy. Anonymising data and use of unique identifiers, or creating classification systems and informing citizens, can help to strike a balance and reduce the risks of and concerns over privacy violations. For instance, in Argentina, the OECD suggested a three-tiered classification, ranging from “confidential information” (only accessible by a judicial authority or the Public Prosecutor’s Office in a legal case) to “information accessible by the Anti-Corruption Office and the Supreme Court and Magistrates Council of the Nation” and “public information” (OECD, 2019[21]). Transparency about use of data can also be a useful tool. In the United Kingdom, the British National Fraud Initiative makes use of “privacy notice” that informs citizens about use of their data, as described in Box B.5.
The Audit Commission’s National Fraud Initiative was launched in 1996 as the United Kingdom’s largest data-matching exercise in relation to fraud. The Serious Crime Act of 2007 enabled bodies, other than those with a mandatory requirement to provide data for the National Fraud Initiative, to volunteer to participate by providing data to the commission (OECD, 2017[24]).
The National Fraud Initiative has enabled participating organisations to prevent and detect more than GBP 300 million fraud and error in the period from April 2016 to March 2018. Approximately 1 200 public and private sector organisations participate in the initiative, among which the public audit agencies in Scotland, Wales and Northern Ireland: each national audit agency carries out data-matching under its own powers, but uses the National Fraud Initiative’s systems, processes and expertise.
To increase transparency around this massive data-matching exercise, the National Fraud Initiative has set out a Code of Data Matching Practice that is followed by all organisations that participate in the Cabinet Office’s data-matching exercises. The code “creates a balance between the important public policy objective of preventing and detecting fraud, and the need to pay due regard to the rights of those whose data are matched for this purpose.” To achieve this goal, the code was informed by the consultation of a range of stakeholders, with the Information Commissioner’s office providing input on data protection.
The code notably requires each institution to publish a privacy notice that informs citizens about the specific datasets used, the way they are collected, the purpose of this data-matching exercise and its legal basis, the institutions with which the data are shared, the retention period for the data, and the rights of citizens including complaints mechanisms.
This example illustrates both the necessity of transparency for integrity actors when implementing anti-fraud programmes and the value of their input to inform the creation of codes of practice that safeguard citizen’s rights.
Source: GOV.UK (2018[25]), National Fraud Initiative, https://www.gov.uk/government/collections/national-fraud-initiative.
Supporting frameworks for the ethical use of data
Integrity actors are well-positioned to inform government-wide policies and dialogue on the ethical use of data and transparency. For example, there are well-known examples of integrity actors, particularly anti-corruption bodies, playing a critical role in promoting transparency in government and advancing the open data agenda. Open data can help prevent fraud and corruption because it gives the public the opportunity to better monitor the flow and use of public money, thereby shedding light on government activities, decisions and expenditures (OECD, 2017[23]). The effectiveness of open data platforms depends on many of the data governance issues discussed, as well as inter-departmental co-ordination. Integrity actors can take the next step to help the public interpret open data by providing analysis and indicators, as illustrated by the example in Colombia in Box B.6.
The Transparency Secretariat of Colombia created a web portal displaying indicators related to: disciplinary, penal and fiscal sanctions; the Open Government Index (Índice de Gobierno Abierto); and the Fiscal Performance Index (Índice de Desempeño Fiscal). These data sources and indicators are co-ordinated and derived from different sources, including the Prosecutor General’s Office (Fiscalía General de la Nación), the Attorney General’s Office (Procuradoría General de la Nación), the supreme audit institution (Auditoría General de la República) and the National Planning Department (Departamento Nacional de Planeación). Additionally, the observatory’s website provides indicators related to transparency and the implementation status of the public anti-corruption policy elaborated by the Transparency Secretariat.
The indicators related to transparency include: a composite index of accountability; a composite index of the quality of the corruption risk maps; an indicator related to the demand and supply of public information; and a composite index on the regional anti-corruption commissions. The indicators of the public anti-corruption policy are composite indexes (based on overall 24 sub-indexes reflecting the objectives of the Colombian policy) showing the progress made related to five strategic priorities: 1) improving the access to and quality of the public information; 2) making the public management tools for preventing corruption more efficient; 3) enhancing social control to prevent corruption; 4) promoting a culture of legality in the state and society; and 5) reducing the impunity related to corrupt practices. All indicators are also available in excel format (open data), which makes the data readily usable for research, comparisons and media reports. Details on the methodology for elaborating the indicators are also provided.
Source: OECD (2019[24]), OECD Integrity Review of Argentina: Achieving Systemic and Sustained Change, https://doi.org/10.1787/g2g98ec3-en.
As the OECD’s working paper on a DDPS explores, integrity actors can play a useful role in supporting the development of normative frameworks that enable a DDPS (van Ooijen, Ubaldi and Welby, 2019[4]). The technical nature of data protection measures has led many countries to create specific bodies to develop such frameworks. For instance, Ireland created the Irish Data Protection Commission and the European Union established the European Data Protection Supervisor, which monitors and ensures the protection of personal data and privacy of individuals. While this is beyond their primary remit, the expertise of integrity actors on ethical issues allows them to identify critical information and to determine priorities in the development of data protection policies and guidelines. Integrity actors can use risk-based approaches to determine which positions are particularly sensitive to integrity risks in order to evaluate the appropriate level of disclosure for different types of public officials depending on the risk-benefit balance (OECD, 2017[24]).
Additionally, the expertise of specific integrity actors, such as ethics offices, can provide useful guidance for institutions that are responsible for developing, implementing and monitoring digital government strategies. Following the example of Hong Kong, China, where the Information Accountability Foundation developed a Model Ethical Data Impact Assessment for the Office of the Privacy Commissioner for Personal Data, integrity actors can support the analysis of how the activities of a DDPS affects citizens (see Box B.8). The Model Ethical Data Impact Assessment looks at the full range of rights and interests of all parties in a data-processing activity to understand how data analytics may impact people in a significant manner, or when data-enabled decisions are being taken without the intervention of people. This type of impact assessment could help public institutions by looking at the rights and interests affected by the data collection, use and disclosure in data-driven activities.
In Hong Kong, China, the Office of the Privacy Commissioner for Personal Data commissioned the Information Accountability Foundation, a think-tank specialised in accountability-based information governance, to conduct a consultancy study with the aim of exploring the core values to guide advanced data-processing activities that are ethical and fair to all stakeholders. Published in 2018, the Ethical Accountability Framework for Hong Kong, China provides an analysis and a model assessment framework concerning the legitimacy of data processing.
One element of this framework is the Model Ethical Data Impact Assessment, a step-by-step guidance tool for organisations to identify the full range of rights and interests of all parties in a data-processing activity where advanced data analytics may impact people in a significant manner or when data-enabled decisions are being taken without the intervention of people. The Model Ethical Data Impact Assessment helps organisations identify the goals of specific data-driven activities, their legal implications, their potential benefits and risks, the level of accountability of decision makers and also covers issues of data accuracy and sensitiveness.
While this impact assessment tool was developed with business partners, it could also help public institutions by looking at the rights and interests impacted by the data collection, use and disclosure in data-driven activities. Integrity actors with expertise in ethics could take part in the creation of similar tools and tailor them to the needs of public institutions to help them find a trade-off between the benefits and the risks implied by the use of data.
Source: Information Accountability Foundation (2018[27]), Ethical Accountability Framework for Hong Kong, China, https://www.pcpd.org.hk/misc/files/Ethical_Accountability_Framework.pdf.
Integrity actors are both beneficiaries and contributors to a DDPS. Many integrity actors have government-wide mandates and they occupy a unique position that gives them visibility into the challenges and solutions across government entities for using data to create public value. In addition, they are often grappling with some of the most pressing issues related to data privacy, security and maintaining trust of citizens with regards to the government’s use of data. As such, integrity actors can be key collaborators for the CoG and the central entities responsible for advancing a DDPS and digital government strategies. As discussed above, elements of the DDPS framework are beyond the remit of integrity actors, or have limited application to their work. Nonetheless, the key components of a DDPS provides a solid foundation for integrity actors to consider both the whole-of-government and institutional factors that facilitate and inhibit their use of data, so that ultimately they are more effective in taking data-driven, or at a minimum, data-informed approaches to prevent and detect fraud and corruption.
References
[9] ACFE (2018), Global Study on Occupational Fraud and Abuse: Government Edition, Association of Certified Fraud Examiners, https://www.acfe.com/uploadedFiles/ACFE_Website/Content/rttn/2018/RTTN-Government-Edition.pdf.
[8] ACFE (2018), Report to the Nations: 2018 Global Study on Occupational Fraud and Abuse, Association of Certified Fraud Examiners, https://www.acfe.com/report-to-the-nations/2018.
[18] Comité national de lutte contre la fraude (2015), Dossier de presse, Comité national de lutte contre la fraude, Paris, https://www.economie.gouv.fr/files/files/directions_services/dnlf/DOSSIER_DE_PRESSE_Comite_National_de_Lutte_contre_la_Fraude-_Mardi_23_juin_2015%281%29.pdf.
[17] Crime and Corruption Commission (2019), Corruption Allegations Data Dashboard, Crime and Corruption Commission, http://www.ccc.qld.gov.au/corruption-prevention/corruption-allegations-data-dashboard/corruption-allegations-data-dashboard-about.
[10] Davenport, T., J. Harris and R. Morison (2010), Analytics at Work: Smarter Decisions, Better Results, Harvard Business School Publishing Corporation.
[6] Dilla, W. and R. Raschke (2015), “Data visualization for fraud detection: Practice implications and a call for future research”, International Journal of Accounting Information Systems, Vol. 16, pp. 1-22, https://doi.org/10.1016/j.accinf.2015.01.001.
[28] European Commission (20014), Guidance for the Commission and Member States on a Common Methodology for the Assessment of Management and Control Systems in the Member States, https://ec.europa.eu/regional_policy/en/information/publications/guidelines/2014/guidance-forthe-commission-and-member-states-on-a-common-methodology-for-the-assessment-ofmanagement-and-control-systems-in-the-member-states.
[3] European Commission (2018), Interoperability of Security and Border Management Systems with Customs Systems: Assessment Report of the Practitioners, European Commission, Brussels, http://www.statewatch.org/news/2019/mar/eu-council-interoperability-customs-5574-19.pdf.
[15] European Commission (2015), Fraud and Error in the Field of EU Social Security Coordination, Publications Office of the European Union, https://ec.europa.eu/social/BlobServlet?docId=18644&langId=en.
[29] European Court of Auditors (2019), Tackling fraud in EU Cohesion Spending: Managing Authorities Need to Strengthen Detection, Response and Coordination, https://op.europa.eu/webpub/eca/special-reports/fraud-in-cohesion-6-2019/en/#chapter3.
[16] Gatto, M. (2015), Making Research Useful: Current Challenges and Good Practices in Data Visualisation, University of Oxford and the Reuters Institute for the Study of Journalism, https://reutersinstitute.politics.ox.ac.uk/sites/default/files/research/files/Making%2520Research%2520Useful%2520-%2520Current%2520Challenges%2520and%2520Good%2520Practices%2520in%2520Data%2520Visualisation.pdf.
[25] GOV.UK (2018), National Fraud Initiative, https://www.gov.uk/government/collections/national-fraud-initiative.
[20] Haute autorité pour la transparence de la vie publique (2018), Règlement intérieur, Haute autorité pour la transparence de la vie publique, Paris, https://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000036970810&dateTexte=&categorieLien=id (accessed on 19 July 2019).
[26] Information Accountability Foundation (2018), Ethical Accountability Framework for Hong Kong, China, Information Accountability Foundation, https://www.pcpd.org.hk/misc/files/Ethical_Accountability_Framework.pdf.
[12] OAG-WA (2016), Audit of Payroll and Other Expenditure Using Data Analytic Procedures, Office of the Auditor General Western Australia, https://audit.wa.gov.au/wp-content/uploads/2016/05/report2016_06-DataAnalytics.pdf.
[5] OECD (2019), Analytics for Integrity: Data-Driven Approaches for Enhancing Corruption and Fraud Risk Assessments, OECD, Paris, http://www.oecd.org/gov/ethics/analytics-for-integrity.pdf.
[2] OECD (2019), Digital Government Review of Argentina: Accelerating the Digitalisation of the Public Sector, OECD Digital Government Studies, OECD Publishing, Paris, https://dx.doi.org/10.1787/354732cc-en.
[21] OECD (2019), OECD Integrity Review of Argentina: Achieving Systemic and Sustained Change, OECD Publishing, Paris, https://doi.org/10.1787/g2g98ec3-en (accessed on 1 July 2019).
[7] OECD (2019), Survey of the OECD Auditors Alliance, OECD, Paris.
[1] OECD (2017), Algorithms and Collusion: Competetion Policy in a Digital Age, OECD, Paris, http://www.oecd.org/daf/competition/Algorithms-and-colllusion-competition-policy-in-the-digital-age.pdf.
[23] OECD (2017), Compendium of Good Practices on the Use of Open Data for Anti-corruption: Towards Data-driven Public Sector Integrity and Civic Auditing, OECD, Paris, http://www.oecd.org/corruption/g20-oecd-compendium-open-data-anti-corruption.htm.
[27] OECD (2017), Internal Control and Risk Management for Public Integrity in the Middle East and North Africa, OECD, Paris, https://www.oecd.org/gov/ethics/corruption-risks-internal-control-mena.pdf.
[24] OECD (2017), OECD Integrity Review of Mexico: Taking a Stronger Stance Against Corruption, OECD Public Governance Reviews, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264273207-en.
[11] OECD (forthcoming), Public Integrity Handbook: Chapter 3 on Strategy.
[14] TCU (2016), “Geotechnologies and monitoring of Sustainable Development Goals by supreme audit institutions”, Revisita do TCU, Vol. 137, https://revista.tcu.gov.br/ojs/index.php/RTCU/issue/view/68/102.
[13] TCU (2016), “InfoSAS: A data mining system for production control of SUS”, Revisita do TCU, Vol. 137, https://revista.tcu.gov.br/ojs/index.php/RTCU/issue/view/68/102.
[22] UK Information Commissioner’s Office (2011), Data Sharing Code of Practice, UK Information Commissioner’s Office, London, https://www.pdpjournals.com/docs/88438.pdf.
[19] US Government Accountability Office (2019), Improper Payments: Selected Agencies Need Improvements in their Assessments to Better Determine and Document Risk Susceptibility, US Government Accountability Office, Washington, DC, https://www.gao.gov/assets/700/696384.pdf.
[4] van Ooijen, C., B. Ubaldi and B. Welby (2019), “A data-driven public sector: Enabling the strategic use of data for productive, inclusive and trustworthy governance”, OECD Working Papers on Public Governance, No. 33, OECD Publishing, Paris, https://dx.doi.org/10.1787/09ab162c-en.
This document, as well as any data and map included herein, are without prejudice to the status of or sovereignty over any territory, to the delimitation of international frontiers and boundaries and to the name of any territory, city or area. Extracts from publications may be subject to additional disclaimers, which are set out in the complete version of the publication, available at the link provided.
https://doi.org/10.1787/059814a7-en
© OECD 2019
The use of this work, whether digital or print, is governed by the Terms and Conditions to be found at http://www.oecd.org/termsandconditions.
