4. Monitoring, evaluating and communicating the implementation of Hungary’s NACS 2023-2025
This chapter assesses Hungary’s proposed processes for monitoring and evaluating the implementation of the NACS 2023-2025. There is scope for Hungary to improve these processes, and this chapter recommends that Hungary could improve coordination mechanisms, clarify funding for actions, and develop indicators for measuring implementation. In addition, the chapter explores how Hungary can allow for flexible adjustments to be made to the NACS and action plan during the reporting period. And it suggests how Hungary could communicate with relevant stakeholders and the public during implementation to inform them of progress made.
To properly gauge the effectiveness of an integrity strategy, it is important to specify monitoring and evaluation arrangements. Monitoring is a continuous function that uses systematic collection of data on specific indicators to provide an indication of the achievement of objectives. Evaluation is the systematic and objective assessment of an ongoing or completed project, programme or policy, its design, implementation and results. It differs from monitoring in that it involves a judgement of the value of the activity and its results (OECD, 2013[1]).
These mechanisms are often institutionalised within the policymaking process for all public policies (Jacob, Speer and Furubo, 2015[2]). Evaluation mechanisms at various stages of the policy process are defined and planned prior to implementing any actions. Having a clear idea of what data will be collected for evaluation, as well as how and when measures taken will be evaluated, informs the design and implementation of actions. It is essential to establish evaluation mechanisms before the implementation phase to ensure measurability, progress reports and accountability. Some data may overlap with what has been collected for the risk identification and assessment procedure in the ex ante stage, but the monitoring and evaluation arrangements have a different purpose: holding the implementing actors accountable for what has been achieved, and how efficiently (OECD, 2020[3]).
Monitoring and evaluation could also generate actionable insights that could help the Medium Term National Anti-Corruption Strategy for 2023-2025 (hereinafter referred to as NACS) to identify what is working and address what is not working, as well as provide information to communicate to stakeholders and the public about progress and results, thus establishing the base for transparency and accountability.
Although interconnected, the differences between monitoring and evaluation are straightforward.
Monitoring corresponds to a routine process of evidence gathering and reporting to ensure that resources are adequately spent, outputs are successfully delivered and milestones and goals are met. For monitoring, it is necessary to systematically collect data on specified indicators to provide the main stakeholders of an ongoing initiative with insights about progress and challenges in its implementation. In the context of Hungary’s NACS, monitoring could enable tracking and, if needed, modifying the different measures considered in the AP.
Policy evaluation is a structured and objective assessment of an ongoing or completed initiative, its design, implementation and results. The goal of policy evaluation is to determine the relevance and fulfilment of objectives, its coherence, efficiency, effectiveness, impact and sustainability, as well as the worth or significance of a policy (OECD, 2020[4]; OECD, 2017[5]). Above all, policy evaluations should trigger learning and enable continuity and coherence over time. One criteria used in OECD’s Public Integrity Indicator for the quality of the strategic framework is precisely whether current strategies have used evidence from evaluations of predecessor strategies to inform their approach.
In turn, communication with both internal and external stakeholders enables accountability, increases the credibility of integrity efforts and stimulates future action. The benefit of monitoring and evaluation for management, policy design, and organisational learning depends significantly on how process and results are communicated (OECD, 2017[5]). As such, communication could be further introduced as an integral part of the NACS and its M&E framework.
An effective anti-corruption strategy and action plan must not only lay out a comprehensive set of substantive reforms, but also indicate the means for ensuring its implementation, monitoring and evaluation as well as communication (OECD, 2020[3]).
An effective monitoring and evaluation system should be incorporated in the continuous update and improvement cycle of the strategic framework (OECD, 2023[6]; OECD, 2021[7]). Thus, public integrity measurement tools such as the OECD Public Integrity Indicators highlight the importance of well-integrated monitoring and evaluation activities within every action plan (OECD, 2023[6]). Table 4.1 shows key elements of effective monitoring practices according to the criteria set in the OECD Public Integrity Indicators (OECD, 2023[6]).
Regarding evaluation practices, the OECD Public Integrity Indicators measuring the OECD Recommendations on Public Integrity, highlight how crucial it is to incorporate the results of system-wide evaluations into every update of strategic objectives. Moreover, effective evaluation practices also engage external non-state actors to secure an unbiased assessment and endow external legitimacy to evaluation exercises. Table 4.2 shows key criteria to ensure transparent and effective evaluation practices (OECD, 2023[6]).
This section assesses the monitoring and evaluation arrangements for the NACS and AP, and makes some proposals for how these arrangements could be improved to ensure the NACS produces the change it is designed to produce.
4.2.1. Clarify who owns co-ordination mechanisms for the Action Plan, how activity will be funded, and how indicators will be used to promote implementation of the NACS and AP
Steering, ownership and co-ordination
Country practices and experiences show that an effective public integrity system requires commitment from the highest political and management levels of the public sector. In particular, this means developing the necessary legal and institutional frameworks and clarifying institutional responsibilities. This is necessary since the promotion of public integrity typically involves many different public sector actors covering the various functions of an integrity system as defined in the OECD Recommendation on Public Integrity (OECD, 2017[8]).
The Recommendation on Public Integrity also invites countries to promote mechanisms for horizontal and vertical co-operation between the different relevant public integrity actors, including, where possible, with and between subnational levels of government. Such co-operation mechanisms can be formal or informal and aim at supporting coherence, avoiding overlaps and gaps, and sharing information and lessons learned from good practices (OECD, 2017[8]). Such arrangements for co-ordination are particularly relevant in the context of the implementation of a national strategy like the NACS.
Arrangements in other countries indicate that, to allow for a proper functioning of such a co-ordination mechanism, it is important that there is a technical support unit with the required mandate, summoning power, capacities and appropriate financial resources to convene the relevant actors, organise necessary meetings and agendas, steer the discussions, and prepare background information. This body could also be responsible for monitoring the implementation by the various agencies responsible for different measures of the AP (see Section 4.2.2).
As previously mentioned, the NACS could make clearer how the responsible entities for implementation were involved in the process of elaborating the strategy. The strategy development process could benefit from the appropriate participation of actors responsible for carrying out any part of the strategy. Where various agencies, departments, units and individuals across different branches of power have responsibilities for implementing elements of a strategy, their acceptance and active support of the strategy’s objectives and actions will be critical for the strategy to succeed (OECD, 2020[3]).
Budget
In the section on financing, the NACS states that the “majority of the directions of actions stipulated in the intervention areas do not project cost requirements that would require budgetary funds to be planned outside the regular operational budgets of the organs concerned.” Officials from the Ministry of Justice and the NPS told the OECD that, in particular, the legislative measures proposed in the AP would not require any new funding commitment.
For some activities, the NACS acknowledges that extra funding will be required. This includes resources for the development and implementation of the information campaign (6.8) envisaged within the “Social integrity” intervention area, the implementation of the corruption perception survey every two years (5.9) included in the “Integrity of public sector organs” intervention area, and the IT developments included in the “Integrity of public sector organs” intervention area (5.5). Officials from the Ministry of Justice and the NPS told us that the Hungarian Government has agreed in principle to allocate budget to the areas which require extra funding. Hungary may wish to consider how objectives and actions will be prioritised in the event of budget reductions and funding shortfalls. Hungary could set a specific requirement in the NACS for implementing authorities to raise resourcing issues in the status reports they are required to submit to the NPS as the monitoring body for the NACS.
Indicators
As set out above, the NACS could improve its use of indicators, baselines, milestones and targets to give implementing authorities a better of sense of what they are expected to deliver, how, and by when. Selecting one or more indicators of progress, choosing a baseline for each indicator, and establishing realistic targets for each implementation indicator element also allows a more in-depth evaluation of progress and delivery. As set out in Section 4.2.2 below, evidence from monitoring or evaluation can enhance targeting and steering of current and future policies.
Officials from the NPS said in consultations with the OECD that they would develop and include a ‘fixed rate indicator’ in the implementation matrix, which would allow them to benchmark measures and enable comparison with future strategies. Hungary could also draw data from different sources for measuring various aspects and levels of the NACS’s implementation. This is because a range of indicators from different sources, including administrative data, public and staff surveys, documents and legislation, can give policymakers a more holistic and integrated view of what it is really happening.
Table 4.3 sets out examples of different data sources that can be used to develop indicators. Hungary could draw from the data sources set out in Section 2.2.2, Chapter 2 above and incorporate indicators, such as those set out below, into their monitoring and evaluation process.
4.2.2. Include a methodology allowing for flexible adjustments of the NACS in the monitoring
The aim of monitoring is to identify real-time challenges related to the implementation of a strategy, to inform decisions and enable adjustments during implementation. To achieve this feed-back loop between monitoring and implementation, clear indicators, mechanisms and procedures to discuss progress made should be established. Monitoring works best if it is part of the day-to-day activity of the agencies responsible for implementing the strategy. A central monitoring body, working outside these implementing agencies, can ensure that relevant information about the overall status of the implementation of the strategy is bundled and analysed to draw a complete and coherent picture that facilitates decision-making, communication and incentives for improvement through benchmarking. Such a central monitoring system can only work if monitoring begins effectively at the level of the implementing agencies, who need to be committed to gather and report the relevant information.
Monitoring the implementation of anti-corruption strategies is essential to draw lessons and to increase the understanding of what works, why, for whom and under what circumstances. This should be a continuous process that allows for the strategy’s ongoing reassessment and revision, in order to establish its adaptability to changing circumstances and the emergence of new challenges, while maintaining its core vision. Although the strategy was developed at a particular point in time responding to specific requirements, it is important to ensure that it is a living document offering a flexible implementation framework that can reflect potential political, organisational and policy changes (UNODC, 2015[12]).
Moreover, a continuous reassessment of the strategy, its objectives and its actions can help achieve real change and results that are more realistic, lowering risk and seizing on windows of opportunity. For example, the Argentinian National Integrity Strategy entrenches flexibility in planning as a guiding principle and provides a dynamic set up, which allows participating bodies to provide information during implementation and to incorporate new actions after the strategy has launched (Jenkins and Camacho, 2022[13]). Indeed, the Anti-Corruption Office of Argentina notes that while the strategy has established objectives and actions, it must be flexible to incorporate new initiatives and actions from public entities to respond to the demands of their fields of competence. For this reason, implementing public entities have the opportunity to make additions or adjustments as they complete their goals. This flexibility makes the strategy a living and ever-expanding document (Oficina Anticorrupción de Argentina, 2021[14]). Hungary could explore methods for an iterative reassessment and adaptation process of the NACS based on information gleaned from monitoring and evaluating activities with the agile involvement of implementing agencies across the public sector.
According to the NACS, the entity responsible for monitoring the AP is the National Protection Service (NPS). The NPS is a special police force under the Ministry of Interior in charge of detecting and preventing crimes committed by law enforcement officials. The NPS has also conducted several awareness-raising initiatives on anti-corruption, ethics and integrity, which have targeted public officials and private companies. The NPS co-operates with the Prime Minister’s Office, which leads on monitoring the Strategy Against Fraud and Corruption for European Union Funds. Based on the monitoring activities conducted by the NPS, the Ministry of Interior is responsible for reporting once a year to the Hungarian Government the status of the NACS’s implementation.
Officials at the Ministry of Justice and the NPS said in consultations that the NPS is in constant communication with the authorities implementing public integrity measures. The NPS requests routine status reports on implementation, and every second and fourth year receives more detailed summary reports. These reports are based on standardised questions and data standards which are then tailored depending on the nature of the commitment and the implementing authority.
Hungary could consider whether the NPS is the most appropriate entity, as a law enforcement agency, for the monitoring of strategy implementation. There could be a risk that monitoring is perceived as a control mechanism intended to ‘name and shame’ agencies not implementing the strategy well enough. In these circumstances, implementing agencies may be less likely to report challenges they encounter, and it may be more likely that agencies seek to ‘game’ the indicators to make implementation look more effective than it is. Instead, monitoring should be established as a joint exercise to analyse and overcome implementation challenges, particularly in an environment in which implementation is spread across departments and agencies.
Hungary could also consider whether the NPS has the resource and convening power to effectively co-ordinate and oversee the implementation of the NACS, given the range of objectives and actions in the strategy which apply to bodies across the public sector, including outside central government. Other OECD countries have established joint units incorporating the public authorities with responsibility for implementing the strategy to improve monitoring and co-ordination. For instance, following publication of the UK’s Anti-Corruption Strategy in 2017 (UK HM Government, 2017[15]), the UK’s Joint Anti-Corruption Unit has been responsible for co-ordinating anti-corruption work across government, supporting the Prime Minister’s Anti-Corruption Champion, and facilitating an Inter-Ministerial Group on corruption which assessed implementation and the changing nature of the corruption threat in the United Kingdom. The NPS could retain a valuable secretariat function, but monitoring may be more effective if Hungary’s Anti-Corruption Task Force, which was established as a joint platform including different implementing authorities, took a more involved role in monitoring as well as evaluation.
Finally, as already highlighted in the previous section and in Section 3.3.1, Chapter 3, the AP does not provide the required level of detail (intermediate steps and milestones) nor the related indicators to enable an effective monitoring process. Again, a more in-depth planning exercise to determine the intermediary steps and milestones and indicators could be undertaken. This planning process could also think about risks that could undermine the achievement of the objectives and the measures included in the AP.
4.2.3. Clarify the methodology for evaluation, defining indicators and including evaluations in the action plan
The information produced by monitoring integrity policies can be used, together with adequate indicators or proxies for the desired outcomes, to evaluate the outcome and impacts of the NACS. Evaluations ask questions beyond the implementation status of a given action, and look instead into questions such relevance, coherence, effectiveness, efficiency, impact or sustainability (Box 4.1). As such, evaluations usually use a broad spectrum of sources of both quantitative and qualitative information to answer these questions and focus on drawing lessons learnt.
According to the NACS, the Anti-corruption Task Force would be responsible for evaluating the implementation of the NACS on an annual basis. These results would be discussed by the Hungarian Government which would then take decisions on whether / how to revise the strategy. The NACS could more clearly lay out the methodology of this evaluation, e.g., following the guidance in Box 4.1, assign budget for carrying out these evaluations, and set out relevant actions in the action plan. In addition, although Hungary committed to adopt a report on the implementation of the NACS by 28 February 2026, which is positive, the NACS itself could be clearer on how this evaluation at the end of the implementation period will work, and how any results will be taken into account in the design of future integrity policies.
In addition, Hungary may wish to consider how the Anti-corruption Task Force’s evaluations could be supported by third-party evaluations, which could provide comparisons with best practice and recommendations for keeping the strategy and Hungary’s anti-corruption work relevant and responsive. For instance, the Romanian NAS 2016-2020 was independently reviewed by the OECD (OECD, 2021[16]).
As a policy evaluation, the goal is not to assess whether every single activity of a given strategy has been implemented to what degree, but rather to look whether the strategy has been able to contribute to the stated goals and objectives. As such, while the level of implementation is a relevant dimension to look at, an evaluation goes beyond the implementation and achievement of outputs to investigate the change to which the strategy has contributed and to what extend the desired results have been achieved and how (OECD, 2017[5]).
As such, the methodology to evaluate the achievement of the envisaged results and the objectives of the NACS could take into account the following dimensions:
Relevance: Was the NACS designed to respond to country needs and priorities? To what extend are the objectives still valid? Do the stakeholders feel a sense of ownership?
Coherence: Was the NACS coherent with other governance reforms and policies in relevant key areas (external coherence)? Were the different objectives of the NACS designed in a way to reinforce one another and create synergies and were the activities relevant to contribute to the achievement of the results and the objectives (internal coherence)?
Effectiveness: To what degree has the NACS achieved the envisaged goals and implemented the activities (level of implementation)? What were the major factors influencing the achievement or non-achievement of the objectives?
Efficiency: How well were the available resources used to achieve the objectives of the NACS? Where the objectives achieved on time?
Impact: What differences did the NACS make? What were the positive changes and could some unintended consequences be observed?
Sustainability: How did the NACS build on earlier efforts to prevent and combat corruption and how likely are the implemented changes to last over time?
Note: The criteria follow the OECD-DAC evaluation criteria.
Source: Adapted from (OECD, 2021[16]), Evaluation of the Romanian National Anti-corruption Strategy 2016-2020, OECD, Paris, https://www.oecd.org/gov/ethics/evaluation-romanian-national-anti-corruption-strategy-2016-2020.pdf.
Aside from monitoring and measuring the benefits of the NACS, communicating progress and results to internal and external stakeholders, including the wider public, does not only enable accountability, but also increases the credibility of integrity efforts by the government and fosters support for the strategy to keep up public pressure for full implementation when it faces political barriers. In their action plan, countries may include information and communication activities, taking into account all new information media such as social media, as well as traditional and tested approaches such as town hall meetings and public hearings (UNODC, 2015[12]).
4.3.1. Plan measures to engage relevant stakeholders and civil society in monitoring implementation, and develop a communications strategy to involve and inform citizens and businesses
The current version of the NACS does not include a communication strategy of the results of the implementation of the NACS and its AP to a wider group of internal and external stakeholders. It includes a mandate to the Ministry of Interior to report to the Hungarian Government the status of the strategy’s implementation, and a mandate to the Anti-corruption Task Force to discuss the evaluation results with the Hungarian Government to define whether the NACS should be revised. However, relevant stakeholders from both the public sector and the whole of society are not targeted with relevant communication activities, missing the opportunity to showcase successes, gain wider support and increase credibility.
Communicating regularly and clearly about progress in the achievement of the NACS’s goals could have several advantages, including building citizens’ trust and increasing general support for the strategy. Hungary could design and implement a communication strategy that focuses on the achievements of the NACS and its AP, showing that change is possible. Such results need to be real and based on the data from monitoring activities and evaluations’ results, to avoid promoting cynicism, decaying trust in government and disconnecting citizens from constructive political engagement. Hungary could consider international good practices for designing a communication strategy (Box 4.2).
When designing a communication strategy, there are several elements that countries should considered to maximise its effectiveness. Below, some general good practices for designing a communication strategy:
Carry out a context analysis: To be able to communicate successfully, it is indispensable to have a clear understanding of the issues that are being addressed, based on evidence rather than impressions.
Define the main objectives of the communication strategy: Based on the context, define the objectives of the communication strategy. This could be done by responding to the following questions: “What is the change or impact you are trying to make? What does success look like?”
Identify the target audiences: Who are you targeting? Whose life do you want to change?
Carefully design the messages to avoid negative impact: Messages should be designed carefully in order to avoid negative or unexpected consequences. For instance, raising the issue of corruption and the costs related to it may lead to undesired consequences. In a context where corruption is already very present in the public debate and media, awareness raising campaigns on corruption may increase the already high awareness for an existing problem and thereby confirming the impression that corruption is widespread, leading to inaction or increasing corruption.
Identify relevant channels that can be used to distribute the messages: When identifying the relevant channels, it may be relevant to use the following guiding questions: How do you engage with our audience? Where are they most receptive? Which format will be more engaging for them? Channels may include television, print media, the website, blogs, social media, and specific networks and partnerships.
Identify the resources and skills needed for the implementation of the strategy.
Assess the results of the communication campaigns and the impact of the strategy: Assess the impact of communications strategy to fine-tune and improve, within a continuous circle of implementation, evaluation and adaptation.
Source: OECD elaboration.
References
[2] Jacob, S., S. Speer and J. Furubo (2015), “The institutionalization of evaluation matters: Updating the International Atlas of Evaluation 10 years later”, Evaluation, Vol. 21/1, pp. 6-31, https://doi.org/10.1177/1356389014564248.
[13] Jenkins, M. and G. Camacho (2022), Core principles for the development of anti-corruption strategies: Practices from around the world, https://www.u4.no/publications/core-principles-for-the-development-of-anti-corruption-strategies.pdf.
[11] Kaptein, M. (2007), Developing and Testing a Measure for the Ethical Culture of Organizations: The Corporate Ethical Virtues Model, ERIM Report series Research in Management, No. ERS-2007-084-ORG, Erasmus Research Institute of Management (ERIM), http://hdl.handle.net/1765/10770.
[6] OECD (2023), OECD Public Integrity Indicators, https://oecd-public-integrity-indicators.org/ (accessed on 5 January 2023).
[16] OECD (2021), Evaluation of the Romanian National Anti-corruption Strategy 2016-2020, OECD, Paris, https://www.oecd.org/gov/ethics/evaluation-romanian-national-anti-corruption-strategy-2016-2020.pdf (accessed on 18 February 2022).
[7] OECD (2021), The OECD Public Integrity Indicators – new measurements for policy makers and actionable change, https://oecd-events.org/gacif2021/onlinesession/72b2f15a-c871-eb11-9889-000d3a20eda5.
[4] OECD (2020), Improving Governance with Policy Evaluation: Lessons From Country Experiences, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/89b1577d-en.
[3] OECD (2020), OECD Public Integrity Handbook, OECD Publishing, Paris, https://doi.org/10.1787/ac8ed8e8-en.
[5] OECD (2017), Monitoring and Evaluating Integrity Policies, Working Party of Senior Public Integrity Officials, GOV/PGC/INT(2017)4, OECD, Paris.
[8] OECD (2017), “Recommendation of the Council on Public Integrity”, OECD Legal Instruments, OECD/LEGAL/0435, OECD, Paris, http://www.oecd.org/gov/ethics/Recommendation-Public-Integrity.pdf.
[1] OECD (2013), Development Results: An Overview of Results Measurement and Management, OECD, Paris, https://www.oecd.org/dac/peer-reviews/Development-Results-Note.pdf.
[14] Oficina Anticorrupción de Argentina (2021), Estrategia Nacional de Integridad: Planificación estratégica de la política de integridad del Poder Ejecutivo Nacional, https://www.argentina.gob.ar/sites/default/files/informe_de_presentacion_de_la_estrategia_nacional_de_integridad.pdf.
[10] Parsons, J. (2013), Indicators of Inputs, Activities, Outputs, Outcomes and Impacts in Security and Justice Programming, Department for International Development (DFID).
[15] UK HM Government (2017), The United Kingdom Anti-corruption Strategy 2017-2022, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/667221/6_3323_Anti-Corruption_Strategy_WEB.pdf.
[9] United Nations (2011), The United Nations Rule of Law Indicators - Implementation Guide and Project Tools, Department of Peacekeeping Operations and Office of the High Commissioner for Human Rights, New York.
[12] UNODC (2015), “The United Nations Convention against Corruption National Anti-Corruption Strategies A Practical Guide for Development and Implementation”, United Nations, https://www.unodc.org/documents/corruption/Publications/2015/National_Anti-Corruption_Strategies_-_A_Practical_Guide_for_Development_and_Implementation_E.pdf (accessed on 19 September 2021).