New Public Governance and Public Value Co-creation: The Case of the European Court of Auditors Environmental Audits
ABSTRACT
This article seeks to bring evidence on public value co-creation between public auditors and auditees, with a focus on the environment, as a matter of pressing interest for the citizens. We analyse data from performance audit reports issued by the European Court of Auditors: the tonality of audit conclusions and the corresponding auditees’ replies and the degree of acceptance and implementation of audit recommendations by the auditees. The analysis is performed using the theoretical perspectives of institutional logics and public value co-creation. We use qualitative and quantitative methods, including sentiment analysis and bivariate analysis. We find evidence that the EU public auditors increased their focus on environmental issues, following an increasingly trending institutional logic of focusing on non-financial topics of public interest. However, the tonality of environment-related audit conclusions and the auditees’ replies does not differ from the non-environment-related ones. We also demonstrate that public auditors co-created public value together with the auditees through a significantly higher acceptance of their recommendations in environmental than in non-environmental audits. The article contributes to the existing literature by using data from a supranational supreme audit institution and by applying an innovative research methodology in the audit field (sentiment analysis). It also offers practitioners valuable insights on performance audit’s evolution and impact. Although the data is limited to one supreme audit institution, this research can open future research avenues to perform comparative analysis with other supreme audit institutions or extend the research scope to content analysis of the substance of audit findings.
Keywords: Value co-creation; Public value; Performance audit; Public audit; Environment; Sustainability.
JEL classification: C10; H11; M42.
La nueva gobernanza pública y la co-creación de valor público: el caso de las auditorías de medio ambiente del Tribunal de Cuentas Europeo
RESUMEN
Este artículo busca aportar evidencias sobre la co-creación de valor público entre los auditores públicos y los entes auditados, poniendo el foco en el medio ambiente como un asunto de interés apremiante para la ciudadanía. Analizamos los datos de los informes de auditoría de gestión emitidos por el Tribunal de Cuentas Europeo: el tono de las conclusiones de auditoría y las correspondientes respuestas de los auditados y el grado de aceptación e implementación de las recomendaciones de auditoría por parte de los auditados. El análisis se realiza desde las perspectivas teóricas de las lógicas institucionales y la co-creación de valor público. Utilizamos métodos cualitativos y cuantitativos, incluido el análisis de sentimiento y el análisis bivariado. Encontramos evidencia de que los auditores públicos de la UE incrementaron su enfoque en temas medioambientales, emitiendo un número cada vez mayor de informes relacionados con el medioambiente, siguiendo una lógica institucional actual de la nueva gobernanza pública de enfocarse en temas no financieros de interés público. No obstante, la tonalidad de los informes de auditoría, las conclusiones y las respuestas de los auditados en auditorias medioambientales no difieren del resto. También demostramos que los auditores públicos cocrearon valor público junto con los auditados a través de una aceptación significativamente mayor de sus recomendaciones en las auditorías ambientales si la comparamos con la aceptación en las no ambientales. El artículo contribuye a la literatura existente utilizando datos de una institución de auditoria supranacional y aplicando una metodología de investigación innovadora en el campo de la auditoría (análisis de sentimiento). También ofrece a los profesionales información valiosa sobre la evolución y el impacto de la auditoría de gestión en el sector público. Aunque los datos se limitan a una única institución auditora, este artículo puede abrir vías de investigación futuras para realizar análisis comparativos con otros tribunales de cuentas o ampliar el alcance de la investigación al análisis de contenido de las conclusiones de auditoría y de las respuestas de los auditados.
Palabras clave: Co-creación de valor; Valor público; Auditoría de gestión; Auditoria pública; Medio ambiente; Sostenibilidad.
Códigos JEL: C10; H11; M42.
1. Introduction
Audit is a key instrument in scrutinising the design and implementation of public actions, assessing performance and enhancing transparency in the public sector (Triantafillou, 2020; Guillamón et al., 2021; Vela-Bargues et al., 2022). Public audit brings value to a wide spectrum of stakeholders (Hay & Cordery, 2021). To maintain their relevance, public auditors need to take an active stance in promoting the debate on hot topics such as the environment, climate change or the United Nations Sustainable Development Goals (SDGs) (Rana et al., 2022). Their work contributes to public value creation through audit recommendations and their acceptance and implementation by the auditees (Gendron et al., 2001). This is particularly important in the case of performance audits (Rana et al., 2022). The auditor’s collaboration with the auditees is essential to ensure the impact of the audit work and can, in fact, positively influence the public value co-creation process between them (Nath et al., 2020). Performance audit has important consequences for public accountability as a pillar of democracy (Pollitt et al., 1999), hence the importance of adding to existing research on the field, especially from a multi-disciplinary perspective (Hay, 2017).
This research seeks to bring evidence on how the interactions between public auditors and auditees during the performance audit process contribute to public value co-creation in a field of great interest for the citizens- i.e., the environment. It covers two research gaps in the literature: i) a need to delve deeper into the little-researched subject of public value co-creation in the performance audit process (Rana et al., 2022; Cordery & Hay, 2019); and ii) the innovative approach to focus on a supranational supreme audit institution (SAI) (Mattei et al., 2021).
The European Court of Auditors (ECA), the guardian of the European Union (EU) finances, is going through a process of transformation seeking to provide its main stakeholders - the citizens - a sounder supervision of the EU public finances. In its 2021-2025 strategy, the ECA intends to enhance its contribution to a better EU by focusing its audit work on topics of pressing interest for the citizens, such as the environment and sustainability, where more value can be created to improve public actions. The ECA intends to achieve these objectives by promoting public service value co-creation through cooperation with peer audit institutions across the EU and with other EU institutional stakeholders (ECA, 2021).
Public service value creation is a concept often used by policymakers and public managers (O’Flynn, 2021), but the literature shows that it is surrounded by a certain degree of controversy (Cluley et al., 2021). In our research, we aim to cover a research gap by offering an original insight into public value co-creation by public auditors and auditees, at the same time analysing the ECA’s evolution following the prevalent institutional logic (IL). Previous research revealed that social issues and value creation are linked (Adams, 2017). Using qualitative methods, we describe collaborative procedures between the EU auditors and their auditees, to better understand the ECA’s contribution to public value co-creation. Using quantitative methods (sentiment and bivariate analysis), we analyse the ECA’s environment-related audit work. Using a database of performance audit observations, we focus on the ECA’s audit outputs - audit conclusions and recommendations - and the feedback received from the auditees in the form of replies to audit conclusions, as well as their acceptance and implementation of recommendations. This is the way how SAIs measure the impact of their work (McCrea & Vada, 1997; INTOSAI, 2022).
Hence, this article aims to contribute to the wide field of value co-creation in the public sector with a focus on a supranational audit institution (Mattei et al., 2021), and also on the institutional theory by providing evidence on a public organisation’s IL adaptation to the prevailing logic (Thornton, 2002; Thornton & Ocasio, 2008). Although the focus is on a single SAI, this analysis could also be relevant for public audit in general (Tidå, 2021) by applying the research methods to data from other SAIs, through individual or comparative analysis (Hay & Cordery, 2020).
After this brief introduction, the study unfolds as follows. Next, we present the theoretical framework in section 2 and the literature review and hypotheses development in section 3. Section 4 describes the data and the research methodology, followed by the presentation of the results in section 5. Finally, section 6 presents the research conclusions, its limitations and future avenues of analysis.
2. Theoretical framework
Our research is based on the IL framework (Alford & Friedland, 1985) together with the important concept of value co-creation in the public sector audit field.
These two theoretical frameworks cover the essential aspects of this research from two perspectives. First, the analysis focuses on environmental performance audits. Considering that the environment is a matter of pressing interest for the citizens, who are the main stakeholders of the public audit sector, IL is a suitable lens because it helps explain how any organisation aims to remain relevant in the public eye (Johnsen, 2019; Lounsbury et al., 2021) - hence, any audit institution -, aligns its strategy and task with its stakeholders’ interests (Parker et al., 2021). In recent times, the public sector has been more inclined to consider public action from a more non-financial perspective, thus enhancing the public value perspective of its actions (O’Flynn, 2007, 2021; Pestoff, 2018). Second, the research focuses on the interactions between public auditors and their auditees, - audit conclusions and the auditees’ replies and audit recommendations and their acceptance and implementation by the auditees -, as suitable proxies for public value co-creation, bearing in mind that the implementation highly depends on the auditors’ follow-up (Aikins, 2012). They represent an example of an inter-organisational collaboration process in the public sector (Abu Hasan et al., 2013), which has been enhanced through more fluid communications between auditors and auditees in recent years (Pierre & de Fine Licht, 2019), as exemplified in the ECA’s case through the adversarial procedure described further in section 3.
2.1. Institutional Logics
IL is a theoretical perspective which provides the core concepts needed to understand how organisations transform themselves by absorbing the evolution and adaptation of the different actors’ beliefs and actions together with internal and external environmental factors which impact their activity (Battilana, 2006; Friedland & Alford, 1991). Thus, IL influences how organisations prioritise certain issues in their activity (Modell et al., 2017; Parker et al., 2021), seeking to gain or maintain legitimacy and relevance, adopting the prevailing logic at any given time (Thornton & Ocasio, 2008; Lounsbury et al., 2021).
In our study, we focus on two ILs adopted by the public administration in recent decades - New Public Management (NPM) and New Public Governance (NPG) (Osborne, 2006; Almqvist et al., 2013; Grossi et al., 2023). The NPM approach has been questioned by scholars as being too inspired in the private sector governance systems, somewhat ignoring the ultimate purpose of the public sector to provide public services and public goods which benefit all the society (Dunleavy et al., 2006; Osborne et al., 2013; Cluley & Radnor, 2020), regardless of the economic status or the purchasing power, by redistributing wealth (Herwartz & Theilen, 2017). Despite these criticisms, new perspectives have arisen on how the public management theory must take a public service-dominant approach (Osborne, 2010) instead of focusing on it as a product as in the classic management theory (Osborne et al., 2013). At the same time, the “Value for Money” concept, intimately connected to the NPM, is also under debate, given the wide spectrum of public sector’s stakeholders, the diversity in the measurements used, or the continuous evolution of the environment in which the actions evaluated are developing (McKevitt, 2015). In the past decades, the public sector IL evolved from the NPM towards a more public value-oriented approach under the NPG logic (Osborne, 2006; 2010), more focused on non-financial aspects such as social issues, SDGs, environment and climate change, transparency, reducing corruption and more oriented towards outcomes and quality control (Almqvist et al., 2013). It enhances the dissemination of non-financial information on environmental or social aspects of public actions (Montesinos & Brusca, 2019), whose impact needs to be scrutinised from a financial and especially non-financial perspective to keep decision-makers and governments accountable (Cordery & Hay, 2022). This new logic is more sensitive to the fact that the public sector does not need to achieve the same competitive advantages as private organisations and that its stakeholders need a wide range of services with sometimes divergent interests which evolve (Ansell & Torfing, 2021). NPG emphasises processes and outcomes, having public value as its core concept (Pestoff, 2018).
2.2 Public value co-creation
The concept of value co-creation, which originated and was intensively researched in marketing - see Grönroos, 2011; Gallarza et al., 2011-, is relatively new when applied to public services (Meynhardt, 2009). Public value co-creation, which may present itself in many forms involving multi-actor actions (Cluley & Radnor, 2020), is considered a transformational concept and a core governance tool (Ansell & Torfing, 2021), albeit depending on societal and governance dynamics (Voorberg et al., 2017). Public value creation is understood and accepted as a democracy-enhancing factor which helps accountability checking (Moore, 2013). It helps to address the public’s needs by directing public sector organisations’ actions towards improving the services provided and adapting them to the evolving needs of the users (Moore, 1995; Bryson et al., 2017).
There are various ways of approaching value co-creation in public services. Firstly, and the most obvious, the involvement of the public service users - the citizens, as ultimate stakeholders - in the process of public value co-creation (Cluley & Radnor, 2020; Osborne et al., 2016). For instance, it is now commonplace for the public administration to require feedback from the citizens on the services provided through surveys after providing an in-person service or on websites after administrative procedures. Furthermore, the public administration’s online presence is increasingly more visible, especially on social media, which provides users with a platform to interact with the institutions and to give immediate feedback on services (Hancu-Budui et al., 2020). These channels offer public organisations a window to explore and understand the citizens’ interests and concerns and to address them. This makes public services much more collaborative and interactive - a distinctive trait of NPG (Baldwin et al., 2019; Sorrentino et al., 2018) - and requires a higher degree of empathy and interpersonal skills from public servants (Steen & Tuurnas, 2018).
Secondly, public services value co-creation may arise from inter-institutional cooperation in public administration to improve the quality of the services provided by the interacting institutions (Bryson et al., 2017; Le Pennec & Raufflet, 2018). In this sense, within the public service systems, different organisations interact, impacting mutually in their activity and evolution and leading to value co-creation in the delivery of public services (Page et al., 2015; Osborne et al., 2016). In this second category of value co-creation steaming from inter-institutional cooperation falls this paper on public sector audit, as the ECA interacts with its auditees as explained in section 3, enhancing the mutual trust, respect and coordination on topics of interest for the society seeking to address public problems through mutual learning (Torfing et al., 2021). It represents a case of co-creation of value with an immediate and future impact on public services, contributing to their sustainability and continuity for the future (Osborne et al., 2015; Osborne et al., 2016).
3. Literature review and hypotheses development
The assessment and evaluation of public policy design and implementation through performance audits in a wide array of topics is a good example of public value co-creation by public audit institutions (Talbot & Wiggan, 2010; Cordery & Hay, 2019). The public value’s core concept is a combination of impact - efficiency and effectiveness - and legitimacy, the latter (legitimacy, hence public trust) being the distinctive characteristic of the public sector (Kelly et al., 2002; Talbot & Wiggan, 2010) and of NPG (Osborne, 2006).
The ECA’s performance audit outputs can be useful to assess how it has contributed to public value co-creation, by focusing on topics of interest for the citizens and by improving public action together with other organisations - the auditees. The value co-creation between the auditors and the auditees is of great interest, given that a performance audit is an evaluation of policy design and actions implementation, producing results directed to point out problems and shortcomings (Pollitt et al., 1999; Reichborn-Kjennerud, 2014). Performance audit offers the auditees a base from which they can make informed decisions on how to tackle and overcome detected failures and how to further improve their processes and strategies (Parker et al., 2021). In addition, if covered by the media, performance audits may stir the debate and draw the attention of politicians. Hence, the implementation of audit recommendations is likelier to happen (Raudla et al., 2016; Johnsen et al., 2019). It may also contribute to improving internal controls in public administration (Reichborn-Kjennerud & Vabo, 2017) and to enhancing public accountability (Gendron et al., 2001; Reichborn-Kjennerud, 2013). By auditing topics such as environmental policies and actions, public auditors can fuel a public debate on issues that concern citizens, maintaining pressure on political actors to act and make decisions for the common good (Prasad, 2018). Thus, public auditors can create public value (Funnell & Wade, 2012), yet sometimes they need to do more to influence public policy (Cordery & Hay, 2019).
3.1. The growing importance of environmental audits
In the past years, the ECA’s approach has gradually shifted in focus from a “financial and compliance audit” towards a more “performance audit oriented” approach, culminating with the adoption of the performance audit manual in 2015 and the issuing of its 2018-2020 Strategy (ECA, 2017). In the document, the ECA expressed the intention of enhancing the added value of its work in the context of the EU financial management, focusing on performance, and producing clearer messages for EU stakeholders to ensure that EU public policies and actions are influenced by audit findings. The Court considers that EU citizens not only expect assurance that public funds are spent correctly, but they also want to know if the intended results have been achieved. Therefore, the ECA emphasizes performance audits to achieve this goal and to create value through EU actions. The ECA undertakes annually a series of performance audits of specific spending or policy areas, and budgetary or management issues whose outcomes are published in Special Reports (SRs). From 1977 (its founding year) to 2020, the ECA has issued a total of 536 SRs, 84% of which were issued after 1996. Thus, the performance audit output has greatly increased, levering its potential to contribute to public value creation through improvements in public policy design and implementation (Morin, 2014; Raudla et al., 2016; Desmedt et al., 2017). This strategy exemplifies under the IL theoretical framework, how the ECA followed an NPM logic in the evolution of its priorities and objectives.
In 2021, the ECA issued its 2021-2025 Strategy (ECA, 2021) which emphasizes value added as a core concept of its activities, together with pressing matters such as the SDGs, with a focus on the environment and climate change. It continues to focus on performance audit as a way of ensuring sound management of taxpayers’ money and of contributing to improving EU public policies, through audit conclusions, recommendations, and subsequent follow-up of their implementation by the auditees. The strategy stresses the objective of strengthening collaboration with peer audit institutions, and other administrations at the EU and member state (MS) level such as parliaments and governments, as a means of increasing the value added of the ECA’s activity. This new strategy exemplifies how the ECA has evolved towards an NPG logic by scrutinising social issues which are at the top of the citizens’ priorities.
The environment and climate change are considered as the main threats to our society by European citizens (Poushter & Huang, 2020). Thus, given that the environment-related assessment is an essential part of performance audit (Huang & Li, 2013), we analyse the ECA’s environment-related performance audits as pressing topics of interest. Environment-related performance audit is an area scarcely researched (Mattei et al., 2021). According to Van Leeuwen (2004) early state-of-the-art study on SAIs’ environmental audits, only half of the SAIs had incorporated environmental issues within their audit scope based on INTOSAI surveys. This evidence is in line with Rika (2009), whose case study concludes that the audit institution researched tends to follow INTOSAI’s practice and guidelines. Existing research has surveyed the trends in global environmental performance audit identifying difficulties the SAIs face while undertaking these audits (Prasad, 2018), or analysed financial performance audits of environmental issues in Germany, confirming a shift in public auditors’ task from mere ex-post checkers to ex-ante evaluators and advisers to policymakers (Weihrich, 2018). Hence, the European SAIs are performing environmental audits, yet their environmental audit reporting shows room for improvement, especially regarding the embeddedness of the SDGs into the audits (Hancu-Budui & Zorio-Grima, 2023). The ECA is the main promoter of environmental audits among the EU audit institutions, being at the same time a debate-stirrer on this topic, hence contributing to increasing actions from other relevant public institutions at the EU and MS level (Hancu-Budui et al., 2020).
On the other hand, extant research has found that depending on the type of the report, the number of pages may change (Roca & Searcy, 2012) or the tonality may affect the impact of information related to sustainability (Vogler & Eisenegger, 2021; Otia & Bracci, 2022). The ECA itself has examined the tonality of its publications concluding it is “slightly on the positive side” (ECA, 2020b). However, one may consider whether our target, the type of SRs of growing importance, i.e., environmental SRs, share or not similar characteristics (in length or tonality) with non-environmental SRs, in case these attributes could be identified as levering the debate.
Considering the above, we propose the following hypotheses:
H1: The ECA increased its performance audit task and its focus on environment-related audits to follow the prevalent IL and create public value.
H2: Environment and non-environment audit reports have different characteristics - different length (pages or words) and different sentiment (tonality scores).
3.2. ECA’s adversarial procedure as a value co-creation instrument
According to the EU Financial Regulations’ provisions, before the issuing of the audit reports, the auditees - the European Commission, EU agencies or other bodies - can reply to the auditors’ findings and recommendations in a process which is known as the adversarial procedure (ECA, 2015). The auditees’ replies are published together with the SRs. Through this procedure, the EU auditors notify the audited organisations of the performance audit’s findings and recommendations. The auditees have six weeks to reply to the draft findings and recommendations, after which the final audit report is issued and published, together with the auditees’ replies. The purpose of the adversarial procedure is to resolve any disagreements regarding the facts and any differences of interpretation between the ECA and the auditees over the audit evidence. This period unfolds as a series of formal and informal exchanges between auditors and auditees, fulfilling the ECA’s principle of ‘no surprises approach’. This principle embodied by reciprocal transparency and professional respect on both sides enhances the trust between auditors and auditees and contributes to effective audit procedures, according the ECA methodology (ECA, 2022). The repeated interactions between auditors and auditees through this procedure represent an interorganisational collaboration (Page et al., 2015) and contributes to public value co-creation consisting of trust, respect, and coordination (Le Pennec & Raufflet, 2018).
During this procedure, depending on the degree of disagreement over facts or audit evidence, both auditors and auditees may vary their observations and replies, respectively, thus contributing to improving public policy and actions (Bryson et al., 2017).
The auditees’ replies on audit recommendations should state if they have been accepted or not, motivating the refusals, and how the auditees intend to implement them. Annex - Table A.1 shows examples of performance audit recommendations issued by the ECA and the replies received by the ECA from the European Commission. Non-binding recommendations, -such are the ECA’s - may not be implemented by public auditees and are often not followed up on by the scrutinising organisations (Van Acker et al., 2015).
As the examples in the Annex - Table A.1 show, the adversarial procedure produces specific actions in auditees which imply an improvement in EU actions and policy design leading thus to public value co-creation between auditors and auditees.
Annex - Table A.2 also presents some examples of audit findings and their replies from auditees, to illustrate the interactions between parties during the adversarial procedure.
The examples of performance audit findings and their replies (Annex - Table A.2) show interaction and synergies between auditors and auditees and prove that the audit process is not a blame allocation process, but a learning one (Furubo, 2011), yielding thus value for the parties involved. Considering that the auditees do not need to express acceptance of audit conclusions, as opposed to audit recommendations, they can reply to the audit findings therefore yielding a value-added exchange with the auditors in various ways. The auditors issue audit conclusions based on factual evidence regarding EU actions’ implementation and policy design. This procedure offers the auditees the opportunity to spot areas for improvement and to reply with either proposed improvements, explanations to why these improvements cannot be implemented or what has been done to address these issues.
Therefore, the exchange of views triggered by the adversarial procedure and its results - i.e., audit conclusions, recommendations and auditees’ replies included in the final performance audit reports- constitute an invaluable source of information for evaluating the performance audit impact and value-added. Not sooner than three years after the issuing of the reports, the auditors may proceed to follow-up on some of the previous recommendations to assess their degree of implementation (ECA, 2020a).
As illustrated by the examples in the Annex (Tables A.1 and A.2), the ECA’s adversarial procedure could be inspiring for other SAIs to implement or improve similar collaborative procedures with the auditees seeking to promote public value co-creation (Ansell & Torfing, 2021; Cluley & Radnor, 2020).
As far as we know, this article is the pioneer in analysing the outcome of the adversarial procedure (i.e., the conclusions, replies and recommendations) reported in the ECA’s SRs, with the two hypotheses that we put forward next:
H3: Accepted recommendations are more likely to be implemented than the rest, which favours the process of public value co-creation both in environment-related and non-environmental audits.
H4: Environment-related performance audits have a higher rate of acceptance and implementation of recommendations than the rest, hence co-creating more value than non-environmental ones.
4. Data and research design
4.1. Data Sample
We analyse all 298 SRs issued by the ECA from 2008 to 2020. The ECA provided us with an initial database with tabulated observations (conclusions, recommendations, auditees’ replies, implementation follow-up) from SRs issued up to mid-July 2019. We updated it with observations from subsequent SRs up to December 2020 - conclusions, recommendations, and auditees’ replies. The implementation follow-up information is gathered from either specific follow-up SRs or from a section of “Follow-up of Special Reports Recommendations” included in the different Annual Reports issued by the ECA, corresponding to the annual EU budget financial and compliance audits.
The SRs have been classified as environment-related (37 reports) and non-environment-related (261 reports). In total, they contain 2,785 audit conclusions and 2,706 audit recommendations, out of which 558 recommendations were followed up by the auditors - note some of the recommendations have been issued too recently to have been followed up yet. The recommendations have been classified by the degree of acceptance and implementation. For example, the first recommendation in Annex Table A.1 (from SR 09/2013) was initially not accepted by the auditee. Still, it was finally fully implemented and the other recommendation (from SR 15/2015) is classified as accepted and fully implemented.
The data is summarized in Table 1.
Table 1. Data summary of the population of reports analysed (2008-2020)
| Total | Environment | Non-environment | |
|---|---|---|---|
| Performance Audit Special Reports (SR) | 298 | 37 | 261 |
| INCLUDING: | |||
| Audit Conclusions | 2,785 | 371 | 2,414 |
| Audit Conclusions and replies kept for the tonality analysis | 2,279 | 291 | 1,988 |
| Audit Recommendations | 2,706 | 322 | 2,384 |
| Followed-up audit recommendations | 558 | 70 | 488 |
4.2. Research design
To address H1 on IL evolution - from an NPM to an NPG approach through more environment-related performance audit work- and public value creation, we perform a bivariate analysis (Spearman’s correlation for quantitative variables) to confirm if the number of environment-related reports follows the trend of an increasing performance audit output.
Next, we deepen our analysis addressing H2 by researching if the environment reports, the conclusions and their replies are different from the rest in length and tonality using sentiment analysis and bivariate analysis (Kruskal-Wallis test for quantitative variables, unequal groups). The IT tools used for H2 analysis are Stata and Python. Stata is used for statistical computations of bivariate analysis. Python is used for computing the reports’ length and for performing sentiment analysis (Gandía & Huguet, 2021) on reports, conclusions, and their replies. For this purpose, we pre-process the texts using the Python libraries PyPDF2, pdfminer and nltk. We analyse the data, extract the text, tokenize, and eliminate StopWords (Zorio-Grima & Carmona, 2019). We use the Python library VADER (Valence Aware Dictionary and Sentiment Reasoner), a lexicon and rule-based sentiment analysis tool (Dey et al., 2018; Alaei et al., 2019) to obtain a tonality score for each text observation (Hadro et al., 2021), to which we will hereinafter refer to as compound score, ranging from -1 (totally negative) to +1 (totally positive)- for instance, and to illustrate how the algorithm works, the sentiment scores obtained for one of the examples in Table A.2 are 0.5569 and 0.4647 in SR 26/2020. All observations with a compound score of 0 are dropped as it is interpreted that the algorithm could not establish a score for them. After dropping the observations with a compound score of 0, out of the 2,785 conclusions, 2,279 were retained for the tonality analysis (table 3) - 291 were environment-related and 1,988 non-environment-related.
To address H3, we perform a bivariate analysis of the 558 followed-up recommendations to assess if their degrees of acceptance and implementation are associated and to establish if the performance audit recommendations have had an impact on auditees’ actions, hence co-creating value (Gendron et al., 2001; Cordery & Hay, 2019). We use the Likelihood-ratio chi2 for categorical variables with less than 5 observations per cell (McHugh, 2013).
To address H4, we analyse the degree of acceptance and the follow-up implementation of the recommendations, proxies for audit impact and effectiveness (McCrae & Vada, 1997), hence for value co-creation. We use the Likelihood-ratio chi2 test for categorical variables to see if there are differences between environmental-related performance audits and non-environmental-related ones. Thus, we bring evidence on how adopting a citizen’s topic of interest into the audits, evolving thus from NPM logic towards NPG logic, leads to a higher acceptance and implementation of audit recommendations, hence increasing public value co-creation from auditors and auditees.
5. Results
Regarding H1 on the ECA’s contribution to value co-creation through environment-related performance audit work, Table 2 shows valuable statistics on SR variables. Panel A summarizes information showing a direct and positive correlation between total performance audit work and environment-related work. Bivariate analysis (Spearman’s correlation) confirms at a significant level that the more performance audits the ECA carried out, the higher the number of environment-related SRs, reaching an all-time high of 35% of all reports in 2020 (Table 2 Panel A). This is also in line with previous comparative research on EU Supreme Audit Institutions’ (SAIs) social media communications (Hancu-Budui et al., 2020) which showed that the ECA’s communication and activity promotion on environmental issues also followed an ascending trend in the last decade, ahead of its peer MS SAIs. This finding confirms that environment-related activities are a focal point of the ECA’s work. Hence, one might conclude that there is evidence of the ECA following the prevalent IL and creating public value by increasingly scrutinising and stirring the public debate on environmental topics, thus pushing for political and management action (Johnsen et al., 2019). This procedure helps to legitimize the institution through its activity outcomes as a public value creator, which enhances trust and accountability (Kelly et al., 2002), as a trait of the ECA’s activity evolution towards a more NPG logic.
To address H2, Table 2 - Panel B provides evidence through the Kruskal-Wallis test that environment SRs are not statistically different from the rest of the SRs in their length or tonality. Moreover, the sentiment analysis of audit conclusions and replies from the auditees (Table 2 - Panel C) does not provide significant evidence of different tonality from the rest of the observations. This result implies that the auditors do not necessarily try to emphasize environmental issues differently from the rest of the EU actions (i.e., as less or more of a problem), yet one can see the importance of environment-related matters by the increasing volume of work performed on the topic, which confirms public value co-creation for the stakeholders (i.e., the auditees and the citizens) through the assessment of pressing public matters.
Table 2. Comparative statistics on environment/non-environment performance audit outputs
| Panel A - Number of performance audit reports | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Year | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 | Total | Environment Vs. Total Reports Spearman’s correlation |
| Total | 12 | 18 | 14 | 16 | 25 | 19 | 24 | 25 | 36 | 23 | 35 | 25 | 26 | 298 | 0.7798*** p= 0.0017 |
| Environment | 0 | 1 | 0 | 1 | 1 | 3 | 3 | 3 | 3 | 3 | 5 | 5 | 9 | 37 | |
| % | 0% | 6% | 0% | 6% | 4% | 16% | 13% | 12% | 8% | 13% | 14% | 20% | 35% | 12% | |
| Non-environment | 12 | 17 | 14 | 15 | 24 | 16 | 21 | 22 | 33 | 20 | 30 | 20 | 17 | 261 | |
| % | 100% | 94% | 100% | 94% | 96% | 84% | 88% | 88% | 92% | 87% | 86% | 80% | 65% | 88% | |
| ***p<0.005 | |||||||||||||||
| Panel B - Statistics on Reports | |||||||
|---|---|---|---|---|---|---|---|
| Variable | Group | N | Mean | Sth. Dev. | Min. | Max. | Environment vs Non-Environment Kruskal Wallis |
| Pages | Environment | 37 | 64.16 | 12.00 | 46 | 93 | 0.0000 p=0.9869 |
| Non-environment | 261 | 65.23 | 18.67 | 16 | 146 | ||
| Words | Environment | 37 | 13785.65 | 4883.47 | 7536 | 32075 | 0.5950 p= 0.4403 |
| Non-environment | 261 | 14703.23 | 5867.03 | 3947 | 40219 | ||
| Sentiment analysis score | Environment | 37 | 0.1485 | 0.1127 | -0.1189 | 0.4102 | 0.9280 p=0.3354 |
| Non-environment | 261 | 0.1717 | 0.1079 | -0.1487 | 0.5768 | ||
| Panel C - Statistics on Sentiment Analysis regarding Conclusions and Replies | |||||||
|---|---|---|---|---|---|---|---|
| Variable | Group | N | Mean | Sth. Dev. | Min. | Max. | Environment vs Non-Environment Kruskal Wallis |
| Conclusions | Environment | 291 | 0.1348 | 0.3114 | -0.9580 | 0.9274 | 0.004 p= 0.9485 |
| Non-environment | 1,988 | 0.1304 | 0.3018 | -0.9179 | 0.9824 | ||
| Replies | Environment | 291 | 0.0868 | 0.3297 | -0.6124 | 0.8934 | 1.460 p= 0.2327 |
| Non-environment | 1,988 | 0.0637 | 0.3095 | -0.6980 | 0.9360 | ||
To address H3 on public audit value co-creation, our sample includes the recommendations which have been followed up - i.e., 558 recommendations which represents 21,73% of the recommendations issued between 2008 and 2020 (see Table 1).
The auditees’ replies were codified into six categories of acceptance and six categories of degree of implementation (Table 3).
Table 3. Statistics on Followed-up recommendations
| Acceptance of recommendations | Recommendations’ implementation | Likelihood-ratio chi2 | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Fully implemented | Implemented in most respects | Implemented in some respects | Not implemented | No longer relevant | Could not verify | Total | % | ||
| Accepted recommendations | 266 | 75 | 74 | 7 | 4 | 7 | 433 | 77.60 | 92.0688*** p=0.0000 |
| Partially accepted recommendations | 15 | 5 | 7 | 0 | 0 | 0 | 27 | 4.84 | |
| Non accepted recommendations | 7 | 3 | 4 | 12 | 0 | 0 | 26 | 4.66 | |
| Recommendations with no reply | 0 | 1 | 1 | 0 | 0 | 0 | 2 | 0.36 | |
| Recommendations for MS | 3 | 3 | 1 | 5 | 0 | 0 | 12 | 2.15 | |
| Others | 27 | 17 | 5 | 6 | 2 | 1 | 58 | 10.39 | |
| Total | 318 | 104 | 92 | 30 | 6 | 8 | 558 | 100 | |
| % | 56.99 | 18.64 | 16.49 | 5.38 | 1.08 | 1.43 | 100 | ||
We observe that 77.60% of the followed-up audit recommendations were accepted by the auditees, and 4.84% were partially accepted, so they may be considered as audit recommendations having a very high impact on the auditees’ actions. Data also show that over 75% of recommendations were either fully implemented (56.99%) or implemented in most respects (18.64%). These results demonstrate at a significant level (Likelihood-ratio chi2, p=0.0000) that performance audit co-created value for stakeholders in the studied period 2008-2020, as the more accepted recommendations achieved higher levels of implementation. This high level of recommendations’ implementation suggests that the auditees took action upon auditors’ findings and improved EU actions. Thus, one can conclude that performance audit has contributed to value co-creation together with the audited entities by improving EU actions and policy design. Our results confirm findings from previous research on public audit impact which analysed the acceptance of recommendations as an added value of performance audits in a collaborative process with auditees (Morin, 2004; 2014; Desmedt et al., 2017; Torres et al., 2019). Considering that the ECA’s recommendations are non-binding for the auditees, their high degree of acceptance and implementation also confirms value co-creation with the auditees, which benefits the citizens through the improvement of the EU actions and policies. This finding is especially interesting and positive for all the stakeholders involved despite previous research, which was rather discouraging when concluding that non-binding recommendations are generally not implemented (Van Acker et al., 2015).
Finally, to address H4, on value co-creation differences of environment-related performance audits as opposed to the rest of the ECA’s performance audit work, we perform bivariate analysis (Likelihood-ratio chi2 for categorical variables) by the degree of acceptance and degree of implementation by the auditees on the 558 followed-up recommendations. Results are presented in Table 4. Out of the 558 followed-up recommendations, 70 were environment recommendations- 59 of those were accepted recommendations (84%), out of which 56 (95%) have been implemented either fully, in most or in some respects. (Table 4 - Panel A). Out of the remaining 488 non-environment recommendations (Table 4 - Panel B), 76.64% were accepted by the auditees. Out of these, 359 (96%) were implemented either fully, in most or in some respects. According to the Likelihood-ratio chi2 test, recommendations were accepted in a significantly different way for environment or non-environment audits, with environment-related recommendations having a higher degree of acceptance. However, no statistical difference is found of differences regarding follow-up implementation of recommendations between environment and non-environment audits (Table 4 - Panel C). The acceptance of recommendations to a higher degree shows value co-creation from the public auditors by identifying shortcomings in policies design and implementation and by proposing actions to auditees to correct them. The higher level of acceptance for environment-related audits also shows that this type of recommendations have a bigger impact than the rest, thus creating more value for the citizens, confirming the NPG logic adopted by the ECA to focus on environment and sustainability topics.
Table 4. Statistics on Followed-up recommendations
| Panel A. Environment recommendations | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Acceptance | Follow-up on recommendations implementation | Likelihood-ratio chi2 | |||||||
| Fully implemented | Implemented in most respects | Implemented in some respects | Not implemented | No longer relevant | Could not verify | Total | % | ||
| Accepted recommendations | 40 | 6 | 10 | 2 | 1 | 0 | 59 | 84.29 | 38.6060*** p=0.001 |
| Partially accepted recommendations | 0 | 1 | 1 | 0 | 0 | 0 | 2 | 2.86 | |
| Non accepted recommendations | 0 | 0 | 1 | 5 | 0 | 0 | 6 | 8.57 | |
| Recommendations with no reply | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 1.43 | |
| Recommendations for MS | 0 | 1 | 1 | 0 | 0 | 0 | 2 | 2.86 | |
| Others | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.00 | |
| Total | 40 | 8 | 14 | 7 | 1 | 0 | 70 | 100 | |
| % | 57.14 | 11.43 | 20.00 | 10.00 | 1.43 | 0.00 | 100 | ||
| ***p<0.005 | |||||||||
| Panel B. Non-environment recommendations | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Acceptance | Follow-up on recommendations implementation | Likelihood-ratio chi2 | |||||||
| Fully implemented | Implemented in most respects | Implemented in some respects | Not implemented | No longer relevant | Could not verify | Total | % | ||
| Accepted recommendations | 226 | 69 | 64 | 5 | 3 | 7 | 374 | 76.64 | 72.2987 *** p=0.000 |
| Partially accepted recommendations | 15 | 4 | 6 | 0 | 0 | 0 | 25 | 5.12 | |
| Non accepted recommendations | 7 | 3 | 3 | 7 | 0 | 0 | 20 | 4.10 | |
| Recommendations with no reply | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0.20 | |
| Recommendations for MS | 3 | 2 | 0 | 5 | 0 | 0 | 10 | 2.05 | |
| Others | 27 | 17 | 5 | 6 | 2 | 1 | 58 | 11.89 | |
| Total | 278 | 96 | 78 | 23 | 5 | 8 | 488 | 100 | |
| % | 56.97 | 19.67 | 15.98 | 4.71 | 1.02 | 1.64 | 100 | ||
| ***p<0.005 | |||||||||
| Panel C. Differences between environment and non-environment recommendations | |||||||
|---|---|---|---|---|---|---|---|
| Acceptance | Type of recommendation | Likelihood-ratio chi2 | Follow-up on recommendations implementation | Type of recommendation | Likelihood-ratio chi2 | ||
| Environment | Non-environment | Environment | Non-environment | ||||
| Accepted recommendations | 59 | 374 | 20.7458*** p=0.001 | Fully implemented | 40 | 278 | 7.9710 p=0.158 |
| Partially accepted recommendations | 2 | 25 | Implemented in most respects | 8 | 96 | ||
| Non accepted recommendations | 6 | 20 | Implemented in some respects | 14 | 78 | ||
| Recommendations with no reply | 1 | 1 | Not implemented | 7 | 23 | ||
| Recommendations for MS | 2 | 10 | No longer relevant | 1 | 5 | ||
| Others | 0 | 58 | Could not verify | 0 | 8 | ||
| Total | 70 | 488 | Total | 70 | 488 | ||
(*)p<0.1; *p<0.05; **p<0.01; ***p<0.005
From these results, we conclude that environment-related recommendations contributed to a higher degree to public value co-creation in EU actions because of significantly higher acceptance of this type of recommendations, but not so on the grounds of differences in implementation. However, note the delay of at least three years of the implementation follow-up and that most of the recommendations on environmental topics were made in recent years (60% of the environment reports being issued from 2017 to 2020, Table 4 Panel A). Hence, future research should revisit this topic to check if environment-related recommendations have been also significantly more implemented than the rest, co-creating higher value together with the auditees thanks to the implementation procedure. This analysis of environment-related recommendations and their implementation is especially relevant because, while environmental audits have been a previous topic of research, as far as we know, no other studies have addressed their impact through the acceptance and implementation of recommendations, which is the most common way for SAIs to assess their impact and public value (INTOSAI, 2022).
6. Conclusions
Our research brings evidence that public audit contributes to value creation for the citizens and to value co-creation with the auditees (Dudau et al., 2019). It also confirms that the ECA adjusted its activity to the prevalent IL, adopting an NPM approach to later evolve towards an NPG logic, in line with the society’s concerns and priorities, in our case, environment-related ones.
In our view, the most relevant finding of this research is the impact that the performance audit has on EU policies and actions, considering the high degree of acceptance and implementation of ECA’s audit recommendations. It proves the usefulness of performance audit in the public sector and adds to findings from previous research (Desmedt et al., 2017; Morin, 2014; Raudla et al., 2016). Value co-creation between auditors and auditees through recommendations’ acceptance and implementation is even more relevant in the case of the ECA because of their non-binding nature. Previous research shows that binding audit recommendations are more likely to be followed than non-binding ones, and even more relevant, in non-binding recommendations, audit institutions do not even perform an implementation follow-up (Van Acker et al., 2015). This poses a question on the auditors’ work impact and value creation for society. However, by systematically reviewing the implementation of its non-binding recommendation, the ECA seeks legitimacy, by expanding its actions beyond its legally established mandate and intertwining its activities with the public’s pressing interests. This confirms again the ECA’s adoption of an NPG logic in pursuing public interests (Osborne, 2006), increasing public audit’s relevance in the scrutiny of public policy and action in a time when citizens do not consider auditors’ work trustworthy (Uman et al., 2023).
Thus, this study reflects on how public audit institutions play a role in increasing public sector value co-creation for the citizens (Cordery & Hay, 2019) by pointing out shortcomings and proposing corrective solutions for public policy and actions in the adversarial procedure. By expanding their scope and increasing their outputs towards the environment, the EU auditors play an increasingly important role in stirring the debate on environmental topics, in line with their communication strategy (Hancu-Budui et al., 2020).
This research contributes to the IL theoretical framework by exemplifying how public institutions adapt its activity to the prevailing logic seeking to legitimise their existence and continuation (Thornton, 2002; Power, 2003). This article contributes to public value co-creation by researching a collaborative process between different organisations, their interactions and its impact on public policy and its implementation. It also adds to research on performance audits in the public sector using data from a supranational audit institution - the ECA - which fills a gap in the existing literature (Mattei et al., 2021). Additionally, it covers a less researched area of performance audit, public value co-creation (Rana et al., 2022; Cordery & Hay, 2019). It brings empirical evidence that the audit work output of a relevant SAI such as the ECA has a significant impact on public actions in the sense that its recommendations are highly accepted and implemented. Furthermore, it confirms results from previous research related to the positive impact of performance audits in the public sector (Reichborn-Kjennerud & Johnsen, 2018) through the implementation of audit recommendations (Torres et al., 2019). Our article also offers practitioners valuable insights on performance audit impact, as the recommendations’ acceptance and implementation is the most common manner of assessing the audits’ impact, as almost 80% of the SAIs surveyed by the INTOSAI Working Group on Environmental Auditing show (INTOSAI, 2022). Additionally, practitioners could find useful the insights on the adversarial procedure as presented in the Annex - Tables A.1 and A.2, if they intend to implement a similar collaborative procedure with the auditees to enhance the impact of their audit results and accountability in the public sector (Lonsdale et al., 2011), at the same time co-creating public value (Page et al., 2015; Le Pennec & Raufflet, 2018).
Thus, these findings should encourage the ECA to continue its task to improve EU actions and to closely collaborate with MS SAIs to enhance the implementation level of audit recommendations made to MS. In addition, our analysis may help other public audit institutions to improve their performance audit activity and their relationship with the auditees, seeking their acceptance and implementing follow-up procedures and reporting on them afterwards to also produce a significant impact on public value, considering that the implementation of audit recommendations depends significantly on the auditors’ follow-up (Aikins, 2012).
The article also makes a methodological contribution by applying sentiment analysis to performance reports conclusions and replies, as a rather innovative technique in audit research. This method may be applied in future research and may be also used by practitioners to evaluate if the sentiment is different depending on the audited topics or evolves over some time and thus contributes to decision-making (Abirami & Gayathri, 2017).
Nonetheless, our research has limitations, as we are using data from only one audit institution. Also, another limitation is that we have not performed a content analysis of the substance of the audit recommendations or the audited-specific topics. This opens new research avenues. Future research may perform content analysis using the ECA data to assess its impact on the auditees, adding to the already existing research based on data from other SAIs (Desmedt et al., 2017; Johnsen et al., 2019; Morin, 2014; Raudla et al., 2016; Torres et al., 2019). Further studies could analyse if auditors adopted a conservative approach to performance audit (Pollitt, 2003) - did they make audit recommendations just on existing policies and regulations or did they further recommend substantial policy changes to impact public actions? Furthermore, our research design could be applied in comparative studies on MS SAIs. Finally, this article could be revisited in the future to see if the degree of implementation of environment-related recommendations is higher than in non-environmental audits once more data becomes available.
Annex
Table A.1. Examples of performance audits recommendations and auditees’ replies
| Special Report | Recommendation | Auditee’s Reply |
|---|---|---|
| SR 09/2013 EU support for governance in the Democratic Republic of the Congo | EU cooperation strategy In coordination with other development partners, notably EU Members States, the Commission and the EEAS should, with a view to programming for the 11th EDF and the design of future EU programmes, (i) pay increased attention to ensuring an appropriate balance of aid between all provinces, especially the poorer ones; (ii) combine support at central level with programmes at the provincial levels that link political and territorial decentralisation with improved natural resource management strategies and infrastructure rehabilitation and development; and (iii) reconsider EU support for improved management of natural resources on the basis of a comprehensive needs assessment. | While sharing the general preoccupations expressed by the Court, the Commission and EEAS do not accept the recommendation of the Court. (i) The Commission will continue to aim at achieving an appropriate balance of aid between all provinces including the poor ones in full coordination with other donors and taking into consideration that the poorest DRC provinces are also the least populated. (ii) The Commission will continue to involve local actors in the implementation of its projects, and within the limit of their capacity. The Commission will also support their capacity building. (iii) The Commission will continue to support the improved management of natural resources provided that it is confirmed as a sector of the 11th EDF and in the context of work sharing arrangements with other donors. |
| SR 15/2015 ACP–EU Energy Facility support for renewable energy in East Africa | Upon completion of complex projects, in particular those involving infrastructure investments, the Commission should: (i) require the implementing partners to provide in their final report an assessment of the potential need for continued technical assistance for operators. (ii) consider the possibility of providing funding for this purpose, for example through an amendment to the grant contract. | The Commission accepts this recommendation. An assessment of any continued technical assistance needs would be useful to identify possible support requirements after the EF funding elapses. Under the current EF programme, it will not be possible to increase the original grant allocation. Nevertheless, the Commission will examine how this might be addressed in order to consolidate the value of the investment made. Moreover, the Commission will examine how to apply the recommendation under new and innovative programmes such as the Electrification Financing Initiative — ElectriFI. |
Table A.2. Examples of performance audits conclusions and auditees’ replies.
| Special Report | Conclusion | Auditee’s Reply |
|---|---|---|
| SR 05/2020 Sustainable use of plant protection products: limited progress in measuring and reducing risks | Applying the IPM principles is mandatory for users, but Member State compliance checks have a limited scope. One reason for the lack of enforcement is that there are no clear criteria as to how users should apply the general principles of IPM or how the authorities should assess compliance (see paragraphs 21 to 27). | Member State authorities shall ensure that professional users comply with this requirement. In order to decide on compliance or non-compliance, Member State authorities should have clear assessment criteria. In line with the subsidiarity principle, converting general principles of IPM into practical criteria is the responsibility of Member States, and the Commission will continue to support Member States in this regard. |
| SR 26/2020 Marine environment: EU protection is wide but not deep | Overall, a framework was in place to protect the marine environment, but the EU’s actions had not restored seas to good environmental status, nor fishing to sustainable levels in all seas. We found that EU action had contributed to progress in the Atlantic where many fish stocks have stabilised and/or improved, but that in the Mediterranean, there were no meaningful signs of progress. | In addition to the significant progress made in the Northeast Atlantic, it is also important to recognise recent efforts and achievements in the Mediterranean. On the EMFF aspects, it is important to note that it is for Member States to target/make use of the available EU funds. Concerning signs of progress in the Mediterranean, the Commission recalls the ambitious strategy adopted in 2017, starting with the MedFish4Ever Declaration and leading to the adoption of a large number of measures at GFCM level and the adoption of the Western Mediterranean MAP. This strategy has yet to produce quantified results in terms of improved state of the stocks, but that is inevitable considering that these changes have been adopted only recently. |