OH-EpiCap: a semi-quantitative tool for the evaluation of One Health epidemiological surveillance capacities and capabilities

Although international health agencies encourage the development of One Health (OH) surveillance, many systems remain mostly compartmentalized, with limited collaborations among sectors and disciplines. In the framework of the OH European Joint Programme “MATRIX” project, a generic evaluation tool called OH-EpiCap has been developed to enable individual institutes/governments to characterize, assess and monitor their own OH epidemiological surveillance capacities and capabilities. The tool is organized around three dimensions: organization, operational activities, and impact of the OH surveillance system; each dimension is then divided into four targets, each including four indicators. A semi-quantitative questionnaire enables the scoring of each indicator, with four levels according to the degree of satisfaction in the studied OH surveillance system. The evaluation is conducted by a panel of surveillance representatives (during a half-day workshop or with a back-and-forth process to reach a consensus). An R Shiny-based web application facilitates implementation of the evaluation and visualization of the results, and includes a benchmarking option. The tool was piloted on several foodborne hazards (i.e., Salmonella, Campylobacter, Listeria), emerging threats (e.g., antimicrobial resistance) and other zoonotic hazards (psittacosis) in multiple European countries in 2022. These case studies showed that the OH-EpiCap tool supports the tracing of strengths and weaknesses in epidemiological capacities and the identification of concrete and direct actions to improve collaborative activities at all steps of surveillance. It appears complementary to the existing EU-LabCap tool, designed to assess the capacity and capability of European microbiology laboratories. In addition, it provides opportunity to reinforce trust between surveillance stakeholders from across the system and to build a good foundation for a professional network for further collaboration.

1. Introduction

In recent years, the One Health (OH) concept has gained momentum, and international efforts have been made to strengthen the implementation of multi-sectoral surveillance to more effectively manage health hazards at the human, animal and environment interface (1). For decades, the Food and Agriculture Organization of the United Nations (FAO), the World Health Organization (WHO), and the World Organization for Animal Health (WOAH, formerly OIE), have been working together to address risks at the human–animal interface. In 2022, the United Nations Environment Programme (UNEP) joined the tripartite collaboration as an equal partner. The first joint plan signed by the quadripartite aims to create a framework to integrate systems and capacity to collectively better prevent, predict, detect, and respond to health threats of humans, animals, plants, and the environment with the objectives of strengthening OH surveillance, early warning and response systems (2).

OH surveillance is defined as a collaborative and systematic collection, validation, analysis, interpretation of data, and dissemination of information collected on humans, animals, and the environment to inform decisions for more effective evidence-based health interventions (3, 4). However, in spite of the efforts of the quadripartite alliance to promote collaboration in surveillance and laboratory networks and overpass professional silos, most surveillance systems remain compartmentalized, with limited interaction across actors in the system (5). For multiple reasons, implementing OH approaches in practice still proves challenging (6) and collaborations between health sectors occur mostly in crisis times (7).

There is a wide range of possible organizational models for collaboration, and its operationalization varies in terms of areas of implementation throughout the surveillance process (8–11). Collaboration is mainly driven by the epidemiological context and surveillance objective and is built according to actors' expectations (5). Regular evaluation of the organization and functionality of collaboration is crucial to assess the surveillance system's capacity and capability to produce relevant information, identify areas for improvement, and optimize added value gained by integrating efforts across sectors.

In recent years, several methods have been developed to assess whether collaborative efforts are appropriate and functional and whether it improves the impact of surveillance systems (12, 13). The Evaluation of Collaboration for Surveillance (ECoSur) tool targets the organization and functioning of multi-sectoral collaborations in a surveillance system (5). It relies on a semi-quantitative approach, with data collection based on interviews of the coordinators of the programs included in the surveillance system, requiring a 1–2-week evaluation period on average (5). The Network for Evaluation of One Health (NEOH) relies on the theory of change to identify the necessary preconditions and actions to be taken to reach long-term goals (14). The whole process is estimated to take 1–2 months and requires interviews of essential actors and stakeholders (13). The OH Assessment for Planning and Performance (OH-APP) focuses on multi-sectoral coordination mechanisms to inform planning and development assistance. The OH-APP complements the WHO Joint External Evaluation by providing specific indicators to measure the maturity of a multi-sectoral coordination mechanism and benchmark its progress toward a sustainable mechanism capable of coordinating multi-sectoral and multi-stakeholder collaboration for preparedness and response to public health threats (https://www.onehealthapp.org/about). Other tools were developed specifically for antimicrobial resistance (AMR) surveillance activities: the Progressive Management Pathway tool for AMR (PMP-AMR), the AMR integrated surveillance system evaluation project (ISSEP) tool, the Assessment Tool for Laboratories and AMR Surveillance Systems (ATLASS) (13) and the Integrated Surveillance System Evaluation (ISSE) framework (2). The different tools appear complementary in terms of evaluation objectives and provide generic science-based guidance for the evaluation of collaboration in surveillance systems. Yet, they also appear quite complex and require a lot of data, time, and human resources (13), limiting their (regular) implementation. There is therefore a need for a user-friendly tool to assess epidemiological surveillance interoperability and capacity across countries, with an aim to be repeatable.

The OH European Joint Programme MATRIX project aimed to produce guidelines and tools applicable at the national level to connect existing surveillance structures and resources, and strengthen integrated surveillance initiatives, ultimately adding value by building on existing resources, and creating synergies among sectors. In this context, we developed a generic evaluation and benchmarking tool (OH-EpiCap), implemented through an interactive online web application, for characterizing, monitoring, and evaluating epidemiological national surveillance capacities and capabilities for OH surveillance. This tool was designed to enable representatives of any surveillance system to conduct an evaluation of the multiple aspects of OH surveillance, in a short time and without requiring an external evaluation team. The evaluation addresses the multisectoral and multidisciplinary efforts to ensure communication, collaboration, and coordination among all relevant actors of the surveillance working locally, nationally, and globally to attain optimal health for people, animals, and our environment (https://extranet.who.int/sph/one-health-operations). Besides identifying areas that could lead to improvements in existing OH epidemiological surveillance capacities, the tool was designed to allow benchmarking (i.e., comparisons) with results from previous evaluations of that surveillance system, or other relevant systems, for example in other countries.

2. Methods

2.1. Identification, definition, and validation of indicators

Existing evaluation tools focusing on multi-sectoral and interdisciplinary collaboration aspects in epidemiological surveillance were used as a basis for the development of the OH-EpiCap tool. Besides, to structure our tool, we considered the format of the EU-LabCap tool, developed to assess bi-annually the capacity and capabilities of European microbiology laboratories (15).

Three dimensions of evaluation were considered in our tool: the organization of the collaborative system, the nature and functioning of collaborations for operational activities, and the impact of collaborations on surveillance (Figure 1). Each dimension was then divided into four targets focusing on specific features of multi-sectoral collaborations, building from the existing evaluation frameworks. Finally, we established standardized indicators defining more accurately each target and we singled out the necessary criteria to support their evaluation. The definition of indicators in each target is available in Hénaux et al. (16).

FIGURE 1

Figure 1. Structural overview of the OH-EpiCap targets, grouped by dimension.

The first dimension, about the organization of the OH surveillance system, includes the following targets and indicators: Target 1.1 Formalization focuses on the common aim of the system, support documentations, coordination roles, and leadership in the OH surveillance system; Target 1.2 Coverage and transdisciplinary addresses whether the surveillance covers all relevant sectors, disciplines, actors, geography, populations and hazards; Target 1.3 Resources addresses aspects related to financial and human resources, sharing of the available operational resources, and training; and Target 1.4 Evaluation and resilience focuses on internal and external evaluations, implementation of corrective measures, and the capacity of the OH surveillance system to adapt to changes.

The second dimension deals with OH aspects in operational activities: Target 2.1 Data collection and methods sharing concerns the level of multi-sectoral collaboration in the design of surveillance protocols, data collection, harmonization of laboratory techniques and data warehousing; Target 2.2 Data sharing addresses data sharing agreements, evaluation of data quality, use of shared data, and the compliance of data with the FAIR principle; Target 2.3 Data analysis and interpretation addresses multi-sectoral integration for data analysis, sharing of statistical analysis techniques, sharing of scientific expertise, and harmonization of indicators; and Target 2.4 Communication focuses on both internal and external communication processes, dissemination to decision-makers, and information sharing in case of suspicion.

The third dimension deals with the impact of the OH surveillance system: Target 3.1 Technical outputs concerns the timely detection of emergence, knowledge improvement on hazard epidemiological situations, increased effectiveness of surveillance, and reduction of operational costs; Target 3.2 Collaborative added value addresses strengthening of the OH team and network, international collaboration and common strategy (road map) design; Target 3.3 Immediate and intermediate outcomes addresses advocacy, awareness, preparedness and interventions based on the information generated by the OH surveillance system; and Target 3.4 Ultimate outcomes focuses on research opportunities, policy changes, behavioral changes and better health outcomes that are attributed to the OH surveillance system.

The organization and definition of the targets and indicators were consolidated and validated through expert consultation. Experts were selected based on previous and ongoing involvement in research activities on OH aspects (e.g., One Health—European Joint Project (OH-EJP) program; Convergence in evaluation frameworks for integrated surveillance of AMR (CoEvalAMR) project) in national veterinary, public and/or environmental health institutes and from EFSA. The experts were asked to review and comment on all the proposed indicators and identify missing information. The list of indicators was refined based on experts' comments and validated with them through a back and forth process. Additional specific modifications were also carried out based on feedback from participants in case studies during the pilot phase (see below).

2.2. Questionnaire and semi-quantitative scoring options

A questionnaire was developed to facilitate the collection of information for the scoring of the indicators, with one question per indicator. A semi-quantitative scale was defined with four levels, describing the level of compliance of the system under examination compared to an ideal situation: higher values suggest better adherence to the OH principle targeted by the indicator (i.e., better integration of sectors) and lower values indicate improvements may be beneficial. In addition, the option of “Not applicable” (NA) was included to take into consideration the case where the indicator would not be relevant to the OH surveillance system under evaluation. The standardized scoring guide, detailing for each individual score, the situation in which that score should be awarded, is available in Hénaux et al. (16).

2.3. Data visualization and web application

A web application was developed (using R shiny and shinydashboard packages) (17, 18) with a user guide describing the different steps for completing the questionnaire and visualizing the results (16). The link to the application is: https://freddietafreeth.shinyapps.io/OH-EpiCap/. The interface enables users to complete the questionnaire interactively (and also to upload the answers from a questionnaire completed previously). Below each question, free text space is provided to add notes or justify the answer provided. These comments are saved and can be also visualized when reviewing the results of the evaluation. The application allows the user to save partially completed questionnaires in csv (human-readable) format, to revisit or complete the answers at a later time. To comply with the European General Data Protection Regulation, the OH-EpiCap team does not collect any data through the application, and the application does not ask any personal or identifying information regarding users or the surveillance system under evaluation. The application is hosted in the cloud with shinyapps.io (www.shinyapps.io), and questionnaire and benchmark data (files) are processed and temporarily stored on an external server for the duration of the user's session only. All data remain inaccessible to other users of the application. Users must save their work locally (i.e., in the machine they are using) before closing the application (to avoid any data loss).

The application facilitates the exploration of the completed (and/or uploaded) assessment and of the results of the evaluation by way of multiple visualizations. The answers to the OH-EpiCap questionnaire are analyzed at the target level for each dimension by averaging the scores across the indicators to get a final score (between 1 and 4), and at the dimension level by averaging target-level outputs (the mean scores over all questions are expressed as a percentage). Results are displayed in the form of interactive radar charts and lollipop plots to identify strengths and weaknesses at both dimension and target levels. Users may hover over data points to explore the breakdown of scores for each target and indicator. At the target level, this option displays for each data point the comments provided by the evaluators during the filling of the related question. Finally, users can download a two-page report (in html format) comprising the graphic outputs and comments highlighting the main strengths and weaknesses of the surveillance system examined. Moreover, the tool also includes a benchmarking functionality to compare results from the ongoing evaluation with a reference set based on results from previous OH-EpiCap evaluations. This reference dataset can be generated from other evaluations that the user has access to, using a specific tab of the web application. This function allows the integration of multiple evaluations (for example, from other countries for the same hazard), thus anonymizing the results for a given system/hazard.

2.4. Process to conduct a OH-EpiCap evaluation

The OH-EpiCap tool was designed to serve as a support for discussion and scoring of the OH aspects by a panel of representatives from the different sectors across the entire surveillance system of a specific hazard. We recommend to identify up to 8–10 participants who have a good knowledge of the system and encompass a range of disciplines and experiences regarding the functioning of collaborations among institutes and programs.

The selected surveillance representatives form an evaluation panel, which gathers during a 4-h workshop to complete the questionnaire, using the online application. For each question, the panel must provide one answer after reaching a consensus. In the case where it is not possible to organize a workshop to conduct the evaluation, the questionnaire may be filled sequentially by the surveillance representatives from each sector, with a back-and-forth process to reach a consensus. Once completed, the online application allows the panel to visualize the outcomes in real-time and to generate a OH-ness profile for the studied system.

2.5. Pilot phase

The OH-EpiCap tool was piloted through eight applications on surveillance systems of specific hazards targeted by the MATRIX consortium, including foodborne and other emerging zoonotic hazards. As a first step, for each surveillance system, a representative was identified directly within the MATRIX participants or their professional networks. Then, a 1-h meeting with the identified surveillance representatives was organized to present the tool and the evaluation process, and to answer questions. Participants were then asked to identify additional surveillance representatives to include in the evaluation panel. When available, a map of the targeted surveillance system (characterizing the institutes involved in the surveillance programs) was used to identify potential representatives. The choice of conducting a workshop or completing the questionnaire sequentially by representatives was left to the participants.

For three study cases, the evaluator panel chose to conduct the evaluation of their surveillance system through a workshop. These study cases focused on:

• Psittacosis surveillance system in Denmark: the workshop was held in person, and gathered seven surveillance representatives, from the public health sector with expertise in laboratory/bacteriology and epidemiology, and from the animal health sector from the official sampling, laboratory, and risk management unit. It lasted 3 h (including a round table of participants and a short introduction to the workshop, the filling of the three dimensions of the questionnaire, the results analysis, and debriefing).

• Salmonella surveillance system in Germany: the workshop was held online and gathered ten representatives, from the public health (Robert Koch Institute), animal health (Friedrich Loeffler Institute), and food safety (German Federal Institute for Risk Assessment—BfR) sectors. It lasted 4 h; the two last targets of the third dimension were not completed during the workshop because of time constraints (and scoring for these indicators was provided at a later stage).

• Campylobacter surveillance system in Sweden: the workshop was held online and gathered five representatives from the public health (Folkhalsomyndigheten), animal health (National Veterinary Institute—SVA), and food safety (Swedish National Food Agency—SLV; Swedish Board of Agriculture) sectors. It lasted 3 h.

These OH-EpiCap evaluations were conducted in the language of the country to facilitate discussions. One or two persons from the MATRIX research team also participated as observers, to identify areas for improvement in the questionnaire and the online application, and to provide additional explanations if needed during the completion of the questionnaire by participants. At the end of the workshop, participants were asked to share their thoughts on the evaluation process, the relevance of the evaluation, and any feedback and comments to improve the tool. A checklist was provided for collecting this information regarding the questionnaire and its implementation (Supplementary material S1).

Other study cases were conducted through completion of the questionnaire (in a Word format), by one to four representatives of the surveillance systems, sequentially:

• AMR surveillance system in Portugal: the questionnaire was completed on the one hand by two representatives from the public health sector (Directorate General for Health—DGS and national health institute—INSA) and on the other hand by an expert from the animal health sector (National Institute for Agricultural and Veterinary Research—INIAV). Subsequently, a representative from the environmental health sector (Portuguese Environment Agency—APA), reviewed and commented the scores proposed by the other representatives.

• AMR surveillance system in France: the questionnaire was completed sequentially by one representative from the animal health and food safety sectors (National Agency for Food, Environmental and Occupational Health Safety—ANSES), and two representatives from the public health sector (Directorate General for Food—DGAL, and the national public health agency—SpF).

• Salmonella surveillance system in France: the questionnaire was completed by a representative from the public health and food safety sectors (ANSES), who is part of the coordination team at the national level.

• Listeria and Salmonella surveillance systems in the Netherlands: these two evaluations were conducted by two representatives from the National Institute for Public Health and the Environment (RIVM), who have a good knowledge of the surveillance across animal health, public health and food safety sectors and existing multi-sectoral collaborations.

Each evaluator spent between 2 and 3 h completing the questionnaire or reviewing and completing a pre-filled questionnaire. Then, the OH-EpiCap team filled the scores in the web application to generate the final report (displaying the results), that was sent back to the surveillance representatives.

2.6. Ethical approval

The MATRIX project obtained ethical approval from the ethical advisors of the One Health European Joint Programme. We informed verbally and through email the participants about the following points: (1) the use of the OH-EpiCap tool and application is voluntary; (2) the OH-EpiCap tool does not collect personal information, to comply with the European General Data Protection Regulation; (3) the web application does not keep the data regarding the OH surveillance system evaluated.

3. Results

3.1. OH-EpiCap report displaying results

Once the questionnaire is completed interactively (i.e., through the R-Shiny application), results from the evaluation are visually summarized. For confidential reasons, specific results and conclusions from the eight study cases are not presented. An example report, generated by the R-Shiny application using simulated data, is provided in Supplementary material S2.

Results are first presented through a radar chart showing average score across the indicators for each target, within the three dimensions (Figure 2), and a lollipop plot (Supplementary material S2) to identify strengths and weaknesses at both dimension and target levels. The graphs are accompanied by a short text, listing the targets demonstrating good adherence to One Health principles and the ones that would most benefit from improvement. Then, the OH-EpiCap report details the results per indicator within each target for each dimension: Organization (Figure 3), Operational activities (Figure 4), and Impact (Figure 5).

FIGURE 2

Figure 2. Example of OH-EpiCap results analyzed at the target level for each dimension (by averaging the scores across the indicators).

FIGURE 3

Figure 3. Example of OH-EpiCap results at the indicator level for each target of dimension 1 (Organization).

FIGURE 4

Figure 4. Example of OH-EpiCap results at the indicator level for each target of dimension 2 (Operations).

FIGURE 5

Figure 5. Example of OH-EpiCap results at the indicator level for each target of dimension 3 (Impact).

3.2. Questions and comments regarding the application of the OH-EpiCap tool

We detailed below the comments and questions raised by surveillance representatives during the meeting of preparation of the evaluation, and during and after the realization of the study cases. A first comment concerned whether the surveillance system targeted for the evaluation could be considered as a OH surveillance system in spite of a lack of formalization or of applicable legislation regarding the collaborations between sectors. We specified that the OH-EpiCap tool was developed for any surveillance system where some collaborations between sectors exist (at any step of the surveillance) even if those ones are not formalized or occur occasionally.

Another comment questioned whether integration could be considered from a system-wide perspective, including multi-sectoral collaborations but also inter-program collaborations even within the same sector (e.g., collaborations between a surveillance program targeting AMR and another one on antimicrobial use, in a specific sector). Although this vision appears different from the OH approach, the tool allows considering different levels of integration; however, such specificity should be clearly stated and understood by all surveillance representatives before the start of the evaluation.

During the filling of the questionnaire, for some indicators, the answers proposed for a specific question appeared to not fit with the OH surveillance system under evaluation or the epidemiological context. When such a comment occurred, we discussed it with the panel of evaluators to determine why the set of situations proposed for a specific indicator did not fit the system under evaluation. The feedback from the evaluators helped refine and complete the answers proposed for some indicators to consider specific OH surveillance contexts and situations not envisaged initially. In addition, the “NA” answer was added to all questions to be used if the question is not relevant for the OH surveillance system under evaluation. Overall, the NA option was selected few times by evaluation panels (between zero and four times among the eight study cases). We also suggested that if the answers proposed for a question did not fit the OH surveillance system under evaluation, the panel could define what would be the ideal situation and score the question accordingly by comparing the current situation to the ideal one. In this case, the panel can specify in the free comment space which alternative answers were considered (this would be useful for further result interpretation and dissemination).

Another question dealt with the amount of data saved in the web application and whether the data is accessible by stakeholders not involved in the evaluation, arguing that some information could be potentially confidential. We made it clear that the web application does not keep any data to comply with the European General Data Protection Regulation. Users must save their work locally (i.e., in the machine they are using) before closing the application (to avoid any data loss). They can also use the options offered by the web application to share the OH-EpiCap results with other stakeholders (as row data in csv format or through a final report in html format).

The last comment underlined the need for more time to further discuss and plan the actions to be taken to improve identified weaknesses. Participants are encouraged to further discuss and investigate underlying issues to improve collaboration in the system during another dedicated workshop.

4. Discussion

We present in this paper the design and the pilot study of the OH-EpiCap tool, which is a semi-quantitative evaluation tool developed for macro analysis of the OH capacities and capabilities of a system for surveillance of a specific hazard. This tool helps, without a priori consideration, characterize how multi-sectoral collaborations operate within surveillance systems. It facilitates the identification of strengths and weaknesses, focusing on the organization and functioning of existing collaborations, and of their impacts on the effectiveness of surveillance. The specific results of the evaluations regarding the strengths and weaknesses of the evaluated surveillance systems will be the topic of another paper.

The OH-EpiCap tool is generic and can be applied to the surveillance of any hazard. Accordingly, the tool was applied to a large range of hazards, including food-borne hazards (Salmonella, Listeria, and Campylobacter), other zoonotic hazards (psittacosis) and AMR. The questionnaire includes specific indicators oriented toward OH preparedness and response and is therefore of interest for surveillance systems targeting emerging or exotic zoonoses. The expert consultation and the pilot phase were beneficial to make the questionnaire more flexible to the diversity of contexts of surveillance, depending on hazards and countries, and to the level of integration of the system. Given that the tool is generic, the importance of clearly specifying the outline of the system under study and the levels of integration considered (e.g., inter-program collaborations), in addition to multi-sectoral integration, is a priority. We encourage the application of the OH-EpiCap tool to other hazards at the human-animal-plant-environment interface, in diverse contexts regarding technical infrastructure, surveillance capacity, and policy support.

Besides, the tool can address any surveillance system, whether it is well-formalized or at a low level of integration, as long as some multi-sectoral collaborations exist at any step of the surveillance even if they are not supported by official regulations, nor formalized through specific agreements and procedures. The formalization of the organization and functioning of the collaborations between sectors is considered an important aspect for OH surveillance (11), and therefore a lack of formalization will lead to low scores in some indicators of the OH-EpiCap tool (in particular in dimension 1). Depending on the aim of the OH surveillance system and if this lack of formalization is considered as an issue, surveillance representatives are encouraged to determine what elements would elevate the current multi-sectoral collaboration level to an official OH surveillance system.

The first step of a OH-EpiCap evaluation process is the identification of the panel of representatives of the surveillance system under study, i.e., who will conduct the evaluation. The composition of the evaluation team must be representative of the whole surveillance system (as much as possible). Thus, the panel should include experts from all sectors involved in the surveillance of the hazard under evaluation, and would encompass a large range of disciplines and experiences regarding the functioning of collaborations among institutes and programs. During the pilot phase, the experts who formed the panel for the OH-EpiCap evaluations encompassed several, if not all, sectors relevant for the surveillance systems, including the public health, animal health, food safety, and environmental health sectors, which aligns with the checklist for one health epidemiological reporting of evidence (COHERE) standards (19). We note that for most hazards evaluated in the pilot phase, the environmental sector is still poorly or not included in the surveillance programs, which represents a challenge to identify an environmental health representative for the evaluation panels. Yet, when relevant for the hazard evaluated, the environment, non-domestic animal, plant, and ecosystem health should also be considered in the scoring of the indicators. A mapping of the surveillance system under study, characterizing the programs and institutes involved in the surveillance for each sector and collected data (20), would help identify surveillance representatives. This panel will then work closely together during the evaluation workshop, with ideally all representatives having the opportunity to express their views during the scoring of the indicators. Therefore, identifying respected and well-known members of the surveillance system under study is an asset to moderate respectful discussion and prevent any stronger opinions from monopolizing the exchanges over the quieter contributors.

The second step consists in the evaluation of the OH epidemiological capacities and capabilities following the three dimensions, through the web app. The evaluation is based on a semi-quantitative method; this is certainly marked by subjectivity, especially in the case of a limited panel of evaluators. Indeed, some indicators might be scored very differently across surveillance representatives with various backgrounds, perceptions, and expectations. Yet, we stress that only one answer can be provided to each question; therefore the surveillance representatives of the evaluation panel must reach a consensus to answer each question (based on their backgrounds, perceptions, and expectations). This constraint of having to reach a consensus for each question, within a standardized set of answers, limits the bias of subjectivity. Another limitation of this tool is that the current implementation assumes that all indicators are of equal importance (i.e., have the same weight). This is obviously a simplification and depending on the context of surveillance and the overall aim of the collaborations among sectors, some aspects of the evaluation may appear more important and should therefore get more focus during the result analysis and interpretation, as well as for prioritizing recommendations.

The organization of the evaluation in three distinct parts (one per dimension) helps the panel to articulate its reflection regarding the OH-ness of their surveillance system. It supports a collective and transparent evaluation approach, and facilitates identification of weaknesses and alternatives. Recommendations and concrete actions to improve the global systems can emerge from this process, facilitating in a second step prioritization among actions to improve OH-ness. The user-friendly web app provides a set of classical graphs (gauges, radar charts, lollipop plots) that enables users to easily visualize and analyze the strengths and weaknesses at the level of the indicators, and also of each target within the three dimensions. We underline the importance of taking careful notes during the workshop. Justifications provided by the panel in the comment spaces during questionnaire completion are displayed on the graph, facilitating the interpretation of the results at the end of the evaluation workshop, and also at a later stage as needed (thanks to the options to upload previously saved questionnaire answers in the web application). A careful documentation of how the questions were interpreted and answered is also recommended to follow changes in the monitoring system over time, through new evaluations by the same panel or by another panel of evaluators.

The pilot study showed that securing a half-day window for the workshop is needed to conduct the evaluation, generate a report, and analyze the results. However, we stress that further discussions regarding prioritization and planning of actions to improve identified weaknesses, should be scheduled at another time. Based on the evaluations conducted, we observed that the tool provides a manageable “first step for action” where there is an interest in upgrading or renewing existing collaborations across surveillance systems. The OH-EpiCap tool provides a macroscopic analysis of the overall organization, functioning and impact of multi-sectoral collaborations. In some cases, it may be relevant to complement the OH-EpiCap approach with a more thorough evaluation of the weaker OH aspects, using evaluation tools dedicated to the functioning and performance of surveillance (21) and/or OH aspects (13). Besides, the OH-EpiCap tool does not assess OH capacities related to laboratory activities; we recommend to consider applying the OH-LabCap tool (developed within the OH-HARMONY-CAP; https://onehealthejp.eu/jip-oh-harmony-cap/) for such aspects.

One important point highlighted by the evaluators is the simplicity of application of the tool, with limited time and human resources, without hindering the quality of the results. Indeed, the evaluation can be conducted through a half-day (3–4 h) workshop, and we suggest limiting the evaluator panel to a maximum of ten representatives. This aligns with recommendations in the literature regarding the sufficient number of representatives (or key informants) to obtain robust information about the evaluated system (22–25). When the evaluation cannot be conducted through a workshop, an evaluation by several experts sequentially or using a Delphi-like approach (i.e., each representative completes the questionnaire, then a facilitator collates and summarizes all responses, and provides the summary back to the participants for cross-checking/validation) can be alternative options to conduct the evaluation (26). These approaches do not enable surveillance representatives to share their views and experiences regarding OH surveillance, in contrast to a roundtable discussion. Therefore, such approaches should be preferred in situations where an evaluation would be requested by policymakers within a short delay, for example during surveys assessing the OH epidemiological capacities of EU countries for a specific hazard, or within a country for a large range of related hazards. As such, the tool will be very complementary to the existing EU-LabCap tool, designed to assess the capacity and capability of European microbiology laboratories (15). We emphasize that the benchmarking module of the OH-EpiCap web app enables each country to compare their results to a reference set that could be generated by the policymakers using a compilation of evaluation results for the same hazard from other countries, or for other hazards from the same country, depending on the context.

5. Conclusion

OH-EpiCap is a generic (i.e., applicable to multi-sectoral surveillance systems of any hazard), interactive (facilitating and supporting discussions among stakeholders from diverse sectors and disciplines), and standalone (thanks to the user-friendly web application) tool developed to conduct macro-level evaluation of epidemiological national capacities and capabilities for OH surveillance. It supports the diagnostic of strengths and weaknesses in multi-sectoral collaborations and helps to identify concrete and direct actions to improve collaborative activities at all steps of surveillance. Besides, this evaluation framework strengthens trust between stakeholders across the systems, building a foundation for professional networks, acculturation to practices in other health sectors and disciplines, and long-term collaborations.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

VH and JP implemented the study. LC, HT, VH, RL, GC, and JP developed the tool. CB, FF, ET, and JP developed the web application. HT, JR, and VH conducted the pilot of the tool. All authors participated in drafting the manuscript and approved the final version.

Funding

This work was supported by funding from the European Union's Horizon 2020 Research and Innovation programme under grant agreement No. 773830: One Health European Joint Programme. Funding sources did not affect the design of this study, data collection, data analysis, decisions on publication, or preparation of the manuscript.

Acknowledgments

We thank the OH-EJP MATRIX consortium, experts from other OH-EJP projects, the CoEvalAMR project, and the European Food Safety Authority (EFSA) and the participants to the OH-EpiCap evaluations for their feedback and support to consolidate and validate the organization of the tool and the definition of the targets and indicators.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2023.1053986/full#supplementary-material

References

1. Johnson I, Hansen A, Bi P. The challenges of implementing an integrated One Health surveillance system in Australia. Zoonoses Public Health. (2018) 65:e229–36. doi: 10.1111/zph.12433

PubMed Abstract | CrossRef Full Text | Google Scholar

2. One Health Joint Plan of Action 2022–2026. FAO; UNEP; WHO; World Organisation for Animal Health (WOAH) (founded as OIE) (2022). Available online at: http://www.fao.org/documents/card/en/c/cc2289en (accessed October 28, 2022).

3. Aenishaenslin C, Häsler B, Ravel A, Parmley EJ, Mediouni S, Bennani H, et al. Evaluating the integration of One Health in surveillance systems for antimicrobial use and resistance: a conceptual framework. Front Vet Sci. (2021) 8:611931. doi: 10.3389/fvets.2021.611931

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Stärk KDC, Arroyo Kuribreña M, Dauphin G, Vokaty S, Ward MP, Wieland B, et al. One Health surveillance – more than a buzz word? Prev Vet Med. (2015) 120:124–30. doi: 10.1016/j.prevetmed.2015.01.019

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Bordier M, Delavenne C, Nguyen DTT, Goutard FL, Hendrikx P. One Health surveillance: a matrix to evaluate multisectoral collaboration. Front Vet Sci. (2019) 6:109. doi: 10.3389/fvets.2019.00109

PubMed Abstract | CrossRef Full Text | Google Scholar

6. PREDICT Consortium, Kelly TR, Machalaba C, Karesh WB, Crook PZ, Gilardi K, et al. Implementing One Health approaches to confront emerging and re-emerging zoonotic disease threats: lessons from PREDICT. One Health Outlook. (2020) 2:1. doi: 10.1186/s42522-019-0007-9

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Mackenzie JS, McKinnon M, Jeggo M. One Health: from concept to practice. In:Yamada A, Kahn LH, Kaplan B, Monath TP, Woodall J, Conti L, , editors. Confronting Emerging Zoonoses. Tokyo: Springer Japan (2014). p. 163–89. Available online at: http://link.springer.com/10.1007/978-4-431-55120-1_8 (accessed September 23, 2022).

Google Scholar

8. Dente MG, Riccardo F, Nacca G, Ranghiasci A, Manuguerra JC, Escadafal C, et al. Strengthening integrated surveillance for arboviruses in the Mediterranean and Black Sea regions in the framework of the One Health approach. Quad Della Soc Ital Di Med Trop E Salut Glob. (2016) 1:41–8.

Google Scholar

9. Hattendorf J, Bardosh KL, Zinsstag J. One Health and its practical implications for surveillance of endemic zoonotic diseases in resource limited settings. Acta Trop. (2017) 165:268-73. doi: 10.1016/j.actatropica.2016.10.009

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Babo Martins S, Rushton J, Stärk KDC. Economics of zoonoses surveillance in a ‘One Health' context: an assessment of Campylobacter surveillance in Switzerland. Epidemiol Infect. (2017) 145:1148–58. doi: 10.1017/S0950268816003320

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Bordier M, Uea-Anuwong T, Binot A, Hendrikx P, Goutard FL. Characteristics of One Health surveillance systems: a systematic literature review. Prev Vet Med. (2020) 181:104560. doi: 10.1016/j.prevetmed.2018.10.005

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Nielsen LR, Alban L, Ellis-Iversen J, Mintiens K, Sandberg M. Evaluating integrated surveillance of antimicrobial resistance: experiences from use of three evaluation tools. Clin Microbiol Infect. (2020) 26:1606–11. doi: 10.1016/j.cmi.2020.03.015

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Sandberg M, Hesp A, Aenishaenslin C, Bordier M, Bennani H, Bergwerff U, et al. Assessment of evaluation tools for integrated surveillance of antimicrobial use and resistance based on selected case studies. Front Vet Sci. (2021) 8:620998. doi: 10.3389/fvets.2021.620998

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Rüegg SR, Nielsen LR, Buttigieg SC, Santa M, Aragrande M, Canali M, et al. A systems approach to evaluate One Health initiatives. Front Vet Sci. (2018) 5:23. doi: 10.3389/fvets.2018.00023

PubMed Abstract | CrossRef Full Text | Google Scholar

15. European Centre for Disease Prevention and Control. EU Laboratory Capability Monitoring System (EULabCap): Report on 2018 Survey of EU/EEA Country Capabilities and Capacities. LU: Publications Office (2020). Available online at: https://data.europa.eu/doi/10.2900/914596 (accessed September 23, 2022).

16. Hénaux V, Henok T, Bogaardt C, Lailler R, Collineau L, Prada J. Deliverable D-JIP-MATRIX-WP4.2 OH-EpiCap tool tutorial. (2022). Available online at: https://zenodo.org/record/7006654 (accessed September 23, 2022).

17. Chang W, Cheng J, Allaire J, Xie Y, McPherson J, Dipert A, et al. Shiny: Web Application Framework for R. R Package Version, 1.7.1. (2021). Available online at: https://CRAN.R-project.org/package=shiny (accessed September 12, 2022).

18. Chang W, Ribeiro BB. Shinydashboard: create dashboards with'Shiny'. R package version 0.7, 2 (2021). Available online at: https://CRAN.R-project.org/package=shinydashboard (accessed September 12, 2022).

19. Davis MF, Rankin SC, Schurer JM, Cole S, Conti L, Rabinowitz P, et al. Checklist for one health epidemiological reporting of evidence (COHERE). One Health. (2017) 4:14–21. doi: 10.1016/j.onehlt.2017.07.001

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Cito F, Amato L, Ågren E, Holmberg M. Deliverable D-JIP-MATRIX-WP2.1 Mapping of the Surveillance Chain for All Hazard Tracks, Cross-Sectorial Linkages. (2022). Available online at: https://zenodo.org/record/5970164#.ZFNrtBVBxPY (accessed September 23, 2022).

21. Calba C, Goutard FL, Hoinville L, Hendrikx P, Lindberg A, Saegerman C, et al. Surveillance systems evaluation: a systematic review of the existing approaches. BMC Public Health. (2015) 15:448. doi: 10.1186/s12889-015-1791-5

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Muellmann S, Brand T, Jürgens D, Gansefort D, Zeeb H. How many key informants are enough? Analysing the validity of the community readiness assessment. BMC Res Notes. (2021) 14:1–6. doi: 10.1186/s13104-021-05497-9

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Morse JM. The significance of saturation. Qual Health Res. (1995) 5:147–9. doi: 10.1177/104973239500500201

CrossRef Full Text | Google Scholar

24. Morse JM. Data were saturated. Qual Health Res. (2015) 25:587–8. doi: 10.1177/1049732315576699

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Kriegner S, Ottersen T, Røttingen JA, Gopinathan U. Promoting intersectoral collaboration through the evaluations of public health interventions: insights from key informants in 6 European countries. Int J Health Policy Manag. (2021) 10:67. doi: 10.34172/ijhpm.2020.19

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Delbecq AL, Van de Ven AH, Gustafson DH. Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes. Glenview, IL: Scott Foresman (1975).

PubMed Abstract | Google Scholar