One Health Evaluation of Antimicrobial Use and Resistance Surveillance: A Novel Tool for Evaluating Integrated, One Health Antimicrobial Resistance and Antimicrobial Use Surveillance Programs

We describe the development, application and utility of our novel, One Health Evaluation of Antimicrobial Use and Resistance Surveillance (OHE-AMURS) tool that we created to evaluate progress toward integrated, One Health surveillance of antimicrobial resistance (AMR) and antimicrobial use (AMU) as a complex system in Canada. We conducted a qualitative inquiry into the current state of policy and programs for integrated AMR/AMU surveillance using explicit and tacit knowledge. To assess the “messy” state of public health surveillance program development, we synthesized recommendations from previous reports by the National Collaborating Centre for Infectious Diseases and the Canadian Council of Chief Veterinary Officers; conducted an environmental scan to find all federal, provincial, and territorial AMR/AMU surveillance programs in Canada; and conducted semi-structured interviews with Canadian subject matter experts. To integrate evidence from these different sources we adapted two published tools to create a new evaluation matrix, deriving 36 components of the ideal integrated AMR/AMU surveillance system. Our two-way matrix tool allowed us to examine seven common, foundational elements of sustainable programs for each component, and assign a stage of development/sustainability ranking for each component according to the matrix definitions. Our adaptable novel tool allowed for granular and repeatable assessment of the many components of a complex surveillance system. The assessment proved robust and exacting to ensure transparency in our methods and results. The matrix allows flexible assignment of program components based on program principles, and stages can be adapted to evaluate any aspect of an AMR/AMU surveillance or other multi-faceted, multi-jurisdictional system. Future refinement should include an assessment of the scope of surveillance components.

Introduction

Monitoring progress in complex systems may require equally complex methods to capture and analyze numerous aspects of whether and how progress is achieved (1, 2). McGill et al., for example, conducted a review of methods and found 74 unique studies that described evaluations of complex systems. The authors concluded that “a range of complex systems methods can be utilized, adapted, or combined to produce different types of evaluative evidence” (3). They went on to note that methodological innovation in systems evaluation may be of particular value in generating “stronger evidence” (3).

One such complex system includes policies, practices, and resources for appropriate surveillance of antimicrobial use (AMU) and antimicrobial resistant organisms [referred to overall as antimicrobial resistance (AMR)] in human, animal, food animal production, and aquaculture. One Health, or the inseparability of the health of humans, animals, and their environment, is an important concept for surveillance that adds to this complexity (4). Assessing Canada's progress to address gaps in integrated, One Health surveillance on AMR and AMU requires interrogation of surveillance systems' objectives and data, analysis of the scope and limitations of such surveillance, and knowledge of policy developments and impediments.

We conducted a review of AMR/AMU surveillance in Canada to understand how well it has developed since reviews in 2014 in the human health realm (5) and in 2016 on animal health (6). The results of our analysis were compiled in a comprehensive report (7) and have been peer-reviewed (8). Integrated AMR/AMU surveillance across humans, animals, and the environment is highly complex and “messy” (9) because surveillance in Canada is done at a variety of sites, under different jurisdictions, with various criteria, for different purposes, and at different levels of sophistication or development. There are at least nine separate surveillance programs in Canada (not including research programs). Some, like the Canadian Integrated Program for Antimicrobial Resistance Surveillance (CIPARS) surveille aspects of human and animal health and food (10), and others, such as the Canadian Nosocomial Infection Surveillance Program (11), Canadian Fed-cattle (beef feedlot) Antimicrobial Surveillance Program (CanFASP) (12) and the Canadian Dairy Network for Antimicrobial Stewardship and Resistance (CaDNetASR) (13) are specific to only certain sectors.

In this paper, we describe our mixed methods approach to collect data on the current state of integrated One Health AMR/AMU surveillance in Canada and the process we used to find and adapt existing frameworks to analyze data gathered from documents, key informants and expert opinions effectively. In particular, we describe a novel tool we developed to assess the information gathered, by considering program scope, resource investment, and tangible progress made in the past seven years. Our purpose is to authentically report (14)—that is, to describe the narrative of our experience—on the iterative nature of the method development to suit the complex systems and allow for objective assessment (15) of AMR/AMU surveillance in Canada.

Context

Antimicrobial resistance is a pressing global threat (16). In Canada, infection and colonization rates of pathogens such as methicillin-resistant Staphylococcus aureus in communities, vancomycin-resistant enterococci in hospitals, and carbapenemase-producing organisms in both settings are increasing (17). Surveillance data for AMR in animal pathogens are mostly non-existent in Canada (8). Antimicrobial resistance surveillance in Canada is considered patchy, particularly compared with the quality of surveillance done in some European countries, and is likely insufficient to deal with the enormity of the threat AMR poses in the near future (18). According to the World Health Organization, integrated, One Health surveillance of AMR/AMU must underpin efforts to protect human, animal, and environmental health (19).

In 2014, the National Collaborating Centre for Infectious Diseases (NCCID) published a report assessing the state of integrated AMR/AMU surveillance in Canada, along with 10 recommendations to address gaps and improve stewardship, surveillance and related policy in Canada (5). Two years later, the Canadian Council of Chief Veterinary Officers (CCVO), published a report evaluating current and alternative models of AMU surveillance for veterinary antimicrobials in the areas of federal, provincial, and territorial (F/P/T) antimicrobial distribution and sales data, veterinary antimicrobial purchase, sales and prescription data, and animal owner antimicrobial purchase and administration data (6). Following a national meeting on AMR surveillance in January 2019 (18), we determined to do a complete review of the current state of surveillance in Canada and to assess progress made since the recommendations made in 2014 and 2016. Our objectives were to: (1) catalog national and provincial AMR/AMU surveillance programs currently operating in Canada and; (2) assess the current state of surveillance to evaluate what progress has been made to address the gaps identified in the earlier reports (5, 6). During this work, it became clear that a reproducible evaluation matrix was needed to transparently evaluate programs in an informative way. Here, we describe our novel tool, One Health Evaluation of Antimicrobial Use and Resistance Surveillance (OHE-AMURS), for evaluating integrated, One Health AMR/AMU surveillance in Canada.

Methods

Our project was a qualitative mixed-methods inquiry into the current state of policy and programs, drawing on explicit (codified, written evidence) and tacit (experiential) knowledge (20). The challenge we faced was to find and adapt research methods and tools to achieve both a program assessment and analysis of policies in combination for a complex One Health surveillance system.

We used three data collection methods, described in detail in the final report to the NCCID (18). Briefly, we first synthesized and updated the recommendations from the previous reports. We then conducted an environmental scan to find all federal, provincial and territorial AMR/AMU surveillance programs in Canada. Thirdly, we conducted semi-structured interviews with Canadian subject matter experts.

Developing a Novel Tool for Analysis

Analysis began with synthesizing and refining the original recommendations from the 2014 report (5) to align with the current surveillance landscape in Canada. Incorporating findings from the 2016 report (6) and information from the environmental scan and interviews, we refined eight overarching requirements for integrated AMR/AMU surveillance. Within each of the requirements, we identified components that define the structures necessary to fulfill the requirement as a whole, based on the existing surveillance programs. While two requirements had one component each, others had five or more, adding up to 36 components in all needed for comprehensive, sustained surveillance of AMR and AMU in humans and animals (Table 1).

TABLE 1

Table 1. Program Requirements and their Components for Integrated, One Health AMR/AMU surveillance in Canada.

Faced with an abundance of information in very different forms and a recognized lack of available, detailed methods and tools to evaluate surveillance programs (21), we searched for an analysis method or tool to enable us to describe the stages of program development for the requirement components systematically, and then to assess surveillance program sustainability with the granularity we required. At the time of this analysis (2019), we did not find any tools that met our specific needs for comprehensive evaluation of all integrated, One Health elements of an AMR/AMU surveillance program. In March 2021, Aenishaenslin et al. published a new Integrated Surveillance System Evaluation (ISSE) framework, which is a conceptual framework for evaluating the integration of One Health surveillance systems for AMR and AMU (22). Very useful for future evaluations, it presented a framework for developing an evaluation protocol but did not provide an explicit tool in itself, thus would not have suited our immediate purpose. In iterative searches of program evaluation literature, we found, adapted and trialed a situation analysis tool for action on AMR, developed by the World Health Organization (WHO) South East Asia Regional Office (SEARO) which addressed progress, that is a “stepwise, incremental approach of a SEARO country toward implementing the WHO Global Action Plan on AMR” (23). Parathon et al. report implementing the tool to assess progress on Indonesia's national AMR prevention and containment, using progress phases categorized as exploration and adoption; programme installation; initial implementation; full operation; and sustainable operation. We adapted the tool to assess the relative stages of program development for integrated AMR/AMU surveillance in Canada, with the slightly refined program development headings of exploration, program adoption, initial implementation, full operation, and sustainable operation.

However, in application, we found that integrated AMR/AMU surveillance programs could have significantly different stages of development across various program aspects so a way to separate discrete program aspects for individualized assessment was required. We identified a public health framework developed by Schell et al. to assess the “sustainability capacity” of public health program components (24). Schell et al. defined sustainability capacity as, “the existence of structures and processes that allow a program to leverage resources to effectively implement and maintain evidence-based policies and activities” (emphasis added). The authors' original nine domains were: political support, funding stability, partnerships, organizational capacity, program evaluation, program adaptation, communications, public health impacts, and strategic planning. We determined we did not have the necessary data or frame of reference to evaluate political support for the components under the eight surveillance program requirements, so substituted evaluation of enabling policy. We also decided not to evaluate public health impacts of AMR/AMU surveillance, in part because of a recent, parallel work assessing the socioeconomic, health systems and health outcome costs of AMR published by the Council of Canadian Academies (25). Thirdly, we did not include program evaluation as there was little information available about this element for national AMR/AMU surveillance. Thus, our modification arrived at seven program elements for our evaluation: funding, organizational capacity, partnerships, program adaptability, communication, strategic planning and enabling policy (Table 2).

TABLE 2

Table 2. Comparison of our One Health Evaluation of Antimicrobial Resistance Surveillance (OHE-AMRU) tool to existing methods that were adapted to fit our purpose.

We combined the five elements of program sustainability with the seven stages of program development (Figure 1) into a final OHE-AMURS matrix with and developed definitions for each (Table 3). This novel matrix allowed us to assess progress made toward the eight requirements for national, integrated AMR/AMU surveillance in Canada.

FIGURE 1

Figure 1. Step-wise process used to develop and use the OHE-AMURS novel tool.

TABLE 3

Table 3. The matrix for the One Health Evaluation—Antimicrobial Use and Resistance Surveillance (OHE-AMURS) tool to evaluate national, integrated antimicrobial resistance and antimicrobial use surveillance program component elements in Canada.

The common program elements for evaluation are in the left column and the rankings for stages of program development are in the top row. Criteria for every element-stage combination rank are defined. Criteria are adapted from Parathon et al. and rankings are adapted from Schell et al. F/P/T, federal/provincial/territorial.

We defined the five stages of development for each of the seven sustainability capacity elements through iterative discussions by the investigation team. We created these definitions in advance of our assessment of program components to ensure objectivity based on the data and information collected.

We assigned one of the five stages of program development to every element of a surveillance program component, based on our analysis of available literature, program reports, and information from the interviews. The investigation team made these assignments through iterative discussion, with justifications provided in a compiled table of results (8). Where we were uncertain, we had follow-up conversations with key experts who were knowledgeable about specific programs. In some cases, where our initial definitions did not allow for clear differentiation between development stages, we refined our definitions and revised any initial assessments to program elements.

Our process and the development of our new assessment tool allowed us to populate our resulting table with additional information to allow replication and interpretation.

Validation by Experts

We compiled a draft summary of our results and distributed it to key interview respondents for their review and validation between December 2019 and January 2020. The reviewers included representatives from the largest national human and animal surveillance programs, co-chairs of national AMR/AMU surveillance task forces, and the authors of the 2014 and 2016 reports. We asked our reviewers to comment on (a) whether our list of surveillance programs was complete; (b) the accuracy of our descriptions of the necessary surveillance program components under each of the requirements; and (c) the completeness and accuracy of our detailed assessments for every development stage and element of the components.

Experts' comments and recommendations were used to amend assignments and rationale for each program assessment component. In some cases, reviewers' rankings of development stage and our justifications were substantively different from those of our investigation team. We followed up with these experts by email and telephone with reviewers and key subject matter experts (February-March 2020) to make adjustments to our findings as applicable.

Final Products and the Value of the Novel Tool

We completed our final analysis of our findings across the eight requirements for integrated, One Health AMR/AMU surveillance in Canada in the summer of 2020 and incorporated additional developments up to October 12, 2020. The importance of developing this novel tool was underscored by our results, as we found that many programs had elements assessed at differing stages of development toward the various recommendations. As well as producing a comprehensive spreadsheet with all rankings and the rationale for those rankings, we created a simplified, color-coded “heat map” that allows for rapid interpretation of the overall state of AMR/AMU surveillance and facilitates comparison across the domains of the tool (7, 8). These products should allow for timely updates and replication of our evaluation.

Discussion and Conclusion

Our objective was to find a suitable method to assess the messy and complex nature of human, animal, crop and aquaculture AMR and AMU surveillance in Canada accurately and adequately, and in a way that could be understood and repeated by others. Other authors have described the challenges of finding tailored methods to evaluate processes and outcomes of complex and messy systems that are based on several sources of data and which must be contextualized (1, 26).

A systematic review of evaluation approaches for surveillance systems highlighted that there are a number for public and animal health (21). The review categorized these approaches as frameworks, guidelines, methods, and tools based on the level of instructional guidance that they provided to the user for implementation. The authors suggested that methods (what components/elements to assess and how to assess them) and tools (practical elements to conduct the assessment) are required for full utilization. However, of the 15 approaches reviewed, only three included such methods and tools for implementation, highlighting the need for practical tool development in this field. None of the reviewed approaches considered the complexity of One Health AMR and AMU surveillance, which includes components from humans, animals, and the environment.

Publications from the international CoEval-AMR Network (27), released in March 2021, parallel the processes we undertook to test and refine evaluation frameworks and tools for AMR/AMU surveillance. Aenishaenslin et al., for example, reported on their development of the ISSE framework and piloted the tool to review one Canadian AMR surveillance program, CIPARS, a sentinel surveillance program for animal and human health (22). The authors began their process with CIPARS program experts to determine an evaluation framework and then conducted the program evaluation in a stepwise fashion. Results included a determination of actions and policy changes arising from the evaluation. However, this framework would not have provided the explicit tool we required to evaluate specific surveillance recommendations and components. Our method and tool did not attempt to comment on actions or next steps for the specific AMR/AMU programs, as our focus was on the overall development and sustainability of programs to achieve effective integrated monitoring. The utility and feasibility of 12 other tools for surveillance or One Health evaluation were assessed by the CoEval-AMR Network project team since 2019 (28). These included AMR/AMU surveillance tools previously reviewed by Nielsen et al. (29) using 10 criteria to rank the utility and feasibility of the frameworks. It would be valuable to use criteria such as Nielsen's to compare the utility of our complex systems assessment tool with other evaluation tools, depending on what is being evaluated. None of these tools were comprehensive that covered all the One Health elements of an integrated AMR/AMU surveillance programs, nor did they allow for the customization of requirements and components that our tool provided.

Our novel OHE-AMURS tool adapted two other systems evaluations to include the elements for assessment with definitions of the sustainability stages. The tool allowed us to assess every component of the eight recommendations for integrated AMR/AMU surveillance with the granularity needed. There was, for example, clear utility in defining a temporal rank of “stage of development,” which also provided a built-in “road map” for what progress would look like and illuminates ultimate goals. The matrix is complex but it allowed us to conduct a nuanced assessment that was robust for iterative review. We found it was ultimately an exacting tool and a way to engage with stakeholders while ensuring transparency in our methods and in the results.

Our tool is also flexible and adaptable and can be used for repeated evaluation of the future state of AMR/AMU surveillance in Canada and other multi-jurisdiction regions. The two-way element-stage of development classification and definitions and lack of pre-determined underlying components provides utility for this tool to be applied to evaluation of other integrated, One Health AMR/AMU surveillance programs. This flexibility was evident when applied to the 36 components and eight requirements across the One Health sectors, which represents the epitome of a “messy” system (9). We did, however, require an iterative process to develop both the matrix definitions and the final assessment of components. As we worked through the evaluation, we realized that some definitions required clarification to allow for adequate discrimination between the stages of development. The review from subject matter experts allowed us to refine our assessments of development for every component. This increased our confidence that we had assessed all of the required components of the integrated AMR/AMU surveillance system with the best available information at the time.

One potential weakness is that in a few instances our evaluation criteria ranked program components highly, but in fact the components were not comprehensive by our own assessment. It became clear that our evaluation tool was missing a means to assess the scope and comprehensiveness of program elements. Some elements ranked highly when using the stage of development definitions, but ultimately were too limited in scope to be considered truly comprehensive and integrated for national AMR/AMU surveillance for Canada. We recommend that for future applications, every program element be evaluated for scope and comprehensiveness using a ranking system such as: Sufficient, Partial, or Insufficient. The criteria should be defined as part of the evolution of this novel tool for future AMR/AMU surveillance program evaluations.

Our novel OHE-AMURS tool was well-suited to provide an assessment of the current status of integrated, One Health AMR/AMU surveillance programs and can be used both for future evaluations at all levels, and for other complex systems with multiple elements that may be at different stages of development. Its flexibility and iterative nature allowed for a comprehensive evaluation of the current state in Canada. With modifications to incorporate assessments of scope and comprehensiveness, OHE-AMURS will serve as a useful tool for future reviews of the AMR/AMU surveillance infrastructure in Canada. Regular evaluations of complex national AMR/AMU surveillance programs are essential for continued momentum to improve monitoring, provide data for action, and mitigate antimicrobial resistance in Canada.

Data Availability Statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author/s.

Author Contributions

MH-B and SO conceived of the paper, wrote the drafts, and made final revisions. LS, AW, and MM-R contributed content and reviewed all drafts. All authors contributed to the article and approved the submitted version.

Funding

Production of this manuscript has been made possible through a financial contribution of the Public Health Agency of Canada for the National Collaborating Centre for Infectious Diseases.

Author Disclaimer

The views expressed herein do not necessarily represent the views of the Agency.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. McGill E, Marks D, Vanessa E, Penney T, Petticrew M, Egan M. Qualitative process evaluation from a complex systems perspective: a systematic review and framework for public health evaluators. PLoS Med. (2020) 17:e1003368. doi: 10.1371/journal.pmed.1003368

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Halley JD, Winkler DA. Classification of emergence and its relation to self-organization. Complexity. (2008) 13:10–5. doi: 10.1002/cplx.20216

CrossRef Full Text | Google Scholar

3. McGill E, Er V, Penney T, Egan M, White M, Meier P, et al. Evaluation of public health interventions from a complex systems perspective : a research methods review. Soc Sci Med. (2021) 272:113697. doi: 10.1016/j.socscimed.2021.113697

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Centers for Disease Control and Prevention. One Health Basics. One Health. CDC. (2018). Available online at: https://www.cdc.gov/onehealth/basics/index.html (accessed April 7, 2021)

5. Grant J, Saxinger L, Patrick D, Keen P. Surveillance of Antimicrobial Resistance and Antimicrobial Utilization in Canada. National Collaborating Center for Infectious Diseases (2014).

Google Scholar

6. Otto SJG, Leger D, Radke B, Anderson M, Pasma T, Nguon S, et al. Non-Human Antimicrobial Use Surveillance in Canada: Surveillance Objectives and Options. vol. 2016. Edmonton, AB: Canadian Council of Chief Veterinary Officers—Antimicrobial Use in Animal Agriculture Committee (2016).

7. Otto SJG, Miazga-Rodriguez M, Saxinger L. Progress on Integrated Antimicrobial Resistance and Antimicrobial Use Surveillance in Canada (2014-2019). Winnipeg, MB: National Collaborating Center for Infectious Diseases (2020). p. 47.

Google Scholar

8. Otto S, Miazga-Rodriguez M, Wierzbowski A, Haworth-Brockman M, Saxinger L. Integrated antimicrobial resistance and antimicrobial use surveillance in Canada: evaluation of current progress (2014-2019). Can J Public Heal Can Vet J. (in press).

Google Scholar

9. Hunter DJ. Role of politics in understanding complex, messy health systems: an essay by David J Hunter. BMJ Br Med J. (2015) 350:h1214. doi: 10.1136/bmj.h1214

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Government of Canada. Background: About the Canadian Integrated Program for Antimicrobial Resistance Surveillance (CIPARS) - Canada. Available online at: https://www.canada.ca/en/public-health/services/surveillance/canadian-integrated-program-antimicrobial-resistance-surveillance-cipars/background.html (accessed June 3, 2021).

11. IPAC. CNISP Publications. Infection Prevention and Control Canada. (2020). Available online at: https://ipac-canada.org/cnisp-publications.php (accessed July 6, 2021).

12. Hannon SJ, Brault SA, Otto SJG, Morley PS, McAllister TA, Journaler CW, et al. Feedlot cattle antimicrobial use surveillance network: a Canadian journey. Front Vet Sci. (2020) 7:596042. doi: 10.3389/fvets.2020.596042

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Sanchez J, Heider L, McClure JT, Tahlan K, Dufour S, Kelton D, et al. Canadian Dairy Network for Antimicrobial Stewardship and Resistance (CaDNetASR): Surveillance of Antimicrobial Use and Resistance to Improve Stewardship Practices and Animal Health on Dairy Farms. Dairy Research Cluster 3. Dairy Farmes of Canada (2018).

14. Conrad D, Opena J. Chapter 4: Authenticity and engagement. In: Assessment Strategies for Online Learning: Engagement and Authenticity. Athabasca: Athabasca University Press (2018).

15. Starr S. Moving from evaluation to assessment. J Med Libr Assoc. (2014) 102:227–9. doi: 10.3163/1536-5050.102.4.001

CrossRef Full Text | Google Scholar

16. World Health Organization (2014). Antimicrobial Resistance: Global Report on Surveillance. World Health Organization. Available online at: https://apps.who.int/iris/handle/10665/112642

Google Scholar

17. Government of Canada. Canadian Integrated Program for Antimicrobial Resistance Surveillance (CIPARS) 2017: Design and Methods. Guelph, ON: Public Health Agency Canada (2020). p. iii, 58.

18. National Collaborating Center for Infectious Diseases. Multi-sectoral Stakeholder Meeting on AMR/AMU Surveillance. Winnipeg, MB: National Collaborating Center for Infectious Diseases (2019).

19. World Health Organization. Global Action Plan on Antimicrobial Resistance. Geneva: World Health Organization (2015).

Google Scholar

20. Kothari AR, Bickford JJ, Edwards N, Dobbins MJ, Meyer M. Uncovering tacit knowledge: a pilot study to broaden the concept of knowledge in knowledge translation. BMC Health Serv Res. (2011) 11:198. doi: 10.1186/1472-6963-11-198

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Calba C, Goutard FL, Hoinville L, Hendrikx P, Lindberg A, Saegerman C, et al. Surveillance systems evaluation: a systematic review of the existing approaches. BMC Public Health. (2015) 15:448. doi: 10.1186/s12889-015-1791-5

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Aenishaenslin C, Häsler B, Ravel A, Parmley EJ, Mediouni S, Bennani H, et al. Evaluating the integration of one health in surveillance systems for antimicrobial use and resistance: a conceptual framework. Front Vet Sci. (2021) 8:611931. doi: 10.3389/fvets.2021.611931

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Parathon H, Kuntaman K, Widiastoety TH, Muliawan BT, Karuniawati A, Qibtiyah M, et al. Progress towards antimicrobial resistance containment and control in Indonesia. BMJ. (2017) 358:j3808. doi: 10.1136/bmj.j3808

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Schell SF, Luke DA, Schooley MW, Elliott MB, Herbers SH, Mueller NB, et al. Public health program capacity for sustainability: a new framework. Implement Sci. (2013) 8:1–9. doi: 10.1186/1748-5908-8-15

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Council of Canadian Academies. When Antibiotics Fail: The Expert Panel on the Potential Socio-Economic Impacts of Antimicrobial Resistance in Canada. Ottawa, ON: Council of Canadian Academies (2019).

Google Scholar

26. Schroeder E, Epp J, Aghdaee M, Cutler H, Sheather R, Lewis P, et al. Evaluating outcomes based care for vulnerable older people: challenges and lessons learned from a complex program evaluation. Int J Integr Care. (2020) 20:A146. doi: 10.5334/ijic.s4146

CrossRef Full Text | Google Scholar

27. Home Page. CoEvalAMR. Available online at: https://coevalamr.fp7-risksur.eu/ (accessed May 25, 2021).

28. Sandberg M, Hesp A, Aenishaenslin C, Bordier M, Bennani H, Bergwerff UB, et al. Assessment of evaluation tools for integrated surveillance of antimicrobial use and resistance based on selected case studies. Front Vet Sci. (2021) 8:663. doi: 10.3389/fvets.2021.620998

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Nielsen LR, Alban L, Ellis-Iversen J, Mintiens K, Sandberg M. Evaluating integrated surveillance of antimicrobial resistance: experiences from use of three evaluation tools. Clin Microbiol Infect. (2020) 26:1606–11. doi: 10.1016/j.cmi.2020.03.015

PubMed Abstract | CrossRef Full Text | Google Scholar