Developing an Evidence-Based Tool for Planning and Evaluating Vaccination Strategies Aimed at Improving Coverage in Elderly and At-Risk Adult Population

Background: Vaccination coverages need to be constantly maintained and improved with the implementation of vaccination strategies. This paper describes the development of an evidence-based tool to guide their planning and evaluation.

Methods: A scoping review was performed in MEDLINE and institutional websites to search for similar available tools. A first version of the tool was developed considering review results and a four-step method used for the control and continuous improvement of processes and products, namely the Deming cycle. A panel of eight experts was then involved in a Delphi study for the finalization of the tool that was eventually discussed in a face-to-face meeting.

Results: The scoping review found only one document and the first version of the tool was composed of 30 items. After the Delphi first round, 11 additional items were suggested and 5 original items amended. After the Delphi second round 41 items were eventually included. During the face-to-face meeting, 7 items were recognized as requisites for setting vaccination strategies, whereas 17 as relevant ones.

Conclusions: Current public health challenges impose the need for evidence-based tools to organize effective vaccination strategies. Our tool is a first proposal which aims to reflect this focus.

Introduction

Vaccinations are among the most relevant public health successes of the nineteenth century. They have the potential to eliminate or, at least, control an important amount of serious diseases, drastically reducing the mortality rate and complications associated with them (1). According to the latest World Health Organization (WHO) data, vaccination nowadays prevents 2–3 million deaths every year (2). In addition to reducing mortality and morbidity rates and limiting the spread of pathogens, vaccines also play an important role in the fight against Antimicrobial Resistance (3, 4).

Although the value and the benefits are recognized globally and many initiatives have been introduced to support vaccinations in recent years, in many countries a decline in vaccination coverage has been observed, with important repercussions on health and at social and economic levels (5). For example, according to the European Centre for Disease Prevention and Control (ECDC), influenza vaccination coverage among older age groups (≥65 years) in Europe (EU) during the 2015–16, 2016–17, and 2017–18 influenza seasons did not reach the EU target of 75% in any of the country that provided data and in many of them it even declined (6). In Italy, in particular, influenza vaccination coverage in elderly showed a considerably and steadily declining trend from 68.3% in the 2005–06 season to 48.6% in the 2014–15 season, turning to increase from that moment on, reaching 54.6% in the last season (2019–20) (7, 8). The same declining trend was registered for Italian vaccination coverage of Polio and Measles-Mumps-Rubella (MMR), especially in 2015 for MMR (85.3%) and in 2016 for Polio (93.3%) causing the political authorities to take measures to tackle the problem (9). The declining trend in vaccination coverages is one of the reasons why epidemics and deaths due to vaccine-preventable diseases (VPDs) still occurs also in developed countries.

The general decline in vaccination coverage can be partly attributable to the so-called vaccine hesitancy, which is known as a delay in acceptance or refusal of vaccines despite the availability of vaccine services (10). Nowadays, vaccine hesitancy represents a real threat to the health and well-being of citizens and communities, influencing the effectiveness of immunization programs all over the world. The determinants of vaccine hesitancy are various and complex and different models have been used over the time to describe the problem. The WHO Working Group on vaccine hesitancy proposed the so-called Complacency, Convenience and Confidence (3C) model. According to it, a possible reason of vaccine hesitancy might be linked to the systems that deliver the vaccination, including the competence and reliability of the own service. In fact, the decision to vaccinate is affected by the quality of the service, no matter if real or perceived, and by the level of delivery of vaccination services in terms of time and place (10). Therefore, the organization of vaccination services could be one of the underlying causes of vaccination hesitancy. In order to overcome this problem, vaccination services should be perceived as acceptable as people are more likely to vaccinate if services are convenient, easily available and without charge, if counseling is offered, and if communication and organization aspects are reasonably arranged. A proper planning of vaccination services is therefore pivotal to ameliorate vaccination attitude (1).

Also, the European Council calls the attention on the need to bring immunization services closer to citizens (11), thus, new vaccination strategies and innovative interventions are needed to promote vaccination and counter unsatisfactory vaccination coverages. Nevertheless, to our knowledge, there is not a validated tool or an institutional guideline to plan and evaluate new vaccination strategies. This paper describes the production of an evidence-based tool that is meant to help developing and appraising vaccination strategies. The tool has been built within an Italian project entitled “Best pRactices improving Vaccination coverage among at-risk adults and Elderly (BRAVE)” whose final aim was to identify and assess strategies set up in Italy in order to promote vaccination among elderly and at-risk adult population.

Methods

The development of the evidence-based tool relied on a scoping review first and on the following application of a consensus development method, namely a Delphi. A face-to-face workshop was eventually organized in Rome on 24th September 2019 in order to discuss the final tool.

Scoping Review and Development of the First Version of the Tool

A preliminary scoping review was performed in MEDLINE to gather evidence on tools already used and validated for planning and evaluating vaccination strategies. The following search terms were used: vaccin^*, strategy, initiative^*, assessment, evaluation, appraisal, grid. The search was carried out in May 2019 by two researchers independently and looked at identifying papers that proposed items for both planning and evaluating vaccination strategies in general. Additionally, other two researchers scanned the ECDC and WHO websites and gray literature. Eventual disagreements were solved by consensus.

In order to develop a first version of the tool, the four phases of the Deming cycle (planning, implementation, check, act) (12), i.e., a method used for the control and continuous improvement of processes and products were also considered as a continuous quality improvement model.

Delphi Process

A multiprofessional panel of eight experts was involved in a 2-round Delphi survey. The Delphi survey was used to collect expert-based judgements and reach a consensus over the topic based on the assumption that a group of experts produce a more valid judgement than a single expert (13).

The experts were selected among individuals with relevant knowledge and experience of vaccination from academia (two Professors of Public Health), governmental/technical institutions (one from the Italian Ministry of Health, one from the National Institute of Health), vaccine delivery (two Directors of Public Health Departments located in tow cities of Center-Southern Italy), the public (two representatives of a non-profit citizens' organization).The two citizens' representatives participated in the survey with a single joint contribution.

The experts were contacted through email from one member of the working group who introduced the objective, the content, and the deadline of the Delphi process.

The experts were asked to express anonymously their agreement in eventually including each item of the first version of the tool through a 4-points Likert scale (“not at all,” “a little,” “quite,” “very much”). Questions were not compulsory; this choice was reasoned by the different background of experts that could have prevented them to have a clear position with respect to each item. The experts were also encouraged to propose new items and amendments to existing ones, if deemed necessary. If at least ≥70% of the experts expressed “quite” or “very much” agreement, the item was included in the final version of the tool. The cut-off of 70% was chosen arbitrarily. In the same way, if at least ≥70% of participants did not agree (“a little” or “not at all” answers) in maintaining the item, it was excluded. In the remaining cases, items were proposed again to be voted in the second round. Similarly, any amendment as well as new items proposed during the first round were submitted to experts' evaluation during the second one. In the second Delphi round, the experts voted “yes” or “no” for the inclusion of the amended and new items in the final version of the tool. As in the first round, the threshold for final inclusion was ≥70% agreement.

Face-to-Face Workshop

The final agreement regarding the items was reached in a face-to-face meeting with the experts. Eventual disagreements were solved by discussion.

Results

The scoping review did not yield any helpful evidence with the exception of a planning checklist for immunization campaign developed within the HProImmune project (the checklist is available at the following website: http://hproimmune.eu/toolkits/files/docs/uk/admin/planning_checklist.pdf). HProImmune (http://www.hproimmune.eu/) was a 3-year project co-funded by the Health Programme of the European Union and aimed to promote immunization among Health Care Workers in Europe. The checklist deals with the planning of immunization campaign and addresses several aspects that were also considered in the development of the first version of our tool, namely the need for a budget and for informational materials as well as the identification and evaluation of indicators.

The first version of the tool was composed of 30 items (Table 1): 13 among them regarded the planning, 7 the implementation, 4 the check and 4 the act phase. Additional two items, that did not pertain to any of the above-mentioned categories, were included among the category “other.”

TABLE 1

Table 1. Items of the first version of the tool and results of the first round of Delphi.

During the first round of the Delphi, there was agreement (≥70% of “quite” or “very much” answers) on all the purposed items (Table 1). All the experts did provide their answers with the exception of few items receiving four or five answers.

As a result of the first round 11 additional items were suggested (Table 2). Furthermore, 4 of the original items were amended (suggested modifications are highlighted in italics in Table 1) and 1 original item was moved from Implementation to Planning section (“Were the health professionals responsible for the initiative identified?”). During the second round all the new/amended items were accepted for the inclusion in the final tool that was eventually composed of 41 items.

TABLE 2

Table 2. New items proposed for the inclusion in the tool after the first round of Delphi.

The face-to-face workshop served as a time for further elaborating the final tool. As a result of the discussion, primary and secondary items were identified with regard to their importance in the planning and evaluation of vaccination strategies. Furthermore, 7 items were labeled as requisites (R) that should be mandatorily met to set up a vaccination strategy. Indeed, these items were rephrased from questions to statements. The final evaluation tool was therefore amended as shown in Table 3.

TABLE 3

Table 3. Final tool.

Discussion

This paper had the aim of developing, through both the review of the available evidence and the consultation of a panel of experts, a tool for planning and evaluating vaccination strategies. The developed tool was also meant to allow the control and continuous improvement of vaccination strategies and therefore covered the four steps of the Deming cycle. The final version of the tool included items that were assigned with different relevance by the experts. Some items were actually considered as requisites, namely mandatory, for the launching and implementation of a vaccination strategy while others were considered either primary or secondary. The primary elements to consider for planning a vaccination strategy should certainly consider the actors to be involved in it (health professionals and, if possible, also representatives of citizens) and the target population to whom vaccination is directed, but also the technological, structural and economic resources necessary for the realization of the initiative. Instead, an interesting aspect deemed required for an effective implementation of the vaccination strategy was the delivery of a free of charge vaccination to the target population.

Taken as a whole, requisite addressed strategies inputs and activities laying the foundations for reaching desirable outputs and outcomes, namely regulations, human and financial resources, facilities, providers' and users' education, data collection -. It is important to point out that two requisites were those concerning the final evaluation of the strategy and the reporting of its results. These two aspects are at the center of frameworks used for program evaluation defined as a “systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future program development” (14). Indeed, the evaluation of health programs gained attention over the years because of the need for public health officials and health decision makers to be accountable. Nevertheless, in order to make public health strategies successful in terms of outputs and outcomes, a set of considerations should be proposed before and our tool was exactly meant to help promoting this task. The relevance of the planning phase already shines through several documents on vaccinations that anyway adopt a national viewpoint (15) or regards specific vaccinations (16, 17). Also, the COVID-19 pandemic calls the attention on the importance to plan and implement effective vaccination strategies. In particular, the European Commission (EC) pinpointed the relevance of planning adequate infrastructures and resources to make delivery and distribution of vaccines ordered, timely and fair (18). The WHO has issued a guidance to support national deployment of COVID-19 vaccines that addresses several aspects; namely, preparedness, planning and coordination, costing and funding, identification of the target population, delivery strategies, supply chain and management of health care waste, human resource management and training, vaccine acceptance and uptake, vaccine safety monitoring and management of adverse events (19). Our tool includes many of the indications suggested by the WHO but it is meant to help both developing and appraising vaccination strategies also at a more local level as it is the latter generally entrusted to deliver vaccination according to national recommendations. A study performed in Tuscany region on the implementation of maternal vaccination against pertussis highlighted meaningful factors that should be taken in mind in the planning and implementation of maternal immunization programs (20). In particular, relevant concerns emerged with respect to accountability, logistic and financial barriers, communication and training shortcomings (20). The authors also suggested some actions to deal with these issues, such as developing “an implementation plan with clear allocation of responsibilities within and across institutions” and provide educational and informational materials and training activities (20). Most of these aspects were considered in our tool. In particular, an adequate planning of activities to inform the target population is fundamental for the implementation of vaccination strategies and could help contrasting the reasons behind vaccine hesitancy (10). In fact, a more organized strategy that also includes information/communication activities could increase vaccine confidence. Similarly, a proper planning and organization of the vaccination strategy could strengthen the perceived convenience of vaccination and counteract vaccination hesitancy.

Beside these issues, also operational concerns should be addressed, namely feasibility and data tracking. These aspects were called into question in particular with respect to maternal vaccination (21, 22) and were included in our tool.

Consequently, we believe that our tool could be a useful support also in the implementation of the Immunization Agenda 2030 (IA2030) (23) that considers the vaccination as “a key contributor to people's fundamental right to the enjoyment of the highest attainable physical and mental health” envisioning a “world where everyone, everywhere, at every age, fully benefits from vaccines to improve health and well-being.” Among the strategic priorities set by the IA2030, coverage & equity goal and life-course & integration deserve a deepening as they call, respectively, for context-specific interventions and approaches to reduce missed opportunities. These two actions, in their turn, draw the attention on the need to develop and evaluate “innovative, locally tailored, evidence-based, people-centered approaches to reach poorly served populations” and to integrate vaccinations into other primary health care services (23). In our opinion, these objectives could be achieved only through a careful planning and evaluation of new vaccination strategies to be implemented at local level.

Our paper has some limitations. The most important one is that the tool was eventually built on experts' opinions as we did not find any useful evidence to develop it. Experts were all Italian, therefore items that were included in the final tool might not be applicable in other contexts, such as low-middle income countries. Furthermore, the experts were selected in an arbitrary number of eight for pragmatic issues and did not include general practitioners (GPs). Their exclusion was linked to the different level of engagement of GPs in the delivery of vaccination across the country. As for the sample size, eight participants is considered an acceptable minimum number (24); furthermore, experts involved in our study were all knowledgeable in the field of vaccination with particular regard to immunization strategies and health policies. It is important to point out that experts panel involved referents engaged in vaccination field but with various position, from academia to health management and governance and also citizens' representatives, thus allowing to have a comprehensive overview of issues that should be considered from different perspective in planning and evaluation of vaccination strategies. Anyway, the tool could be further improved with its testing in the field at national and international level. With this respect a feasibility study on its application could be helpful.

An innovative aspect of our study is represented by the application of the Deming cycle for the development of our tool. We are in fact in the context of health planning defined as a process of organizing decisions and actions to be undertaken in the present in order to achieve particular goals in the future (25). The health planning process is divided into four phases such as: conception, planning, action and evaluation (26). In this field, the use of the Deming Cycle is considered the best practice to improve the quality of systems in healthcare (12) and, for the first time, the Deming cycle has been applied to the vaccination field, providing an important methodological basis for the creation of our tool.

In conclusion, vaccinations represent a fundamental public health intervention to be supported with every mean to prevent the entire population from the burden of infectious diseases that are potentially and easily countered. Therefore, new vaccination strategies and innovative interventions are needed to promote vaccination and counter unsatisfactory vaccination coverages. To ensure this result evidence-based tools to organize effective vaccination strategies are needed. Our tool is a first proposal aimed to support this important objective.

Data Availability Statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding authors.

Author Contributions

GEC, WR, and CdW conceptualized, designed, and supervised the study. AT, EC, SM, LS, and VC performed the data collection. AT, EC, SM, and LS analyzed the data and wrote the original draft of the manuscript that was revised and edited by GEC and CdW. All authors revised the manuscript and approved its submission for publication, and certifying that the work is original and their own. The expert Panel was involved in the definition of the grid.

Funding

The “Best pRactices improving Vaccination coverage among at-risk adults and Elderly (BRAVE)” was a project of V.I.H.T.A.L.I. supported with an unconditional grant from Seqirus S.r.l.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

BRAVE Project Expert Panel

Michele Conversano, Valeria Fava, Antonio Gaudioso, Stefania Iannazzo, Roberto Ieraci, Patrizia Laurenti, Giovanni Rezza, and Stefano Vella.

References

1. World Health Organization. Global Vaccine Action Plan 2011-2020. Geneva: WHO Press (2013).

Google Scholar

2. Immunization Coverage. World Health Organization. Available online at: https://www.who.int/news-room/fact-sheets/detail/immunization-coverage (accessed July 30, 2020).

Google Scholar

3. Vaccines Europe. Vaccines Europe Paper: The Role of Vaccination in Reducing Antimicrobial Resistance (AMR). Available online at: http://www.vaccineseurope.eu/wp-content/uploads/2016/11/VE-policy-paper-on-the-role-of-vaccines-in-reducing-AMR-2016-FIN.pdf (accessed July 30, 2020).

Google Scholar

4. World Health Organization. Global Action Plan on Antimicrobial Resistance. Draft Resolution With Amendments Resulting From Informal Consultations. Sixty-Eighth World Health Assembly. Available online at: https://apps.who.int/iris/bitstream/handle/10665/253204/A68_ACONF1Rev1-en.pdf?sequence=1&isAllowed=y (accessed July 30, 2020).

Google Scholar

5. Calabrò GE, Tognetto A, Carini E, Mancinelli S, Sarnari L, Colamesta V, et al. Strategies to improve vaccination among at-risk adults and the elderly in italy. Vaccines. (2020) 8:358. doi: 10.3390/vaccines8030358

PubMed Abstract | CrossRef Full Text | Google Scholar

6. European Centre for Disease Prevention and Control (ECDC). Seasonal Influenza Vaccination and Antiviral Use in EU/EEA Member States – Overview of Vaccine Recommendations For 2017–2018 and Vaccination Coverage Rates For 2015–2016 and 2016–2017 Influenza Seasons. (2018). Available online at: https://www.ecdc.europa.eu/sites/default/files/documents/seasonal-influenza-antiviral-use-2018.pdf (accessed July 30, 2020).

Google Scholar

7. Osservatorio Nazionale Sulla Salute Nelle Regioni Italiane. Rapporto Osservasalute 2019. Stato di salute e qualità dell'assistenza nelle regioni italiane. Roma: Osservatorio Nazionale sulla salute nelle regioni italiane (2020).

Google Scholar

8. Epicentro. Istituto Superiore di Sanità, Italy. Available online at: https://www.epicentro.iss.it/influenza/coperture-vaccinali (accessed July 30, 2020).

Google Scholar

9. Rezza G. Mandatory vaccination for infants and children: the Italian experience. Pathog Glob Health. (2019) 113:291–6. doi: 10.1080/20477724.2019.1705021

PubMed Abstract | CrossRef Full Text | Google Scholar

10. World Health Organization. Report of the Sage Working Group on Vaccine Hesitancy. (2014). Available online at: https://www.who.int/immunization/sage/meetings/2014/october/1_Report_WORKING_GROUP_vaccine_hesitancy_final.pdf (accessed July 30, 2020).

Google Scholar

11. European Commission. Proposal For a Council Recommendation on Strengthened Cooperation Against Vaccine Preventable Diseases. Available online at: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM%3A2018%3A244%3AFIN (accessed July 30, 2020).

Google Scholar

12. Taylor MJ, McNicholas C, Nicolay C, Darzi A, Bell D, Reed JE. Systematic review of the application of the plan–do–study–act method to improve quality in healthcare. BMJ Qual Saf. (2014) 23:290–8. doi: 10.1136/bmjqs-2013-001862

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Niederberger M, Spranger J. Delphi technique in health sciences: a map. Front Public Health. (2020) 8:457. doi: 10.3389/fpubh.2020.00457

PubMed Abstract | CrossRef Full Text | Google Scholar

14. U.S. Department of Health and Human Services Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta: Centers for Disease Control and Prevention (2011).

Google Scholar

15. Expanded Programme on Immunization (EPI) of the Department of Immunization Vaccines and Biologicals. WHO-UNICEF Guidelines for Comprehensive Multi-Year Planning for Immunization. Geneva: WHO (2014).

Google Scholar

16. World Health Organization. Polio global eradication initiative. Best practices for planning a vaccination campaign for an entire population. Available online at: https://www.who.int/polio-transition/documents-resources/best-practices-microplanning-entire-population.pdf?ua=1 (accessed July 30, 2020).

Google Scholar

17. Riphagen-Dalhuisen J, Frijstein G, van der Geest-Blankert N, Danhof-Pont M, de Jager H, Bos N, et al. Planning and process evaluation of a multi-faceted influenza vaccination implementation strategy for health care workers in acute health care settings. BMC Infect Dis. (2013) 13:235. doi: 10.1186/1471-2334-13-235

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Communication From the Commission to the European Parliament and the Council. Preparedness For COVID-19 Vaccination Strategies and Vaccine Deployment. Available online at: https://ec.europa.eu/health/sites/default/files/vaccination/docs/2020_strategies_deployment_en.pdf (accessed May, 2021).

Google Scholar

19. Guidance on developing a national deployment and vaccination plan for COVID-19 vaccines. Geneva: World Health Organization (2020).

Google Scholar

20. Mazzilli S, Tavoschi L, Lopalco PL. Factors affecting the implementation process of pertussis [Tdap] immunization in pregnant women in an Italian region: a qualitative study. Front Public Health. (2020) 8:120. doi: 10.3389/fpubh.2020.00120

CrossRef Full Text | Google Scholar

21. Kochhar S, Edwards KM, Ropero Alvarez AM, Moro PL, Ortiz JR. Introduction of new vaccines for immunization in pregnancy - programmatic, regulatory, safety and ethical considerations. Vaccine. (2019) 37:3267–77. doi: 10.1016/j.vaccine.2019.04.075

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Fleming JA, Munthali A, Ngwira B, Kadzandira J, Jamili-Phiri M, Ortiz JR, et al. Maternal immunization in Malawi: a mixed methods study of community perceptions, programmatic considerations, and recommendations for future planning. Vaccine. (2019) 37:4568–75. doi: 10.1016/j.vaccine.2019.06.020

PubMed Abstract | CrossRef Full Text | Google Scholar

23. World Health Organization. Immunization Agenda 2030: A Global Strategy to Leave No One Behind. Available online at: https://www.who.int/immunization/IA2030_draft_4_WHA.pdf?ua=1 (accessed July 30, 2020).

Google Scholar

24. Hallowel MR, Gambatese JA. Qualitative research: application of the Delphi method to CEM research. J Constr Eng Manag. (2010) 136:1–9. doi: 10.1061/(ASCE)CO.1943-7862.0000137

CrossRef Full Text | Google Scholar

25. Liebler JG, Levin RE, Rothman J. Management Principles For Health Professional. 2nd ed. St. Gaithersburg: Aspen Publishers (1992).

Google Scholar

26. Damiani G, Specchia ML, Bellentani M, Visca M, Ricciardi W. Principi e metodologie di base della programmazione sanitaria. In: Manuale di programmazione e organizzazione sanitaria. 3rd ed. Idelson-Gnocchi (2018). p. 49–54.

Google Scholar