Country score tool to assess readiness and guide evidence generation of immunization programs in aging adults in Europe

Objectives: Delaying of policies for immunization of aging adults, low vaccine uptake, and the lack of supportive evidence at the national level could diminish the value in health and economics of such programs. This study aims to develop a “country score tool” to assess readiness and to facilitate evidence generation for aging adult immunization programs in Europe, and examine the comprehensiveness, relevance, acceptability, and feasibility of the tool.

Methods: The tool was developed in two phases. First, a modified Delphi process was used to construct the tool. The process included a literature review, stakeholder consultations, and a three-round Delphi study. The Delphi panel included researchers, supra-national and national decision-makers of immunization programs recruited from five countries, using snowball sampling method. The consensus was predefined at the agreement rate of 70%. Pilot testing of the tool was conducted in the Netherlands, Germany, Serbia, and Hungary involving researchers in the field of health technology assessment. After assessing the countries' readiness, researchers evaluated four features, namely comprehensiveness, relevance, acceptability, and feasibility of the tool via an online survey that included 5-scale Likert questions. The percentages of affirmative answers including “agree” and “totally agree” choices were presented.

Results: The review identified 16 tools and frameworks that formed the first version of our tool with 14 items. Eight experts were involved in the Delphi panel. Through three Delphi rounds, four items were added, one was dropped, and all others were amended. The consensus was achieved on the tool with 17 items divided into decision-making and implementation parts. Each item has a guiding question, corresponding to explanations and rationales to inform assessment with readiness scores. Eight researchers completed the pilot testing. The tool was rated as comprehensive (75%), relevant (100%), acceptable (75%), and feasible (88%) by participants.

Conclusion: Through a thorough and transparent process, a country score tool was developed helping to identify strengths, weaknesses, and evidential requirements for decision-making and implementation of immunization programs of aging adults. The tool is relevant for different European contexts and shows good comprehensiveness, acceptability, and feasibility.

1. Introduction

Burden of vaccine preventable diseases (VPDs) is threatening healthcare systems not only by an increasing number of aging people (50 years and older) but also through the age-related decline of the immune system (1, 2). An extreme burden was demonstrated with the COVID-19 pandemic when a sudden increase in cases, morbidity and mortality became an urgent problem (3). Besides COVID-19, other VPDs significantly impact on aging population. The issue is exemplified by European data with ~40,000 deaths annually caused by influenza with 90% occurring among elderly (4), 73 hospitalizations per 100,000 seniors every year for community-acquired pneumococcal pneumonia (5), and almost a third of adults developing herpes zoster over lifetime (6).

As a preventive strategy, vaccination has been reported to be effective in limiting the severity, reducing morbidity and mortality of infections in aging population (7–9). Those health benefits enable them to contribute to nations' social and economic development as well as perform various valuable functions including childcare, providing financial and emotional support to families (10). Along with health and social benefits, immunization programs have been proved to be cost-effective (11–13). Despite those potential benefits in health, social and economic aspects, vaccines are underutilized in aging population characterized by delayed policy implementation and low vaccine uptakes. Less than two third of European countries had policies for pneumococcal vaccine and most countries had no vaccination policies against herpes zoster (14). Although influenza vaccination programs were in place all over Europe, only one country achieved the vaccine uptake target in 2018/2019 of 75% recommended by World Health Organization (WHO) (15, 16). During the same season, half of the countries had <35% of older people vaccinated against influenza; the lowest coverage was <1% (16). Therefore, countries need to strengthen their immunization programs for aging population by focusing on both decision-making and implementation aspects.

Required elements for both decision-making and strategies for implementation have been investigated among European countries. The most important component for making decision on vaccine introduction is evidence, including international literature and country-specific evidence (17). The former can be done by a standardized systematic approach using available supporting tools, for instance, GRADE and CAPACITI (18, 19). The latter element includes, but is not limited to, setting-specific evidence of the burden of disease and health economic evaluation. However, in many countries, there is no transparent guidance that presents various methodological requirements (17). It suggests a need for a holistic assessment to first identify the gaps in healthcare systems concerning aging adult immunization programs, and second, to identify methods for future research enabling decision-making and implementation (20). This assessment could be sufficiently defined by a concept of a “readiness assessment.” By definition, readiness assessment can help to identify the potential challenges when implementing new processes within a current organizational context and affords the opportunity to remedy these gaps either before, or as part of, the implementation plan (21).

Although the two concepts of “decision-making” and “implementation” are commonly used across publications (22–24), they are poorly defined. That could be driven from the complexity and dynamic nature of introducing vaccines into different settings. A recent study identified key features of vaccine market access pathways across the European Union (EU) and the United Kingdom. They included horizon scanning, early advice, initiation of assessment, recommendations from advisory groups for vaccine introduction and funding, final decision, National Immunization Program (NIP) inclusion, and procurement (25). Drawing on that pathway, we defined decision-making as the process of generating recommendations to include vaccines to the national immunization program and possibly reimbursed by National Immunization Technical Advisory Group (NITAG) and/or Health Technology Assessment Body (HTAB) (or equivalents); and implementation as the process of proceeding with the recommendations including resourcing, making final decisions of vaccine introduction, procurement, and sustainability.

Various published works exist that help to assess a country's readiness with regards to immunization programs; however, there are limitations in terms of target diseases, population, and transferability in different European contexts. First are readiness assessment tools for COVID-19 vaccine introduction, which might be though inapplicable for non-pandemic vaccines, given the special priority of resources for COVID-19 (26, 27). Second, a majority of guidelines and publications focus on vaccination of the general populations (28, 29), or specifically for children (30) or during pregnancy (31). Despite immunization programmes' common aspects, those of aging population have specific requirements including a need for adult vaccine working groups (22), evidence on potential gains from immunization, and relevant infrastructure to deliver immunization service to this population (20). Therefore, it is important to consider those existing guidelines as a foundation, but unique requirements for aging adults should be added to the assessment tool targeting this population. To partly fill in this gap, an “Evidence-Based Tool for Planning and Evaluating” was considered to be the first proposal up to 2021 aiming at improving coverage for the elderly by planning and monitoring vaccination strategies (32). However, the ability to support making policies and possibilities to generate evidence in immunization programs are not in scope of the tool. Additionally, the transferability of the tool to different European countries is limited when its development solely considered the Italian context.

All those gaps in the literature raise a need for a tool that can simultaneously assess countries' readiness and facilitate evidence generation to support decision-making and implementation processes of immunization programs in European countries targeting aging population. This work aims to (1) develop a country score tool, which can help to achieve those purposes and (2) to test the comprehensiveness, relevance, acceptability, and feasibility of the tool.

2. Materials and methods

The country score tool was developed in two phases (Figure 1). In phase 1, a modified Delphi process was used. This method concerns a group consensus strategy that systematically uses literature review, opinions of the research team (labeled steering group) and the judgment of experts within the research field to reach agreement (33, 34). Phase 2 involved pilot testing in four countries including the Netherlands, Germany, Hungary, and Serbia.

FIGURE 1

Figure 1. Development process of the country score tool.

2.1. Phase 1: Tool development using a modified Delphi process

2.1.1. Development of an item pool and the first version of the tool

We conducted a comprehensive literature review in August 2021 to identify existing readiness assessment tools and frameworks for immunization programs with regards to decision-making and implementation in general populations and adult populations separately. Publications that specifically focused on aging population were separated from the adult-related search. To define aging population, the cut-off of 50-year-old and over was applied as suggested in published work regarding vaccination to promote healthy aging globally (35, 36). Details of the literature review including inclusion, exclusion criteria, and search strategy are presented in Supplementary material 1. All components and sub-components of the identified tools and frameworks were extracted for the development of an item pool. COVID-19 related aspects were excluded in the development process. This feature enables the tool to be used for non-pandemic vaccination programs in aging population in the future after the COVID-19 pandemic era.

We structured the tool along the two lines of decision-making and implementation. The foundation for item generation was set up based on (1) a published systematic review of 116 studies on national decision-making for the introduction of new vaccines and (2) WHO guidance on principles and considerations for adding a vaccine to a national immunization program (28, 30). Next, items in the item pool were linked to this foundation. Similar components were grouped, and new components were added. Disease-specific or vaccine-specific items were excluded from the item bank afterwards.

The structure of the tool was generated by integrating a checklist approach and a scoring approach. As a checklist, the country readiness tool presented each item in the item pool as a question supported by definitions and rationales (31). In addition, the readiness level in each item was defined by a scoring system of three values including 0, 1, and 2 (22). No weighting system was implied in order to provide flexibility for countries when using the tool according to particular infrastructures and the national priorities.

2.1.2. Stakeholder consultation

Consultation with stakeholders occurred in December 2021-January 2022 to first determine the relevance and importance of a set of components identified in the previous step, and to identify additional components. Participants were recruited from VITAL consortium comprising 17 public partners from 11 countries in Europe together with seven biopharmaceutical companies in a European project named Vaccines and InfecTious diseases in the Aging population (37).

All participants were provided with the first version of tool and asked for individual feedback before participating in an online meeting where all disagreements among individuals and suggestions for additional items were discussed. Polls were used during the meeting to support reaching agreements. The meeting was recorded and later transcribed via Microsoft Teams. The second version of the tool was generated according to the agreement.

2.1.3. Three-round Delphi study

Delphi panelists were non-VITAL members who have experience in one or more of the predefined fields/positions: (1) Ministry of Health; (2) NITAG (or equivalent); (3) Insurer; (4) HTAB (or equivalent); (5) Representatives of: Healthy aging, General practitioners (GPs), Pharmacists, Nursing homes; (6) World Health Organization; and/or (7) Researcher in related fields including, but not limited to, Infectious diseases, Immunization, Epidemiology, Health Economics and Public Health. Snowball sampling method was used to recruit participants. This method involves identifying index individuals and asking them to refer other persons suitable for the study (38). Responses were collected via a Qualtrics survey using Qualtrics software (39).

Figure 2 presents the item evaluation process following the three-round Delphi study. In the first and second rounds, panelists were required to evaluate all items and give their opinions if the item should be “included as is,” “included with edits” or “dropped” from the tool; provide their suggestions for amendments, or for additional items or reasons for the exclusion. Consensus was predefined as a rate of agreement or disagreement above 70%. Items that reached the consensus value were included directly in the final version without further evaluation. Other items that did not reach the consensus or were subjects for edits or were newly added were transferred to the next round. Within 1 week after the evaluation period, panelists received the result and the survey link for the next round. Panelists who did not participate in one round were allowed to join the following round after he or she agreed with the current report.

FIGURE 2

Figure 2. Item generation process through a three-round Delphi study.

In the third round, panelists were required to evaluate revised items resulting from round 2 by choosing “include the item” or “drop the item.” Suggestions for edits or new items were not applicable. Consensus for inclusion was defined with a cut-off of 70%. The final result of the Delphi study was made available for all participants 2 weeks after the third round.

2.2. Phase 2: Pilot testing

An e-pilot test was conducted in four European countries: Netherlands (NL), Germany (GER), Serbia (SER), and Hungary (HU) to (i) test comprehensiveness, relevance, acceptability, and feasibility of the tool (40); (ii) finalize the tool; and (iii) present preliminary assessment of the countries' readiness. Tool version 3 (electronic version in excel) was tested in a small sample of the target audience—two researchers per country in the field of health economics/ health policy. These criteria were made based on considering the feasibility of the project and the ability of participant to assess different aspects of the tool rather than the importance of their roles in the topic area. Having a good understanding of various dimensions in the vaccine introduction process, starting from measuring the burden of diseases to financing and delivering the intervention, health economic/ health policy participants are a relevant proxy of the multidisciplinary audience of the tool. Snowball sampling method was used to recruit participants.

First, participants were asked to complete the readiness assessment in their countries and provide with corresponding evidence for their answers. Collected readiness scores were re-evaluated by the research team when conflicting answers existed between participants evaluating the same country. The final scores were determined based on provided evidence.

Second, participants took an online post-pilot assessment evaluating the tool followed on a five-point scale (strongly disagree, disagree, neutral, agree, and strongly agree) to determine the comprehensiveness, relevance, and acceptability of the tool. The feasibility of the tool was determined based on the ability to complete the readiness assessment and the ability to come up with one research idea that aimed to address identified problems based on the readiness scores. In addition, participants were also asked if they had any additional feedback to improve the abovementioned features and the tool as a whole. Based on the feedback, adaptation was followed to define the final version of the tool. Suggestions for major changes or inclusion of new items were inapplicable due to the extensive need for re-evaluation subsequently. Responses were collected via a Qualtrics survey (39).

3. Results

3.1. Phase 1: Tool development

3.1.1. Development of the item pool and the first version of the tool

From the literature review, 16 tools and frameworks were identified. After the data extraction process, an item pool was formed that included 65 items (Supplementary material 1). The first version of the tool included eight and six items in the decision-making and implementation parts respectively. Table 1 provides a summary of the item generation process throughout the whole process, from the development of the item pool until the pilot test.

TABLE 1

Table 1. Changes in the number of items during the item generation process of the country score tool.

3.1.2. Stakeholder consultation

Eleven members participated in the plenary meeting. All proposed items were evaluated to be relevant and important with some suggestions for edits. Overall, it was suggested to specify which vaccines to be used as criteria to assess the strength of current national immunization programs. Furthermore, experts pointed out the importance of having guidelines or strategies based on local evidence in the country with regards to vaccine acceptability among aging adults. The suggestion led to adding one item to the decision-making part.

3.1.3. Three-round Delphi study

The first round was completed by seven panelists. During the second and third rounds, all members completed the evaluation (N = 8). Description of the Delphi panel is presented in Table 2.

TABLE 2

Table 2. Description of the Delphi panel.

Availability of national guidance or communication campaigns for generating acceptance and demand for vaccines in aging adults, together with availability of healthy aging strategies, were considered to be important by 100% of panelists in round 1. In contrast, availability of alternative measures for prevention and control of VPDs in aging adults was suggested to be dropped by 14% of panelists in round 1, subjected for re-evaluation in the next rounds, and finally excluded. Moreover, the panel suggested and agreed on adding three items related to (1) basic requirements for NITAGs (or equivalent), (2) hard-to-reach groups (e.g., illiterate individual and migrants), and (3) healthcare professionals' involvement in providing vaccine recommendations to aging adults. Thirteen items that did not reach the consensus in round 1 were re-evaluated over round 2 and/or round 3. Generally, the panel suggested using universal terms to optimize the transferability of the tool in different contexts, given the diversity in policy and organizational setup of immunization programs among European countries. Furthermore, suggestions for rewording were proposed by panelists with regards to comprehensiveness of the tool as a whole and scoring system of each item that can cover all possibilities to assess readiness.

3.2. Phase 2: Pilot testing

3.2.1. Testing four features of the tool: Comprehensiveness, relevance, acceptability, and feasibility

Eight researchers participated and completed the readiness assessment and the post-pilot survey (Response rate 100%, N = 8). Table 3 shows the results of the post-pilot assessment in which a percentage of participants evaluating different features of the tool are presented.

TABLE 3

Table 3. The post-pilot assessment result.

The tool was evaluated to be comprehensive to capture all essential aspects of the readiness assessment by 75% of participants. In addition, 88% of participants perceived all components well-explained. All participants rated the tool to be relevant in their countries. In terms of acceptability, 75% of participants indicated their acceptance with time required and the future use of the tool. Moreover, 100% of participants suggested that the tool and the readiness results would be useful to their colleagues in health economics, and other fields including public health, epidemiology, and health politics. The feasibility of the tool was accepted by more than 85% of participants.

3.2.2. Final version of the tool

Participants of the pilot provided their feedback to redraft some of the sentences or provide more information for a better understanding on three items regarding existing vaccination programs (2 items) and healthy aging strategies. The final version of the tool consists of 17 items divided into decision-making and implementation parts. Each item has a guiding question, corresponding to explanations and rationales to inform assessment with readiness scores. Due to the large amount of information condensed in the tool, rationales and explanations of all items are presented separately in Supplementary material 2. The tool with guiding questions and readiness scores is presented in Table 4 of this manuscript.

TABLE 4

Table 4. The final version of the country score tool (^*) and readiness scores in four pilot countries.

The readiness of countries with regards to decision-making is assessed by the availability of sufficient infrastructure, fully functioning supportive groups and relevant strategies. First, the national surveillance system of VPDs should be in place to provide baseline epidemiological data on VPDs as well as to measure the impact of vaccines on disease incidences, morbidity, and mortality. Moreover, previously successful vaccine introductions for aging adults are indicative of the relevant infrastructure of the NIP for this target population. Second, the involvement of relevant stakeholders is crucial for countries to make decisions on vaccine introductions. While a NITAG (or equivalent) is a basic need for an NIP, the availability of adult vaccine workgroups is a specific requirement for aging adult immunization. Such groups could be in place as standalone groups or initiated by involving relevant experts into recommending bodies where vaccination of aging adults is considered. Third, national immunization strategies and healthy aging strategies covering aging adult vaccination serve as a foundation to direct new vaccine introductions given the many health issues and resource constraints that need to be tackled. Additionally, communication strategies, adjusted by local evidence on vaccine demand and acceptability, are needed to ensure high vaccine uptake. If the evidence is not available, corresponding studies at the local or national level need to be conducted to guide policies.

In alignment with the decision-making part, items in the implementation part of the tool further assess how countries have proceeded with vaccine recommendations and identify how the current system can facilitate future vaccine adoption for aging populations. By reviewing vaccine financing aspects, decision-makers might consider if expanding immunization financing mechanisms would be needed for the long-term financial sustainability of the NIP. Furthermore, a high level of government advocacy toward recommended vaccines plays an important role in increasing vaccine uptake. Healthcare professionals (HCP) are essential stakeholders who are actively involved in the care of aging adults being influential in commencing and completing vaccination schedules. The active involvement of this group contributes to the success of the NIP by increasing the uptakes of not only newly introduced vaccines but also routine vaccines. Other aspects, including centralization of vaccine invite/reminder/recall, easy access to supplies of vaccines and vaccine registries should be in place. Extra effort and resources need to be allocated to engage “hard-to-reach” aging adults to be vaccinated. They are often at high-risk for infectious diseases but face access barriers to vaccination due to various reasons, for instance, distance from vaccination centers, healthcare provider discrimination, and legal restrictions.

3.2.3. Readiness assessment results

Readiness scores for the four countries involved in the pilot testing are presented in Table 4. Although the scores cannot be considered as an official description of the readiness per country, the pilot assessment briefly reflects the current settings and performance of the national immunization programs in those countries. Overall, all countries demonstrated sufficient surveillance of at least three VPDs including influenza, pneumococcal disease, herpes zoster and/or pertussis. On the contrary, efforts to encourage hard-to-reach aging adults to be vaccinated were absent in all countries. In addition, availability of adult vaccine working groups and involvement of healthcare professionals were generally lacking. The Netherlands scored highly in items related to implemented vaccines including surveillance, local evidence of vaccine demand, and acceptability, the number of vaccines that have been introduced, and government advocacy to support vaccine introductions. However, supportive stakeholders, policies and strategies toward aging adult immunization programs were not fully in place. In contrast, Germany showed full preparations in those aspects which aligned with high scores in the item assessing the ease to vaccinate in the implementation part. Hungary was the only country of the four that did not have a NITAG. In addition, the country scored lowly in related items including having a national immunization plan, vaccine recommendations, and implementation in general. A similar pattern in implementation was seen in Serbia. There was no evidence on centralization of vaccine invite/reminder/recall in aging adults, together with active involvement of healthcare professionals. Although Serbia scored 0 in items related to existing immunization program and national guidance for communication, crucial stakeholders and national policy and surveillance systems were fully in place.

4. Discussion

This work has developed a tool to assess countries' readiness with regards to immunization programs in aging adults that can be used either before, or as part of the implementation process. Particularly, results from the readiness assessment can help to identify areas for improvement of the current programs as well as methodologies for future research to support decision-making and implementation of such programs in Europe.

In comparison with existing literature on this topic, our country score tool presents numerous unique features. First, unlike the purpose of other scoring systems such as archetype analysis aiming to categorize countries based on their characteristics (22), our tool serves as a readiness assessment which means that higher scores always indicate a higher level of readiness mentioned in the items. Second, although we propose a set of universal items that could be transferable among European countries, we consider each country to be unique. Thus, individual items in the tool are not necessarily equally important in all settings and subsequently no requisite labels are applied, which is opposite to the approach of a tool for planning and evaluating vaccination strategies in a published work (32). Additionally, the tool offers a great degree of flexibility for national authorities to specifically define the cut-offs of age for the national immunization programs considering the conceptualized cut-off of 50-year-old and over. Third, while some existing tools or guidance documents try to support countries in developing a comprehensive immunization program from zero (30, 41), our tool focuses on pinpointing important gaps in the current national immunization programs. By identifying those gaps, decision-makers can generate direct actions to strengthen current infrastructures and organizations such as setting up surveillance systems for common VPDs or forming working groups of aging adult vaccines. Fourth, aiming to not solely facilitate actions, our tool can help to go one step further to generate evidence needed for better decision-making. For that reason, the target audience of the tool is not only country-level decision-makers but also, researchers in different fields. Epidemiological research could be needed when countries score low in surveillance or having no evidence in monitoring vaccine coverage rates. In addition, when studies on vaccine demand and acceptability are lacking, behavioral research can help to generate tailored interventions or corresponding guidance to support implementation of the program. Furthermore, health economic research could be useful to address different gaps identified from the tool separately or simultaneously. For example, incorporating cost-effectiveness and value of information analyses can facilitate making well-informed policies regarding reimbursement of vaccination programs and setting priorities for future research.

In terms of content, our tool inherits from previous works to include well-known components aiming to support immunization programs; for instance: surveillance of VPDs, performance of existing programs, and public health priority of diseases (22). In addition, conducting research addressing the attitudes toward vaccines was considered crucial before the introduction of any vaccines to identify if low initial acceptance will exist or reasons for anti-vaccine sentiments (30). Nevertheless, it is important to turn knowledge into action. Therefore, one item was added to our tool to assess the availability of national guidance or communication campaigns for generating acceptance and demand for vaccines in aging adults. Regarding implementation, one novel item, relating to hard-to-reach populations, was suggested to be added and finally reached the consensus after the Delphi study. The item suggests to clearly define this population to inform strategy, planning, and resource determinations for target interventions to remove existing barriers to vaccinate (42). Besides confirmatory findings, contradictory judgments were found in this project. First, the item related to alternative measures for prevention and control of VPDs was excluded from our tool although it was considered as one of the criteria for decision-making in existing guidelines (30, 43). Those measures include, but are not limited to, treatment with antibiotics, antiviral therapies, or other medicines which could have a synergy with immunization programs. However, it might be irrelevant to include that aspect in the readiness assessment because those alternatives should be in place regardless of the availability of vaccines. Second, the involvement of healthcare workers by providing vaccine recommendation to aging adults was not initially included in the tool because of a non-significant association between this factor and vaccination coverage rates (VCR) of influenza in the elderly (44). However, considering that the success of one vaccination program is not necessarily comparable with others, the item was added by the Delphi panel.

Regarding the development process of the tool, the modified Delphi process provided many advantages. Firstly, the Delphi technique was an appropriate method when there was a lack of agreement, incomplete knowledge, uncertainty or lack of evidence (45). Compared to the conventional design, a modified Delphi approach allowed an active involvement of a steering group that performed a literature review in the problem area instead of open-ended discussion in the initial Delphi round (34). That helped to provide a comprehensive item pool for the following steps. Moreover, the internal VITAL stakeholder consultation was organized as an extra effort to ensure that comprehensiveness, relevance, and transferability of the tool were considered at the early phase of the development process. In addition, during the Delphi study, the use of a web-based platform allowed for anonymity and the inclusion of international experts. Additionally, predefined consensus criteria and a transparent procedure helped to reduce bias related to opinions of the steering group which was recognized as a disadvantage of the given study design (34). Furthermore, the comprehensiveness, relevance, and potential transferability of the tool were preliminary indicated in the pilot which has not been done in the development of existing tools. Finally, readiness scores of the four pilot countries showed that countries with higher scores including Germany and the Netherlands, had more vaccines being scheduled in the national immunization programs for aging adults, as well as higher vaccine coverage rates. That is indicative of the face validity of the tool.

Despite the thorough process, the study has some limitations which are primarily related to the small sample size in the Delphi study. There is no standard size of the panel members. It depends on the complexity of the topic and heterogeneity of the panel (34). With a similar topic, a Delphi panel of 8 participants was considered to be sufficient (32). The second limitation concerns the pilot testing with regards to the choice of countries, selection criteria for participants, and the sampling method. We selected four countries—partners of the VITAL project based on a convenience sampling approach. However, this set of four might be unrepresentative for European countries in general, and therefore might hamper the generalization of our findings. Regarding the selection criteria, although national authorities and researchers are both primary target audience of the tool, the inclusion of solely researchers can ensure the feasibility of the project but might affect the accuracy of the readiness assessment. Furthermore, the snowball sampling method could potentially introduce bias as the readiness assessment for one country might be performed by people working in the same organization. However, the difference between answers of two participants per country indicated that bias was unlikely to happen. The difference could be explained by the fact that not all of participants had strong background and experience in immunization. That suggests the ease to use of the tool when non-experts in the field could independently complete the tool with a reasonable level of accuracy. The third limitation comes with regards to the inclusions of items of the tool. Although the development process was evidence-based, the item inclusions could be subjective as indirect consequences of expert opinions from previous works and directly from our Delphi study. Therefore, it is uncertain that the effort spent to fill all gaps would always result in better decision-making. Future research should attempt to quantify the impact of individual items on the outcome of immunization programs measured by vaccine coverage rates. Besides, we suggest future research moving forward with the implication of the tool. First, it is important to conduct comprehensive field testing by using the tool in the appropriate target audience, being decision-makers, advisors and experienced researchers in immunization programs of aging adults across the majority of European countries. Second, digitalizing the tool could facilitate the usage of the tool on a large scale. Subsequentially, collection of readiness assessment data in many countries could help to quantify the association between readiness scores of individual item and vaccine coverage rates. Therefore, a precisely evidence-based readiness assessment could be developed to better inform decision-making at the national or even supranational level with regards to aging immunization programs.

5. Conclusion

Decision-making and implementation of immunization programs in aging adults in European countries need to be strengthened based on a holistic assessment and using relevant evidence. Our extensive consultations and iterative development process have resulted in an innovative tool that serves as a readiness assessment and simultaneously helps to identify methodologies for future research to support decision-making and implementation of such programs. The tool is considered to be comprehensive, relevant, acceptable, and feasible in four countries. This indicates the transferability and applicability of the tool in Europe. We suggest future effort of this work to focus on digitalizing and comprehensive field testing of the tool across the majority of Europeans countries. Moreover, additional studies should be conducted to quantify the impact of individual items in the readiness assessment on vaccine coverage rates.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding authors.

Ethics statement

According to the Dutch Law on Medical Research Involving Human Subjects (WMO), ethical approval is not required for research that does not involve participation by patients or use of patients' data (46). All expert participants gave informed consent before they took part in the study.

Author contributions

TP conducted the literature review and wrote the first draft of the manuscript. JvdS, EB, and TP contributed to the development of the item pool, organized stakeholder consultation, Delphi study, and pilot test. MP has made a direct and intellectual contribution to the first version of the tool, and the stakeholder consultation. BN, TÁ, CW, DS, and NN substantially contributed to the final version of the tool. All authors read, edited, and approved the final manuscript.

Funding

This work was conducted as part of the Vaccines and InfecTious diseases in the Aging population (VITAL) project which has received funding from the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement No. 806776. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Acknowledgments

We thank all VITAL members for their advice, feedback, and collaboration throughout the process. We thank Dr. Peter G. Braun at Central Medical Library—University of Groningen for his assistance in our literature search, Desi Burghoorn and Cian Mitchell for a language check in the final version of the country score tool. We thank Delphi panelists and pilot participants for their invaluable contributions.

Conflict of interest

MP reports grants and personal fees from various pharmaceutical industries, all outside the submitted work. EB is an employee of and holds shares in the GSK group of companies. TP was employed by Asc Academics.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2022.1080678/full#supplementary-material

Supplementary material 1. Literature review and the item pool.

Supplementary material 2. Supportive part of the country score tool.

References

1. Esposito S, Franco E, Gavazzi G, de Miguel AG, Hardt R, Kassianos G, et al. The public health value of vaccination for seniors in Europe. Vaccine. (2018) 36:2523–8. doi: 10.1016/j.vaccine.2018.03.053

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Talbird SE, La EM, Carrico J, Poston S, Poirrier J-E, DeMartino JK, et al. Impact of population aging on the burden of vaccine-preventable diseases among older adults in the United States. Hum Vacc Immunother. (2021) 17:332–43. doi: 10.1080/21645515.2020.1780847

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Kooli C. COVID-19: public health issues and ethical dilemmas. Ethics Med Public Health. (2021) 17:100635. doi: 10.1016/j.jemep.2021.100635

PubMed Abstract | CrossRef Full Text | Google Scholar

4. European Centre for Disease Prevention and Control. Seasonal Influenza. Factsheet for health professional. Available online at: https://www.ecdc.europa.eu/en/seasonal-influenza/facts/factsheet (accessed February 03 2022).

Google Scholar

5. Welte T, Torres A, Nathwani D. Clinical and economic burden of community-acquired pneumonia among adults in Europe. Thorax. (2012) 67:71–9. doi: 10.1136/thx.2009.129502

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Harpaz R, Ortega-Sanchez IR, Seward JF. Prevention of Herpes Zoster: recommendations of the advisory committee on immunization practices (ACIP). Morb Mortal Wkly Rep Recommend Rep. (2008) 57:1–30. Available online at: https://www.cdc.gov/mmwr/preview/mmwrhtml/rr5705a1.htm

PubMed Abstract | Google Scholar

7. McElhaney JE, Beran J, Devaster J-M, Esen M, Launay O, Leroux-Roels G, et al. AS03-adjuvanted versus non-adjuvanted inactivated trivalent influenza vaccine against seasonal influenza in elderly people: a phase 3 randomised trial. Lancet Infect Dis. (2013) 13:485–96. doi: 10.1016/S1473-3099(13)70046-X

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Yin M, Huang L, Zhang Y, Yu N, Xu X, Liang Y, et al. Effectiveness and safety of dual influenza and pneumococcal vaccination versus separate administration or no vaccination in older adults: a meta-analysis. Expert Rev Vacc. (2018) 17:653–63. doi: 10.1080/14760584.2018.1495077

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Bonten MJ, Huijts SM, Bolkenbaas M, Webber C, Patterson S, Gault S, et al. Polysaccharide conjugate vaccine against pneumococcal pneumonia in adults. N Engl J Med. (2015) 372:1114–25. doi: 10.1056/NEJMoa1408544

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Beard JR, Officer A, de Carvalho IA, Sadana R, Pot AM, Michel J-P, et al. The World report on ageing and health: a policy framework for healthy ageing. Lancet. (2016) 387:2145–54. doi: 10.1016/S0140-6736(15)00516-4

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Ting EEK, Sander B, Ungar WJ. Systematic review of the cost-effectiveness of influenza immunization programs. Vaccine. (2017) 35:1828–43. doi: 10.1016/j.vaccine.2017.02.044

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Shao Y, Stoecker C. Cost-effectiveness of pneumococcal vaccines among adults over 50 years old in low- and middle-income countries: a systematic review. Expert Rev Vacc. (2020) 19:1141–51. doi: 10.1080/14760584.2020.1874929

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Chiyaka ET, Nghiem VT, Zhang L, Deshpande A, Mullen PD, Le P. Cost-effectiveness of herpes zoster vaccination: a systematic review. Pharmaco Econ. (2019) 37:169–200. doi: 10.1007/s40273-018-0735-1

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Tan L. Adult vaccination: now is the time to realize an unfulfilled potential. Hum Vacc Immunother. (2015) 11:2158–66. doi: 10.4161/21645515.2014.982998

PubMed Abstract | CrossRef Full Text | Google Scholar

15. European Centre for Disease Prevention and Control. Seasonal Influenza Vaccination and Antiviral Use in EU/EEA Member States – Overview of Vaccine Recommendations for 2017–2018 and Vaccination Coverage Rates for 2015–2016 and 2016–2017 Influenza Seasons. Stockholm: ECDC (2018).

Google Scholar

16. World Health Organization. New data reveals significant gaps in influenza immunization of those who need it most in Europe 2020. Available online at: https://www.who.int/europe/news/item/15-12-2020-new-data-reveals-significant-gaps-in-influenza-immunization-of-those-who-need-it-most-in-europe (accessed July, 2022).

17. European Centre for Disease Prevention and Control. Current Practices in Immunisation Policymaking in European Countries. Stockholm: ECDC (2015).

Google Scholar

18. Botwright S, Giersing BK, Meltzer MI, Kahn AL, Jit M, Baltussen R, et al. The CAPACITI decision-support tool for national immunization programs. Value Health. (2021) 24:1150–7. doi: 10.1016/j.jval.2021.04.1273

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Duclos P, Durrheim DN, Reingold AL, Bhutta ZA, Vannice K, Rees H. Developing evidence-based immunization recommendations and GRADE. Vaccine. (2012) 31:12–9. doi: 10.1016/j.vaccine.2012.02.041

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Sauer M, Vasudevan P, Meghani A, Luthra K, Garcia C, Knoll MD, et al. Situational assessment of adult vaccine preventable disease and the potential for immunization advocacy and policy in low- and middle-income countries. Vaccine. (2021) 39:1556–64. doi: 10.1016/j.vaccine.2021.01.066

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Health Resources and Service Administration (HRSA). Readiness Assessment & Developing Project Aims. U.S. Department of Health and Human Services (HHS) (2011).

Google Scholar

22. Privor-Dumm L, Vasudevan P, Kobayashi K, Gupta J. Archetype analysis of older adult immunization decision-making and implementation in 34 countries. Vaccine. (2020) 38:4170–82. doi: 10.1016/j.vaccine.2020.04.027

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Burchett HE, Mounier-Jack S, Griffiths UK, Mills AJ. National decision-making on adopting new vaccines: a systematic review. Health Policy Plann. (2012) 27(Suppl. 2):ii62–76. doi: 10.1093/heapol/czr049

PubMed Abstract | CrossRef Full Text | Google Scholar

24. World Health Organization. Regional Office for Europe, European Observatory on Health Systems and Policies & Nolte, Ellen. How Do We Ensure That Innovation in Health Service Delivery and Organization Is Implemented, Sustained and Spread? Copenhagen: World Health Organization. Regional Office for Europe (2018).

Google Scholar

25. Laigle V, Postma MJ, Pavlovic M, Cadeddu C, Beck E, Kapusniak A, et al. Vaccine market access pathways in the EU27 and the United Kingdom - analysis and recommendations for improvements. Vaccine. (2021) 39:5706–18. doi: 10.1016/j.vaccine.2021.07.040

PubMed Abstract | CrossRef Full Text | Google Scholar

26. World Health Organization. World Experts and Funders Set Priorities for COVID-19 Research 2020. Available online at: https://www.who.int/news/item/12-02-2020-world-experts-and-funders-set-priorities-for-covid-19-research (accessed May 01, 2022).

Google Scholar

27. World Health Organization. Country Readiness for COVID-19 Vaccines 2021. Available online at: https://www.who.int/news-room/feature-stories/detail/country-readiness-for-covid-19-vaccines (accessed May 01, 2022).

28. Donadel M, Panero MS, Ametewee L, Shefer AM. National decision-making for the introduction of new vaccines: a systematic review, 2010–2020. Vaccine. (2021) 39:1897–909. doi: 10.1016/j.vaccine.2021.02.059

PubMed Abstract | CrossRef Full Text | Google Scholar

29. González-Lorenzo M, Piatti A, Coppola L, Gramegna M, Demicheli V, Melegaro A, et al. Conceptual frameworks and key dimensions to support coverage decisions for vaccines. Vaccine. (2015) 33:1206–17. doi: 10.1016/j.vaccine.2014.12.020

PubMed Abstract | CrossRef Full Text | Google Scholar

30. World Health Organization. Principles and Considerations for Adding a Vaccine to a National Immunization Programme: From Decision to Implementation and Monitoring. Geneva: World Health Organization (2014).

Google Scholar

31. Giles ML, Mason EM, Lambach P, Mantel C. Maternal immunization country readiness: a checklist approach. Hum Vacc Immunother. (2020) 16:3177–83. doi: 10.1080/21645515.2020.1750248

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Calabrò GE, Carini E, Tognetto A, Mancinelli S, Sarnari L, Colamesta V, et al. Developing an evidence-based tool for planning and evaluating vaccination strategies aimed at improving coverage in elderly and at-risk adult population. Front Public Health. (2021) 9:658979. doi: 10.3389/fpubh.2021.658979

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Monguet JM, Trejo A, Martí T, Escarrabill J. Health consensus: a digital adapted delphi for healthcare. Int J User Driven Healthc. (2017) 7:27–43. doi: 10.4018/IJUDH.2017010103

CrossRef Full Text | Google Scholar

34. Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: how to decide its appropriateness. World J Methodol. (2021) 11:116–29. doi: 10.5662/wjm.v11.i4.116

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Esposito S, Principi N, Rezza G, Bonanni P, Gavazzi G, Beyer I, et al. Vaccination of 50+ adults to promote healthy ageing in Europe: the way forward. Vaccine. (2018) 36:5819–24. doi: 10.1016/j.vaccine.2018.08.041

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Privor-Dumm LA, Poland GA, Barratt J, Durrheim DN, Deloria Knoll M, Vasudevan P, et al. A global agenda for older adult immunization in the COVID-19 era: a roadmap for action. Vaccine. (2021) 39:5240–50. doi: 10.1016/j.vaccine.2020.06.082

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Van Baarle D, Bollaerts K, Del Giudice G, Lockhart S, Luxemburger C, Postma MJ, et al. Preventing infectious diseases for healthy ageing: the VITAL public-private partnership project. Vaccine. (2020) 38:5896–904. doi: 10.1016/j.vaccine.2020.07.005

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Kennedy-Shaffer L, Qiu X, Hanage WP. Snowball sampling study design for serosurveys early in disease outbreaks. Am J Epidemiol. (2021) 190:1918–27. doi: 10.1093/aje/kwab098

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Qualtrics. Qualtrics Version January 2022. Provo, UT, USA (2022). Available online at: https://www.qualtrics.com (accessed May 9, 2022).

40. de Vet HCW, Terwee CB, Mokkink LB, Knol DL. Measurement in Medicine: A Practical Guide. Cambridge: Cambridge University Press (2011). doi: 10.1017/CBO9780511996214

CrossRef Full Text | Google Scholar

41. World Health Organization. WHO-UNICEF Guidelines for Comprehensive Multi-Year Planning for Immunization: Update September 2013. Geneva: World Health Organization (2014).

Google Scholar

42. Poudel A, Lau ETL, Deldot M, Campbell C, Waite NM, Nissen LM. Pharmacist role in vaccination: evidence and challenges. Vaccine. (2019) 37:5939–45. doi: 10.1016/j.vaccine.2019.08.060

PubMed Abstract | CrossRef Full Text | Google Scholar

43. World Health Organization. Country-led Assessment for Prioritization in Immunization (CAPACITI): Decision-Support Framework Version 2.1 (2021).

44. Blank PR, Schwenkglenks M, Szucs TD. The impact of European vaccination policies on seasonal influenza vaccination coverage rates in the elderly. Hum Vacc Immunother. (2012) 8:318–25. doi: 10.4161/hv.18629

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Powell C. The Delphi technique: myths and realities. J Adv Nurs. (2003) 41:376–82. doi: 10.1046/j.1365-2648.2003.02537.x

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Central Committee on Research Involving Humans. Legal Framework for Medical Scientific Research: Your Research – Is it Subject to the WMO or Not? (2021). Available online at: https://english.ccmo.nl/investigators/legal-framework-for-medical-scientific-research/your-research-is-it-subject-to-the-wmo-or-not (accessed February 18, 2022).

Google Scholar