External validation of the parental attitude about childhood vaccination scale

Introduction: Internal validation techniques alone do not guarantee the value of a model. This study aims to investigate the external validity of the Parental Attitude toward Childhood Vaccination (PACV) scale for assessing parents’ attitude toward seasonal influenza vaccination.

Methods: Using a snowball sampling approach, an anonymous online questionnaire was distributed in two languages (English and Arabic) across seven countries. To assess the internal validity of the model, the machine learning technique of “resampling methods” was used to repeatedly select various samples collected from Egypt and refit the model for each sample. The binary logistic regression model was used to identify the main determinants of parental intention to vaccinate their children against seasonal influenza. We adopted the original model developed and used its predictors to determine parents’ intention to vaccinate their children in Libya, Lebanon, Syria, Iraq, Palestine, and Sudan. The area under the curve (AUC) indicated the model’s ability to distinguish events from non-events. We visually compared the observed and predicted probabilities of parents’ intention to vaccinate their children using a calibration plot.

Results: A total of 430 parents were recruited from Egypt to internally validate the model, and responses from 2095 parents in the other six countries were used to externally validate the model. Multivariate regression analysis showed that the PACV score, child age (adolescence), and Coronavirus disease 2019 (COVID-19) vaccination in children were significantly associated with the intention to receive the vaccination. The AUC of the developed model was 0.845. Most of the predicted points were close to the diagonal line, demonstrating better calibration (the prediction error was 16.82%). The sensitivity and specificity of the externally validated model were 89.64 and 37.89%, respectively (AUC = 0.769).

Conclusion: The PACV showed similar calibration and discrimination across the six countries. It is transportable and can be used to assess attitudes towards influenza vaccination among parents in different countries using either the Arabic or English version of the scale.

1. Introduction

Influenza viruses are a major threat to global health, causing high rates of morbidity and mortality (1). Influenza burden fluctuates from year to year, depending on which viruses are circulating and the number of people infected. The severity of illness can range from mild to severe. High-risk groups, such as pregnant women, young children, older adults, and those with chronic medical conditions or compromised immune systems, are more likely to develop complications (2). Globally, seasonal influenza infects up to 20% of the population during winter, resulting in up to 650,000 annual deaths from influenza-related respiratory diseases. Furthermore, influenza has a large economic impact in terms of expenses and lost working hours (3).

Influenza viruses infect the nose, throat, and lungs and are easily transmitted among children because they frequently touch their nose, eyes, and mouth and touch each other while playing (3, 4). Furthermore, there are many interactions between parents/caregivers and children, including holding hands, picking up, feeding, changing diapers, and other activities (4). Influenza continues to have a serious impact on the morbidity and mortality of children and continues to increase annually. Young children, children with chronic illnesses, and household members are more vulnerable to influenza-related illnesses. Despite this, seasonal influenza vaccination rates in childhood remain low (5, 6).

The most vital step in preventing seasonal influenza infection is vaccination (3, 7). A crucial first step in avoiding the spread of influenza among healthcare workers (HCWs), patients, high-risk populations, and children is to increase vaccination rates. Recently, the Centers for Disease Control and Prevention (CDC) recommended that all people aged 6 months and older, including HCWs, patients, and residents of long-term care homes, should receive vaccinations unless otherwise specified (7). Seasonal influenza vaccination is available in the fall and provides protection during the influenza season (November to April) (4). Full protection from the moment of vaccine administration typically takes 2 weeks. Children under the age of 9 years require two doses 4 weeks apart during the first year of immunization. Seasonal influenza vaccines are not recommended for infants below 6 months as breast milk protects against various respiratory infections, including influenza (8). In addition, their safety and immunogenicity have not yet been approved, especially if administered with other vaccines (9).

Despite the fact that yearly influenza vaccination is recommended for all children aged 6 months to 18 years, little is known about the level of parental hesitation to vaccinate their children against seasonal influenza worldwide (10). The World Health Organization (WHO) lists vaccine hesitancy (VH) as one of the leading 10 causes of global health threats in 2019 (11). VH refers to a delay in acceptance or refusal of immunization, despite the availability of vaccination services. VH is complicated and varies across time, region, and type of vaccine. It is affected by complacency, constrain, and confidence in the vaccine and its service delivery (12).

The Parent Attitudes about Childhood Vaccines (PACV) survey is a reliable technique that has been successfully used in several countries to identify parental VH (13, 14). The PACV has been internally validated in English and Arabic versions (10, 13, 15). Unlike internal validation, external validation aims to assess the performance of a risk-prediction model for new individuals (new dataset). External validation aims to investigate whether the developed model can accurately predict similar but distinct individuals outside the original setting (16). It reveals the degree of heterogeneity and the extent to which the model can be generalized outside the development set. External validation measures the predictive accuracy of the developed model under different circumstances, indicating its transportability to other individuals at different times (temporal validation) or in different countries (geographic validation) (16, 17). A model is said to be “transportable” if it continues to perform well in a population distinct from the one for which it was initially designed (18).

Few studies have evaluated external validation (17, 19, 20). However, it is insufficient to validate the prediction model internally and indicate its success in predicting the outcome of interest. Furthermore, internal validation techniques alone do not guarantee the value of a model (21, 22). Assessing the external validity of prediction models is vital to verify their performance in other samples, as they usually perform better in development samples than in other new or different samples. Thus, in addition to assessing internal validity, the performance of prediction models should be validated in new individuals before use in practical studies. We hypothesized that the PACV is valid for the assessment of parental attitude toward vaccination. In this context, this study aimed to externally validate the PACV scale in six countries in the Eastern Mediterranean Region (EMR).

2. Materials and methods

2.1. Study design

A cross-sectional survey was conducted from September 8 to October 15, 2022, among the general population from seven countries in EMR using an online questionnaire.

2.2. Study population and sampling methods

The snowball sampling approach was used to include individuals who met the following eligibility criteria: parents aged 18 or older, with one or more children aged 6 months to 19 years, with a mobile phone or computer, who were able to self-complete the survey, and who resided in one of the following randomly selected EMR countries (Egypt, Libya, Lebanon, Syria, Iraq, Palestine, Sudan).

2.3. Data collection

The survey was created using Google Forms and distributed in two language versions (English and Arabic) via social media channels (Facebook and Twitter), WhatsApp, and emails. Before data collection, a pilot test was conducted to assess the feasibility and comprehensibility of the questionnaire. The feasibility and accessibility of the web application were also evaluated. Each data collector was asked to provide at least two responses to calculate the time required to complete the survey. The time spent by the respondents to fill in the questionnaire was 5–12 min. Only minor edits, including linguistic corrections, were made based on the respondents’ recommendations (Supplementary File 1).

2.4. External validation process

2.4.1. Development of the PACV questionnaire in the original sample

The development and scoring technique of the PACV have been explained elsewhere (13). In summary, 230 parents of children aged 19–35 months were recruited for this study. The domains of PACV (safety, efficacy, and attitude and behavior) explained 70% of parental intention to vaccinate children. The Cronbach’s alphas for the three domains were 0.74, 0.84, and 0.74, respectively. The overall PACV score was calculated by adding the weighted scores for each item. The total score ranges from 0 to 100 points. Participants were classified as hesitant (≥50 points) or non-hesitant (<50 points). The equation for the developed model was not available. Therefore, the internal validity of the PACV was tested to develop an equation that could be used for external validation.

2.4.2. Steps of external validation of PACV

We collected data from seven countries to verify the geographic validation of the PACV tool for predicting parents’ intention to vaccinate their children against seasonal influenza. First, we estimated the predictors of parents’ intention to vaccinate their children using data collected from Egypt (internal validation). Second, we adopted the original model developed in the first step and utilized its predictors to predict parents’ intention to vaccinate their children in the remaining countries: Libya, Lebanon, Syria, Iraq, Palestine, and Sudan (external validation). The predictive performance of the adopted model was measured when it was applied to the second group by quantifying the main aspects of discrimination and calibration. We followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) Statement, a reporting guideline for research creating or validating a multivariable prediction model (Supplementary File 2).

2.5. Sample size calculation

Internal validation requires a minimum sample size of 300 as a rule of thumb (23), and our study exceeded this requirement with a sample size of 430 in the internal validity group. The predicted PACV model was developed using data from Egypt.

For external validation, it is recommended to have at least 100 events and 100 non-events to ensure accurate and precise estimates of performance measures, and even larger sample sizes (a minimum of 200 events and 200 non-events) to derive flexible calibration curves (24, 25). To validate the PACV tool externally in six countries, we collected data from 2095 respondents, with 399 cases in Libya, 301 cases in Lebanon, 386 cases in the Syrian Arab Republic, 389 cases in Iraq, 389 cases in Palestine, and 231 cases in Sudan. We chose to develop the predicted model using data from Egypt because we had previously internally validated the Arabic version of the PACV tool using Egypt’s data in a previous publication, which indicated the validity and reliability of the PACV instrument in Arabic language (15).

2.6. Statistical analysis

Logistic regression was used to predict parents’ intention to vaccinate their children, using multiple predictor variables, including the PACV score. The outcome variable is binary, “one” for parents who intended to vaccinate their children against seasonal influenza and “zero” for parents who did not intend to vaccinate their children against seasonal influenza. Binary logistic regression was used to calculate the predicted outcomes. The probability equation for the logistic prediction model takes the following form. $p\left(probability\right)=\frac{1}{1+e^{-\left({\beta }_0+PI\right)}}$

where ${\beta }_0$ = the intercept and the prognostic index (PI) =  ${\displaystyle {\sum }_{i=1}^n{\beta }_i}{X}_i,$ $\beta =\ln \left(odds\mathrm{}ratio\right)$ .

The prognostic index (PI) is the main component of the prediction equation. The PI is a linear predictor calculated by summing the model’s predictors ( $X_s$ ) multiplied by their regression coefficients $\mathrm{(}{\beta }_i\mathrm{}\mathrm{)}.X$s denotes the independent variables that include the PACV score and the characteristics of both parents and children. To develop an accurate predictive model, we checked logistic regression assumptions before running the test. We investigated the linearity between the PACV score, mother’s age, number of children, birth order, and logit of parents’ intention outcomes. Smoothed scatter plots indicated that all continuous variables were relatively linearly associated with parents’ intention on the logit scale (Supplementary Figure S1). Cook’s distance and standardized residuals were measured to check the influential values. Cook’s distance was used to determine the most extreme observations (Supplementary Figure S2). However, we did not have influential observations because their absolute standardized residuals were lesser than three, as indicated in Supplementary Figure S3. Moreover, our model was free from multicollinearity because the variance inflation factor was <5.

To assess the internal validity of the model, we utilized a more advanced machine learning technique and resampling methods to select various samples and refit the model for each sample. Fitting the developed model to each new sample produced additional information regarding the variability of the developed model’s fit. We used the most common methods known as cross-validation techniques. The samples were selected using leave-one-out cross-validation (LOOCV) and K-Fold Cross-Validation (K-fold CV) techniques. The average prediction error was then estimated from the drawn samples to assess the performance of the developed model (26). Further details can be found in Supplementary File 3.

To evaluate the external validity of the predictive model, we used common assessment metrics and methods. First, we computed the prediction accuracy rate, which refers to the proportion of correctly predicted observations. Conversely, the prediction error rate refers to the proportion of incorrectly predicted observations. Confusion matrices were used to determine the proportions of type I and type II errors. Type I errors occur when the predictive model incorrectly predicts parents who do not intend to vaccinate their children in the intending group. In contrast, type II error refers to incorrectly assigning parents in the intending group to the group that includes parents who lack the intention of vaccination.

Furthermore, we measured the sensitivity and specificity metrics that summarize the model’s overall performance. The sensitivity of the predicted model was measured using the true-positive rate (TPR), which is the proportion of parents in the intended group correctly predicted by the developed model. For comparison, the specificity of the predicted model was measured using the true-negative rate (TNR), which is the proportion of parents in the non-intending group who were correctly predicted by the developed model. Therefore, the false-positive rate (FPR) is the proportion of parents in the non-intending group that are incorrectly predicted in the intending group. The FPR is the complement of specificity.

To visualize the predictive model performance, we used the receiver operating characteristic area under the curve (ROC-AUC), which shows the sensitivity against “1-specificity” at various values of the probability cutoff. The AUC was calculated to summarize the overall performance of the predictive model, which indicates the ability of the model to distinguish events from non-events (i.e., discrimination). Additionally, we visually compared the observed and predicted probabilities of parents’ intention to vaccinate their children using a calibration plot. The 45° line indicates perfect agreement between the predicted and observed probabilities (calibration). Overprediction results in points above the diagonal line, whereas underprediction results in points below the diagonal line.

2.7. Considerations of ethics

This study was part of a larger project that aimed to evaluate parents’ seasonal influenza vaccine hesitancy in the EMR (27). The Ethics Committee of the Faculty of Medicine at Alexandria University, Egypt approved this study (IRB no. 0305688). The study followed the Helsinki Declaration and the Ethics Committee guidelines to ensure anonymity, confidentiality, and voluntary participation. All participants provided written informed consent before taking part in the study. The collected information was stored in a coded format on a secure computer that was only accessible to the principal investigator.

3. Results

3.1. Groups used to develop and validate the model

Table 1 shows the differences in average values of other predictors, including parents’ socio-demographic characteristics (mother age, education, employment, residence, and parents’ previous influenza vaccination) and child characteristics (child’s birth order, whether the child had a chronic disease, whether the child got sick from influenza last year, whether the child got influenza vaccination last year, and whether the child got routine vaccination), between the development and validation samples. Notably, parental intention toward the influenza vaccine differed significantly between the development and validation groups.

TABLE 1

Table 1. Characteristics of parents and their children in development and validation samples.

3.2. The model development

The null deviance of the prediction model that included only the intercept to quantify parents’ intention to vaccinate their children was 499.40. When fitting the parents’ intention model with all independent variables, including the PACV variable, the residual deviance was 360.27, and it was 432.62 if we excluded the PACV score. The Akaike Information criterion (AIC) was 410.27 if the PACV score was included and 480.62 if it was excluded. Including the fit of PACV improved the model and was highly significant (Table 2).

TABLE 2

Table 2. The predictive PACV model.

3.3. Internal validation of the model

The proportion of parents’ intention that have been correctly classified was 79.3% [95% CI:75.5–83.1], while the classification error was 20.7% [95% CI:16.9–24.5]. The two cross-validation approaches, LOOCV and k-fold CV, yielded similar results, indicating that the PACV model had high predictive power, where the prediction error rate did not exceed 16.82%.

As shown in Figure 1, the AUC value was 0.846, indicating better performance of the predicted model. The calibration plot also demonstrates the accuracy of the calibration, where most points close to the diagonal line show better calibration. The TPR (sensitivity) was 92.06%, and the TNR (specificity) was 44.35%, respectively.

FIGURE 1

Figure 1. ROC curve and calibration plot for the PACV predictive model.

3.4. External validation

3.4.1. Model discrimination

The external validity of the PACV model was evaluated using the model developed to predict the validation group. The PACV-developed model correctly predicted 69.4% [95% CI:67.8–70.8] of parents’ intention in the validation group, while its prediction error was 30.6% [95% CI:29.2–32.2]. Based on the confusion matrix, the PACV model incorrectly predicted 24.3% [95% CI:22.9–25.7] of parents in the intending group (Type I error) and 6.3% [95% CI:5.5–7.1] in the non-intending group, including parents who lack the intention of vaccination (Type II error). The TPR (sensitivity) was 89.64% [95% CI:88.6–90.6], and the TNR (specificity) was 37.9% [95% CI:36.3–39.5]. The ROC curve indicates that the TPR increases faster than the FPR, and the AUC value was 0.769, indicating the better performance of the predicted model.

3.4.2. Model calibration

The calibration plot shows the validity of the predictive model, where most points are close to the diagonal line (45° line; Figure 2). Therefore, we can explore whether the PACV model performs better in all selected countries.

FIGURE 2

Figure 2. ROC curve and calibration plot for the validation group.

3.5. External validation across different countries

The proportion of correctly predicted observations differed across countries, as indicated in Table 3. The highest accuracy rate for the PACV predictive model was 78.92% in Iraq, and the lowest rate was 69.10% in Lebanon. Type I error was the highest in Palestine and the lowest in Iraq, whereas Type II error was the highest in Sudan and the lowest in Palestine. Figures 3–8 display the ROC curves and calibration plots for the PACV-predicted models. The model performs better as the AUC in all selected countries is large; the AUC has a greater value and is far from the diagonal line. Calibration plots indicated that the PACV prediction model performed better and had better predictive accuracy in the validation samples of the countries, where most points were close to the diagonal line.

TABLE 3

Table 3. Performance assessment measures of the predictive classification model.

FIGURE 3

Figure 3. Roc curve and calibration plot for the PACV predictive model in Iraq.

FIGURE 4

Figure 4. Roc curve and calibration plot for the PACV predictive model in Lebanon.

FIGURE 5

Figure 5. Roc curve and calibration plot for the PACV predictive model in Palestine.

FIGURE 6

Figure 6. Roc curve and calibration plot for the PACV predictive model in Sudan.

FIGURE 7

Figure 7. Roc curve and calibration plot for the PACV predictive model in Libya.

FIGURE 8

Figure 8. Roc curve and calibration plot for the PACV predictive model in Syria.

4. Discussion

With the increase in vaccine-preventable diseases, introduction of new vaccines, propagation of misinformation, and lack of coverage during the last decade, VH has been identified as a major public health challenge. This has resulted in a flood of scientific literature on VH in the realms of public health, biomedicine, and social science. Much of this research used different tools, such as the 5C scale and PACV, to assess VH. However, the overwhelming amount of data indicates that the quality of reporting in prediction model research is inadequate. This may provide incorrect information about the population’s attitude towards vaccination, and consequently affect the implementation of preventive measures to control infectious diseases. The implementation of these inappropriate measures may have deleterious effects on global health.

Furthermore, with the emergence of many infectious diseases such as coronavirus disease 2019 (COVID-19) and monkeypox, in addition to the ongoing risk of infection from circulating diseases such as seasonal influenza, there is a crucial need to develop and evaluate the validity of tools that assess VH among parents towards childhood vaccination. Although PACV has been used worldwide to assess parental VH, its external validity has not been examined. Therefore, the current study is the first to assess the external validity of the PACV scale across countries in two languages (Arabic and English) among 2095 participants from the general population.

Because of the unavailability of the equation of an internally validated model, we first developed the model, which was then externally validated. Internal validation of the model showed that PACV could effectively assess VH among parents. The model was developed in Egypt using a sample of 430 parents and it showed that PACV was a significant predictor of parents’ intention to vaccinate their children against influenza. In addition, the internal validity of the model showed good discrimination (AUC = 0.845, sensitivity = 92.06%, and specificity = 44.35%) and high predictive power (the prediction error rate did not exceed 16.82).

Unbiased validation of previous research findings is a fundamental scientific principle (28). After creating a prediction model, it is strongly recommended to test the model’s performance using data from individuals other than those used for model development. External validation involves using the previous model (the published regression formula) to make predictions for each individual in the new dataset and comparing them with the observed outcomes (16). Interestingly, many tools have been proven invalid when externally validated after being applied to thousands of patients. For instance, the Chronic Obstructive Pulmonary Disease (COPD) diagnostic questionnaire was created to improve the efficiency and precision of COPD diagnosis in primary care by distinguishing between individuals with and without airflow limitation. However, the COPD diagnostic questionnaire’s ability to differentiate between people with and without COPD was inadequate when externally validated, with an AUC of 0.65, sensitivity of 89.2%, and specificity of 24.24% (20). Similarly, Yu et al. (29) found that the vast majority of studies (70 of 86, 81%) that assessed the external validation of deep learning for radiologic diagnosis reported at least a decrease in external performance compared to internal performance. Almost half (42 of 86, 49%) reported a modest decrease (0.05 on the unit scale), and almost a quarter (21 of 86, 24%) reported a significant decrease (0.10 on the unit scale) in external performance compared with internal performance of the tool (29).

The external validation of a model depends primarily on its discrimination and calibration, which are essential for determining whether patients who experience an outcome have a higher expected risk than those who do not. For discrimination purposes, it makes little difference whether the absolute predicted risk is 8% or 80%, as long as the patient with the desired outcome has a higher risk (21).

In this study, external validity findings revealed that the discriminative ability of PACV was good. Moreover, the current study showed that the AUC of the six countries included in the external validation ranged from 70.3% in Libya to 83.3% in Lebanon. This result suggests that the model’s discrimination across countries with different economic incomes was acceptable.

A diagnostic test that can distinguish patients with and without a specific condition should have high sensitivity and specificity. The current study found that the sensitivity ranged from 86.7 to 90.9%, whereas the specificity ranged from 35.8 to 48.3%. This suggests that the developed model correctly identified nearly nine out of 10 parents intending to vaccinate their children against influenza (true-positive). At the same time, the questionnaire correctly classified nearly half of the participants as having no intention of getting their children vaccinated (true-negative).

TNR, which is the proportion of parents in the non-intending group who were correctly predicted by the developed model. The calibration of a model is defined as the degree of agreement between the predicted probabilities and observed outcomes (30). In the current study, the overall calibration and cross-country calibration of the model were good. This implies that the absolute expected outcome matches the observed risks. This finding is evident in the calibration plot of the entire dataset and at a country-based level.

Despite significant differences between countries in terms of mothers’ level of education, maternal age, place of residence, parents’ influenza vaccination, childbirth order, child status of having a chronic disease, received the influenza vaccine in the last year, the child received routine vaccination, and intention to receive influenza vaccination, PACV demonstrated good performance in diagnosing parental hesitancy about seasonal influenza vaccination. Accordingly, PACV can be used to identify VH among parents in different situations across different countries. Additionally, the current study emphasizes the importance of external validation of a newly designed diagnostic instrument before its inclusion in recommendations and clinical practice.

4.1 Strength and limitations

Although PACV has been widely used to assess parents’ attitudes toward different vaccines, its external validity has not been established until now. Therefore, our study is the first to evaluate the external validity of PACV among a large sample of the general population from six countries using both English and Arabic versions of the questionnaire.

However, this study had several limitations that should be considered in the future. First, the regression equation of the originally developed model was not available for use as a reference standard to compare our results. Therefore, we developed the model and tested its internal validity using the Egyptian dataset. Then, we used the developed equation to test the external validity of PACV on another larger dataset of six countries in the EMR. Second, the data were collected through an online survey. However, in most countries, web surveys have become the primary method of collecting data, surpassing face-to-face and computer-assisted telephone interviewing (CATI). This shift has been further accelerated by the COVID-19 pandemic, which has made traditional modes of data collection challenging. In fact, web surveys still encounter significant obstacles in obtaining probability-based samples that represent the general population, as a certain subset of the population with internet access and smartphones is targeted. However, according to 2022 statistics, a large population of the included countries had access to the Internet and were using social media platforms. For example, 94.5% of Egyptians had smartphones, nearly three-fourths of Egyptians had access to the Internet, and about 50% of them used social media platforms (31). Similar statistics have been reported in Iraq, where the number of social media users has increased by over 90%, as well as in Libya and Saudi Arabia (32). Third, the adopted web-based surveys rely on nonprobability survey designs, which are not considered the gold standard in survey sampling, unlike probability-based design. Nevertheless, nonprobability web surveys can still prove valuable in certain circumstances. For instance, nonprobability samples can help offset known biases in probability-based web survey samples by deliberately targeting underrepresented respondent profiles. In this study, we included a certain group of the population based on their representation (those living in mountains and deserts) to ensure the appropriate representation of all population categories. Additionally, to overcome the selection bias, we tried to include responses in proportion to each sector’s presentation despite using a non-probability sampling method. This is evident in Table 1, where there is nearly equal representation of different children age groups and parents’ working status in both the development and validation groups. Furthermore, we considered the population’s presentation based on their residence; most of respondents were living in urban areas, followed by rural areas, and then deserts and mountains. Finally, the cross-sectional survey itself has many inherent limitations, including the difficulty of determining whether the exposure or the outcome arrived first. Respondents’ recall and social acceptability biases are all examples of biases that might occur in the current study. Nonetheless, a cross-sectional design was the best design for addressing the study hypothesis.

5. Conclusion

Based on the findings of this study, the PACV model is a useful tool for assessing parental attitudes towards vaccination, regardless of the language used. The model has good discrimination and calibration, making it an effective tool for evaluating VH among parents in different countries. Policymakers and researchers can use the PACV model to assess and understand parental attitudes towards vaccination. Utilization of this tool can be extended to include pediatricians and other healthcare professionals. Identifying the determinants of parental attitudes is crucial, as it can help to increase vaccination acceptance and coverage by reducing VH. Therefore, we recommend the use of PACV as a valid tool to assess parents’ attitudes toward vaccination and to promote vaccination uptake among children.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by the Ethics Review Committee of Faculty of Medicine of Alexandria University (IRB no. 0305688). The patients/participants provided their written informed consent to participate in this study.

Author contributions

RG and NF: conceptualization and methodology. RG, AE, SA-R, SA, and NY: data analysis. RG, AE, SA-R, and SA: data curation. RG, ST, and SE: writing—original draft preparation. RG, ST, SE, NY, and NF: writing—review and editing. RG, SA, TA, and BK: visualization. All authors contributed to the article and approved the submitted version.

Acknowledgments

This study was supported by Princess Nourah bint Abdulrahman University Researchers Supporting Project number (PNURSP2023R177) Princess Nourah bint Abdulrahman University, Riyadh, Saudi Arabia.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as potential conflicts of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2023.1146792/full#supplementary-material

Abbreviations

AIC, Akaike Information criteria; AUC, Area under the curve; COPD, Chronic obstructive pulmonary disease; COVID-19, Coronavirus disease 2019; EMR, East Mediterranean Region; LOOCV, Leave-one-out cross-validation; PACV, Parental attitudes about childhood vaccines; PI, Prognostic index; ROC, Receiver operating characteristic; TNR, True-negative rate; TPR, True-positive rate; FPR, False-positive rate; VH, Vaccine hesitancy.

References

1. Tyrrell, CS, Allen, JLY, and Gkrania-Klotsas, E. Influenza: epidemiology and hospital management. Medicine (Baltimore). (2021) 49:797–804. doi: 10.1016/j.mpmed.2021.09.015

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Coleman, BL, Fadel, SA, Fitzpatrick, T, and Thomas, S-M. Risk factors for serious outcomes associated with influenza illness in high- versus low- and middle-income countries: systematic literature review and meta-analysis. Influenza Other Respir Viruses. (2018) 12:22–9. doi: 10.1111/irv.12504

PubMed Abstract | CrossRef Full Text | Google Scholar

3. World Health Organization (WHO) (2018). Influenza (Seasonal) Available at: https://www.who.int/news-room/fact-sheets/detail/influenza-(seasonal) (Accessed December 23, 2022).

Google Scholar

4. Influenza in children. Paediatr Child Health. (2005) 10:485–7. doi: 10.1093/pch/10.8.485

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Derdemezis, C, Markozannes, G, Rontogianni, MO, Trigki, M, Kanellopoulou, A, Papamichail, D, et al. Parental hesitancy towards the established childhood vaccination programmes in the COVID-19 era: assessing the drivers of a challenging public health concern. Vaccines. (2022) 10:814. doi: 10.3390/vaccines10050814

CrossRef Full Text | Google Scholar

6. Strelitz, B, Gritton, J, Klein, EJ, Bradford, MC, Follmer, K, Zerr, DM, et al. Parental vaccine hesitancy and acceptance of seasonal influenza vaccine in the pediatric emergency department. Vaccine. (2015) 33:1802–7. doi: 10.1016/j.vaccine.2015.02.034

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Centers for Disease Control and Prevention (2021). Prevention strategies for seasonal influenza in healthcare settings. Available at: https://www.cdc.gov/flu/professionals/infectioncontrol/healthcaresettings.htm (Accessed December 23, 2022).

Google Scholar

8. Centers for Disease Control and Prevention (2021). Influenza (Flu). Available at: https://www.cdc.gov/breastfeeding/breastfeeding-special-circumstances/maternal-or-infant-illnesses/influenza.html (Accessed March 17, 2023).

Google Scholar

9. Moriarty, LF, and Omer, SB. Infants and the seasonal influenza vaccine. Hum Vaccin Immunother. (2014) 10:2721–8. doi: 10.4161/hv.29669

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Abd Halim, H, Abdul-Razak, S, Md Yasin, M, and Isa, MR. Validation study of the parent attitudes about childhood vaccines (PACV) questionnaire: the Malay version. Hum Vaccin Immunother. (2020) 16:1040–9. doi: 10.1080/21645515.2019.1674112

PubMed Abstract | CrossRef Full Text | Google Scholar

11. World Health Organization (WHO) (2019). Ten threats to global health in 2019. Available at: https://www.who.int/news-room/spotlight/ten-threats-to-global-health-in-2019 (Accessed March 17, 2023).

Google Scholar

12. MacDonald, NE. Vaccine hesitancy: definition, scope and determinants. Vaccine. (2015) 33:4161–4. doi: 10.1016/j.vaccine.2015.04.036

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Opel, DJ, Taylor, JA, Mangione-Smith, R, Solomon, C, Zhao, C, Catz, S, et al. Validity and reliability of a survey to identify vaccine-hesitant parents. Vaccine. (2011) 29:6598–605. doi: 10.1016/j.vaccine.2011.06.115

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Elkhadry, SW, Salem, TAEH, Elshabrawy, A, Goda, SS, Al, BHA, Youssef, N, et al. COVID-19 vaccine hesitancy among parents of children with chronic liver diseases. Vaccine. (2022) 10:2094. doi: 10.3390/vaccines10122094

CrossRef Full Text | Google Scholar

15. ElSayed, DA, Bou Raad, E, Bekhit, SA, Sallam, M, Ibrahim, NM, Soliman, S, et al. Validation and cultural adaptation of the parent attitudes about childhood vaccines (PACV) questionnaire in Arabic language widely spoken in a region with a high prevalence of COVID-19 vaccine hesitancy. Trop Med Infect Dis. (2022) 7:234. doi: 10.3390/tropicalmed7090234

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Moons, KGM, Kengne, AP, Grobbee, DE, Royston, P, Vergouwe, Y, Altman, DG, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart. (2012) 98:691–8. doi: 10.1136/heartjnl-2011-301247

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Wessler, BS, Nelson, J, Park, JG, McGinnes, H, Gulati, G, Brazil, R, et al. External validations of cardiovascular clinical prediction models: a large-scale review of the literature. Circ Cardiovasc Qual Outcomes. (2021) 14:e007858. doi: 10.1161/CIRCOUTCOMES.121.007858

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Debray, TPA, Vergouwe, Y, Koffijberg, H, Nieboer, D, Steyerberg, EW, and Moons, KGM. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol. (2015) 68:279–89. doi: 10.1016/j.jclinepi.2014.06.018

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Laas, E, Zacharopoulou, C, Montanari, G, Seracchioli, R, Abrão, MS, Bassi, MA, et al. External validation of the SF-36 quality-of-life questionnaire in Italian and Brazilian populations to select patients with colorectal endometriosis for surgery. J Minim Invasive Gynecol. (2015) 22:378–83. doi: 10.1016/j.jmig.2014.06.005

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Kotz, D, Nelemans, P, van Schayck, CP, and Wesseling, GJ. External validation of a COPD diagnostic questionnaire. Eur Respir J. (2008) 31:298–303. doi: 10.1183/09031936.00074307

CrossRef Full Text | Google Scholar

21. Ramspek, CL, Jager, KJ, Dekker, FW, Zoccali, C, and van Diepen, M. External validation of prognostic models: what, why, how, when and where? Clin Kidney J. (2021) 14:49–58. doi: 10.1093/ckj/sfaa188

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Royston, P, Moons, KGM, Altman, DG, and Vergouwe, Y. Prognosis and prognostic research: developing a prognostic model. BMJ. (2009) 338:b604. doi: 10.1136/bmj.b604

CrossRef Full Text | Google Scholar

23. Rouquette, A, and Falissard, B. Sample size requirements for the internal validation of psychiatric scales. Int J Methods Psychiatr Res. (2011) 20:235–49. doi: 10.1002/mpr.352

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Van Calster, B, Nieboer, D, Vergouwe, Y, De Cock, B, Pencina, MJ, and Steyerberg, EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol. (2016) 74:167–76. doi: 10.1016/j.jclinepi.2015.12.005

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Collins, GS, Ogundimu, EO, and Altman, DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med. (2016) 35:214–26. doi: 10.1002/sim.6787

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Sohil, F, Sohali, MU, and Shabbir, J. An introduction to statistical learning with applications in R. Stat Theory Relat Fields. (2022) 6:87–7. doi: 10.1080/24754269.2021.1980261

CrossRef Full Text | Google Scholar

27. Fadl, N, Al Awaidy, ST, Elshabrawy, A, Makhlouf, MSAH, Ibrahim, SA, Abdel-Rahman, S, et al. Determinants of parental seasonal influenza vaccine hesitancy in the Eastern Mediterranean region: A cross-sectional study. Front. Public Health. (2023) 11. doi: 10.3389/fpubh.2023.1132798

CrossRef Full Text | Google Scholar

28. Siontis, GCM, and Ioannidis, JPA. Response to letter by Forike et al.: more rigorous, not less, external validation is needed. J Clin Epidemiol. (2016) 69:250–1. doi: 10.1016/j.jclinepi.2015.01.021

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Yu, AC, Mohajer, B, and Eng, J. External validation of deep learning algorithms for radiologic diagnosis: a systematic review. Radiol. Artif Intell. (2022) 4:1–9. doi: 10.1148/ryai.210064

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Steyerberg, EW. Clinical prediction models:A practical approach to development, validation, and updating. Cham: Springer International Publishing (2019).

Google Scholar

31. NAOS Solutions (2022). Egyptians and digital: 2022 report. Available at: https://naos-solutions.com/digital-egypt-blog/egyptians-and-digital-uses/ (Accessed March 17, 2023).

Google Scholar

32. Datareportal (2022). DIGITAL 2022: LIBYA. Available at: https://datareportal.com/reports/digital-2022-libya (Accessed March 17, 2023).

Google Scholar