HLS19-DIGI - a new instrument for measuring digital health literacy: development, validation and associations with determinants and health outcomes in 13 countries

Introduction: Digital health information sources are playing an increasingly prominent role in health promotion, public health and in healthcare systems. Consequently, digital health literacy skills are likewise becoming increasingly important.

Methods: Using a concept validation approach, the aim of the study was to validate a digital health literacy measure applied in the European Health Literacy Survey 2019–2021 (HLS₁₉) of the WHO M-POHL Network, analyzing data from 28,057 respondents from 13 European countries.

Results: The scale displayed high internal consistency. Confirmatory factor analysis (CFA) strengthened the hypothesized one-factor structure. In most countries, the data displayed acceptable fit to the unidimensional Rasch partial credit model (PCM). Pearson correlation with a measure of general health literacy showed sufficient discriminant validity, and a social gradient was found. Testing for predictive validity showed that the scale score predicts health-related outcomes.

Discussion: The study shows that considerable proportions of the general adult populations across countries in Europe have limited DHL skills. The level of DHL has direct potential consequences for some forms of health service utilization, in some countries. Implications of the study include recommendations for improving digital health literacy, promoting organizational health literacy and quality assurance for digital health information and resources.

1 Introduction

The increasing availability and use of health-related digital resources such as electronic health records, telehealth initiatives, digital health applications, and interactive communication options with public health agencies and health care providers (e.g., for making appointments or reporting medical/test results), places increasing demands on peoples’ abilities to use these applications and resources. Health care organizations and governments often promote or initiate and develop digital resources that require users to access, understand, appraise, and apply health information, as well as specific skills such as navigating digital health information. Simultaneously, the process of digitalization has enhanced the amount of online health information, as well as the number and variety of channels that are used for disseminating this information. More recently, artificial intelligence (AI) has entered the digital arena with tools such as ChatGPT and others. Due to the ubiquitous nature of digital communication, commercial companies and individuals are also seeking the public’s attention through digital channels, including social media. As a result, more interest-driven, manipulative, or simply false information is disseminated to the public, requiring particularly critical and analytical skills from the public and individual users (1).

It is therefore incumbent upon researchers, practitioners, and policy makers to recognize and realize the importance of understanding and improving people’s proficiency in using digital resources for managing their health conditions and for promoting their health. Concerns have been raised about whether increased digitalization of health services will help make health resources available for more people, or whether it will increase health inequalities by creating a digital divide (2). To answer this question, national population surveys should include validated measures of digital health literacy.

As a relatively new concept, digital health literacy (DHL) is distinct from digital literacy (3) in that it refers to a person’s specific ability to process health-related information and re-sources obtained from digital sources. It is also distinct from eHealth literacy. While eHealth and digital health are often used interchangeably and are closely related, eHealth focuses mainly on healthcare, while DHL is broader and also refers to mHealth (mobile), artificial intelligence (AI) and other emerging areas of innovation and information technology (4).

1.1 Existing research and measures of digital health literacy

The need to assess people’s use of electronic sources to access and process health-related information was first recognized by Norman and Skinner (5), who proposed the term eHealth literacy to refer to “the ability to seek, find, understand, appraise health information from electronic sources and apply the knowledge gained to addressing or solving a health problem.” In accordance with this definition, Norman and Skinner developed the eHealth Literacy Scale (eHEALS), a self-report measure assessing a person’s skills to find and evaluate health information on the Internet, which at the time was the primary online source for health information. Subsequently, eHealth literacy has been commonly assessed in health-related studies (6), and, together with media health literacy, has been conceptualized as the interplay of personal, situational, and contextual factors in the processing of health information from online sources and their interactions (2). A plethora of studies have examined media health literacy and eHealth literacy throughout the lifespan, from childhood and adolescence (7–9) through adulthood and among the older adult (10, 11). Additionally, it has been examined with respect to cultural transition (12), to specific health conditions (13), and to specific health behaviors (14), although studies on performance-based eHealth literacy are still scarce (15, 16). Empirically, general health literacy and eHealth literacy were found to be distinct but related concepts (17).

Yet, the rapid development of digital health sources beyond the Internet warrants new thinking about the definition and measurement of DHL regarding the use and application of digital health resources at the population level. While eHEALS has been used extensively to assess the self-reported consultation of online health information, the past decade has seen a proliferation of digital health information. This has broadened the potential for applying digital tools for health, but also the scope of digital skills needed to use these tools. These skills are at the core of the DHL concept. A recent study refers to DHL as the individual, social, technical, critical, and analytic skills that are required for searching, finding, understanding, evaluating, and applying digitally available health information (18). According to Bittlingmayer et al. (19), DHL refers not only to navigation in the digital space and to the use of digital information sources and their evaluation (e.g., health apps, social media, information sites on the Internet), but also to the individual “option spaces” that arise with increasing digitalization and the related availability of health information.

To capture these skills, a more comprehensive self-report scale (DHLI), which also addresses the interactive task of adding content, was developed, and validated in the Netherlands (16). While this scale captures various aspects of DHL, it has not been applied or field-tested in large samples, except for its use in an international cross-sectional survey among university students in the specific context of the COVID-19 pandemic (20).

1.2 Rationale for developing a new digital health literacy measure

Valid and reliable knowledge about people’s DHL can help health authorities to discern the experiences and difficulties of patients and of the wider population, including the extent to which they perceive health information provided through digital sources as apparent, accessible, understandable, correct, and applicable. Yet, while DHL has been assessed in some countries in the European Region (21, 22), these assessments mainly used the eHEALS or similar instruments. As such, they have not looked at DHL in a broader societal context, nor provided relevant associations with other aspects of health literacy or examined DHL in a broader international context. Apart from national studies conducted as part of the M-POHL Health Literacy Survey 2019–2021 (HLS₁₉) (23–25), no national health literacy survey to date has assessed and compared DHL to general health literacy on an international level. Moreover, there are no studies that have examined factors and covariates associated with DHL as determinants or its association with health promoting behavior, healthy lifestyles, early detection of diseases, use of health services, or self-care in the case of long-term illness.

The European Health Literacy Survey (HLS₁₉) under the auspices of the WHO Action Network on Measuring Population and Organizational Health Literacy (M-POHL) (26) offered an opportunity to develop and validate a new measure for DHL, to report on DHL at the general population level, and to study its association with general health literacy, with sociodemographic, socioeconomic, and other variables and with possible health-related consequences and outcomes.

This article is part of a series of papers, introducing new health literacy tools that have been developed, applied, and tested through the HLS₁₉ study (27–30). The aim of this series is to use the data collected in HLS₁₉ to examine the psychometric properties of the newly developed health literacy tools and different aspects of their validity. In order to derive overarching and comparable conclusions about the HLS₁₉ tools, the articles in this series address similar research questions and use the same analytical procedures.

2 Materials and methods

2.1 Development of the HLS₁₉-DIGI instrument

The DHL instrument developed for the HLS₁₉ survey, named HLS₁₉-DIGI, is based on the DHLI measure (16) but aligned more strongly with the concept and model of general health literacy proposed by the HLS-EU study (27) and promoted by M-POHL. This model defines health literacy as the ability to access, understand, appraise, evaluate, and apply health information. Compared to the DHLI, the HLS₁₉-DIGI adds the dimension of understanding digitally accessed health information and eliminates redundancy on the topic of applying health information. In addition, the scope and diversity of digital health resources was broadened to include use and interaction with social media, health apps, wearables, personal health records, and interaction with healthcare providers. Thus, the HLS₁₉ concept of DHL for promoting health includes the ability to search for, access, understand, appraise, validate, and apply online health information, and the ability to formulate and express questions, opinions, thoughts, or feelings when using digital devices.

HLS₁₉-DIGI consists of 10 items measuring two aspects of DHL: (1) the ability to search for, access, understand, appraise, and apply online health information measured by 8 items, namely HLS₁₉-DIGI-HI e.g.: “How easy or difficult is it for you to use the right search terms and queries to find the health information you are looking for?,” or “…to judge whether the information you find online is reliable?”(Figure 1); and (2) the ability to clearly formulate questions, opinions, thoughts, or feelings when interacting by typing or posting information on a digital device (measured by 2 items, namely HLS₁₉-DIGI-INT: “How easy or difficult is it for you to clearly formulate your written message when communicating with a health provider,” and “… to express your opinion, thoughts or feelings, ask a question in posting on social media including online forums”). In alignment with the format of other scales of the HLS₁₉ survey, respondents are asked to rate each item on a four-point rating scale ranging from 4 = very easy to 1 = very difficult where a higher score indicates higher DHL.

Figure 1

Figure 1. Percentage of respondents who responded "very difficult" or "difficult" to the items measuring the ability to access, understand, appraise and apply health information from digital sources (ranked by the overall percentage), for each country.

Six additional items assess the frequency with which one uses various digital sources of health information, including websites, social media, digital devices related to health or health care, or digital interaction with the health system (such as online video consultations, digital personal health records and prescriptions, health related apps on mobile phone, etc.). For each of these sources, respondents are asked to indicate how many days, in a typical week, they use them for obtaining health-related information, with response categories ranging from “less than once per week,” “1–3 days per week,” “4–6 days per week,” “once a day,” to “more than once per day.” A mean score ranging from 1 (“less than once per week” or not relevant) to 5 (“more than once per day”) is calculated as a relative measure for the frequency of use of health-related digital resources.

We have adapted, shortened and extended the content of the DHLI tool, translated it into several languages and tested it, and this paper reports on the psychometric testing. The questionnaire was developed in English, and subsequently translated into 12 languages (Arabic, Czech, Danish, Dutch, French, German, Hebrew, Hungarian, Norwegian, Portuguese, Russian and Slovakian) by the HLS₁₉ national teams and data collection agencies. Two forward translations were performed by 10 countries [Austria, Belgium (Dutch translation), Switzerland (German translation), Germany, Denmark, Hungary, Italy, Norway, and Slovakia], and one forward translation into Czech and Portuguese. Back translation was performed for five languages (Czech, Hebrew, Arabic, Russian and Norwegian). Countries with common languages collaborated in the translation process. For the German version, one forward translation was conducted by the national researchers of the Austrian, German and Swiss team, and one by the German national data collection agency, after which both versions were compared, and consensus was reached. For the French version, a translation into French was made by the language service of the Swiss Federal Office of Public Health (FOPH), reviewed by different experts, and consent reached between the Swiss, French and Belgian teams.

2.2 Data collection

Data on DHL were collected as national surveys for HLS₁₉ in Austria (AT), Belgium (BE), Czech Republic (CZ), Denmark (DK), France (FR), Germany (DE), Hungary (HU), Ireland (IE), Israel (IL), Norway (NO), Portugal (PT), Slovakia (SK) and Switzerland (CH), resulting in a total of 28,057 respondents aged 18 and over, based on multi-stage random sampling or quota sampling procedures. Country specific sample sizes ranged from 1,000 to 4,487. Data collection involved a variety of methods and sampling procedures, including computer-assisted personal interviewing (CAPI), pen-and-paper personal interviewing (PAPI), computer-assisted telephone interviewing (CATI), and computer-assisted web interviewing (CAWI) (Table 1). Some countries applied different data collection methods for different subpopulations. Participation in a CAWI interview requires some familiarity with digital media and was used exclusively in BE, DK, and FR. In CH, CZ, and IL data were collected through CAWI for part of the sample, complemented by other methods.

Table 1

Table 1. Data collection characteristics for digital health literacy by country.

The HLS₁₉ study protocol required national samples to be weighted by gender, age groups, population density, and geographical areas/units, based on national census data to increase representativeness.

In addition to the 10 items of the HLS₁₉-DIGI and the six additional items on the use of digital health information sources, data on the following sociodemographic and socioeconomic variables were collected: gender, age (in years), self-perceived social status (from 1 = “lowest self-perceived social status” to 10 = “highest self-perceived social status”) (27), the highest level of completed education (lower secondary or less: ISCED 0–2; higher secondary: ISCED 3; above secondary: ISCED 4–8), employment status (employed and unemployed or retired), and financial deprivation (“difficulty to pay all bills at the end of the month,” scored on a scale from 4 = “very easy” to 1 = “very difficult”) and whether or not the respondent has training in a health profession. To test for differential item functioning (DIF), we dichotomized variables on education level (ISCED 0–3 and 4–8) and social status (levels 1–4 and levels 5–10), and age categories were computed. More detailed information can be seen in Supplementary Table 1. Respondents were also asked to rate their self-reported general health on a scale ranging from very good to very bad, and to report limitations due to health problems (“severely limited”, “limited but not severely”, “not limited at all”), the existence of a long-term/chronic illness (lasting for 6 months or more: more than one, one, no) and their use of healthcare services (number of visits to GP/family doctor and number of specialist consultations in the past year) (13). General health literacy (GEN-HL) was measured, using the HLS₁₉-Q12 self-assessment scale (26), which is a 12-item revised short version of the HLS-EU-Q47 (27), providing for a comprehensive operationalization of health literacy by measuring the perceived difficulty to access, understand, appraise and apply health information in three health domains (healthcare, prevention and health promotion). A four-point rating scale (4 = “very easy,” 3 = “easy,” 2 = “difficult,” and 1 = “very difficult”) was offered for the HLS₁₉-Q12.

2.3 Data analysis

In a first step, perceived difficulties to access, understand, appraise, and apply online health information were analyzed for all countries by calculating the percentage of respondents who answered “very difficult” or “difficult” for each item. The scores of HL measures were calculated based on dichotomized as well as polytomous answer categories. To compensate for the potential resulting loss of information, item sets of categorical variables are also described by the average percentage response patterns (APRP), or the average percentage of how often a response category was selected within an item set. Given a data matrix consisting of m categorical items with k identical response categories for n respondents, an APRP is a compositional measure consisting of k percentages that describe for each of the k response categories how often the n respondents selected the respective category on average when answering the m categorical items (26). APRP were calculated separately for the eight items measuring the ability to access, understand, appraise, and apply online health information, and for the two items measuring the ability to interact with digital health resources.

Post-stratification weights were applied to the univariate, bivariate, and regression analyses described below. The base weights were calculated by the national teams and differ depending on the survey procedure. For country-by-country analyses, the effective weights were scaled so that their sum equals the number of valid cases in a country dataset. For analyses across all participating countries, the weights are rescaled so that the sum of weights by country equals 1,000, i.e., the countries have equal weight.

Next, scale scores measuring each of the two dimensions of the HLS₁₉-DIGI were calculated by combining the respondents’ replies to the eight items related to accessing, understanding, appraising, and applying health information from digital sources HLS₁₉-DIGI-HI, and separately the two items measuring the difficulty to interact with digital sources HLS₁₉-DIGI-INT. Following Pelikan et al. (27), two types of scores are calculated and compared with respect to their performance. First, the “d-type score,” which is based on a count of the dichotomized items obtained by combining the “easy” and “very easy” and “difficult” and “very difficult” categories. The advantage of dichotomization is that it reduces the potential inequalities due to extreme responses in subpopulation which could benefit for example comparisons within countries or international comparisons, besides being easier to communicate the meaning of the score. Yet the disadvantage of dichotomization may be the loss of information. In order to address this issue, “p-type” scores were also calculated, as the mean of the numeric values ranging from 1 to 4. Both scores are scaled to the range of 0 to 100. The higher the score, the higher the level of DHL. Scores were computed only for respondents who had answered at least 80% of the items.

Spearman correlations were calculated among the eight items measuring the first DHL dimension (HLS₁₉-DIGI-HI = i.e., the ability to search for, access, understand, appraise, and apply online health information), for the dichotomized items for all countries. Internal consistencies of the scales were tested for all countries using the Cronbach alpha coefficient, and the unidimensionality of the scales was tested using confirmative factor analysis (CFA) and Rasch analyses, based only on the polytomous items, respectively. For the latter, the data per country were tested against the unidimensional Rasch Partial Credit Model (PCM) (28). For each country, overall data-model fit, single item fit, the ordering of response categories, response dependency, one-dimensionality, and differential item functioning (DIF) were evaluated using an amended sample size of n = 480, corresponding to 20 respondents for each of 24 thresholds or n = 240 (10 respondents per threshold) for certain countries (28). These analyses were only performed for the first dimension (eight-item) of the tool (HLS₁₉-DIGI-HI), as it would not be appropriate to perform them on the two-item scale measuring the difficulty of interacting with digital sources (HLS₁₉-DIGI-INT). A detailed description of the methodological approach can be found in Griese et al. (29) and Finbraten et al. (30).

Finally, to investigate the determinants and health outcomes of DHL, the association of the HLS₁₉-DIGI-HI subscale with sociodemographic and socioeconomic variables, general health literacy as measured by the HLS₁₉-Q12, perceived health status and health care use was assessed by calculating Spearman’s ρ correlations and multiple linear regression analysis. Variables included in the regression models were: age, education (ISCED 0–8), self-perceived social status (1–10), financial deprivation, self-reported general health status, GP visits and specialist consultations.

3 Results

3.1 Use of digital information sources for health

Table 2 shows the proportion of responses, for all countries combined, regarding the use of different digital sources to obtain health information. Among those using digital resources for health, on average, websites are the most frequently consulted digital source on health information, followed by social media, digital interaction with the health system, a digital device related to health or health care, a health app on a mobile phone, and other.

Table 2

Table 2. Use of different digital resources to obtain health information (% of respondents for all 13 countries combined).

3.2 Perceived difficulty to process digital health information

Figure 1 shows, for each item, the percentage of respondents who replied “very difficult” or “difficult” to the eight items measuring the ability to access, understand, appraise, and apply health information from digital sources, for each country (weighted equally). Percentage distributions for all four categories, for each item, are available in the Supplementary material of the International Report of the HLS₁₉ Consortium (26). Although there is a considerable variation in perceived difficulty across countries, the ranking of the combined two difficulty response categories shows similar patterns across countries, with few exceptions. The percentages of combined “difficult” and “very difficult” responses range from 21.8% for item 1 (“to use the proper words or search query to find the information you are looking for”) to 54.1% for item 4 (“to judge whether the information is reliable”). For both items, the percentage of respondents finding these tasks (very) difficult is the lowest in Norway (9.5 and 30.6%) and highest in Germany (38.5 and 82.6%, respectively).

Figures 2, 3 show the APRP for the two DHL dimensions of the HLS₁₉-DIGI (i.e., the ability to access, understand, appraise, and apply online health information, and to interact with digital devices related to health, respectively) for each country. On average, nearly 40% of the respondents across all countries reported difficulties in processing digital health information (i.e., they responded “difficult” or “very difficult”), with Norway having the lowest proportion (22%) and Germany the highest (58%). Figure 3 shows that, on average, more than a quarter of respondents (29%) reported difficulties interacting with digital devices for health, but again, the numbers vary by country. Respondents from Portugal reported the lowest difficulties (11%) and respondents from Slovakia the highest (48%). In contrast, 51% of respondents experience interacting with digital devices for health as easy, with scores varying between 35% in Ireland and 78% in Portugal, and 20% as very easy, with scores varying from 10% in Slovakia to 38% in Norway.

Figure 2

Figure 2. Average Percentage Response Patterns (APRP) for the item set measuring the ability to access, understand, appraise, and apply online health information, for each country (equally weighted).

Figure 3

Figure 3. Average Percentage Response Patterns (APRP) for item set measuring the ability to interact with digital health devices, for each country (equally weighted).

3.3 Psychometric properties of the HLS₁₉-DIGI scales

Spearman correlations among the eight items measuring the HLS₁₉-DIGI-HI dimension of the HLS₁₉-DIGI (i.e., the ability to access, understand, appraise, and apply online health information) ranged between 0.37 and 0.66 for all countries combined. The same levels of correlation are found for individual countries (26).

Cronbach’s alpha (α) for the HLS₁₉-DIGI-HI items are above 0.70 for all countries, with an average α of 0.83 for the dichotomized items and 0.89 for the polytomous items (Table 3), indicating a reliable scale. The two HLS₁₉-DIGI-INT items were highly correlated.

Table 3

Table 3. Internal consistencies (Cronbach alpha) and fit indices for the one-factor confirmatory factor model for HL-DIGI-8 items, for each country and mean of all countries (equally weighted).

On average, the HLS₁₉-DIGI-HI and DIGI-INT scores are moderately correlated (with a Pearson correlation coefficient of r = 0.48 for the d-type score and r = 0.55 for the p-type score), but the level of correlation differs considerably across countries (from 0.21 to 0.60 for the d-type score and 0.24 to 0.68 for the p-type score).

Table 3 shows the fit indices for the single-factor confirmatory factor model for the HLS₁₉-DIGI-HI scale. The indices generally indicate an acceptable fit for all countries, although the standardized root mean square residual (SRMSR, assuming a 0.08 threshold value) is too high in three countries and the lower bound of the confidence interval of the root mean square error of approximation (RMSEA, with 0.05 as threshold) is too high in all countries, suggesting a possible misfit between the observed data and the single-factor model. However, the other fit indices (comparative fit index, Tucker-Lewis index, goodness of fit and adjusted goodness of fit index) overall indicate a sufficient fit between the observed covariance matrix and the model implied covariance matrix for all countries.

Using an amended sample size of n = 480 the data showed an overall sufficient fit to the Rasch model (χ² statistic) for data collected in Austria, Germany, Ireland, Norway, and Switzerland. We observed acceptable fit for the data collected in Belgium and in the Czech Republic, and, with a further reduced sample size of n = 240, also for the data collected in Denmark, Hungary, Israel, and Portugal. A principal component analysis (PCA) of Rasch model residuals, combined with dependent t-tests to identify possible subscales, revealed that the HLS₁₉-DIGI-HI scale could be considered sufficiently unidimensional for most countries. The HLS₁₉-DIGI-HI scale was also considered well-targeted in most countries. The thresholds, and thus the response categories, were ordered and functioned well.

There is no significant statistical dependency between pairs of items, which means that no items are “too similar” and therefore redundant. However, item 5 “to judge whether health-related information is offered with commercial interests” tends to under-discriminate in most countries and thus does not strictly correspond to the latent trait that is underlying the scale. Some items displayed DIF for person factors, such as age and gender, but there was no consistent pattern across countries. Furthermore, the scale did not measure invariantly across countries, as the item threshold locations or item “difficulty order” varied between countries. The latter may be ascribed to DIF for country or language.

3.4 Content and concurrent validity of the HLS₁₉-DIGI scales

Both scales of the HLS₁₉-DIGI are supported by a theory-based model that justifies the selection of items for the measures. Table 4 shows the distribution of the DHL score measuring the ability to access, understand, appraise, and apply digital health information for each country. For the d-type score, the mean score across all countries (equally weighted) is 62.3 (on a scale from 0 to 100), varying between 41.8 (DE) and 78.7 (NO). For most countries, the distribution is left skewed with the 75% percentile starting at the maximum value, indicating a ceiling effect for the measure. For the p-type score, the mean score across all countries (equally weighted) is 56.6 (on a scale from 0–100), varying between 44.1 (DE) and 68.6 (NO). The p-scores are less skewed than the d-scores.

Table 4

Table 4. Mean, median, quartiles, and standard deviation of the HLS₁₉-DIGI-HI score, for each country and for all countries (equally weighted).

In terms of concurrent validity, Pearson correlations between the ability to access, understand, appraise, and apply digital health information measured by the HLS₁₉-DIGI-HI and general health literacy measured by the HLS₁₉-Q12 varies, for the d-type scores, between 0.44 (BE) and 0.67 (IL), with a mean of 0.53. For the p-type scores, the correlation coefficients range from 0.52 (BE, CH, HU) to 0.72 (IL) with a mean of 0.58. This confirms the assumption that both concepts are strongly related, but also sufficiently distinct to warrant separate measures. Across countries, the Spearman correlation between the scale and a score for the use of digital resources is ρ = 0.15, ranging from 0.04 (for BE) to 0.31 (for DE). So, for most countries, the perceived ease of using digital health information sources is moderately related to the use of such sources.

3.5 Determinants of DHL

Correlational analyses indicate that in most HLS₁₉ countries the score on the HLS₁₉-DIGI-HI scale measuring the ability to access, understand, appraise and apply digital health information is moderately correlated with having had training in a health profession (Spearman ρ = 0.14 for all countries, varying between 0.03 for AT and 0.18 for SK). The same applies to self-perceived social status in society (ρ = 0.13, ranging from 0.00 in AT to 0.25 in SK), age (ρ = −0.11, ranging from 0.03 in IL to −0.25 in DE), and education level (ρ = 0.07, varying from 0.03 in the CZ to 0.31 in SK). The relationship with education is more or less linear, meaning that the score on the HLS₁₉-DIGI-HI scale increases with educational level. In contrast, the correlation with gender (overall ρ = 0.01) is very low for all countries.

Supplementary Table 2 shows the deviation from the population mean d-score for the scale measuring the ability to access, understand, appraise, and apply digital health information for selected subpopulations that are potentially vulnerable or disadvantaged regarding digital health literacy. Notably, older people show, on average, the largest deviation from the general population (−12), followed by those with a (very) bad self-perceived health (−11.4), financial deprivation (−8.7), low education (−7.8), and low social status (−6.8). Those who often visit their GP/family doctor also score markedly lower (−6.9). A somewhat lower level of deviation from the general population score is seen for people with limitations due to health problems (−4.7) and for those suffering from a long-term/chronic illness (−3.2). For most subpopulations the variation between countries is considerable.

To identify possible associations among the determinants of DHL, a series of regression models (Supplementary Table 3) were tested with the 8-item HLS₁₉-DIGI-HI scale as the dependent variable. A first model including the core social variables of age, gender, education level, self-perceived social status and financial deprivation as independent variables explains 6% of the variance in DHL for all countries weighted equally, with the explained variance varying from 2% (BE) to 23% (SK). The best predictor of HLS₁₉-DIGI-HI for all countries is financial deprivation (overall ß = −0.15; significant for 11 countries with ß values between −0.08 and − 0.27), followed by age (overall ß = −0.13; significant for 6 countries with ß values between −0.15 and − 0.26) and self-perceived social status (overall ß = 0.08; significant for 10 countries with ß values between 0.05 and 0.13). Education and gender have, on average, a smaller effect on the level of DHL, with lower overall ß values (0.03) and a significant effect in only a few countries. As such, the results suggest a social gradient for DHL, albeit with an inconsistent pattern across countries. A second model (not shown in the tables), in which use of digital resources is added as an independent variable, gives a small increase of the explained variance to 7% (ranging between 2% in BE and 24% in SK). Again, financial deprivation (overall ß = −0.14), age (overall ß = −0.12) and social status (overall ß = 0.08) are the strongest predictors, along with use of digital sources (overall ß = 0.11), while gender and education level have little influence (overall ß = 0.02).

3.6 Outcomes of DHL

The importance of DHL for health can be inferred from its relationship with a number of health outcomes. Significant Spearman correlations are found between the scores on the HLS₁₉-DIGI-HI scale measuring the ability to access, understand, appraise, and apply digital health information and self-perceived health (ρ = −0.18 for all countries, with ρ varying between −0.10 in IE and − 0.28 for DE), limitations due to illness or health problems (ρ = 0.18 for all countries), and long-term illness or health problems (ρ = −0.11 for all countries).

The contribution of DHL to health outcomes can be derived from a series of multiple regression analyses using the HLS₁₉-DIGI-HI score in combination with gender, age, education, self-perceived social status, and financial deprivation as independent variables (Supplementary Table 4). When self-perceived health is entered as the outcome variable, the model explains 15% of the variance for all countries combined (R² ranging between 10% in IE and 26% in SK). Although age (overall ß = 0.24, ranging from 0.07 to 0.38, significant for all countries except BE) and self-perceived social status (overall ß = −0.17, ranging from −0.29 to −0.09, significant for all countries) are stronger predictors of self-perceived health, DHL also significantly contributes to the regression in nine out of the 13 countries participating in the study. The overall ß = −0.10, with ß values for individual countries ranging from −0.05 to −0.14.

However, when more objective outcome measures are considered such as the reported number of GP/family doctor visits or specialist consultations, DHL becomes less important as a determinant. For GP/family doctor’s visits, the model explains 6% of the variance for all countries combined (R² ranging between 4% in IL and 14% in DE). On par with financial deprivation (average ß = 0.08, significant for 10 countries with ß between −0.01 and 0.17), DHL is the third most important contributor to the regression model, with ß values ranging from −0.01 to −0.10 (overall ß = −0.08). For specialist consultations, the variance explained by the model is only 5% (R² ranging between 1 and 11%), and DHL’s contribution to the regression is only significant in SK (ß = −0.08) and an overall ß of −0.07. Thus, the level of DHL has direct potential consequences for some forms of health service utilization, in some countries.

4 Discussion

DHL is gaining momentum in public health and health service research, policy and practice. The rapid development of digital health information sources and resources beyond the Internet has widened the potential for the digitalization of health services, but also created a need to address the necessary digital skills of individuals to use these resources. Aligned with these developments, DHL refers to the individual, social, technical, critical, and analytic skills that a person needs in order to find, access, understand, evaluate, and apply digitally available health information (18). While existing research on DHL tends to take a narrow view of the topic, limiting it to finding and evaluating health-related information on the Internet, we detected a need to examine the subject more broadly in terms of the skills needed to navigate the digital space, access and use digital information sources such as health apps including wearables, social media, information websites, evaluate them, and take decisions on one’s health based on the options offered. A systematic bibliometric analysis on DHL did not take into consideration the validation of psychometrics and the sub-skills embedded in the term DHL (31).

This article described the conceptual background, development, and validation of a new instrument to measure DHL at the population level and investigated its determinants and associations with health outcomes. The instrument, named HLS₁₉-DIGI, builds on the conceptual model, definition and dimensions of general health literacy proposed by the HLS-EU consortium (32) to measure the perceived ability of respondents to find, understand, appraise, and apply specifically digitally available health related information, and interact with digital information sources. Its validation is based on data of the European Health Literacy Survey 2019–2021 (HLS₁₉) undertaken by the WHO Action Network on Measuring Population and Organizational Health Literacy (M-POHL) (26), involving approximately 28,000 participants in 13 countries in measuring DHL.

The instrument consists of 10 items measuring two aspects of DHL: the ability to access, understand, appraise/validate, and apply online health information (HLS₁₉-DIGI-HI), and the ability to clearly formulate questions, opinions, thoughts, or feelings when interacting by typing or posting information on a digital device (HLS₁₉-DIGI-INT). With regard to the first aspect, our results demonstrate that the 8-item scale measuring this dimension is sufficiently robust and valid. The internal consistency, confirmatory factor analysis and Rasch analysis confirmed its unidimensional structure for the data of all countries. The reliability of the scale, based on Cronbach’s alpha, is sufficiently high, with values ranging from 0.77–0.87 for the d-score and 0.84–0.92 for the p-score. The thresholds, and thus the response categories, are ordered and well-functioning. The correlation with general health literacy is high but not extreme (r = 0.55 for p-type scores and 0.48 for d-type scores), suggesting that both scales measure parts of the same construct, but are independent enough to be treated as different entities. One item tends to discriminate somewhat poorly across countries and some items display DIF for country or language, which could limit international comparisons. Future research could investigate whether minor changes to items like, e.g., 6 “to visit different websites to check whether they provide similar information about a topic” or 5 “to judge whether the information is offered with commercial interests” (Figure 1) could improve the model fit. Also, in some countries, based on the d-type score the scale displays a ceiling effect, with more than 25% scoring the highest value.

Regarding the second component (HLS₁₉-DIGI-INT), the two items measuring difficulties in clearly formulating questions, opinions, thoughts or feelings when posting information on a digital device, are highly correlated, and, in some countries, do not differentiate well, suggesting that the items should be adjusted. The two items are, on average, moderately correlated (d-type score r = 0.48 and p-type score r = 0.55), but this differs considerably between countries. In the next M-POHL Health Literacy Survey 2024–2026 (HLS₂₄), a refined measure may be considered for this aspect of DHL.

Using the HLS₁₉-DIGI-HI, it was possible to document the perceived difficulty to process digital health information at the population level. Thus, it was seen that a significant part of the population – between 22 and 58%, depending on the country – finds it difficult to find, access, understand, evaluate, and apply digitally available health information. On average, 8% of the respondents (percentages ranging from 2 to 18% for the different countries) found it very difficult, and another 25% found it difficult (percentages ranging from 20 to 40%).

The most difficult tasks are to judge/appraise whether the information is offered with commercial interests (item 5), or whether information is reliable (item 4). Using proper words or search queries to find information one is seeking or visiting different websites to check whether they provide similar information is considered less difficult. This resonates with results from other (health) literacy research that with the ubiquitous stream of information in today’s society, the challenge is less to find information, but rather to discern whether it is correct, complete, and useful, aligned with the concept of critical health literacy (33).

In terms of its determinants, a social gradient for DHL is seen for all countries, but to a considerably different degree for different countries and indicators. The strongest social predictors of DHL are, on average, financial deprivation, age, and self-perceived social status in society. Perceived social status has acknowledged as a contribution to other aspects in a population’s social gradient (34). Education and gender are less significant, and only in some countries. Regarding education, ISCED levels were handled as a continuous variable. Different ISCED levels refer to different intervals of “years of education,” and these intervals also differ somewhat across countries. Thus, in future studies the measure may best be continuous such as number of years in an educational setting. General health literacy is a very strong predictor of DHL for all countries, and use of digital resources for most countries. Older people, people with poor health, and people with a low education level are most probable to have low DHL skills, which confirms the findings of other studies (16). This finding supports the conclusion of studies that have stressed the importance of interventions that address DHL in socially disadvantaged groups (35).

Regarding its importance for health and health care use, it was seen that DHL correlates with self-perceived health and limitations due to long-term illness, and that in most countries it is a determinant of self-perceived health along with age and self-perceived social status, but that its impact on healthcare use (i.e., GP visits and specialist consultations) is generally less strong.

Future analyses based on the study can examine the association between DHL and health behavior indicators measured in HLS₁₉ (smoking, physical activity, BMI, alcohol, fruit and vegetable consumption).

4.1 Limitations

Differences in sampling and data collection limit the extent of country comparisons. Web-based surveys may have biased the results toward over-reporting of high DHL. For some countries, the non-response rates for several items were markedly higher than for other HLS₁₉ measures. this may be partly due to the fact that people cannot evaluate something they do not do in everyday life, namely if they do not use the Internet to search for health information. As a result, the results may not be representative of all country samples. For example, in Ireland (IE), the group with missing data have a higher mean age than the total sample and a lower mean level of education. Furthermore, some of the countries collected data prior to the COVID-19 pandemic, others during the pandemic and some following the major pandemic waves. This diversity may reduce the extent to which participants in different countries can be compared, since the pandemic may have influenced the extent to which populations became digitally health literate. Yet, the measures across countries seems to behave rather uniformly, despite different data collecting modes, and the general picture of associations between determinants and DHL and with outcome measures also suggest that our conclusions are well supported. As is the case in all cross-sectional surveys association between variables does not infer causation. The next cycle of the M-POHL HLS survey can consider ways to mitigate these limitations.

5 Conclusions and implications

A compact, conceptually sound new instrument to measure DHL was validated for 13 languages in 13 countries, showing acceptable psychometric properties. The study shows that considerable proportions of the general adult populations across 13 countries in Europe have limited DHL skills, that the level of DHL shows a social gradient, and that it is associated with indicators of health status. Professionals and policy makers in public health, health promotion and healthcare services should recognize the difficulties that large groups of society, especially the most vulnerable, have in accessing, understanding, appraising, validating and applying health information from digital sources and interacting with digital services. This challenge will become more critical as healthcare systems continue to transition into digital avenues of communication and information sharing with patients and communities (36). Addressing these difficulties should include organizational health literacy (OHL) action such as quality assurance of digital health resources (37), promoting critical DHL skills and expanding research to include populations from all ages in the life course. Finally, as artificial intelligence emerges and becomes more accessible for use in healthcare and public health (38), DHL skills will become increasingly important among the population-at-large to leave no one behind.

Author’s note

The authors are part of the WHO Action Network on Measuring Population and Organizational Health Literacy (M-POHL Network). The M-POHL Network is initiator of the HLS₁₉.

Data availability statement

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found at: https://m-pohl.net/Design_Methods.

Ethics statement

In each country, it has been ensured that ethical requirements have been met in accordance with the country/institutional ethics regulation and requirements. For more information about ethical considerations, data protection, and informed consent by country, please see the International Report on the methodology, results, and recommendations of the European Health Literacy Population Survey 2019–2021 (HLS19) of M-POHL (26).

Author contributions

DL-Z: Conceptualization, Formal analysis, Funding acquisition, Investigation, Methodology, Supervision, Validation, Writing – original draft, Writing – review & editing, Project administration. SB: Conceptualization, Data curation, Funding acquisition, Investigation, Methodology, Writing – original draft, Writing – review & editing, Formal analysis. ÉB: Writing – original draft, Writing – review & editing, Investigation, Methodology. HB: Writing – original draft, Writing – review & editing, Investigation, Methodology. LB: Writing – original draft, Writing – review & editing. SMG: Writing – original draft, Writing – review & editing, Investigation, Methodology. HS: Writing – original draft, Writing – review & editing, Formal analysis, Investigation, Methodology. SG: Writing – original draft, Writing – review & editing, Investigation. RG: Conceptualization, Data curation, Formal analysis, Methodology, Writing – original draft, Writing – review & editing. LG: Writing – original draft, Writing – review & editing, Methodology, Investigation. ØG: Formal analysis, Methodology, Writing – original draft, Writing – review & editing. ZKl: Writing – original draft, Writing – review & editing, Investigation, Methodology. ZKu: Writing – original draft, Writing – review & editing, Investigation, Methodology. CL: Writing – original draft, Writing – review & editing, Formal analysis, Investigation, Methodology. TL: Writing – original draft, Writing – review & editing, Conceptualization, Data curation, Formal analysis, Investigation, Methodology. JM: Writing – original draft, Writing – review & editing, Investigation, Methodology. DM: Writing – original draft, Writing – review & editing, Investigation, Methodology. DS: Writing – original draft, Writing – review & editing, Investigation, Methodology, Supervision. CR: Writing – original draft, Writing – review & editing, Investigation. KS: Writing – original draft, Writing – review & editing, Conceptualization, Investigation, Methodology. CS: Methodology, Writing – original draft, Writing – review & editing, Conceptualization, Funding acquisition, Investigation, Supervision. MT: Writing – original draft, Writing – review & editing, Investigation, Methodology. MV: Writing – original draft, Writing – review & editing, Investigation, Methodology. JP: Conceptualization, Writing – original draft.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. This research received no external funding, but the data collection was supported either by ministries of health, university, public health institutes, or insurance funds in the different countries. AT: The Austrian Health Literacy Survey was commissioned and financed by the Austrian Federal Health Agency and the Federation of Austrian Social Insurance Institutions. BE: Data collection for the Belgian study was funded by the MLOZ Health Insurance Company. BG: Medical University—Sofia, Faculty of Public Health and AGREEMENT No. D-125/24.06.2020, PROJECT No. 8418/21.11.2019, “GRANT-2020,” MU-Sofia. CH The Swiss Federal Office of Public Health FOPH with a financial contribution from the foundation Health Promotion Switzerland. CZ: Data col-lection was funded by (all seven) Czech health insurance funds. DE: The German study was funded by the German Federal Ministry of Health, grant number: Kapitel 1504 Titel 54401, ZMV I 1–2518 004 (HLS-GER 2). DK: The distribution of the questionnaire was funded by Aalborg University. FR: The research was supported by the National Public Health agency (Santé Publique France, 21DPPA040-0) and by Ligue contre le cancer (LIGUE2019). HU: The research was support-d by the Ministry of Human Capacities, Hungary (IV/956–740 4/2020/EKF). The preparation of this paper was supported by the János Bolyai Research Scholarship of the Hungarian Academy of Sciences (BO/00933/22/5). IE: Department of Health, Dublin. IL: The Israel study was funded by the Israel Ministry of Health, Clalit Health Services and the University of Haifa Israel. IT: The Italian participation to the Action Network on Measuring Population and Organizational Health Literacy (M-POHL Network) was funded by the Istituto Superiore di Sanità, Rome, Italy; the implementation of the Health Literacy Survey-HLS₁₉ in Italy was funded by the National Centre for Disease Prevention and Control (CCM) of the Italian Ministry of Health and coordinated by the Department of cardiovascular, endocrine–metabolic diseases and aging of the Istituto Superiore di Sanità, Rome, Italy. NO: The Norwegian HLS₁₉ was commissioned and financed by the Norwegian Ministry of Health and Care Services. The Norwegian Directorate of Health funded the data collection and the administrative costs for the whole project, while Oslo Metropolitan University and Inland Norway University of Applied Sciences contributed with the scientific workforce. PT: Direção-Geral da Saúde, Lisbon. RU: The Russian study was supported by the Ministry of Health of the Russian Federation and WHO Europe. SI: The national survey of Health literacy in Slovenia took place within the framework of the project increasing health literacy in Slovenia-ZaPiS, which is co-financed by the Republic of Slovenia to the amount of 20% of the value and the European Union from the European Social Fund to the amount of 80% of the value. SK: Public Health Authority of the Slovak Republic, Bratislava.

Acknowledgments

This article is dedicated to the loving memory of Professor Jürgen Pelikan, who initially conceptualized and founded the WHO Action Network on Measuring Population and Organizational Health Literacy (M-POHL) and was a driving force in promoting this study and publication. The authors would like to acknowledge The International Coordination Centre (ICC) of the HLS₁₉ Project for project coordination. We also want to thank all national HLS₁₉ team members who contributed to the HLS₁₉ Project by contributing to the chapter on Digital Health Literacy in the International Report or/and providing country data for analyses.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2025.1472706/full#supplementary-material

References

1. Paige, S, Stellson, M, Krieger, JL, Anderson-Lewis, C, Cheong, JW, and Stopka, C. Proposing a transactional model of eHealth literacy: concept analysis. J Med Internet Res. (2018) 20:e10175. doi: 10.2196/10175

PubMed Abstract | Crossref Full Text | Google Scholar

2. Levin-Zamir, D, and Bertschi, I. Media health literacy, eHealth literacy, and the role of the social environment in context. Int J Environ Res Public Health. (2018) 15:1643. doi: 10.3390/ijerph15081643

PubMed Abstract | Crossref Full Text | Google Scholar

3. Visser, M. Digital literacy and public policy through the library lens. Maine Policy Rev. (2013) 22:104–13. doi: 10.53558/WCUZ5068

Crossref Full Text | Google Scholar

4. World Health Organization. Digital health resolution. Geneva: World Health Assembly (2018).

Google Scholar

5. Norman, CD, and Skinner, HA. eHEALS: the eHealth literacy scale. J Med Internet Res. (2006) 8:e27. doi: 10.2196/jmir.8.4.e27

PubMed Abstract | Crossref Full Text | Google Scholar

6. Karnoe, A, and Kayser, L. How is eHealth literacy measured and what do the measure-ments tell us? A systematic review. Knowl Manag E-Learn. (2015) 7:576–600.

Google Scholar

7. Faux-Nightingale, A, Philp, F, Chadwick, D, Singh, B, and Pandyan, A. Available tools to evaluate digital health literacy and engagement with eHealth resources: a scoping review. Heliyon. (2022) 8:e10380. doi: 10.1016/j.heliyon.2022.e10380

PubMed Abstract | Crossref Full Text | Google Scholar

8. Livingstone, S, Lemish, D, Lim, SS, Bulger, M, Cabello, P, Claro, M, et al. Global perspectives on children’s digital opportunities: an emerging research and policy agenda. Pediatrics. (2017) 140:S137–41. doi: 10.1542/peds.2016-1758S

PubMed Abstract | Crossref Full Text | Google Scholar

9. Shu-Ching, Y, Yi-Fang, L, Chia-Hsun, C, et al. The associations among individual factors, eHealth literacy, and health-promoting lifestyles among college students. J Med Internet Res. (2017) 19:e15. doi: 10.2196/jmir.5964

PubMed Abstract | Crossref Full Text | Google Scholar

10. Leung, A, Ko, P, Chan, KS, Chi, I, and Chow, N. Searching health information via the web: Hong Kong Chinese older adults' experience. Public Health Nurs. (2007) 24:169–75. doi: 10.1111/j.1525-1446.2007.00621.x

PubMed Abstract | Crossref Full Text | Google Scholar

11. Choi, NG, and Dinitto, DM. The digital divide among low-income homebound older adults: internet use patterns, eHealth literacy, and attitudes toward computer/internet use. J Med Internet Res. (2013) 15:e93. doi: 10.2196/jmir.2645

PubMed Abstract | Crossref Full Text | Google Scholar

12. Levin-Zamir, D, Leung, YMA, Dodson, S, Rowlands, G, Peres, F, Uwamahoro, N, et al. Health literacy in selected populations: individuals, families, and communities from the international and cultural perspective. Inf Serv Use. (2017) 37:131–51. doi: 10.3233/ISU-170834

Crossref Full Text | Google Scholar

13. Aaby, A, Friis, K, Christensen, B, Rowlands, G, and Maindal, HT. Health literacy is associated with health behaviour and self-reported health: a large population-based study in individuals with cardiovascular disease. Eur J Prev Cardiol. (2017) 24:1880–8. doi: 10.1177/2047487317729538

PubMed Abstract | Crossref Full Text | Google Scholar

14. Aharony, N, and Goldman, R. E-health literacy and the vaccination dilemma: an Israeli perspective. Inf Res. (2017) 22:751.

Google Scholar

15. Neter, E, and Brainin, E. eHealth literacy: extending the digital divide to the realm of health information. J Med Internet Res. (2012) 14:e19. doi: 10.2196/jmir.1619

PubMed Abstract | Crossref Full Text | Google Scholar

16. van der Vaart, R, and Drossaert, C. Development of the digital health literacy instrument: measuring a broad spectrum of health 1.0 and health 2.0 skills. J Med Internet Res. (2017) 19:e27. doi: 10.2196/jmir.6709

PubMed Abstract | Crossref Full Text | Google Scholar

17. Neter, E, Brainin, E, and Baron-Epel, O. Group differences in health literacy are ameliorated in ehealth literacy. Health Psychol Behav Med. (2021) 9:480–97. doi: 10.1080/21642850.2021.1926256

PubMed Abstract | Crossref Full Text | Google Scholar

18. Kolpatzik, K, Mohrmann, M, and Zeeb, H. Digitale Gesundheitskompetenz in Deutschland. Berlin: KomPart (2020).

Google Scholar

19. Bittlingmayer, UH, Dadaczynski, K, Sahrai, D, van den Broucke, S, and Okan, O. Digital health literacy—conceptual contextualization, measurement, and promotion. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. (2020) 63:176–84. doi: 10.1007/s00103-019-03087-6

PubMed Abstract | Crossref Full Text | Google Scholar

20. Dadaczynski, K, Okan, O, Messer, M, Leung, AY, Rosário, R, Darlington, E, et al. Digital health literacy and web-based information-seeking behaviors of university students in Germany during the COVID-19 pandemic: cross-sectional survey study. J Med Internet Res. (2021) 23:e24097. doi: 10.2196/24097

PubMed Abstract | Crossref Full Text | Google Scholar

21. Zrubka, Z, Brito Fernandes, Ó, Baji, P, Hajdu, O, Kovacs, L, Kringos, D, et al. Exploring eHealth literacy and patient-reported experiences with outpatient care in the Hungarian general adult population: cross-sectional study. J Med Internet Res. (2020) 22:e19013. doi: 10.2196/19013

PubMed Abstract | Crossref Full Text | Google Scholar

22. European Commission. (2014) “European citizens' digital health literacy” Flash Eurobarometer 404 Report. Available at: http://publications.europa.eu/resource/cellar/fd42f9e7-937c-41f3-bf03-4221b2db712b.0001.04/DOC_1 (Accessed July 1, 2024).

Google Scholar

23. Schaeffer, D, Berens, EM, Vogt, D, Gille, S, Griese, L, Klinger, J, et al. Health literacy in Germany – findings of a representative follow-up survey. Dtsch Arztebl Int. (2021) 118:723–9. doi: 10.3238/arztebl.m2021.0310

Crossref Full Text | Google Scholar

24. Le, C, Finbråten, HS, Pettersen, KS, Joranger, P, and Guttersrud, Ø. Befolkningens helsekompetanse, del I the International Health Literacy Population Survey 2019–2021. Oslo, Norway: The Norwegian Directorate of Health (2021).

Google Scholar

25. Levin-Zamir, D, Laron, M, Artom, T, and Baron-Epel, O. Health literacy in Israel – HLS19 results of National Survey 2019-2021. Israel: Health Literacy Survey (2023).

Google Scholar

26. The HLS19 Consortium of the WHO Action Network M-POHL. International report on the methodology, results, and recommendations of the European health literacy population survey 2019–2021 (HLS19) of M-POHL. Vienna: Austrian National Public Health Institute (2021).

Google Scholar

27. Pelikan, JM, Link, T, Straßmayr, C, Waldherr, K, Alfers, T, Bøggild, H, et al. Measuring comprehensive, general health literacy in the general adult population: the development and validation of the HLS19-Q12 instrument in seventeen countries. Int J Environ Res Public Health. (2022) 19:14129. doi: 10.3390/ijerph192114129

PubMed Abstract | Crossref Full Text | Google Scholar

28. Guttersrud, Ø, Le, C, Pettersen, KS, and Finbråten, HS. Rasch analyses of data collected in 17 countries - a technical report to support decision-making within the M-POHL consortium. Vienna, Austria: International HLS19 (2023).

Google Scholar

29. Griese, L, Finbråten, HS, Francisco, R, De Gani, SM, Griebler, R, Guttersrud, Ø, et al. HLS19-NAV—validation of a new instrument measuring navigational health literacy in eight European countries. Int J Environ Res Public Health. (2022) 19:13863. doi: 10.3390/ijerph192113863

PubMed Abstract | Crossref Full Text | Google Scholar

30. Finbråten, HS, Nowak, P, Griebler, R, Bíró, É, Vrdelja, M, Charafeddine, R, et al. The HLS19-COM-P, a new instrument for measuring communicative health literacy in interaction with physicians: development and validation in nine European countries. Int J Environ Res Public Health. (2022) 19:11592. doi: 10.3390/ijerph191811592

PubMed Abstract | Crossref Full Text | Google Scholar

31. Yang, K, Hu, Y, and Qi, H. Digital health literacy: bibliometric analysis. J Med Internet Res. (2022) 24:e35816. doi: 10.2196/35816

PubMed Abstract | Crossref Full Text | Google Scholar

32. Sørensen, K, Van den Broucke, S, Fullam, J, Doyle, G, Pelikan, J, Slonska, Z, et al. Health literacy and public health: a systematic review and integration of definitions and models. BMC Public Health. (2012) 12:1–13. doi: 10.1186/1471-2458-12-80

PubMed Abstract | Crossref Full Text | Google Scholar

33. Abel, T, and McQueen, D. Critical health literacy and the COVID-19 crisis. Health Promot Int. (2020) 35:1612–3. doi: 10.1093/heapro/daaa040

PubMed Abstract | Crossref Full Text | Google Scholar

34. Adler, NE, Epel, ES, Castellazzo, G, and Ickovics, JR. Relationship of subjective and objective social status with psychological and physiological functioning: preliminary data in healthy white women. Health Psychol. (2000) 19:586–92. doi: 10.1037/0278-6133.19.6.586

PubMed Abstract | Crossref Full Text | Google Scholar

35. Cheng, C, Beauchamp, A, Elsworth, GR, and Osborne, RH. Applying the electronic health literacy lens: systematic review of electronic health interventions targeted at socially disadvantaged groups. J Med Internet Res. (2020) 22:e18476. doi: 10.2196/18476

PubMed Abstract | Crossref Full Text | Google Scholar

36. Palumbo, R, Nicola, C, and Adinolfi, P. Addressing health literacy in the digital domain: insights from a literature review. Kybernetes. (2022) 51:82–97. doi: 10.1108/K-07-2021-0547

Crossref Full Text | Google Scholar

37. Sentell, T, Foss-Durant, A, Patil, U, Taira, D, Paasche-Orlow, MK, and Trinacty, CM. Organizational health literacy: opportunities for patient-centered care in the wake of COVID-19. Qual Manag Health Care. (2021) 30:49-60.60. doi: 10.1097/QMH.0000000000000279

PubMed Abstract | Crossref Full Text | Google Scholar

38. Nutbeam, D, and Lloyd, JE. Understanding and responding to health literacy as a social determinant of health. Annu Rev Public Health. (2021) 42:159–73. doi: 10.1146/annurev-publhealth-090419-102529

Crossref Full Text | Google Scholar