Katherine Jennifer Kelly1*
Alyson Campbell2Anja Salijevic1
Sarah Doak1Laurie Michael1
William Montelpare1- 1Health Research Network, Department of Applied Human Sciences, University of Prince Edward Island, Charlottetown, PE, Canada
- 2Faculty of Nursing, University of Prince Edward Island, Charlottetown, PE, Canada
Introduction: Effective communication of COVID-19 information involves clear messaging to ensure that readers comprehend and can easily apply behavioral recommendations. This study evaluated the readability, understandability, and actionability of public health resources produced by the four provincial governments in Atlantic Canada (New Brunswick, Newfoundland and Labrador, Nova Scotia, and Prince Edward Island).
Methods: A total of 400 web-based resources were extracted in June 2022 and evaluated using the Flesch-Kincaid Grade Level, CDC Clear Communication Index, and the Patient and Education Materials Assessment Tool. Descriptive statistics and a comparison of mean scores were conducted across provinces and type of resources (e.g., text, video).
Results: Overall, readability of resources across the region exceeded recommendations, requiring an average Grade 11 reading level. Videos and short form communication resources, including infographics, were the most understandable and actionable. Mean scores across provinces differed significantly on each tool; Newfoundland and Labrador produced materials that were most readable, understandable, and actionable, followed by New Brunswick.
Discussion: Recommendations on improving clarity of COVID-19 resources are described. Careful consideration in the development of publicly available resources is necessary in supporting COVID-19 knowledge uptake, while reducing the prevalence of misinformation.
1. Introduction
Since the onset of the COVID-19 pandemic, there has been an immeasurable need for scientific information related to virus transmission and prevention. The path of this information from research to policy to practice has proceeded under the attentive eye of the public. Communicating best practices about COVID-19, including the importance of handwashing, vaccination, and wearing masks, has been vital to disease prevention and protecting individuals who are considered more vulnerable to severe outcomes (Basu and Dutta, 2008; Ng et al., 2022). However, misunderstandings and barriers to evidence-based information have, in part, led to an info-demic scattered with misinformation (Solomon et al., 2020).
Ensuring the accessibility of COVID-19 public health messaging requires consideration of the general public's ability to “access, understand, and use information to promote and maintain good health”, otherwise known as health literacy (Nutbeam, 2008). General guidance on the recommended reading level of publicly available health information suggests that material should not exceed a grade 5–8 reading level, or that of an 11- to 12-year old (The National Work Group on Literacy Health, 1998; Cotugna et al., 2005). Various tools exist to assess the readability of written materials, including (but not limited to) the Flesch-Kincaid Grade Level (Kincaid et al., 1975) and the Simple Measure of Gobbledygook (Fitzsimmons et al., 2010).
Beyond readability, factors related to health literacy include the ability to understand and actionize information. Understandability refers to the degree to which health information can be easily comprehended. Actionability of materials emphasizes one's ability to identify what to do with information and how to execute recommended behaviors (Agency for Healthcare Research Quality, 2020, p. 1).
Social distancing and isolation measures throughout the pandemic led to a reliance on the Internet and other virtual media modalities (e.g., television) to convey evidence-based messaging about pandemic measures. Recent analyses of materials from national (Worrall et al., 2020) and regional (Khan et al., 2020; Mani et al., 2021) health authorities, such as the US Centers for Disease Control (Bothun et al., 2022), suggest that much of the available information on COVID-19 is unreadable to the average person. This accessibility barrier has significant implications for equity, particularly with regards to the disproportionate rate of severe outcomes from COVID-19 in Black, Indigenous, and other minority ethnicities (Khan et al., 2020).
Misinformation related to COVID-19 is considered an important factor in the perpetuation of vaccine hesitancy, which is the delay or refusal to become vaccinated (MacDonald and SAGE Working Group on Vaccine Hesitancy, 2015). Inappropriate readability of health information material may be an influential factor in vaccine hesitancy (Mani et al., 2021; Okuhara et al., 2022), as material requiring an advanced reading level could be misconstrued (Szmuda et al., 2020). High levels of health literacy are positively associated with vaccine confidence (Duong et al., 2021), such that low health literacy may predict vaccine hesitancy (Lorini et al., 2018), and may be a mediating factor between health system distrust and COVID-19 vaccine uptake (Turhan et al., 2021). As a result, improving health literacy, such as through increased access to clear, evidence-based information, may improve uptake of the COVID-19 vaccine (Zhang et al., 2021).
Observations that publicly available COVID-19 information is unclear and exceeds appropriate readability levels have been made in public health departments in the United States (Mani et al., 2021), government and community organizations across English speaking countries (Worrall et al., 2020); and international health authorities, such as the World Health Organization (Yeung et al., 2022). This lack of clarity and accessibility impacts effective information dissemination to the public. Despite the inclusion of Canada in a combined international assessment of content, to our knowledge, the evaluation of material from Canadian provincial governments has not been investigated.
We chose to focus our evaluation of COVID-19 information within the four provinces in Atlantic Canada: Nova Scotia (NS), New Brunswick (NB), Newfoundland and Labrador (NL), and Prince Edward Island (PEI). The extent to which materials produced by Canadian government entities is readable, understandable, and actionable is currently unknown. This region comprises two islands (NL and PEI) that successfully limited inter-provincial travel during the pandemic, requiring residents to frequently consult provincial government materials for unique guidance on leaving and re-entering each province. At the height of travel restrictions that required special permission to cross provincial boundaries, government officials arranged an “Atlantic bubble” that allowed free movement between the four Atlantic provinces. However, residents were required to apply for passes to move within this “bubble”; thus, provincial government websites were the primary source of information for Atlantic Canadians. Analysis of the readability of these websites is important in gauging the suitability of information to Atlantic Canadians during the pandemic.
This study aims to systematically extract and assess COVID-19 related public health information created and disseminated by provincial government entities in Atlantic Canada. The following three questions guided this research. (1) What is the readability of COVID-19-related public health information created by each of the provincial governments in Atlantic Canada? (2) To what extent is public health information on COVID-19 understandable and actionable? (3) Are there measurable differences in the readability, understandability, and actionability of materials according to material type and targeted audience?
2. Materials and methods
2.1. Data extraction and evaluation
We reviewed the publicly available resources for COVID-19 information that was created by each provincial government in the four provinces of Atlantic Canada. The research methodology followed the approach used by Mani et al. (2021) to evaluate COVID-19 information distributed in the United States.
A data extraction tool was developed and piloted using Microsoft Excel. We consulted with an expert librarian in health research to develop our search strategy. Two reviewers (SD and KK) systematically searched for information related to COVID-19 on the provincial government and health authority websites for each province. Reviewers followed a process of identifying the COVID-19 information section of each website, including the Frequently Asked Questions (FAQ) or Questions and Answers (Q&A) sections. Specifically, reviewers extracted all resources available on the main COVID-19 hubs of each website and continued to extract items from links within the website until a maximum of 100 resources were procured. Three independent searches using the terms “COVID,” “COVID-19,” and “coronavirus” were also conducted using the websites' internal search bars to ensure that all relevant information was captured. Identified resources were evaluated against an inclusion and exclusion criteria. We considered any type of resource, including webpages, infographics, and videos that primarily focus on topics related to COVID-19 and were provided in English. Resources were excluded if the central focus was not on COVID-19 or in a language other than English. We only included resources that were produced by provincial entities in the four Canadian provinces; links to information outside of these government websites (e.g., to the Public Health Agency of Canada) were not included. A maximum of 100 items were extracted from each provincial government and/or health authority website due to resource limitations.
Reviewers cataloged resources in an audit form using Microsoft Excel and assigned an identification number to each item. We extracted the following information from identified resources (Supplementary material 1): Name of source, date of resource publication; province; date publication update (if applicable), title, topic, hyperlink, type of source (e.g., print and web, infographic, video, etc.), and primary audience (e.g., parents/guardians, general public).
2.2. Scoring and interpretation
We analyzed written and printable resources using three validated health literacy tools: the Flesch-Kincaid Grade Level (FKGL) (Kincaid et al., 1975); the CDC Clear Communication Index (CDC Index) (Centers for Disease Control Prevention, 2021); and the Patient Education Materials Assessment Tool (PEMAT) (Agency for Healthcare Research Quality, 2020). Four independent reviewers (KK, AS, SD, LM) applied these tools to the information resources identified during data extraction. Specifically, two reviewers applied the FKGL to the material (KK, AS), two reviewers applied the PEMAT (KK, SD), and two reviewers applied the CDC Index (AS, LM). A pilot test of the first 20 items from each province, using each of the three tools, was conducted to ensure calibration between the reviewers. Inter-rater reliability was calculated during the pilot test using the intraclass correlation coefficient (ICC) to evaluate the measure of agreement between the two evaluators (Winer, 1971). This analysis resulted in an ICC coefficient of 0.96 indicating excellent agreement (Koo and Li, 2016). Disagreements were resolved through discussion or consultation with a fifth reviewer (WM).
2.2.1. Readability
Readability was evaluated using the FKGL, which provides a measure of readability in the form of the estimated grade level required to understand material (Kincaid et al., 1975). The formula used to calculate the score is as followed: (0.39 × total words/total sentences) + (11.8 × total syllables/total words) – 15.59. Scores under 8, denoting a grade 8 reading level, are considered universally accessible with regards to readability (The National Work Group on Literacy Health, 1998; Worrall et al., 2020).
The FKGL was assessed using a built-in tool on Microsoft Word. Written content from included materials were copied to a Microsoft Word document and analyzed. As the FKGL is based on length of syllables and considers length of sentences in its marker of readability (Kincaid et al., 1975), special care was made to ensure the preservation of the original material's integrity when copying from its original source to a Microsoft Word document.
2.2.2. Understandability and actionability
The CDC Index and PEMAT were used to evaluate the understandability of materials, and the PEMAT was used to determine actionability. The CDC Index is a research-based tool that evaluates the clarity and understandability of communication materials (Centers for Disease Control Prevention, 2021) and the PEMAT assesses the understandability and actionability of health-related materials (Agency for Healthcare Research Quality, 2020). We applied the Full CDC Index to print and web-based materials (e.g., webpages) and the Modified CDC Index to videos and short text-based items (e.g., infographics).
The CDC Index assesses items across seven sections: (1) main message and call to action; (2) language; (3) information design; (4) state of the science; (5) behavioral recommendations; (6) numbers; and (7) risk. Questions across each section are answered by either yes or no and scored by 1 (yes) or 0 (no). For example: “Does the material contain at least one visual that conveys or supports the main message?” Scores from individual sections are summed into a final score that is divided by 11 and multiplied by 100. Total scores above 90% indicate that the material is easy to understand and use; scores at or below 89% suggest that information resources should be revised according to items that scored 0 (Centers for Disease Control Prevention, 2021).
The PEMAT is a guide that assesses understandability and actionability of patient education materials (Agency for Healthcare Research Quality, 2020). We applied the PEMAT for Printable Materials (PEMAT-P) to text-based items (e.g., webpages) and the PEMAT for Audiovisual Materials (PEMAT-A/V) to audiovisual items (i.e., videos). The PEMAT-P (Shoemaker et al., 2014) has 26 questions separated into two sections: understandability and actionability. Similar to the CDC Index, the understandability section is further sub-divided into 6 sub-sections: (1) content; (2) word choice and style, (3) use of numbers, (4) organization; (5) layout and design; and (6) use of visual aids. Actionability is assessed by questions such as, “The material breaks down any action into manageable, explicit steps”. The PEMAT-A/V (Shoemaker et al., 2014) follows a similar structure with two main sections for understandability and actionability, where the former is further divided into 5 sections: (1) content; (2) word choice and style; (3) organization; (4) layout and design; and (5) use of visual aids. Questions in the PEMAT follow the form of statements which are scored using 0 (disagree), 1 (agree), or N/A (not applicable) scheme. Scores are tallied across sections into two final scores, each for understandability and actionability. Higher PEMAT scores indicate more understandable or actionable material; the PEMAT does not provide thresholds in scores for understandability or actionability.
2.3. Statistical analysis
Data were analyzed using SAS Studio Education Analytics Suite (SAS, 2013). Descriptive statistics were first calculated for each evaluation test (FKGL, CDC Index, and PEMATu for understandability and PEMATa for actionability), across (1) the four Atlantic provinces, (2) resource type (web or print, infographic, video, pamphlet, or poster), and (3) target audience (e.g., the general public, parents/teachers). The calculations included means ± standard deviation, confidence intervals, and frequency distributions. Next, a comparison of mean scores for each of the test responses was compared between the provinces, and on selected types of resources (e.g., web or print), and the audience which received the information (e.g., general public), using a general linear model ANOVA with post-hoc least squares means tests.
Bland-Altman plots were used to show agreement between scores reported for the CDC Index, a measure of a reader's level of understanding, and the PEMAT score for understandability, within each province. Bland-Altman is a statistical approach to quantifying bias and agreement of continuous variables (Altman and Bland, 1983).
3. Results
Data reported in this study are based on an initial screening of 400 public health resources published between March 2020 and June 2022. The information was extracted between June 15 and June 27, 2022 (see Supplementary material 2). Items were scored using the FKGL, CDC Index, and PEMAT. A total of N = 390 resources were screened across the four provinces. Ninety-six items were screened for the province of NS and for NL, while 99 items were screened for the provinces of NB and for PEI. The 10 remaining resources were lost as a result of broken internet links between the data extraction and the evaluation stages.
A review of resource type was based on the total of 390 items across the four Atlantic provinces. The results showed that most resources were print and web-based (N = 274), followed by videos (N = 44), posters (N = 38), infographics (N = 28), and pamphlets (N = 8). Figure 1 presents the distribution of resource type by province.
Figure 1. Material type organized by province.
Ten target audiences were identified across the four provinces: general public (N = 351), parents, guardians, or teachers (N = 10), domestic travelers (N = 8), current or recently discharged hospital patients (N = 5), pregnant or lactating individuals (N = 4), children or youth (N = 3), domestic employers (N = 3), health care professionals (N = 2), seniors (N = 2), and informal caregivers (N = 2).
3.1. Readability
The FKGL test, which provides an estimate of readability, was applied to 345 print and web resources, posters, infographics, and pamphlets. Across the provinces, the average readability of resources were 11.26 ± 3.69. Print and web-based resources scored the highest in readability (12.18 ± 3.24), followed by pamphlets (9.78 ± 0.87), posters (7.87 ± 3.53), and infographics (6.05 ± 3.86). With regards to target audience, the readability of resources for the three largest categories were 11.33 ± 3.72 (general public); 8.34 ± 2.26 (parents, guardians, or teachers); and 13.88 ± 1.78 (domestic travelers).
The average FKGL score for print and web-based resources showed that material from PEI yielded the highest score for readability (12.43 ± 2.61) while the province of NL had the lowest average score (10.15 ± 3.48) (see Table 1). FKGL scores were significant across provinces, f(3, 344)= 6.44, p < 0.01. Post-hoc analysis revealed that the average score for resources from PEI (12.43 ± 2.61) were significantly higher than NB (10.64 ± 3.61, p < 0.01) and NL (10.15 ± 3.48, p < 0.01), but not NS (11.84 ± 5.07, p = 0.3, ns). Similarly, scores from NS were statistically different from NB (p < 0.05) and NL (p < 0.01), but not PEI. The observed difference in scores between NB and NL were not statistically significant (p = 0.4).
Table 1. Mean readability, understandability, and actionability score of COVID-19 public health material across the four Atlantic provinces.
3.2. Understandability
3.2.1. CDC Index
The CDC Index was applied to all 390 resources to provide a measure of understandability. The CDC Full Index was applied to print and web resources and pamphlets (N = 282), and the modified version of the CDC Index was applied to videos and short text-based resources, including posters and infographics (N = 108). The average score for videos was the highest on the CDC Index (69.53 ± 18.07%), followed by posters (69.12 ± 19.85%), infographics (60.36 ± 26.63%), pamphlets (58.30 ± 6.12%), and print and web-based resources (55.34 ± 17.07%).
Resources targeting children or youth (N = 3) scored highest for understandability on the CDC Index (78.67 ± 14.02%), followed by pregnant or lactating individuals (N = 4, 74.3 ± 19.82%), health care professionals (N = 3, 64.33 ± 8.39%), domestic employers (N = 3, 61.03 ±12.33%), domestic travelers (N = 8, 60.16 ± 19.50%), parents, guardians, and teachers (N = 20, 59.24 ± 19.65%), and then the general public (N = 337, 57.62 ± 20.10%).
The combined average score across provinces on the CDC Index was 58.03 ± 18.87%. Results from the ANOVA showed a significant difference in scores across provinces, f(3, 386) = 15.06, p < 0.01. Post-hoc analysis revealed that CDC Index scores of resource understandability from NL (65.08 ± 16.90%) were significantly higher than scores from PEI (60.79 ± 18.01%, p < 0.01); NS (47.42 ± 14.00%, p < 0.01); and NB (58.81 ± 20.26%), p < 0.05.
3.2.2. PEMAT: understandability
The PEMAT-P and PEMAT-A/V were applied to 389 resources, resulting in one score each for understandability and actionability. The PEMAT-P was applied to all text-based resources, including print and web resources, posters, infographics, and pamphlets (N = 345), and the PEMAT-A/V was applied to videos (N = 44). Posters scored the highest on the PEMAT for understandability (91.51 ± 9.44%), followed by infographics (88.96 ± 8.92%), videos (86.57 ± 11.31%), pamphlets (84.71 ± 2.43%), and print and web-based resources (71.94 ± 14.83).
Resources targeting seniors (N = 2) scored the highest for understandability on the PEMAT-U (93.8 ± 0%), followed by children and youth (N = 3, 81.82 ± 18.19%), domestic travelers (N = 8, 79.53 ± 9.54%), current or recently discharged hospital patients (N = 5, 79.15 ± 16.40%), health care professionals (N = 3, 76.91 ± 13.35%), pregnant or lactating individuals (N = 4, 76.91 ± 0.01%), and then the general public (N = 338, 76.40 ± 17.98%).
The average score of resources across all four provinces using the PEMAT-P or PEMAT-A/V for understandability was 59.91 ± 16.49%. Average scores categorized by province are presented in Table 1. Scores for the PEMAT differed significantly across provinces [f(3, 388) = 15.03, p < 0.01]. Post-hoc analysis showed that scores for understandability from PEI (68.63 ± 20.27%) were significantly lower than NB (80.22 ± 13.96%) and NL (83.43 ± 11.98), p < 0.01. Further, NS (73.60 ± 19.75%) were also significantly lower than NL (83.43 ± 11.98, p < 0.01).
3.2.3. PEMAT: actionability
The PEMAT-P and PEMAT-A/V were applied to resources to determine scores of actionability. Videos scored the highest on the PEMAT's score for actionability (75.35 ± 21.85%), followed by pamphlets (70.00 ± 4.71%), infographics (68.62 ± 26.58%), posters (63.02 ± 26.87%), and print and web-based resources (46.39 ± 25.54%).
Resources targeting seniors (N = 2) scored the highest for actionability on the PEMAT (100 ± 0%), followed by children and youth (N = 3, 75.56 ± 21.43%), health care professionals (N = 3, 66.67 ± 30.55%), informal caregivers (N = 2, 60 ± 0%), parents, guardians, and teachers (N = 21, 59.37 ± 26.49%), and the general public (N = 338, 51.85 ± 31.41%).
The average score of actionability across all resources and provinces was 52.40 ± 29.80%. Individual scores by province were significantly different, f(3, 388) = 8.13, p < 0.01. Post hoc analysis showed that actionability scores from NB (58.92 ± 32.28%) were significantly higher than PEI (46.73 ± 27.63%) and NS (43.29 ± 32.41%), p < 0.01. Moreover, actionability scores from NL (60.67 ± 26.89%) were significantly higher than PEI (46.73 ± 27.63%) and NS (43.29 ± 32.41%), however were not different from NB (58.92 ± 32.28%), p = 0.68, ns.
3.2.4. Agreement between the CDC Index and PEMAT
Agreement in scores of understandability using the CDC Index and the PEMAT score for understandability were evaluated using Bland-Altman plotting. Results from this analysis are presented in Figure 2. The results support a high degree of agreement as indicated by the cluster of plots within the 95% confidence range. The graph also reports an average difference between the percent scores from the CDC Index and PEMAT score for understandability s μ = 18.06 ± 20.72.
Figure 2. Bland-Altman plot for inter-rater agreement analysis (n = 389). Limits of agreement are shown as solid, green lines, and the average difference between means (as a dotted red line).
4. Discussion
Publicly available resources providing COVID-19 information were extracted from the websites of four provincial health authorities in Atlantic Canada. We applied three validated tools of health literacy (FKGL, CDC Index, and PEMAT) to the information presented by each province in order to estimate readability, understandability, and actionability of each resource.
Overall, the evaluated resources exceeded recommended readability levels, with PEI and NS scoring the lowest in measures of understandability and actionability. From the total items evaluated (N = 390), the majority were print and web-based resources (N = 274, 70%). Short form communication resources, including videos (N = 44), posters (N = 38), and infographics (N = 28), consisted of approximately 28% of all resources evaluated. Similarly, most resources targeted the general public (N = 351, 90%), followed by parents, guardians, and teachers (N = 10, 3%), and domestic travelers (N = 8, 2%). Scores of understandability and actionability were highest among resources targeting specific audiences, including children and youth, pregnant or lactating individuals, health care professionals, and seniors, in contrast to those broadly targeting the general public. Given the universal impact of the pandemic, and the imminent need to share COVID-19 information, it is not surprising that many resources were developed for a wide audience, rather than a specific population. However, as revealed in this study, the broader the audience to which the information is distributed, the greater likelihood is the risk to readability, understandability, and actionability.
Resources across the four provinces required an average reading level for grade 11. When separated by province, NL had the lowest readability score, requiring a reading level for grade 10, while PEI had the highest score, requiring a reading level for grade 12. Importantly, the average scores in readability across all four provinces exceeded the recommended threshold of grade 8, suggesting that resources observed in this study were not universally accessible from at a general literacy level (Worrall et al., 2020). These findings are consistent with previous studies investigating the readability of COVID-19 health information using the FKGL (Basch et al., 2020; Khan et al., 2020; Worrall et al., 2020; Valizadeh-Haghi et al., 2021).
COVID-19, as a communicable disease described by epidemiologists and public health officials, poses unique challenges because the terminology used to describe events associated with the disease is complex. Often the syntax of the information used to develop knowledge within society includes multisyllabic words (e.g., variant, incubation) that represent complex concepts (e.g., period of communicability, mode of transmission). This complexity undoubtedly influenced the increase in the reading level required for several of the reported resources. With regard to knowledge translation, involving consumers in the development of public health resources is crucial to ensuring that information is shared in a way that can be easily understood to enhance their uptake (Tugwell, 2007; Graham et al., 2013). Approximately 60% of Canadian adults do not possess adequate health literacy skills (Canadian Council on Learning, 2008); these rates may be disproportionately higher for racial and ethnic minorities, particularly in the context of language and culture (Hasnain-Wynia and Wolf, 2010). Resources observed in this study reflected the reading levels of high school (grades 10 to 12), however, Atlantic Canada has the lowest functional literacy levels across the country (Statistics Canada, 2012). Inclusion of consumers in the development of materials may promote a process conducive to ensuring equity, as well as clarity, understandability, and actionability of content across diverse populations.
Scores of understandability were measured using the CDC Index and PEMAT score for understandability. We evaluated the level of agreement between these two measures on the concept of understandability using a measure of agreement using the graphical approach suggested by Bland and Altman. Our findings illustrated the level of agreement between the scores on each of the CDC Index and PEMAT for understandability of the identified information resources. While the resources from NL scored the highest in understandability on both evaluation tools, none of the provinces met the criteria for a “passing” grade (i.e., over 90%) on the CDC Index.
With regard to interpretation of the PEMAT scores for understandability, these estimates are not compared to a threshold but rather between samples. Here we observed that the resources from NL (83%) demonstrated a higher level of understandability when compared to resources from NB (80%), NS (74%), or PEI (69%). Moreover, NL (61%) and NB (60%) had the highest scores for actionability on the PEMAT, and were statistically different from PEI (47%) and NS (43%). Together, these findings suggest that COVID-19 communication materials from NL and NB were more understandable and actionable than material from NS and PEI.
One explanation for these findings may be that NL and NB produced a greater variety of resources in comparison to the type of information resources produced by the provinces of PEI and NS (Table 1). Specifically while NL and NB produced a lower percentage of print and web-based resources when compared with the percentage volume produced by PEI and NS, the provinces of NL and NB produced more short text and video information resources, thereby adding to the variety of information from which the consumer could access. When separated by type of resource, short form communication resources, including infographics and posters scored at an appropriate level of readability (grades 6 and 8, respectively). Similarly, posters scored the highest on the PEMAT for understandability (92%), followed by infographics (89%), and videos (87%). Not surprisingly, given its emphasis on audiovisual communication, videos scored the highest on actionability (74%). In all cases, web and print-based resources scored the lowest in understandability and actionability. Taken together, these findings suggest that short-form and audiovisual materials, including infographics, posters, and videos, may be better suited to promoting the understandability and actionability of health information (Scott et al., 2009; Reid et al., 2017; Campbell et al., 2022).
In the current study, NL not only scored the highest on measures of readability, understandability, and actionability, but it is also that province that led the country in COVID-19 vaccine uptake, across all age groups (Government of Canada, 2022). The association between readability of disease and vaccine-related information and vaccine uptake is unclear (Okuhara et al., 2022); however, there is evidence that interventions aimed at improving vaccine literacy may impact the likelihood to participate in immunization programs (Owais et al., 2011; Zhou et al., 2020; Siddiqui et al., 2022). Reducing barriers to health information to promote equity across diverse groups within the population requires effective communication strategies using principles of health literacy (Logan et al., 2015). While beyond the scope of this study, communication of health information includes consideration of content accessibility, such as through validated closed captioning, cultural appropriateness, and accurate multi-language translations.
When there is a lack of understanding toward health conditions, or treatments, worse outcomes can occur, including prolonged illness and increased resource use (Taddio et al., 2013; Morrison et al., 2014; May et al., 2018; Cunningham et al., 2022). Poor understanding of health information can affect one's ability to act on behavioral recommendations and make informed decisions (Kutner et al., 2006). One way to mitigate this, is to have robust health information available on the internet, in a variety of formats that are easy to understand and based on evidence. However, more research is required to fully understand the most effective way to deliver health information, including information format, dissemination avenues and timing of delivery (Campbell, 2021). Improving the level of consumer knowledge of health conditions, like COVID-19, may encourage better health decisions, reduce stress and encourage productive conversations with health providers by prompting questions (Yardi et al., 2018; Cunningham et al., 2022). Furthermore, when clinicians are equipped with credible and effective sources of information that can be easily shared with patients and families, improved provider-patient relationships may develop. This is crucial during a time where the spread of misinformation is abundantly prevalent (Solomon et al., 2020).
Throughout the pandemic, the Atlantic region of Canada was hailed as a “success story” as a result of reporting so few COVID-19 cases in comparison to other jurisdictions (Wu and Mackenzie, 2021). Yet this success was not random. On the contrary, the lack of COVID cases was credited, in part, to strict travel limitations and inter-provincial collaboration between public health authorities (Cameron-Blake et al., 2021). Ever-shifting quarantine mandates and constant changes to gathering limits required residents to frequently consult public health resources on provincial government websites. Individuals that failed to adhere to public health measures faced large fines and in some cases police intervention. The intention of these measures was to reduce case numbers and protect the health care system; however, a consequence of distributing information using a rapid communication format was that the information may have been unclear, and less understandable by many individuals in Atlantic. These barriers to the understandability and actionability of resources can have a notable impact on underserved populations, who are often overlooked in health communication strategies (Kelley et al., 2015). Ensuring equitable access to health information begins during the development of resources.
4.1. Recommendations based on observations
The health literacy tools used in the current study evaluated COVID-19 resources on the readability, understandability, and actionability of content. Overall, the information resources identified in this study scored beyond the recommended upper threshold for appropriate levels of literacy within a general population. As such, the resources required necessary to ensure understandability and actionability. In this section, we provide recommendations to public health departments with regard to the development of publicly accessible health information based on our observations in the current study (see Table 2).
Table 2. Recommendations in the creation of public health materials.
Among the greatest impediments to readability of the resources identified in the present study was the overuse of acronyms, medical jargon, and passive voice. Acronyms reduce clarity of material and therefore should be avoided unless necessary. Medical terms should only be used in materials to familiarize the reader with an important concept or terminology (Centers for Disease Control Prevention, 2021), and always defined in plain language. For example, ambiguity regarding the similarities and differences between a “PCR test” and “rapid test”; while use of these terms is important to familiarize the public with multiple diagnostic tools, resources should be clear about the meaning of these terms and steps to actionize behaviors associated with their use. Finally, using an active voice provides a clear and direct mode of communication, reducing ambiguity and improving understandability (Greene, 2013).
The use of numbers and assumptions regarding readers' ability to conduct mental calculations also impacted the understandability of material in this study. Numbers should always be defined and used in a way that are familiar to the reader (e.g., whole numbers instead of decimals). In this context, material describing time (e.g., to self-isolate or complete a COVID-19 test) was rarely defined in a conversation that is commonly understood by the public (e.g., 96 h is 4 days). If demonstrating numbers where calculations are necessary, it is recommended that the material provides examples or appropriate conversions to reduce ambiguity or confusion (Centers for Disease Control Prevention, 2017).
We recommend that public health entities apply the tools used in this study to ensure that materials are accessible and easily comprehended by members of the public. For example, the FKGL is a simple and easy-to-use measure of readability that can be easily applied using a built-in tool on Microsoft Word (Kincaid et al., 1975). The resulting score (equivalent grade reading level) can be easily interpreted and adjusted through document revision. Current recommendations suggest that publicly-accessible materials should not exceed a grade 5 to 8 reading level (Cotugna et al., 2005). The CDC's Everyday Words for Public Health Communication tool is a searchable thesaurus for translating common public health jargon into plain language (Centers for Disease Control, n.d.); this resource simplifies the process of ensuring appropriate readability of public health materials.
Standardization of the health information that is delivered online or otherwise, from trusted authorities is needed to ensure access to current, evidence-based information (Jayasinghe et al., 2020). It is imperative that we improve knowledge translation strategies to improve the health literacy of consumers. For example, knowledge translation strategies that include consumers as co-creators who bring their unique perspectives, knowledge, or lived experiences may improve uptake and understanding of health materials (Shieh and Hosei, 2008; Cunningham et al., 2022).
5. Limitations
Our study attempted to collect an inclusive and representative sample of COVID-19 public health resources from across the four Atlantic provinces. A significant limitation of this study, however, is that only English resources were collected. New Brunswick has two provincial health authorities, Horizon Health and Vitalité, the latter of which provides services and resources in French. Excluding French materials may have impacted the generalizability of these findings to New Brunswick and throughout the region. Moreover, resources were collected during a narrow time frame in June and July 2022, which posts limitations to the study. Although no limits were placed on the date of resource publication, it is possible that the same collected in the current study did not reflect the quality of material at every stage of the pandemic. For example, the actionability of material may have been impacted by rapidly changing recommendations (Caballero et al., 2020).
Material in the current study was evaluated individually according to the modality of the content (i.e., text vs. audiovisual). As a result, some items were evaluated independent of its context. For example, a video embedded on a webpage with text were evaluated as separate items, yet the intent of this presentation may have been to provide readers with different presentations of the information. These multimedia educational interventions (i.e., text-based information combined with audiovisual material or images) may promote increased clarity, understanding, and actionability of content (Coulter and Ellins, 2007; Ritzert, 2015).
Most of the resources captured within this study (N = 338, 86.89%) targeted the general public, which resulted in a skewed dataset when compared to specific audiences (e.g., children and youth, seniors, etc.). Our findings suggest that resources targeting specific audiences may be more understandable and actionable than those broadly targeting the general public, however further research is needed to explore differences among public health resources targeting various cohorts within the population.
Although we attempted to conduct a comprehensive search of public health websites in the Atlantic region, internal algorithms and our geographic location may have biased the identification of resources on which our results are based. While we acknowledge that our searches may have been influenced by our web browsers, we attempted to mitigate this issue by collecting a large sample of 100 items from each province. Finally, we focused our study on evaluation of web-based COVID-19 resources. Although physical resources were less common during the pandemic, alternative modes of communication, including through radio and television, may have produced different results regarding the clarity, understandability, and actionability of material. For example, resources provided by provincial governments may have been clarified through traditional media sources, including the Canadian Broadcasting Corporations. For example, PEI has identified local television and radio media as integral to the success of varicella (Sweet et al., 2003) and meningococcal (Sweet et al., 2003) immunization campaigns. Future research might consider investigating the impact of the media in promoting the understandability and actionability of public health information. Future research might also consider extending the research lens nationally across regions in Canada (Mani et al., 2021) to explore differences in readability, understandability, and actionability of different types of materials (e.g., videos).
6. Conclusion
Throughout the COVID-19 pandemic, rapid developments in our understanding of disease prevention, transmission, and treatment required public health officials to quickly produce a large volume of education and awareness materials. Communication of current, evidence-based information is essential to protect public health and health systems; however, careful consideration of the thresholds related to readability, understandability, and actionability are imperative to developing and disseminating resources that are used effectively. Furthermore, producing resources that are clear and easily understood is essential to building and maintaining public trust. Understanding the appropriateness of materials is also essential to supporting COVID-19 communication for information uptake and knowledge development, while reducing the prevalence of misinformation (Campbell, 2021).
Data availability statement
The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.
Author contributions
KK, AC, and WM contributed to the conception and design of the study. KK and SD organized the database. KK, AS, SD, and LM equally extracted and evaluated data. WM performed the statistical analysis. KK wrote the first draft of the manuscript. AC wrote sections of the manuscript. All authors contributed to manuscript revision, read, and approved the submitted version.
Funding
This project was funded by the Immunization Partnership Fund from the Public Health Agency of Canada (contribution agreement #2122-HQ-000319).
Acknowledgments
We would like to acknowledge Kim Mears, Health Sciences and Scholarly Communications Librarian from the University of Prince Edward Island, for providing consultation with the research team on the search strategy.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fcomm.2023.1006784/full#supplementary-material
References
Agency for Healthcare Research Quality (2020). The Patient Education Materials Assessment Tool (PEMAT) and User's Guide. Available online at: https://www.ahrq.gov/health-literacy/patient-education/pemat.html (accessed July, 2022).
Altman, D. G., and Bland, J. M. (1983). Measurement in medicine: the analysis of method comparison studies. Statistician 32, 307–317. doi: 10.2307/2987937
Basch, C. H., Mohlman, J., Hillyer, G. C., and Garcia, P. (2020). Public health communication in time of crisis: readability of on-line COVID-19 information. Disaster Med. Public Health Prep. 14, 635–637. doi: 10.1017/dmp.2020.151
Basu, A., and Dutta, M. J. (2008). The relationship between health information seeking and community participation: the roles of health information orientation and efficacy. Health Commun. 23, 70–79. doi: 10.1080/10410230701807121
Bothun, L. S., Feeder, S. E., and Poland, G. A. (2022). Readability of COVID-19 vaccine information for the general public. Vaccine 40, 3466–3469. doi: 10.1016/j.vaccine.2022.04.096
Caballero, A. E., Ceriello, A., Misra, A., Aschner, P., McDonnell, M. E., Hassanein, M., et al. (2020). COVID-19 in people living with diabetes: an international consensus. J. Diabetes Complicat. 34, 107671. doi: 10.1016/j.jdiacomp.2020.107671
Cameron-Blake, E., Breton, C., Sim, P., Tatlow, H., Hale, T., Wood, A., et al (2021). Variation in the Canadian provincial and territorial responses to COVID-19. Nat. Hum. Behav. Available online at: https://www.bsg.ox.ac.uk/sites/default/files/2021-03/BSG-WP-2021-039.pdf
Campbell, A. (2021). The Development and Evaluation of an Innovative Knowledge Translation Tool about Pediatric Concussion. Alberta: University of Alberta Libraries. Available online at: https://www.bsg.ox.ac.uk/sites/default/files/2021-03/BSG-WP-2021-039.pdf
Campbell, A., Hartling, L., Plourde, V., and Scott, S. (2022). Parental knowledge, self-confidence, and usability evaluation of a web-based infographic for pediatric concussion: Multi method study. JMIR Pediat Parent. 5, e36317. doi: 10.2196/36317
Canadian Council on Learning (2008). Health Literacy in Canada: A Health Understanding. Available online at: http://www.en.copian.ca/library/research/ccl/health/health.pdf (accessed July, 2022).
Centers for Disease Control (n.d.). Everyday Words for Public Health Communication. Available online at: https://tools.cdc.gov/ewapi/termsearch.html (accessed July 2022).
Centers for Disease Control Prevention (2017). Index Design Guides. Available online at: https://www.cdc.gov/ccindex/index-design-guides.html (accessed July 21, 2022).
Centers for Disease Control Prevention (2021). The CDC Clear Communication Index. Available online at: https://www.cdc.gov/ccindex/index.html (accessed July, 2022).
Cotugna, N., Vickery, C. E., and Carpenter-Haefele, K. M. (2005). Evaluation of literacy level of patient education pages in health-related journals. J. Community Health 30, 213–219. doi: 10.1007/s10900-004-1959-x
Coulter, A., and Ellins, J. (2007). Effectiveness of strategies for informing, educating, and involving patients. Br. Med. J. 335, 24–27. doi: 10.1136/bmj.39246.581169.80
Cunningham, C., Sung, H., Benoit, J., Conway, J., and Scott, S. D. (2022). Multimedia knowledge translation tools for parents about childhood heart failure: environmental scan. JMIR Pediatr. Parent. 5, e34166. doi: 10.2196/34166
Duong, T. V., Lin, C.-Y., Chen, S. C., Huang, T.-K., Koan, O., Dadczynski, K., et al. (2021). Oxford COVID-19 vaccine hesitancy in school principals: impacts of gender, well-being, and coronavirus-related health literacy. Vaccines 9, 985. doi: 10.3390/vaccines9090985
Fitzsimmons, P. E., Michael, B. D., Hulley, J. L., and Scott, G. O. (2010). A readability assessment of online Parkinson's disease information. J. R. Coll. Physicians Edinb. 40, 292–296. doi: 10.4997/JRCPE.2010.401
Government of Canada (2022). COVID-19 Vaccination in Canada. Available online at: https://health-infobase.canada.ca/covid-19/vaccination-coverage/ (accessed July, 2022).
Graham, I. D., Straus, S. E., and Tetroe, J. (2013). Knowledge Translation in Health Care: Moving From Evidence to Practice. 2nd Edn. Toronto, ON: John Wiley & Sons.
Greene, A. E. (2013). “Favor the active voice,” in Writing Science in Plain English, ed A. E. Greene (Chicago, IL: University of Chicago Press), 22–28.
Hasnain-Wynia, R., and Wolf, M. S. (2010). Promoting health care equity: is health literacy a missing link? Health Serv. Res. 45, 897–902. doi: 10.1111/j.1475-6773.2010.01134.x
Jayasinghe, R., Ranasinghe, S., Jayarajah, U., and Seneviratne, S. (2020). Quality of online information for the general public on COVID-19. Patient Educ. Couns. 103, 2594–2597. doi: 10.1016/j.pec.2020.08.001
Kelley, M. S., Su, D., and Britigan, D. H. (2015). Disparities in health information access: results of a county-wide survey and implications for health communication. Health Commun. 31, 575–582. doi: 10.1080/10410236.2014.979976
Khan, S., Asif, A., and Jaffery, A. E. (2020). Language in a time of COVID-19: literacy bias ethnic minorities face during COVID-19 from online information in the UK. J. Racial Ethnic Health Disparities 8, 1242–1248. doi: 10.1007/s40615-020-00883-8
Kincaid, J. P., Fishburne, J., Robert, P. R., Richard, L. C., and Brad, S. (1975). Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease formula) for Navy Enlisted Personnel. US Dept of the Navy. Chief of Naval Technical Training: Naval Air Station Memphis.
Koo, T. K., and Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J. Chiropractor Med. 15, 155–163. doi: 10.1016/j.jcm.2016.02.012
Kutner, M., Greenberg, E., Jin, Y., Paulsen, C., and White, S. (2006). The Health Literacy of America's Adults: Results From the 2003 National Assessment of Adult Literacy. Available online at: http://nces.ed.gov/pubs2006/2006483_1.pdf (accessed July, 2022).
Logan, R. A., Wong, W. F., Villaire, M., Daus, G., Parnell, T. A., Willis, E., et al (2015). Health Literacy: A Necessary Element for Achieving Health Equity. Washington, DC: Institute of Medicine. Available online at: http://www.nam.edu/perspectives/2015/Health-literacy-anecessary-element-for-achieving-health-equity doi: 10.31478/201507a (accessed July, 2022).
Lorini, C., Santomauro, F., Donzellini, M., Capecchi, L., Bechini, A., Boccalini, S., et al. (2018). Health literacy and vaccination: a systematic review. Hum. Vaccin. Immunother. 14, 478–488. doi: 10.1080/21645515.2017.1392423
MacDonald N. E. SAGE Working Group on Vaccine Hesitancy. (2015). Vaccine hesitancy: Definition, scope and determinants. Vaccine. 33, 4161–4164. doi: 10.1016/j.vaccine.2015.04.036
Mani, N. S., Ottosen, T., and Fratta, M. (2021). A health literacy analysis of the consumer-oriented COVID-19 information produced by ten state health departments. J. Med. Library Assoc. 109, 422–431. doi: 10.5195/jmla.2021.1165
May, M., Brousseau, D. C., Nelson, D. A., Flynn, K. E., Wolf, M. S., Lepley, B., et al. (2018). Why parents seek care for acute illness in the clinic or the ED: the role of health literacy. Acad. Pediatr. 18, 289–296. doi: 10.1016/j.acap.2017.06.010
Morrison, A. K., Schapira, M. M., Gorelick, M. H., Hoffmann, R. G., and Brousseau, D. C. (2014). Low caregiver health literacy is associated with higher pediatric emergency department use and nonurgent visits. Acad. Pediatr. 14, 309–314. doi: 10.1016/j.acap.2014.01.004
Ng, C. C.-W., Wong, E. L.-Y., Ho, K.-F., Cheung, A. W.-L., Wong, S. Y.-S., and Yeoh, E.-K. (2022). Association between information dissemination and compliance with preventive measures during the coronavirus disease pandemic in Hong kong working population: cross-sectional survey. Challenges 13, 17. doi: 10.3390/challe13010017
Nutbeam, D. (2008). The evolving concept of health literacy. Soc. Sci. Med. 67, 2072–2078. doi: 10.1016/j.socscimed.2008.09.050
Okuhara, T., Ishikawa, H., Ueno, H., Kato, M., and Kiuchi, T. (2022). Readability assessment of vaccine information: a systematic review for addressing vaccine hesitancy. Patient Educat. Counsel. 105, 331–338. doi: 10.1016/j.pec.2021.05.039
Owais, A., Hanif, B., Siddiqui, A. R., Agha, A., and Zaidi, A. K. M. (2011). Does improving maternal knowledge of vaccines impact infant immunization rates? A community-based randomized-controlled trial in Karachi, Pakistan. BMC Public Health 11, 239. doi: 10.1186/1471-2458-11-239
Reid, K., Le, A., Norris, A., Scott, S. D., Hartling, L., and Ali, S. (2017). Development and usability evaluation of an art and narrative-based knowledge translation tool for parents with a child with pediatric chronic pain: multi-method study. J. Med. Internet Res. 19, e412. doi: 10.2196/jmir.8877
Ritzert, B. (2015). Multimedia educational interventions for consumers about prescribed and over-the-counter medications. Public Health Nurs. 32, 186–188. doi: 10.1111/phn.12102
SAS (2013). Statistical Analysis Software. Users' Guide Statistics Version 9.4. Cary, NC: SAS Institute Inc.
Scott, S. D., Hartling, L., and Klassen, T. P. (2009). The power of stories: using narratives to communicate evidence to consumers. Nursing Womens Health 13, 109–111. doi: 10.1111/j.1751-486X.2009.01401.x
Shieh, C., and Hosei, B. (2008). Printed health information materials: evaluation of readability and suitability. J. Community Health Nurs. 25, 73–90. doi: 10.1080/07370010802017083
Shoemaker, S. J., Wolf, M. S., and Brach, C. (2014). Development of the Patient Education Materials Assessment Tool (PEMAT): A new measure of understandability and actionability for print and audiovisual patient information. Patient Educ. Counsell. 96, 395–403. doi: 10.1016/j.pec.2014.05.027
Siddiqui, F. A., Padhani, Z. A., Salam, R. A., Aliani, R., Lassi, Z. S., Das, J. K., et al. (2022). Interventions to improve immunization coverage among children and adolescents: a meta-analysis. Pediatrics 149, e2021053852. doi: 10.1542/peds.2021-053852D
Solomon, D. H., Bucala, R., Kaplan, M. J., and Nigrovic, P. A. (2020). The “Infodemic” of COVID-19. Arthrit. Rheumatol. 72, 1806–1808. doi: 10.1002/art.41468
Statistics Canada (2012). Skills in Canada: First Results from the Programme for the International Assessment of Adult Competencies (PIAAC). No. 89-555-X (Ottawa, ON).
Sweet, L., Gallant, P., Morris, M., and Halperin, S. A. (2003). Canada's first universal varicella immunization program: lessons from Prince Edward Island. Can. J. Infect. Dis. 14, 41–44. doi: 10.1155/2003/904351
Szmuda, T., Özdemir, C., Ali, S., Singh, A., Syed, M. T., and Sloniewski, P. (2020). Readability of online patient education material for the novel coronavirus disease (COVID-19): a cross-sectional health literacy study. Public Health 185, 21–25. doi: 10.1016/j.puhe.2020.05.041
Taddio, A., Shah, V., Leung, E., Wang, J., Parikh, C., Smart, S., et al. (2013). Knowledge translation of the HELPinKIDS clinical practice guideline for managing childhood vaccination pain: usability and knowledge uptake of educational materials directed to new parents. BMC Pediatr. 13, 23. doi: 10.1186/1471-2431-13-23
The National Work Group on Literacy and Health (1998). Communicating with patients who have limited literacy skills. J. Fam. Pract. 46, 168–176.
Tugwell, P. S. (2007). Knowledge translation for effective consumers. Phys. Ther. 87, 1728–1738. doi: 10.2522/ptj.20070056
Turhan, Z., Dilcen, H. Y., and Dolu, I. (2021). The mediating role of health literacy on the relationship between health care system distrust and vaccine hesitancy during COVID-19 pandemic. Curr. Psychol. 41, 8147–8156. doi: 10.1007/s12144-021-02105-8
Valizadeh-Haghi, S., Khazaal, Y., and Rahmatizadeh, S. (2021). Health websites on COVID-19: are they readable and credible enough to help public self-care? J. Med. Library Assoc. 109, 75–83. doi: 10.5195/jmla.2021.1020
Worrall, A. P., Connolly, M. J., O'Neill, A., O'Doherty, M., Thorton, K. P., McNally, C., et al. (2020). Readability of online COVID-19 health information: a comparison between four English speaking countries. BMC Public Health 20, 1635. doi: 10.1186/s12889-020-09710-5
Wu, H., and Mackenzie, J. (2021). Dual-gendered leadership: gender-inclusive scientific-political public health communication supporting government COVID-19 responses in Atlantic Canada. Healthcare 9, 1345. doi: 10.3390/healthcare9101345
Yardi, S., Caldwell, P. H., Barnes, E. H., and Scott, K. M. (2018). Determining parents' patterns of behaviour when searching for online information on their child's health. J. Paediatr. Child Health 54, 1246–1254. doi: 10.1111/jpc.14068
Yeung, A. W. K., Wochele-Thomas, T., Eibensteiner, F., Klager, E., Hribersek, M., Parvanov, E. D., et al. (2022). Official websites providing information on COVID-19 vaccination: readability and content analysis. JMIR Public Health Surveill. 8, e34003. doi: 10.2196/34003
Zhang, H., Li, Y., Peng, S., Jiang, Y., Jin, H., and Zhang, F. (2021). The effect of health literacy on COVID-19 vaccine hesitancy: the moderating role of stress. medRxiv. 40, 4473–4478. doi: 10.1101/2021.06.16.21258808
Keywords: health literacy, knowledge translation, communication COVID-19, health information, health communication
Citation: Kelly KJ, Campbell A, Salijevic A, Doak S, Michael L and Montelpare W (2023) Evaluation of the readability, understandability, and actionability of COVID-19 public health messaging in Atlantic Canada. Front. Commun. 8:1006784. doi: 10.3389/fcomm.2023.1006784
Received: 29 July 2022; Accepted: 24 April 2023;
Published: 12 May 2023.
Edited by:
Ke Jiang, Elon University, United StatesReviewed by:
Salima Meherali, University of Alberta, CanadaRuobing Li, Stony Brook University, United States
Copyright © 2023 Kelly, Campbell, Salijevic, Doak, Michael and Montelpare. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Katherine Jennifer Kelly, kjkelly@upei.ca