Systematic Review on Information Technology Approaches to Evaluate the Impact of Public Health Campaigns: Real Cases and Possible Directions

Evaluating the success of a public health campaign is critical. It helps policy makers to improve prevention strategies and close existing gaps. For instance, Brazil's “Syphilis No!” campaign reached many people, but how do we analyze its real impact on population awareness? Are epidemiologic variables sufficient? This study examined literature on using of information technology approaches to analyze the impact of public health campaigns. We began the systematic review with 276 papers and narrowed it down to 17, which analyzed campaigns. In addition to epidemiological variables, other types of variables of interest included: level of (i) access to the campaign website, (ii) subject knowledge and awareness, based on questionnaires, (iii) target population's interest, measured from both online search engine and engagement with Social Network Service, and (iv) campaign exposure through advertising, using data from television commercials. Furthermore, we evaluated the impact by considering several dimensions such as: communication, epidemiology, and policy enforcement. Our findings provide researchers with an overview of various dimensions, and variables-of-interest, for measuring public campaign impact, and examples of how and which campaigns have used them.

Introduction

Public communications campaigns (PCC) are strategic to communicating important information to the public (1). In the context of this work, PCCs for Health Care will be called public health campaigns (PHC). PHC aims to influence behavior either on an individual level, through direct messages, or collectively with policies that inspire change (1).

PHC aims to promote awareness, increase knowledge, encourage the adoption of desirable attitudes and behaviors, and contribute to individual and collective health decision-making (2). PHCs are often sponsored by policy makers, and offer preventive recommendations addressing serious health problems, such as sexually transmitted diseases, alcohol, tobacco, and obesity, as well as the dangers of automobiles, guns and pharmaceuticals (3).

For example, the “Sífilis Não!” (Syphilis No!) campaign, communicated the importance of syphilis prevention, and recommended rapid syphilis testing or VDRL test, conducted at Basic Health Units (UBS) of the Unified Health System (SUS) in Brazil. The Ministry of Health sponsored the campaign, to fight the syphilis epidemic in the country. Between Nov 2018 and May 2019, a variety of content reached large audiences through various media platforms (e.g., television, radio, and online), as well as outdoor media, such as billboards and posters, and print media, such as magazines and newspapers.

Campaign evaluations serve several useful purposes. They help policy makers improve prevention strategies and close existing gaps. Analysis can guide the development of better campaigns, to improve the impact or intensify the message. Understanding the effectiveness of a campaign can help to better direct the use of funds and effort. To assess levels of engagement with a given topic on a platforms such as the Internet, researchers are increasingly using computational approaches. A variety of such techniques and technologies have been used to assess the impact of public health campaigns. Subsequently, a secondary study is beneficial both to understand the current state-of-the-art and to guide future research in this domain.

Accordingly, we performed a systematic literature review (SLR) according to Kitchenham and Charters (4) and Petersen et al. (5) to: (1) characterize scientific literature related to using information technology approaches to evaluate the impact of public health campaigns, and (2) summarize the knowledge gained from understanding the techniques employed, variables of interest analyzed, and the metrics used to validate these results. Results of this study help identify gaps and provide a direction to appropriately position new research activities in this domain.

Mass media campaigns have long been a tool for promoting public health (6) and there are many systematic literature reviews in this domain. Some of them focus on specific public campaigns: Te et al. (7) identifies social media health campaigns against the consumption of sugary drinks; Jones and Salazar (8) describes the use of Social Network Services (SNS) in the context of primary HIV prevention; Yadav and Kobayashi (9) assesses newer evidence from quantitative studies on the effectiveness of mass media campaigns for reducing alcohol-impaired driving (AID) and alcohol-related crashes; Vega and Roland (10) describes the social marketing approaches used to increase syphilis awareness in eight US cities.

Conversely, some studies do not target a specific public, such as: Shi et al. (11) aims to differentiate SNS from more traditional health communication approaches; Jacob et al. (12) determines the costs, benefits, and overall economic value of communication campaigns; Robinson et al. (13) evaluates the effectiveness of health communication campaigns that use multiple channels, including mass media, and distribute health-related products; Randolph et al. (14) discusses the effects of three particular campaign strategies, entertainment education, law enforcement, and mass media.

Dorfman et al. (1) proposed a taxonomy of communication campaigns, which includes three axes: purpose, scope, and maturity. By purpose, the authors understand that the goal of a campaign should directly affect the individuals, or the collective policies, that shape social behavior. Scope refers to the most visible part of a communication campaign: its size and extent (In what region? In what period? Is it local, state or national?). Maturity can come over time (campaigns can be unique events or could last for years). It may become more formal, with clearer goals, well-developed materials, and deeper integration into an organization's overall activities.

This taxonomy was important to characterize a public campaign, but we missed other properties that we believe to be relevant. Such properties, presented in Table 1, in addition to Dorfman et al.'s taxonomy, helped us to identify public campaigns within the scope of our work. A catalog with the identification of the primary studies of our research is provided in Appendix 1, based on all identified properties.

TABLE 1

Table 1. Properties added to the taxonomy proposed by Dorfman et al (1).

Methods

The research methodology for conducting this systematic literature review (SLR) was performed from the guidelines proposed by Kitchenham and Charters (4) and Petersen et al. (5). Although the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines (15, 16) were not used in this study, the guidelines used follow similar stages, such as: (i) articles were identified through a database search; (ii) articles were screened according to the selection criteria concerning title and abstract; (iii) the eligibility of each full-text article was rated according to predetermined criteria (quality assessment); (iv) approved articles were included in the systematic review.

Research Questions

The goal of this research is to analyze information technology approaches that assess the impact of public health communications campaigns as a health care promotion strategic method. Thus, we framed the research questions using the PICOC (Population, Intervention, Comparison, Outcome, and Context) criteria as suggested by Kitchenham and Charters (4).

• Population: public health campaigns.

• Intervention: the use of technology information to evaluate health campaigns.

• Comparison: characterization of the measures used to support the evaluation of health campaign.

• Outcomes: characterization of the produced output, emergent techniques, overlooked areas and open issues.

• Context: any case where technology information approaches were used to evaluate health campaign.

This leads to the following research questions (RQs):

• RQ1: When and where have the studies been published?

• RQ2: What are the characteristics (topic area, location and year) of the campaigns?

• RQ3: What variables of interest were analyzed to assess the impact of public health campaigns?

• RQ4: What techniques or tools have been used to support campaign impact analysis?

Search Process

Scopus database was the main source of scientific papers for the current study. We selected this digital library because it contains publications from major journals and conference proceedings, and it is one of the largest curated bibliographic abstract and citation databases of research literature (17). Moreover, recent bibliographic research indicated it as the most comprehensive and user-friendly database (18, 19).

In May 2020, we conducted a preliminary search with a set of terms related to the research topics based on Population and the Intervention previous defined. The search generated many results. However, few were relevant. We repeated this process until we found a search string with a set of relevant results. To maximize recall, we did not include the comparison, outcomes, and context criteria in the search term. We considered the context criterion in Inclusion Criteria (IC), while the outcomes and the comparison criteria defined our data extraction strategy.

The set of search terms included three aspects: (i) “campaign^*” AND (ii) (“public health” OR “communicable disease” OR “disease transmission” OR “transmitted infection” OR “disease outbreak^*” OR “illness outbreak^*” OR “infectious disease^*” OR “disease surveillance” OR “disease epidemiology”) AND (iii) (“impact” OR “correlation” OR “assess” OR “effectiveness” OR “efficacy” OR “evaluation” OR “analysis”).

The automated search looked for these terms in titles, abstracts, and keywords, in the Computer Science area. To increase the quality of the studies found, they had to be published in a peer- reviewed journal. In addition, a backward snowball search (20) was performed by scanning the bibliographies of all selected papers.

Inclusion and Exclusion Criteria

The systematic review included studies that met the following two inclusion criteria: (IC1) The study discusses the effectiveness of using information technology approaches to measure public health campaigns; (IC2) The paper evaluates a health care campaign.

Regarding the exclusion criteria (EC), studies were excluded if they presented at least one of the following criteria: (EC1) The study is not written in English; (EC2) The study is an older version of other study already considered; (EC3) The study is a short publication (<5 pages) or poster; (EC4) The study is a gray literature (For example: technical reports, white papers, and work in progress); (EC5) Paper full text is not available for download.

Paper Selection

Figure 1 shows the steps and quantities of papers returned in the entire search process. The process of selection and classification of the studies was performed by the first author of this article and the second author reviewed the process to avoid research bias.

FIGURE 1

Figure 1. Flow diagram of the study selection process and results.

Titles and abstracts of papers resulting from the automated search were evaluated, rejecting papers that were obviously not relevant (Stage 1-identification). If eligibility was unclear from the abstract, the study was retained for further evaluation (Stage 2–screening). Subsequently, in Stage 3-eligibility, the full texts were read to confirm the criteria and submitted to Stage 4–included for data extraction.

Fifty-one Studies Were Analyzed in Stage 2 and 18 in Stage 3. However, Only five Documents Were Included for Data Extraction. A Complete Analysis of 18 Documents Was Necessary to Understand the Research Context and to Apply the Inclusion and Exclusion Criteria More accurately.

Those five documents served as basis for a subsequent analysis, through their own references, using the backward snowball technique. Thus, 225 articles were identified from citation searching (Stage 1), and screened (Stage 2) in order to verify the inclusion and exclusion criteria. In Stage 3, 19 articles were selected for complete reading and, finally, 12 articles met the determined criteria. Accordingly, 17 studies (five identified via database plus 12 identified from snowball technique) were selected for data extraction (listed in Appendix A).

Study Quality Assessment

Quality assessment is essential in systematic reviews to determine the rigor and relevance of the primary studies and should be applied in a similar way across the different types of studies identified (21).

To assess the quality of the studies, we developed a survey on six characteristics of communication campaigns, Table 1 (3rd to 8th rows). In addition, we only consider studies classified as Evaluation Research, as defined by Wieringa et al. (22) that is: techniques, methods, tools or other solutions are implemented and evaluated in practice.

For each question, the study's quality was evaluated as “Yes,” “Partially,” or “No,” and scored with the values 1, 0.5, and 0, respectively. We noted that the overall quality of the studies was good, ranging from 0.75 to 1. Table 2 shows the quality assessment criteria found for the 17 selected studies. In Appendix A, these data are listed in detail.

TABLE 2

Table 2. Quality scores.

Data Extraction

To extract data from the identified primary studies, we developed a template (Table 3) to register the main information in a spreadsheet. Each data item addresses a Research Question (RQ), Quality Assessment (QA) or an overview of the study.

TABLE 3

Table 3. Data extraction form.

Threats to Validity

The ability to choose digital libraries that will accurately represent a study can influence the validity of an SLR. Scopus has one of the largest abstracts and citation databases for research literature. Researchers consider it a high-quality, versatile, and respected research database (19).

This review created more restrictive filters for the area (Science Computer) and source type (Journals), to avoid an exhaustive review. Accordingly, some studies–although relevant–may not be included. We mitigated this limitation by scanning the reference list through a backward snowballing technique (20) of identified papers. This technique added 225 papers for the analysis of titles and abstracts, of which 19 were included in the final selection.

Regarding data extraction, the process of selection and classification of the studies was done by one researcher and all data extracted was checked by the other, and subsequently rechecked by the first researcher. At the end, we discussed the results and when doubts remained, a third researcher was consulted, to make the final decision.

Results

In total, 276 papers were found by the automatic and snowball searches. However, after applying the inclusion/exclusion criteria, 17 papers remained in the set of relevant papers. We believe that these 17 papers present the necessary criteria to answer the research questions.

RQ1: When and where have the studies been published? Historically, the first publication was in 2003. Then, in 2008. After 3 years with no results found, publications returned in 2011, where there were annual publications until 2019. The countries carrying out this research were: Australia (n = 6), United States (n = 6), United Kingdom (n = 3), China (n = 1), and Norway (n = 1). Most studies were published in health and communication journals as shown in Table 4.

TABLE 4

Table 4. Sources identified in primary studies, ordered by year.

RQ2: Which topic areas were addressed to the campaign, location, and year? In the 17 selected studies, the topic areas addressed were Anti-Smoking [n = 5, (S03, S06, S07, S09, S10)], Children Vaccination [n = 1, (S01)], Newborn screening and biobanking programs [n = 1, (S14)], Overweight and Obesity [n = 3, (S05, S16, S17)], and Sexual Health Care [n = 7, (S02, S04, S08, S11, S12, S13, S15)].

Figure 2 shows the distribution of thematic areas targeted to the campaign in the period from 1995 to 2018. Each rectangle covers a period on a campaign topic by country. Each country is designated by a color. Australia (green) covered long-term campaigns in the Anti-Smoking, Overweight and Obesity, and Sexual Health Care areas. The United States (blue) had campaigns in the areas of Anti-smoking, Newborn screening and biobanking programs, and Sexual Health Care. England (orange) in Sexual Health Care (Chlamydia) and Children Vaccination. Norway (yellow) had a short-term campaign in the Sexual Health Care area.

FIGURE 2

Figure 2. Distribution of topic areas addressed to campaign by range of period.

When the dates were not clearly specified in the article, an internet search was carried out to find approximate values. For example: In S01, influenza season (2013/2014) meant the period from 02/09/2013 to 13/04/2014 (It was dominated by the circulation of influenza A(H1N1) pdm09 virus, in Europe). When the articles specified only the month and year of the campaign, we considered the first (starting) and last (end) day of the month. One study (S07) did not report a period and due to this it was not shown in the Figure 2.

RQ3: What variables of interest were analyzed to assess the impact of public health campaigns? The 17 studies evaluated several types of variables, such as: amount of access to the campaign website, awareness and knowledge of the subject through the questionnaires, interest of the population through the online search engine, engagement through the SNS, and advertising exposure data of television commercials.

Most studies sought to cross-check the data obtained with socioeconomic and census data. The data source used to obtain the variables can be categorized as follows: questionnaire (n = 10), social network service (n = 10), Sexually Transmitted Infection (STI) testing (n = 2), television commercials (n = 2), campaign website (n = 1), pharmaceutical products (n = 1), smoke-free restaurant laws (n = 1), sociodemographic data (n = 1), tobacco prices (n = 1). Table 5 shows data sources, variables of interest, dimensions, and methods used by each study.

TABLE 5

Table 5. Data sources and variables of interest analyzed per study.

Despite the variety of variables analyzed, we can observe a predominance related to i) assessing user engagement with the campaign subject through SNS, such as Facebook and Twitter (S01, S02, S03, S04, S08, S09, S10, S11, S12, S14); and ii) assessing public knowledge, attitudes and behavior through questionnaires (S04, S05, S06, S07, S08, S12, S13, S15, S16, S17), before and/or after campaign dissemination.

SNS were used to recruit participants to answer the questionnaires (S05, S08, S13), assessing the reach and knowledge of the impacted people. The authors argued that this approach produced good results due to accurate user geolocation and demographic data (e.g., gender, age, education, employment status, language spoken).

In addition, some studies (S02, S04, S05, S08, S12, S14) highlighted that Facebook advertisements significantly helped to increase both the number of people visiting a campaign webpage, and the number of users who could potentially improve their knowledge and health care behavior, because of the campaign.

Televised ads were assessed in two anti-tobacco campaigns using broadcasting time and its Nielsen rating (https://www.nielsen.com/ca/en/about-us/). S03 used TV ratings correlated with social media (Twitter) while S06 correlated TV ratings with questionnaire, tobacco prices, population use of pharmaceutical smoking cessation products, and smoke-free restaurant laws. In both cases, the results indicated success in the approaches used to assess the campaign impact.

Regarding STI Testing, S04 and S12 indicate that their campaign achieved its goal, i.e., to change sex care behavior and reach a large number of people. In fact, in the post-campaign period, the demand for test orders increased.

Grouping the variables of interest in dimensions (areas of context) provided better understanding about which dimension was most explored. Our findings show three dimensions: communication (n = 20), epidemiology (n = 10), and public policy (n = 2).

Communication dimension was assigned when the variable of interest sought to assess engagement on the Internet/social media or surveys asked the user knowledge about the health campaign. Epidemiology dimension was assigned when the variable of interest sought to assess patterns of health and disease conditions in a defined population. Public policy dimension was detected when the variable of interest sought to assess control policies, such as changes in taxes on products, and new laws.

RQ4: What techniques or tools have been used to support campaign impact analysis? Considering that social networking services (Facebook, Twitter, and Instagram) were widely explored in the studies, the main tools were the dashboards of the respective sites (explored by S02, S04, S08, S11, S12, S14). SNS dashboards often provide enough data to verify the engagement of privately hosted websites.

Nevertheless, analyzing user-generated content on a social network, such as Twitter, and inferring correlations with campaign data requires a detailed evaluation of the content. Accordingly, the studies S03, S09, and S10 obtained data from Twitter through a licensed data provider (http://www.gnip.com). This vendor provides real-time access to 100% of all tweets and meta-data. These analyses and correlations were supported by general linear models used to assess the relationship between demographics data and campaign awareness. Furthermore, Google analytics was used to identify the number of unique users, total visits, page views, unique views, and average visit duration for each page of the campaign website.

Regarding software for statistical analysis and using data science to explore, visualize, model and make inferences in the data, studies S12, S13 and S16 used Stata (https://www.stata.com/) while studies S06, S08, and S17 used Statistical Analysis System–SAS (https://www.sas.com/enca/software/stat.html). NVivo (https://www.qsrinternational.com/nvivo/home), a qualitative data analysis software, was used by S05 for textual analysis of the questionnaires and by S11 to capture the campaign page content, using a simple counting method to describe user and moderator content, discussion thread patterns, and moderator intervention. S12 used a web-based survey administration and data management application called REDCap (https://www.project-redcap.org/).

In S05, all statistics analyzes were conducted with SPSS Statistics 22 (https://www.ibm.com/support/pages/spss-statistics-220-available-download), a software platform that offers advanced statistical analysis, a vast library of machine learning algorithms, text analysis, open-source extensibility, integration with big data and seamless deployment into applications.

S01 proposed a statistical framework for transforming user-generated content published on web platforms to an assessment of the impact of a health-oriented intervention and S07 used a statistical technique named ANOVA (analysis of variance) to test all variables defined in the study.

Discussion

This paper assessed how researchers evaluate public health campaigns. Based on the studies found and the answers to the research questions, we observed that the studies sought to assess campaign impact through quantitative and qualitative analysis, using mainly SNS, especially Facebook. The popularity of social platforms such as Facebook–in the United States and worldwide–serves to justify this approach (40, 41). Facebook ads can reach and engage large, well-specified populations at relatively low cost (42, 43).

In addition, SNS are a rich source of data that can provide responses to PHC impact assessments. This may be because data is available to users through the dashboards, where it is possible to observe quantitative values. Detailed data can be easily obtained from some vendors, by paying fees, which facilitates data collection, allowing such methods and tools to do the “heavy lifting” (sometimes paid, such as Stata, SAS, NVivo or SPSS). A potential threat related to this approach is to identify real engagement of users. Some studies reported this issue (S4, S5, S8, S11, S14), which raised the following question: What is the real motivation to engage on campaign interventions pages on social networks? What about the “spiral of the silence” (44) which suggests that interventions can change the attitudes and behaviors of real interested users behind the screen, even without their interaction?

It is intriguing to consider the paradox of trying to reach a target audience through the SNS to raise awareness about an infectious disease, since some parts of the population cannot be reached through technologies and the Internet. In Brazil, for example, internet use reached 152 million people, representing 81% of the population. This percentage drops to 67% when we looked at the lower social classes (45). However, at the same time, it is likely that no communication channel will be able to reach 100% of the target audience by itself. The technological bias present in many campaigns is clear when only digital channels are activated, mainly excluding lower social classes, homeless people and people deprived of liberty, who in turn are part of the target audience of this type of campaign.

In addition, questionnaires targeting impacted users was another effective form of evaluating campaign impact. This approach is useful for measuring target audience awareness, attitudes and behavior, and has been used in eight out of 17 primary studies.

In contrast to most of the studies analyzed, article S15 reported that advertisements through paper-based methods, text messaging and social networking sites were not effective. In this study, only 28 rural youth contacted the health care service (campaign's goal) over the 11-month period. Twenty young people were reached by nurses (15 from school nurses, five from community health nurses), six through the campaign webpage, one through Facebook and one through the student diary. No clients were recruited through other advertising methods. Arguably, the limited publicity could be because the target audience were young people living in rural areas in Victoria, Australia. Therefore, although SNS have a great potential to provide data to analyze the impact of campaigns, attention must be paid to the public and the conditions of access and web-behavior of that public.

Despite the widespread use of social networking sites, only one study (S01) sought a second source of data based on user-generated Internet content (Microsoft's Bing search engine). The study aimed to introduce a complementary framework for evaluating the impact of a targeted intervention, such as a vaccination campaign against an infectious disease, through statistical analysis of user-generated content submitted on web platforms (Twitter and Bing search engine). The results from Twitter data demonstrated less sensitivity across similar controls relative to Bing data, suggesting a greater reliability.

In general, we observed a gap in assessing the impact of public health campaigns, regarding the use of online data (i.e., online news) and others user-generated Internet content. Google Search and Yahoo should be used in addition to Bing search engine to check the increase of news related to the campaign. In addition, a variety of questions should be asked, such as what can the analysis of this content bring? Are campaigns helping to grow spontaneous news on the related topic? When evaluating a population over time, Google Trends is arguably an excellent tool to demonstrate how the population has sought to know about a certain topic (46–48). However, it was not explored in the context of these works.

Moreover, databases of scientific papers such as Springer Nature, Wiley Blackwell, Taylor and Francis, IEEE, American Physical Science and Elsevier and its indexers such as Scopus, Web of Science (WoS) and Google Scholar could be used to demonstrate the interest of the academy in developing research on the campaign theme in a comparison of time (before, during and after the campaign). These variables of interest can show a new dimension, Education.

Health campaigns are common worldwide. Nevertheless, few studies have reported concrete campaign data. We would like to encourage future research to better specify analyzed data in their studies showing properties raised by Dorfman et al. (1) and complemented for us, to enable a comparison on health campaigns analysis, as present in Appendix A, stimulating the Campaign dimension.

Notably, by using variables of interest that were grouped into three dimensions (as discussed in this review), we analyzed the reach of the “Syphilis No!” campaign (49), assessing data related to the campaign, online news, search engine activity, online courses, serological tests, medication distribution and case notification rates. Results of this analysis show positive changes over time in communication, education, and epidemiological surveillance dimensions, especially after the campaign propagation.

Conclusions

The paper provided an overview of studies that evaluate the impact of public health campaigns. The analysis focused on identifying variables of interest, techniques and tools used. We discussed the results, presented new questions, and discovered unexplored variables of interest.

Public health campaigns play a strategic role promoting awareness, increasing knowledge, and encouraging the target population to adopt desirable attitudes and behaviors. As observed, its impact must be measured in several dimensions such as: i) communication (engagement in social networks, questionnaires assessing the user's knowledge about the campaign), ii) epidemiology (disease screening test, cases' notification, questionnaires assessing the user's knowledge about the disease/health issues), and iii) policy enforcement (law strategies on health promotion).

This multidimensional analysis provided a complete evaluation of public health campaigns, to understand its scope, find correlations between different variables of interest and expand possibilities for analysis.

Accordingly, we hope this work will motivate future researchers to explore other variables of interest and dimensions that are possibly overlooked in assessing the impact of public health campaigns based on multidimensional aspects.

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Author Contributions

RP, LS, and VK: conceptualization. RP, LS, RV, and VK: methodology. RP, LS, RV, VK, CG, CO, and JL: validation and writing–review and editing. RP and LS: formal analysis. RP: investigation, data curation, and writing–original draft preparation. LS, RV, and VK: supervision. RV: project administration and funding acquisition. All authors contributed to the article and approved the submitted version.

Funding

Brazilian Ministry of Health–Number's project: 54/2017.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

This research was carried out in cooperation agreement between Federal University of Rio Grande do Norte and Athabasca University. We express our sincere gratitude to all leaders that was involved with this agreement.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2021.715403/full#supplementary-material

References

1. Dorfman L, Ervice J, Woodruff K. Voices For Change: A Taxonomy of Public Communications Campaigns and Their Evaluation Challenges. Washington, DC: Communications Consortium Media Center, Media Evaluation Project (2002).

Google Scholar

2. Bucchi M, Trench B. Health campaign research: Enduring challenges and new developments. In: Routledge handbook of public communication of science and technology. Routledge (2014). p. 214–29.

Google Scholar

3. Freudenberg N. Public health advocacy to change corporate practices: implications for health education practice and research. Health Educ Behav. (2005) 32:298–319. doi: 10.1177/1090198105275044

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Kitchenham B, Charters S. Guidelines For Performing Systematic Literature Reviews in Software Engineering. Keele; Keele University (2007).

Google Scholar

5. Petersen K, Feldt R, Mujtaba S, Mattsson M. Systematic mapping studies in software engineering. EASE. (2008) 8:68–77. doi: 10.14236/ewic/EASE2008.8

CrossRef Full Text | Google Scholar

6. Noar SM. A 10-year retrospective of research in health mass media campaigns: where do we go from here? J Health Commun. (2006) 11:21–42. doi: 10.1080/10810730500461059

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Te V, Ford P, Schubert L. Exploring social media campaigns against sugar-sweetened beverage consumption: a systematic search. Cogent Medicine. (2019) 6:1607432. doi: 10.1080/2331205X.2019.1607432

CrossRef Full Text | Google Scholar

8. Jones J, Salazar LF. A review of hiv prevention studies that use social networking sites: implications for recruitment, health promotion campaigns, and efficacy trials. AIDS Behav. (2016) 20:2772–81. doi: 10.1007/s10461-016-1342-9

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Yadav RP, Kobayashi M. A systematic review: effectiveness of mass media campaigns for reducing alcohol-impaired driving and alcohol-related crashes. BMC Public Health. (2015) 15:857. doi: 10.1186/s12889-015-2088-4

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Vega MY, Roland EL. Social marketing techniques for public health communication: a review of syphilis awareness campaigns in 8 US cities. Sex Transm Dis. (2005) 32:S30–6. doi: 10.1097/01.olq.0000180461.30725.f4

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Shi J, Poorisat T, Salmon CT. The use of social networking sites (SNSS) in health communication campaigns: review and recommendations. Health Commun. (2018) 33:49–56. doi: 10.1080/10410236.2016.1242035

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Jacob V, Chattopadhyay SK, Elder RW, Robinson MN, Tansil KA, Soler RE, et al. Economics of mass media health campaigns with health-related product distribution: a community guide systematic review. Am J Prev Med. (2014) 47:348–59. doi: 10.1016/j.amepre.2014.05.031

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Robinson MN, Tansil KA, Elder RW, Soler RE, Labre MP, Mercer SL, et al. Mass media health communication campaigns combined with health-related product distribution: a community guide systematic review. Am J Prev Med. (2014) 47:360–71. doi: 10.1016/j.amepre.2014.05.034

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Randolph KA, Whitaker P, Arellano A. The unique effects of environmental strategies in health promotion campaigns: a review. Eval Program Plann. (2012) 35:344–53. doi: 10.1016/j.evalprogplan.2011.12.004

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the prisma statement. PLoS Med. (2009) 6:e1000097. doi: 10.1371/journal.pmed.1000097

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (prisma-p) 2015 statement. Syst Rev. (2015) 4:1–9. doi: 10.1186/2046-4053-4-1

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Baas J, Schotten M, Plume A, Côté G, Karimi R. Scopus as a curated, high-quality bibliometric data source for academic research in quantitative science studies. Quantitative Science Studies. (2020) 1:377–86. doi: 10.1162/qss_a_00019

CrossRef Full Text | Google Scholar

18. Harzing AW, Alakangas S. Google scholar, scopus and the web of science: a longitudinal and cross-disciplinary comparison. Scientometrics. (2016) 106:787–804. doi: 10.1007/s11192-015-1798-9

CrossRef Full Text | Google Scholar

19. Mongeon P, Paul-Hus A. The journal coverage of web of science and scopus: a comparative analysis. Scientometrics. (2016) 106:213–28. doi: 10.1007/s11192-015-1765-5

CrossRef Full Text | Google Scholar

20. Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. (2005) 331:1064–5. doi: 10.1136/bmj.38636.593461.68

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Wohlin C, Runeson P, Neto PAdMS, Engström E, do Carmo Machado I, De Almeida ES. On the reliability of mapping studies in software engineering. J Syst Softw. (2013) 86:2594–610. doi: 10.1016/j.jss.2013.04.076

CrossRef Full Text | Google Scholar

22. Wieringa R, Maiden N, Mead N, Rolland C. Requirements engineering paper classification and evaluation criteria: a proposal and a discussion. Requir Eng. (2006) 11:102–7. doi: 10.1007/s00766-005-0021-6

CrossRef Full Text | Google Scholar

23. Grandpre J, Alvaro EM, Burgoon M, Miller CH, Hall JR. Adolescent reactance and anti-smoking campaigns: a theoretical approach. Health Commun. (2003) 15:349–66. doi: 10.1207/S15327027HC1503_6

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Wakefield MA, Durkin S, Spittal MJ, Siahpush M, Scollo M, Simpson JA, et al. Impact of tobacco control policies and mass media campaigns on monthly adult smoking prevalence. Am J Public Health. (2008) 98:1443–50. doi: 10.2105/AJPH.2007.128991

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Gamage DG, Fuller CA, Cummings R, Tomnay JE, Chung M, Chen M, et al. Advertising sexual health services that provide sexually transmissible infection screening for rural young people–what works and what doesn't. Sex Health. (2011) 8:407–11. doi: 10.1071/SH10144

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Bull SS, Levine DK, Black SR, Schmiege SJ, Santelli J. Social media–delivered sexual health intervention: a cluster randomized controlled trial. Am J Prev Med. (2012) 43:467–74. doi: 10.1016/j.amepre.2012.07.022

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Ahmed N, Jayasinghe Y, Wark JD, Fenner Y, Moore EE, Tabrizi SN, et al. Attitudes to chlamydia screening elicited using the social networking site facebook for subject recruitment. Sex Health. (2013) 10:224–8. doi: 10.1071/SH12198

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Syred J, Naidoo C, Woodhall SC, Baraitser P. Would you tell everyone this? Facebook conversations as health promotion interventions. J Med Internet Res. (2014) 16:e108. doi: 10.2196/jmir.3231

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Emery SL, Szczypka G, Abril EP, Kim Y, Vera L. Are you scared yet? Evaluating fear appeal messages in tweets about the tips campaign. J Commun. (2014) 64:278–95. doi: 10.1111/jcom.12083

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Lampos V, Yom-Tov E, Pebody R, Cox IJ. Assessing the impact of a health intervention via user- generated internet content. Data Min Knowl Discov. (2015) 29:1434–57. doi: 10.1007/s10618-015-0427-9

CrossRef Full Text | Google Scholar

31. Dowshen N, Lee S, Lehman BM, Castillo M, Mollen C. Iknowushould2: Feasibility of a youth-driven social media campaign to promote STI and HIV testing among adolescents in Philadelphia. AIDS Behav. (2015) 19:106–11. doi: 10.1007/s10461-014-0991-9

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Platt T, Platt J, Thiel DB, Kardia SL. Facebook advertising across an engagement spectrum: a case example for public health communication. JMIR Public Health Surveill. (2016) 2:e27. doi: 10.2196/publichealth.5623

PubMed Abstract | CrossRef Full Text | Google Scholar

33. O'Hara BJ, Grunseit A, Phongsavan P, Bellew W, Briggs M, Bauman AE. Impact of the swap it, don't stop it Australian national mass media campaign on promoting small changes to lifestyle behaviors. J Health Commun. (2016) 21:1276–85. doi: 10.1080/10810730.2016.1245803

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Allem JP, Escobedo P, Chu KH, Soto DW, Cruz TB, Unger JB. Campaigns and counter campaigns: reactions on twitter to e-cigarette education. Tob Control. (2017) 26:226–9. doi: 10.1136/tobaccocontrol-2015-052757

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Gabarron E, Luque LF, Schopf TR, Lau AY, Armayones M, Wynn R, et al. Impact of facebook ads for sexual health promotion via an educational web app: a case study. Healthcare Policy and Reform. (2019) 990–1003. doi: 10.4018/978-1-5225-6915-2.ch045

CrossRef Full Text | Google Scholar

36. Zhan Q, Zhang J, Philip SY, Emery S, Xie J. Inferring social influence of anti-tobacco mass media campaign. IEEE Trans Nanobioscience. (2017) 16:356–66. doi: 10.1109/TNB.2017.2707075

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Kite J, Gale J, Grunseit A, Bellew W, Li V, Lloyd B, et al. Impact of the make healthy normal mass media campaign (phase 1) on knowledge, attitudes and behaviours: a cohort study. Aust N Z J Public Health. (2018) 42:269–76. doi: 10.1111/1753-6405.12779

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Kite J, McGill B, Freeman B, Vineburg J, Li V, Berton N, et al. User perceptions of the make healthy normal campaign facebook page: a mixed methods study. Soc Media Soc. (2018) 4:1–10. doi: 10.1177/2056305118794639

CrossRef Full Text | Google Scholar

39. Nadarzynski T, Burton J, Henderson K, Zimmerman D, Hill O, Graham C. Targeted advertisement of chlamydia screening on social media: a mixed-methods analysis. Digital Health. (2019) 5:1–10. doi: 10.1177/2055207619827193

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Capurro D, Cole K, Echavarr ia MI, Joe J, Neogi T, Turner AM. The use of social networking sites for public health practice and research: a systematic review. J Med Internet Res. (2014) 16:e79. doi: 10.2196/jmir.2679

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Madden M, Lenhart A, Cortesi S, Gasser U. Pew Internet and American Life Project. Washington, DC: Pew Research Center (2010).

Google Scholar

42. Platt J, Platt T, Thiel D, Kardia S. ‘Born in michigan? you're in the biobank': Engaging population biobank participants through facebook advertisements. Public Health Genomics. (2013) 16:145–58. doi: 10.1159/000351451

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Ramo DE, Prochaska JJ. Broad reach and targeted recruitment using facebook for an online survey of young adult substance use. J Med Internet Res. (2012) 14:e28. doi: 10.2196/jmir.1878

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Hampton KN, Rainie H, Lu W, Dwyer M, Shin I, Purcell K. Social Media the “Spiral of Silence” (PewResearchCenter). Washington, DC: Pew Research Center (2014). Available online at: https://www.pewresearch.org/internet/2014/08/26/social-media-and-the-spiral-of-silence/

Google Scholar

45. UOL T. Brasil chega a 152 mi de usuários de internet; idosos estão mais conectados. (2021). Available online at: https://www.uol.com.br/tilt/noticias/redacao/2021/08/18/tic-domicilios-2020-idosos-usaram-mais-internet-uso-de-smart-tv-cresceu.htm (accessed November 20, 2021).

Google Scholar

46. Dreher PC, Tong C, Ghiraldi E, Friedlander JI. Use of google trends to track online behavior and interest in kidney stone surgery. Urology. (2018) 121:74–8. doi: 10.1016/j.urology.2018.05.040

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Glynn RW, Kelly JC, Coffey N, Sweeney KJ, Kerin MJ. The effect of breast cancer awareness month on internet search activity-a comparison with awareness campaigns for lung and prostate cancer. BMC Cancer. (2011) 11:442. doi: 10.1186/1471-2407-11-442

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Ling R, Lee J. Disease monitoring and health campaign evaluation using google search activities for HIV and aids, stroke, colorectal cancer, and marijuana use in Canada: a retrospective observational study. JMIR Public Health Surveill. (2016) 2:e156. doi: 10.2196/publichealth.6504

PubMed Abstract | CrossRef Full Text | Google Scholar

49. de Morais Pinto R, de Medeiros Valentim RA, Fernandes da Silva L, Lima GFdMS, Kumar V, Pereira de Oliveira CA, et al. Analyzing the reach of public health campaigns based on multidimensional aspects: the case of the syphilis epidemic in brazil. BMC Public Health. (2021) 21:1–13. doi: 10.1186/s12889-021-11588-w

PubMed Abstract | CrossRef Full Text | Google Scholar