Online Information of Vaccines: Information Quality, Not Only Privacy, Is an Ethical Responsibility of Search Engines

The fact that Internet companies may record our personal data and track our online behavior for commercial or political purpose has emphasized aspects related to online privacy. This has also led to the development of search engines that promise no tracking and privacy. Search engines also have a major role in spreading low-quality health information such as that of anti-vaccine websites. This study investigates the relationship between search engines' approach to privacy and the scientific quality of the information they return. We analyzed the first 30 webpages returned searching “vaccines autism” in English, Spanish, Italian, and French. The results show that not only “alternative” search engines (Duckduckgo, Ecosia, Qwant, Swisscows, and Mojeek) but also other commercial engines (Bing, Yahoo) often return more anti-vaccine pages (10–53%) than Google.com (0%). Some localized versions of Google, however, returned more anti-vaccine webpages (up to 10%) than Google.com. Health information returned by search engines has an impact on public health and, specifically, in the acceptance of vaccines. The issue of information quality when seeking information for making health-related decisions also impact the ethical aspect represented by the right to an informed consent. Our study suggests that designing a search engine that is privacy savvy and avoids issues with filter bubbles that can result from user-tracking is necessary but insufficient; instead, mechanisms should be developed to test search engines from the perspective of information quality (particularly for health-related webpages) before they can be deemed trustworthy providers of public health information.

Introduction

The World Health Organization lists vaccine hesitancy as one of the top 10 threats to global health in 2019 (1), requiring ongoing global monitoring. Despite the fact that the 1998 study, which incorrectly suggested that the MMR vaccine could cause autism in children and prompted anti-vaccine beliefs (2), has now been discredited, misinformation and, indeed, disinformation¹ about vaccines continues to be published on the Internet, perpetuating such beliefs.

It has been suggested that this misinformation plays a role in the current low uptake of vaccines in developed countries (3). Understanding whether this is actually the case, and how to address this issue, is crucial as web-based sources of health information may be instrumental to solve the sustainability challenge currently facing health systems across the globe (4).

The accuracy of information provided by a website is a key indicator of its overall information quality (IQ). The broader aspects of IQ have been the subject of many studies (5), but health IQ and trustworthiness of the sources have only partially been characterized (6). Studies looking at the influence of variations in eHealth literacy levels (7) and trust in different sources of online health information² indicate that the relationship is not linear in all cases, i.e., higher health IQ does not result automatically in higher perceived levels of trustworthiness. For example, in a study by Chen et al. (8), 618 people were recruited to complete a survey which tested their eHealth literacy level and asked them to identify which of 25 sources of health information they used and how much they trusted each source. The study showed that people with lower eHealth literacy were less likely to trust medical websites (typically higher IQ) and more likely to trust social media, blogs, and celebrity webpages (typically lower IQ) (8).

This might seem a spurious result, were it not for the fact that research has shown that those with high eHealth literacy assess more accurately the credibility and relevance of online health information, whereas those with low eHealth literacy often struggle to locate and understand eHealth information (9). This difficulty lowers their self-efficacy (10), distorts their perception of source credibility, and impacts negatively perceived trustworthiness (9), ultimately creating a need for individuals with low eHealth literacy to find an alternative means of determining trustworthiness in online sources of information. One such alternative is social endorsement. Visible social endorsement, e.g., “likes,” (11) enables those with low eHealth literacy to determine trust based on the bandwagon heuristic and assume that, if the source has already been deemed valid by others, then it is safe for them to trust it too (10, 12). Traditional medical websites afford those with low eHealth literacy no such alternative means of determining credibility and trust.

This suggests that those who are more vulnerable to the real-world effects of both disinformation and misinformation (e.g., declining to vaccinate their children) are more likely to rely on online sources with lower health IQ, which are more prone to spread such inflammatory and inaccurate information. Personalization of online search results may favor this phenomenon and lead to a vicious cycle where the more one searches and reads dis- and misinformation about vaccines, the less one finds and reads scientific information on the same topic.

At the same time, there are increasing concerns about the privacy risks associated with Internet search engines storing potentially sensitive and private health information contained within users search histories, combining it with additional information collected for tracking purposes, and using these data for commercial or other purposes (13, 14). This creates a public push back against the idea that search engines or public health providers should interfere in the results people see when searching for health information online. This makes it hard to address concerns about health disinformation/misinformation in a way that is at least socially acceptable if not ideally preferable (15). The UK's National Health Service (NHS) discovered this when it announced it would team up with Amazon to use Amazon Alexa as a voice-activated assistant that would automatically search the NHS website and respond with guaranteed high-IQ content from NHS.UK to user voice queries such as “Alexa, how do I treat a migraine.” This resulted in a public outcry over privacy infringement (16), with advocacy groups raising concerns on handing data from a public healthcare system to a private foreign company. This raises the question whether, in the context of online vaccination information, it is possible at all to balance concerns about user privacy and IQ.

The objective of this study was to address the research question: “What is the current relationship between search engines' approach to privacy and the scientific quality of the information they return?” For this purpose, we used the example of information returned by different search engines after a search on vaccines and autism. The topic was chosen not only because of its primary importance in public health, as described above, but also because the assessment of IQ is straightforward, based on the wide scientific consensus on vaccine safety and of the lack of a causal relationship between vaccines and autism. The study was performed searching the phrase “vaccines autism” in different languages (English, French, Italian, and Spanish) and comparing a wide range of search engines including the main ones (Google, Bing, Yahoo), those branded as “privacy-savvy” (Duckduckgo, Ecosia, Qwant, Mojeek, Swisscows), and some country-specific ones (Arianna and Virgilio, that do not have a specific no-tracking policy).

Methods

The term “vaccines autism” was used (in French “vaccins autism,” in Italian “vaccini autism,” in Spanish “vacunas autism.” Searches in English were done from Falmer, Sussex, United Kingdom; in Italian from Urbino, Italy; in French from Bruxelles, Belgium; and in Spanish from Barcelona, Spain. Each search was done using a logged-out Chrome browser cleared of cookies and previous search history so that the only identification was the IP address and its geolocalization. Additionally, when available, the local version of each search engine was used (e.g., Google.co.uk and Google.it). For searching Google.com, automatic redirection to Google.co.uk was avoided by using the URL Google.com/ncr (no-country-redirect).

The first 30 URL results from each search engine result page (SERP), excluding those marked as advertisements, were transferred to a spreadsheet. Pages that contained no information, aggregators, and indexes were excluded. Websites were then visited and the content of each page was coded as vaccine-positive, -negative or -neutral, depending on the stance taken on the connection between vaccines and autism.

Webpages recommending vaccination and/or negating the link with autism were coded as “vaccine-positive.” Those promoting vaccine hesitancy, cautioning about the risk of autism or openly anti-vaccine, were coded as “vaccine-negative.” Additionally, webpages that claimed further studies needed to be conducted to clarify the link between vaccines and autism were also coded as “vaccine-negative,” as previous research (17) has shown that users perceive this as confirmation of the fact that vaccine safety has not been proven. Webpages simply reporting the history of the anti-vaccine movement or a related legal debate were coded as “vaccine-neutral.” Examples of positive, negative, and neutral webpages are provided in Table 1.

TABLE 1

Table 1. Example of classification of webpages.

Coding was completed by two raters for each language, and inter-rater agreement was calculated with GraphPad, which uses equations 18.16–18.20 from Fleiss (18). On a sample of 59 webpages in English, agreement was 85%, with a Kappa of 0.669 (standard error, 0.077) and a 95% confidence interval from 0.518 to 0.820, a strength of agreement considered to be “good” (18). In Italian, agreement was 90%, with a Kappa of 0.818 (standard error, 0.067) and a 95% confidence interval from 0.687 to 0.950, a strength of agreement considered to be “very good.” In Spanish, agreement was 83%, with a Kappa of 0.609 (standard error, 0.098) and a 95% confidence interval from 0.418 to 0.801, a strength of agreement considered to be “very good.” In French, agreement was 89% with a Kappa of 0.746 (standard error, 0.091) and a 95% confidence interval from 0.568 to 0.924. Disagreements were resolved by further discussion with a third rater to reach an agreement.

When frequencies of vaccine-negative webpages were compared across different search engines, we used a two-tailed Fisher's test corrected for multiple comparison using the method of Benjamini, Krieger, and Yekutieli and a false discovery rate of 5%. Statistical analysis was performed using GraphPad Prism 8.3.0 for Windows (GraphPad Software, San Diego, CA).

The list of URLs for all searches and their coding is provided in Supplementary File 1.

Results

Figure 1 shows the ranking of positive (green), neutral (yellow), and negative (red) websites returned by the different search engines in English, French, Italian, and Spanish.

FIGURE 1

Figure 1. Stance on vaccines in webpages returned by different search engines in four languages. The top 30 webpages returned in the SERPs are shown. Green, vaccine-positive, yellow, vaccine-neutral, red, vaccine-negative.

Because the purpose of this study was to assess the ranking of misinformation, we compared the frequency of vaccine-negative webpages across the different search engines. Table 2 shows that Google is consistently returning less misinformation, although the Italian and Spanish versions of Google, as well as its UK English version, returned more vaccine-negative webpages than the English-US version (Google.com). Other search engines return more vaccine-negative webpages with some, like Mojeek in English and French or Arianna and Virgilio in Italian, more likely to rank higher webpages with misinformation.

TABLE 2

Table 2. Vaccine-negative webpages in different SERPs.

In English, the frequency of vaccine-negative webpages in Yahoo, Bing, Duckduckgo, Swisscows, and Mojeek was significantly higher than in Google.com, with Mojeek also significantly higher than Google.co.uk. In Italian, all SERPs had a higher proportion of vaccine-negative webpages compared with Google.it, but this was not statistically significant, although it was if compared with international English google.com (5% FDR). The two Italian-only search engines (Virgilio and Arianna) returned the highest number of negative pages in Italian.

In French, all search engines returned a significantly higher number of vaccine-negative webpages than Google in French. In Spanish, there were, on average, less vaccine-negative results. All the search engines tested had a higher proportion of vaccine-negative results but this was not statistically significant when compared with local Google or with Google.com.

A common feature was that the SERPs of all search engines providers contained a higher proportion of vaccine-negative results than those obtained from Google.com. However, even the localized versions of Google (Google.co.uk, Google.it and Google.es) returned more negative pages than the US/international English Google.com.

Discussion

The results indicate that currently privacy-enhancing search engines often give more visibility to webpages promoting vaccine hesitancy or with a clear anti-vaccine position than Google. This is in agreement with the findings from a recent study by the Economist which analyzed 175,000 news links returned by Google to demonstrate that the search engine's algorithm favors trustworthy publications (19). Reputation and trustworthiness are key factors included in Google's ranking algorithm. In 2019, Google published its search quality evaluation guidelines,³ which define webpages containing information that may affect the users' health or financial stability, as “your money your life” (YMYL) pages. These guidelines reveal that when rating YMYL webpages, Google looks at the three criteria of Expertise-Authority-Trustworthiness (E-A-T) and states:

“High E-A-T information pages on scientific topics should be produced by people or organizations with appropriate scientific expertise and represent well-established scientific consensus on issues where such consensus exists.”

Thus, in the case of Google, the bandwagon heuristic mechanism of assessing medical information credibility is working in favor of promoting IQ. This is perhaps because those responsible for driving up the “reputation” of specific websites by, for example, linking to them, have higher levels of eHealth literacy and therefore act as pseudo-gatekeepers protecting those with lower eHealth literacy from poor IQ results by ensuring that websites providing inaccurate online health information have a low ranking in the search results.

In the case of privacy-preserving engines, this appears not to be working—potentially because they do not track “reputation” factors, which could be seen as proxies for IQ and credibility, e.g., clicks and bounce-rate, and so have no gatekeepers (pseudo or otherwise) working to limit the circulation of misinformation. To our knowledge, the algorithms used by these “alternative” search engines are not public—Qwant states that they use their own algorithms⁴—but despite this, we found a large overlap between the SERP of Qwant and those of Bing and Ecosia. Likewise, Ecosia and Swisscows showed a similar overlap (up to 70%) with Bing, while Ecosia and Qwant often had a complete overlap. This means that it is not possible to check what factors determine the outcome of the decision making processes of the search engines algorithms and, hence, identify those elements that contribute to circulate misinformation.

It should be mentioned, however, that also non-privacy-savvy search engines, like Bing or Yahoo, often returned more vaccine-negative results than Google, also stressing that IQ is independent on privacy policy. More importantly, even localized versions of Google (UK English, Italian, and Spanish) returned more vaccine-negative results than Google.com or the one in French, indicating that many factors, including the use of non-English language and/or localization can affect the IQ of the results.

Even without being able to check the exact mechanisms for the overall poorer results, the results suggest that, currently, decisions made by search engines to prioritize privacy preservation may have a negative impact on the IQ of results returned to users in health contexts. The pledge to provide independent and unbiased results or not to promote or hide websites based on political or moral interests can be seen as ethically ambiguous, in view of the potential consequences of pointing to scientifically unsound health information.

Medical ethics requires that patients give informed consent before treatment and must, therefore, be informed accurately of the risks and benefits associated with treatment, something that is not possible if the search engine provider used by an individual is returning results with low IQ. From this consequentialist perspective, it is inherently unethical choosing not to interfere with a search engine's ranking algorithm to ensure “manually” that results of higher IQ are prioritized, while those of lower IQ are suppressed. This builds on arguments already made in the wider literature about algorithm ethics (20, 21). For example, the Association for Computing Machinery (ACM) code of ethics and professional conduct includes, as a general principle, “avoid harm” along with that of “respect privacy” (22), which aligns with the Hippocratic oath. Specifically, the implication that the current design of privacy-preserving search engine algorithms underestimates the need for evaluation of IQ fits into the wider discussion about algorithmic design and how to ensure design decisions are made to protect and incorporate key values such as IQ. It is necessary, but insufficient, to design search engine algorithms that index purely on “relevance,” they must also be designed to index on quality. The challenge lies in the ability to do this in a way that balances the need to accept different perspectives (particularly those that are rooted in different cultural, religious, or social ideals), while also filtering for IQ. Supporters of such arguments, including the authors, note that this requirement necessitates making the workings of search engine algorithms more transparent (23) to ensure their ethical compliance.

Providing information on vaccines that is based on misinformation or disinformation (including studies whose data or conclusions have been shown to be wrong) is a deceptive practice, which goes against the basic tenets of medical and business ethics (24). This is in line with those who argue that the promotion of “alternative medicine” is unethical because it lacks the evidence and transparency of clinical efficacy and should be considered “false advertising” (25). Online service providers have a moral responsibilities to ensure that users access health information that is scientifically validated (26). Misinformation and disinformation concerning healthcare circulating on the internet can have severe consequences and lead to widespread harm. Consider the example of South African president Mbeki, who delayed introducing anti-retroviral drugs in favor of alternative medicine based on information obtained from HIV-denialist internet websites, a decision estimated to have resulted in over 300,000 deaths (27). More recently, a cancer patient died in China after following an alternative, non-approved therapy they found using a search engine (28). In response, Chinese authorities issued new rules that require search engines to provide “objective, fair, and authoritative results.”

Another important aspect of IQ is its role in the context of informed consent, which is a central aspect of medical ethics, along with right to privacy and minimizing harm. It has been pointed out by Shahvisi (29) that to be informed about a treatment means not only to have knowledge but also to have an understanding of the treatment, and understandability is one of the basic dimensions of IQ (30). This raises the issue of the responsibility of search engines in the context of the existing health information to make informed and autonomous choices. If people make autonomous decisions based on the information obtained on the Internet (something that is often incorrectly called “doctor google” effect), one may argue whether search engines have a responsibility to provide high quality information. This is clearly a new challenge in medical ethics, and has been discussed, for instance, in the context of online information on medical tourism (31). The efforts made by the big internet companies, from search engines to social media, to pay particular attention to the IQ of health information indicates that they are well aware of the responsibility that comes with their role.

Focusing on data privacy without addressing the aspect of health IQ and its role in informed consent is a deficiency in the smaller search engines that would ultimately impact on the process of informed consent. This study, although limited to the information on vaccines, highlights an unregulated gray area for which search engines and regulators should be ethically responsible and that will need being addressed.

Moral responsibility follows on the level of harm that misinformation and disinformation on healthcare may cause. It could be argued that there is a greater onus on Google—and other commercial search engine providers—than on alternative search engines to take these considerations into account when designing or interfering with algorithms for the purposes of promoting ethical compliance, given their larger market share. Google currently has over 90% of the worldwide market share⁵ and therefore has the potential to indirectly “cause harm,” through the potential promotion of low IQ webpages on vaccinations, to a great many more people than any of the alternative providers.

However, studies have shown that those who hold anti-authoritarian views, openness to (potentially) controversial opinions, and an interest in alternative medicine are, in some cases, already more likely to hold vaccine-hesitant beliefs (17). These are also the individuals who are most likely to use alternative search providers given their sensitivity to privacy and tracking concerns. Therefore, while the alternative providers might reach a proportionally smaller audience—they are reaching an audience that is already more receptive to anti-vaccine information (32) and therefore more vulnerable to its effects. In other words, unless the alternative providers take steps to rank vaccine-related search results according to IQ, they may cause greater harm to those they do reach, meaning that the net negative impact is still greater even though the number of individuals they reach is smaller.

Conclusion

Our analysis shows that while it may well be technically possible to design a search engine that manages to balance privacy-preservation with the promotion of high IQ material, this is currently not the case. The current relationship between privacy-preserving design features of search engines and the IQ of the results they return is inverse (although not proportionally). In instances where this can have harm public health, as in the example we have provided of the promotion of anti-vaccine misinformation, not intervening to alter the design of the algorithm—even if this means sacrificing some degree of user privacy—can lead to severe harm for a large population of users and is, therefore, unethical.

Designing a search engine that is privacy savvy and avoids issues with filter bubbles that can result from user-tracking may be a good thing, and in fact this aspect is, at least in part, regulated from the perspective of data protection—which is primarily interpreted as data security rather than data privacy. Our study suggests that this is necessary but insufficient, and instead mechanisms should be developed to test search engines from the perspective of IQ (particularly for YMYL webpages) before they can be deemed trustworthy providers of health information.

Data Availability Statement

All datasets generated for this study are included in the article/Supplementary Material.

Author Contributions

PG and MG designed the research. PG, PB, GC, AC, MG, MN, AP-B, PS, and TV performed the research and analyzed the data. PG, JM, MT, and LF analyzed the data and wrote the paper. All authors contributed to the article and approved the submitted version.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fmed.2020.00400/full#supplementary-material

Footnotes

1. ^Misinformation is incorrect information that stems from human error e.g., a lack of fact checking, whereas disinformation is purposefully and deliberately incorrect. Both matter in this context because whilst the original 1998 study may have been an example of misinformation, it is possible that maleficent actors are now purposefully spreading disinformation for the purpose of undermining public health.

2. ^eHealth literacy is defined by Norman and Skinner (7) as the ability to seek, find, understand, and appraise online health information and apply the knowledge gained to addressing or solving a health problem. This is why we use the term eHealth literacy even though this article focuses exclusively on the IQ of online health information —not broader sources of eHealth information such as electronic health records or wearable devices.

3. ^https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityevaluatorguidelines.pdf

4. ^https://medium.com/qwant-blog/web-indexation-where-does-qwants-independence-stand-8eab4f7856f8 (Archived at: https://web.archive.org/web/20190627090553/https://medium.com/qwant-blog/web-indexation-where-does-qwants-independence-stand-8eab4f7856f8).

5. ^https://gs.statcounter.com/search-engine-market-share

References

1. World Health Organization. Ten Threats to Global Health in 2019. (2019). Available online at: https://www.who.int/emergencies/ten-threats-to-global-health-in-2019 (archived at https://web.archive.org/web/20190627111040/https://www.who.int/emergencies/ten-threats-to-global-health-in-2019) (accessed June 27, 2019).

Google Scholar

2. Wakefield AJ, Murch SH, Anthony A, Linnell J, Casson DM, Malik M, et al. Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. Lancet. (1998) 351:637–41. doi: 10.1016/S0140-6736(97)11096-0

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Stahl JP, Cohen R, Denis F, Gaudelus J, Martinot A, Lery T, et al. The impact of the web and social networks on vaccination. New challenges and opportunities offered to fight against vaccine hesitancy. Med Mal Infect. (2016) 46:117–22. doi: 10.1016/j.medmal.2016.02.002

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Martin GP, Sutton E, Willars J, Dixon-Woods M. Frameworks for change in healthcare organisations: a formative evaluation of the NHS Change Model. Health Serv Manage Res. (2013) 26:65–75. doi: 10.1177/0951484813511233

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Illari P, Floridi L (editors). Information quality, data and philosophy. In: The Philosophy of Information Quality. Cham: Springer (2014). p. 5–23.

Google Scholar

6. Al-Jefri M, Evans R, Uchyigit G, Ghezzi P. What is health information quality? Ethical dimension and perception by users. Front Med. (2018) 5:260. doi: 10.3389/fmed.2018.00260

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Norman CD, Skinner HA. eHEALS: the ehealth literacy scale. J Med Internet Res. (2006) 8:e27. doi: 10.2196/jmir.8.4.e27

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Chen X, Hay JL, Waters EA, Kiviniemi MT, Biddle C, Schofield E, et al. Health literacy and use and trust in health information. J Health Commun. (2018) 23:724–34. doi: 10.1080/10810730.2018.1511658

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Paige SR, Krieger JL, Stellefson ML. The influence of ehealth literacy on perceived trust in online health communication channels and sources. J Health Commun. (2017) 22:53–65. doi: 10.1080/10810730.2016.1250846

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Kim J, Park HA. Development of a health information technology acceptance model using consumers' health behavior intention. J Med Internet Res. (2012) 14:e133. doi: 10.2196/jmir.2143

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Borah P, Xiao X. The importance of 'likes': the interplay of message framing, source, and social endorsement on credibility perceptions of health information on facebook. J Health Commun. (2018) 23:399–411. doi: 10.1080/10810730.2018.1455770

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Huo C, Zhang M, Ma F. Factors influencing people's health knowledge adoption in social media: the mediating effect of trust and the moderating effect of health threat. Library Hi Tech. (2018) 36:129–51. doi: 10.1108/LHT-04-2017-0074

CrossRef Full Text | Google Scholar

13. Chiou L, Tucker C. Search engines and Data Retention: Implications for Privacy and Antitrust. Massachusetts Institute of Technology Sloan School Working Paper 5094–14. Cambridge, MA: National Bureau of Economic Research (2017).

Google Scholar

14. Esteve A. The business of personal data: Google, Facebook, and privacy issues in the EU and the USA. Int. Data Privacy Law. (2017) 7:36–47. doi: 10.1093/idpl/ipw026

CrossRef Full Text | Google Scholar

15. Floridi L, Taddeo M. What is data ethics? Philos Trans Royal Soc A. (2016) 374. doi: 10.1098/rsta.2016.0360

CrossRef Full Text | Google Scholar

16. Moskvitch K. Amazon's Alexa Deal With the NHS Can Help Patients but at What Cost? Wired. (2019). Available online at: https://www.wired.co.uk/article/amazon-alexa-nhs (accessed August 11, 2019).

Google Scholar

17. Browne M, Thomson P, Rockloff MJ, Pennycook G. Going against the herd: psychological and cultural factors underlying the 'vaccination confidence gap'. PLoS ONE. (2015) 10:e0132562. doi: 10.1371/journal.pone.0132562

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Fleiss JL, Levin B, Paik MC. Statistical Methods for Rates and Proportions. Hoboken, NJ: John Wiley & Sons (2013).

Google Scholar

19. Economist T. Seek and you shall find. Google rewards reputable reporting, not left-wing politics. Economist. (2019) 431:85.

Google Scholar

20. Taddeo M, Floridi L. The debate on the moral responsibilities of online service providers. Sci Eng Ethics. (2015) 1–29.

PubMed Abstract | Google Scholar

21. Mittelstadt BD, Allo P, Taddeo M, Wachter S, Floridi L. The ethics of algorithms: mapping the debate. Big Data Soc. (2016) 3:2053951716679679. doi: 10.1177/2053951716679679

CrossRef Full Text | Google Scholar

22. Association for Computing Machinery. ACM Code of Ethics and Professional Conduct. (2018). Available online at: https://www.acm.org/code-of-ethics (archived at https://web.archive.org/web/20190627114921/https://www.acm.org/code-of-ethics) (accessed June 27, 2019).

Google Scholar

23. Tufekci Z. Algorithmic harms beyond Facebook and Google: emergent challenges of computational agency. Colo Tech LJ. (2015) 13:203.

Google Scholar

24. Hasnas J. Teaching business ethics: the principles approach. J Business Ethics Educ. (2013) 10:275–304. doi: 10.5840/jbee20131014

CrossRef Full Text | Google Scholar

25. Macdonald C, Gavura S. Alternative medicine and the ethics of commerce. Bioethics. (2016) 30:77–84. doi: 10.1111/bioe.12226

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Taddeo M, Floridi L. The debate on the moral responsibilities of online service providers. Sci Eng Ethics. (2016) 22:1575–603. doi: 10.1007/s11948-015-9734-1

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Chigwedere P, Seage GR 3rd, Gruskin S, Lee TH, Essex M. Estimating the lost benefits of antiretroviral drug use in South Africa. J Acquir Immune Defic Syndr. (2008) 49:410–5. doi: 10.1097/QAI.0b013e31818a6cd5

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Ouyang Y. Student's death highlights gaps in China's health regulations. Lancet Oncol. (2016) 17:709. doi: 10.1016/S1470-2045(16)30164-4

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Shahvisi A. No understanding, no consent: the case against alternative medicine. Bioethics. (2016) 30:69–76. doi: 10.1111/bioe.12228

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Lee YW, Strong DM, Kahn BK, Wang RY. AIMQ: a methodology for information quality assessment. Information Manage. (2002) 40:133–46. doi: 10.1016/S0378-7206(02)00043-5

CrossRef Full Text | Google Scholar

31. Penney K, Snyder J, Crooks VA, Johnston R. Risk communication and informed consent in the medical tourism industry: a thematic content analysis of Canadian broker websites. BMC Med Ethics. (2011) 12:17. doi: 10.1186/1472-6939-12-17

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Bean SJ. Emerging and continuing trends in vaccine opposition website content. Vaccine. (2011) 29:1874–80. doi: 10.1016/j.vaccine.2011.01.003

PubMed Abstract | CrossRef Full Text | Google Scholar