Understanding news-related user comments and their effects: a systematic review

Kubin, Emily; Merz, Pascal; Wahba, Mariam; Davis, Cate; Gray, Kurt; von Sikorski, Christian

doi:10.3389/fcomm.2024.1447457

REVIEW article

Front. Commun., 23 July 2024

Sec. Media Governance and the Public Sphere

Volume 9 - 2024 | https://doi.org/10.3389/fcomm.2024.1447457

Understanding news-related user comments and their effects: a systematic review

Emily Kubin^1,2

Pascal Merz¹

Mariam Wahba²

Cate Davis²

Kurt Gray²

Christian von Sikorski¹^*

¹Department of Psychology, University of Kaiserslautern-Landau (RPTU), Landau, Germany
²Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, Chapel Hill, NC, United States

There has been growing interest in research on news-related user comments. Here we conduct the first systematic review of this literature—quantitatively and qualitatively (248 studies)—that covers the entire communication process (content analyses, surveys, experiments). Results indicate a focus on online news articles (vs videos) and little consideration for major social media platforms (Instagram, TikTok). Research often assesses incivility in comments but offers conflicting conclusions on the actual level of incivility in comment threads—and seldom considers how to effectively combat any incivility. We propose four priorities for future work: more comparative and longitudinal approaches; exploring social media and video content; examining platform design, content moderation and artificial intelligence; and implementing measures to reduce incivility and protect the integrity of journalism.

Introduction

User-generated content (UGC) represents an ever-increasing proportion of media content where users evaluate, disseminate and comment on news (Bruns, 2005; Hermida and Thurman, 2008). User comments are the most prominent form of UGC and user participation (Stroud et al., 2016b) and are usually presented below news articles, thus reaching the same audience as professionally journalistic content (Springer et al., 2015).

From the perspective of democratic theory, user participation in comment sections was welcomed in the early 2000s, when online newspapers included user comments underneath news content (Santana, 2011). By 2013, 90% of news platforms in the United States provided a comment section on their website (Stroud et al., 2016a). Many celebrated the benefits of comment sections, suggesting it would usher an era of deliberative democracy (Habermas, 1962; Engelke, 2020) and create new public spaces where people easily and quickly voice their opinions—allowing citizens to understand what others think about issues (Lee and Jang, 2010).

On the other hand, others have been pessimistic (including journalists; Bergström and Wadbring, 2015) arguing that user comments do not enrich public discourse and comment sections do not automatically promote healthy democratic exchange. User comments can spread negativity (Winter et al., 2015), misleading or false information (Anspach and Carlson, 2020), and extreme populist ideas (Blassnig et al., 2019). Further, comments can include uncivil language (Anderson et al., 2014) that insults others (i.e., dark participation, Frischlich et al., 2019).

Researchers also recognized these user comment sections impact others who read comment threads—swaying attitudes (Lee and Jang, 2010), risk perceptions (Anderson et al., 2014), public opinion perceptions (Lee et al., 2021) and behaviors (Hsueh et al., 2015). Importantly, many recognized user comments can hurt the reputations of journalists and media platforms—making them seem less credible and trustworthy (Walther et al., 2010; von Sikorski and Hänelt, 2016; Naab et al., 2020). Some news organizations like USA Today, Reuters and NPR have responded by dropping comment sections from their websites.

Given the disconnect between what many hoped news-related user comments could do to help society and public discourse (Habermas, 1962, 2022; Engelke, 2020), and findings suggesting news-related user comments may be detrimental, it is imperative to comprehensively assess current understandings of news related user comments (and their effects).

Over the past two decades, the number of quantitative studies focusing on news-related user comments has exploded and the large number of results now obscures central findings. Researchers have begun to present initial overviews of sub-areas of the field (e.g., on the content of user comments), but an up-to-date and systematic overview, from the content to the effects of user comments, is lacking.

Researchers have taken first steps in systematically examining findings related to user comments. Naab and Sehl (2017), conducted a systematic review of content analyses regarding user comments published in nine academic journals between 2004 and 2012. They explored processes such as the theoretical frameworks employed by scholars and the modality examined (e.g., text vs. moving image). Reimer et al. (2023) conducted another systematic review of content analyses of user comments, primarily focused on methodology. They found an overabundance of analyses from Anglo-American newspapers and a focus on incivility.

While these systematic reviews provide valuable insights into user comments research, and others have added to the literature through non-systematic reviews of user comments (e.g., Ksiazek and Springer, 2018; Naab and Küchler, 2023), we suggest it is essential to systematically re-assess the current state of knowledge in the literature. Given the fast paced and changing media market (Newman et al., 2023), it is essential to re-examine current understandings of user comments frequently. However, previous systematic reviews included papers published by 2012 in nine pre-selected academic journals (Naab and Sehl, 2017) and by 2016 (Reimer et al., 2023)—meaning any recent insights have yet to be considered within a systematic review. Further, the systematic reviews focused on current understandings of content analyses (i.e., what comments look like) and less so on many other questions related to comments. For example, what commenting is related to, and the effects of commenting.

In the current research we are the first to conduct a broad systematic review—quantitatively and qualitatively oriented—where we consider the contents of user comments, motives that predict commenting, demographic factors like gender, news use, and levels of political participation that correlate with user comments, and effects of news-related user comments. See Table 1. Additionally, we report our systematic review in-line with PRISMA guidelines for writing reviews (Page et al., 2021).

Table 1

Table 1. Comparison of previous systematic reviews of user comments as compared to current systematic review.

While past reviews provide meaningful insights, questions regarding news-related user comments remain. In the current research, we close these gaps and answer three research questions. RQ1 is quantitatively oriented and asks: How can the current state of research on news-related user comments be characterized in terms of (a) the development of the research over time, (b) the demographics of samples, (c) and the methods used? RQ2 is qualitatively oriented and asks: What do we know about news-related user comments regarding (a) the content of comments (e.g., are user comments frequently uncivil?) (b) what factors correlate with commenting (e.g., is commenting associated with news use?) and (c) the effect of comments (e.g., can user comments shape others’ attitudes?). The third research question (RQ3) is both quantitatively and qualitatively oriented and asks: How are news-related user comments examined in the literature?

Methods

For this systematic review, we conducted analyses using a 2-stage approach. In Stage 1, we determined which papers fit the criteria for being included in our review using a systematic coding approach. In Stage 2, we systematically coded papers (a quantitative approach), and extracted key findings from each paper (a qualitative approach). Further details about each step are found below.

Stage 1: determining inclusion criteria

To answer our research questions, we conducted a systematic literature search using Web of Science (aligned with previous systematic reviews; see Ahmed and Matthes, 2017; von Sikorski, 2018; Tsfati et al., 2020; Kubin and von Sikorski, 2021) to identify potentially relevant articles. Our search terms focused on user-generated comments and news media. Example search terms include: “user comment,” “reader comment,” “online comment,” and “news,” “news article,” “press coverage.” We searched for articles that referenced both user comments and news—checking article titles, abstracts, and the keywords of articles. Through this search process (conducted on April 4th, 2022),¹ 553 articles were identified that contained a combination of keywords from both categories.

Our Web of Science search was complemented by also considering articles from other available reviews about news-related user comments (Ksiazek and Springer, 2018; Reimer et al., 2023). In this step, we included articles that were accessible through Web of Science that were not already included in our search query (N = 57).² Reasons why these papers were not included related to divergent terminology used by authors (e.g., using the term “post” rather than “comment” in the abstract; Colley and Maltby, 2008). This resulted in a total of 610 papers that were further examined regarding whether they should be included in this review.

In a next step, two trained research assistants independently assessed a total of 610 papers (based on instructions provided in a codebook; see Appendix A) to determine which papers would be included in the systematic review. Articles were included in the sample when: (a) the article was in English, (b) the article was quantitative in nature and published in an academic journal, and (c) the article focused on the content of comments, constructs that correlated with user comments or the effects of news-related user comments. To ensure reliability, approximately 10% of the identified research papers were assessed by both research assistants (there was 89.29% agreement between research assistants). Based on our exclusion criteria we identified 258 articles that should be coded in this systematic review (See Figure 1).

Figure 1

Figure 1. Flow chart of paper identification, inclusion determination, coding criteria, reliability analysis and the methodology utilized in papers included in systematic review.

Stage 2: coding included papers

Two research assistants coded the included papers (N = 258) on a variety of dimensions following a systematic codebook (see Appendix B). Both research assistants were extensively trained³ and read through each paper—coding a variety of quantitative and qualitative categories for each relevant study within a paper. Each research assistant was responsible for independently coding half of all papers (i.e., approximately 130 papers each). Key quantitative categories include year of publication, study method, country of sample, the news source, type of news content, whether the study was focused on news or social media, if on social media—which social media site was used, and whether the paper explicitly studied incivility.⁴ Research assistants also reported the topics being considered within the study (e.g., what topic the news content discussed). A more qualitative category was related to key findings from each study.

To ensure reliability, both coders and the 1st author coded the same 24 papers. There was high reliability in codes between the 1st author and RA 1 (κ = 0.93),⁵ the 1st author and RA 2 (κ = 0.97), and between RA 1 and RA 2 (κ = 0.89). Reliability analysis indicated interrater reliability was moderately to strongly reliable. More information about reliability can be found in Appendix C. All articles included in this review, their codes, and codebooks can be found here: https://osf.io/swx76/?view_only=30aa778596aa4c469634fe84a3de63bb. We encourage other scholars to use these materials in future research on user comments.

During the coding process, we recognized some papers (N = 27) that were originally included in this systematic review did not actually qualify (e.g., because the article was more qualitatively focused than originally thought or because the comments were not news-related). This was determined after a more careful and complete read through during the coding process—231 papers (248 studies) were included in analyses.

Results

We present both quantitative and qualitative results. Due to space limitations, we cannot report every finding from every paper, but provide a broad scope for understanding the key areas of agreement (and disagreement) between scholars, key methods and perspectives, and overarching patterns (and gaps) in current understandings of news-related user comments.

Quantitative analyses

Year of publication

We answer RQ1a (i.e., assessing development of research over time), by assessing when articles on news-related user comments were published, results revealed growing interest in this research. The earliest article was published in 2008—since then there has been exponential growth in publications. See Figure 2.

Figure 2

Figure 2. Number of publications on news-related user comments per publication year. For papers published as Early Access papers on the day of retrieval (April 4th 2022), we checked when those papers were published first online and included that information in the graph above. Of these early access papers; 4 papers were published first online in 2020, 8 in 2021, and 9 in 2022. We do not depict the papers published in 2022 as we only collected papers published by April 4th 2022–thus we had incomplete data for 2022. Therefore 4 published papers (and 9 early access papers) are excluded from this figure.

Sample characteristics

To assess RQ1b (i.e., the demographics of samples in user comment research), we next focused on where samples came from. We found an overabundance of research conducted in Western societies—especially in the United States (N = 100) and Germany (N = 46), but also found numerous studies in South Korea (N = 18). While there was a lot of research conducted within Western societies––there was still great diversity in samples (e.g., in India, Israel, China, Russia, and Chile)—though there were few samples from the Global South. See Figure 3.

Figure 3

Figure 3. Distribution of samples across countries. Red circles represent samples from that country, the larger the circle the more samples from that country. 13 studies did not provide clear information regarding the country the sample came from and 7 studies included worldwide samples (e.g., recruiting participants through the Internet). These studies were not included in the above map.

Methodology

Answering RQ1c (i.e., regarding methodology), we found that studies exploring news-related user comments frequently used content analyses (131 studies, 52.82%), followed by experiments (109 studies, 43.95%), and then surveys (7 studies, 2.82%).

News source

To answer RQ3 (i.e., how news-related user comments are examined in the literature), we explored what kinds of news sources (and their comments) were examined. Most studies (221 studies, 89.52%) solely considered news articles (and their accompanying user comments). Of which, only 49 (22.17%) assessed effects in non-Western samples. Thirteen studies (5.24%) considered news videos (e.g., TV content) and their accompanying comments, though only 2 of these studies came from non-Western samples (15.38%). Some studies (13 studies, 5.24%), considered a variety of news sources within a given study (e.g., consider both news articles and TV news).

Traditional news vs. social media news content

To further answer RQ3, we coded for whether researchers considered comments under news content on a news site or social media. Results indicated 72.18% of studies (N = 179) focused solely on traditional news content, 19.35% of studies (N = 48) focused solely on news media content on social media (e.g., news article posted to Facebook) and 6.85% of studies (N = 17) considered both social media and traditional news media content.

Social media platforms

Of the studies considering news content on social media (N = 65), most assessed news shared on Facebook (N = 52, 80.00%), though only 5 of those studies that focused on Facebook used non-Western samples (11.54%)—a serious gap in the literature given Facebook’s much higher prevalence in non-Western societies (Statista, 2024). Six studies considered content on Twitter (9.23%) (only 1 of which was a non-Western sample), only 2 studies (3.08%) considered news-related user comments on Instagram (1 was a non-Western sample), and 10 studies explored content on YouTube (15.38%), 2 of which focused on non-Western samples. There were several studies (4 studies, 6.15%) that used less well-known social media platforms (e.g., Vkontakte; Toepfl and Litvinenko, 2021). No studies explored user comments under news content on TikTok. These results (in part) answer RQ3 by explaining the contexts and places where user comments are frequently assessed.

Outcome measures

We also coded whether or not studies focused on questions related to the incivility of user comments—believing this could be a key outcome variable. Results suggested a sizeable proportion of studies did research incivility (N = 72, 29.03%). This outcome measure is discussed more in the qualitative section.

Qualitative analyses

In addition to our quantitative analyses, we also took a more qualitative approach in understanding the topics and themes of included studies to further assess RQ2 and RQ3.

Diversity in news and user comment topics

There were a variety of topics discussed in the news content (and comments) across studies—providing some partial insights in RQ3 (i.e., how user comments are examined in the literature). Topics ranged from political (elections; Saldana and Rosenberg, 2020) to apolitical topics (celebrity news; Van den Bulck and Claessens, 2014; sports; Waddell, 2020). We outline some key topics frequently studied below.

Many studies considered hot-button issues in the news. Examples include climate change (e.g., Walter et al., 2018), gun policy (Stroud et al., 2015; Gearhart et al., 2020), and immigration (e.g., Blassnig et al., 2019). Many studies also considered social issues like LGBTQ adoption rights (Wang, 2020) and abortion access (e.g., Chen and Ng, 2017).

There was a heavy focus on health-related topics. Many considered vaccinations (Masullo et al., 2021), mental health care (Cabrera et al., 2018), and disease outbreak (Rodin et al., 2019). Some news content focused on scandals (von Sikorski and Hänelt, 2016).

Finally, many studies considered news content (and comments) about science communication and education. Examples of science communication include reports on research about differences between men and women’s brains (O’Connor and Joffe, 2014) and diabetes (Vehof et al., 2019). In terms of education, scholars considered laws for schools (Lee, 2012), lawsuits against schools (Sherrick and Hoewe, 2018) and appropriate conduct between educators and students (Jahng, 2018). Taken together, and answering RQ3, there is a great variety of contexts where user comments are analyzed.

Key results from user comments research

To understand the overarching themes of results across studies, and to answer RQ2, we next break down results into three subsections: the content of comments (RQ2a), what factors correlate with comments (RQ2b), and the effects of comments (RQ2c).

The content of user comments

To answer RQ2a, we first qualitatively examined all content analyses–exploring key findings related to the contents of comments.

Incivility

Many content analyses explored incivility, examining what drives incivility and the frequency of incivility in comments. Some research suggests when there are more sources referenced within a news article and there is journalistic participation in comment sections, people leave fewer uncivil comments (e.g., Ksiazek, 2018). Incivility was common under news stories about politics (Szabo et al., 2021). Further, there was consistent evidence that anonymity on platforms drives incivility (Ksiazek, 2015).

Others explored whether incivility varies by platform. Researchers found conservative news and local news (as compared to liberal and national news) tends to include more uncivil comments (Su et al., 2018). Others found uncivil commenting may be more common in conservative news (vs. liberal news) (Chen et al., 2019). These results suggest conservative media may insight greater incivility in user comments.

Another important consideration that had little attention is whether there are societal differences in commenting uncivilly. In one study, comments under Chilean news were found to be especially uncivil (and more so than comments under news from the Global North; Saldana and Rosenberg, 2020). In another analysis, uncivil comments occurred more often in majoritarian (rather than consensus-based) democracies (Jakob et al., 2023)⁶. These results indicate societal differences may be a key factor for understanding the incivility of comments—but more research is needed to explore these differences.

A looming question is whether incivility is increasing over time and in its pervasiveness. Again, there has been little research focused on this longitudinal perspective, though in one study in Hungary between 2017 and 2019, researchers found incivility in comments remained stable over time (Szabo et al., 2021).

Some scholars suggest incivility in news-related user comments is not widespread (e.g., Aji and Sapto, 2020). In one study, comments posted under news videos on YouTube showed higher levels of civility than hostility (Ksiazek et al., 2015). However, other research suggests incivility is common in online discussions (e.g., San Pascual, 2020). Others find that incivility is more common than other more deliberative forms of commenting (Chen et al., 2020). These results suggest an ongoing debate in scholarly research regarding incivility.

Positive and negative comments

Some content analyses examined the positive and negative sentiment of comments. For example, people responded more positively to news articles when those articles discuss cures for diabetes as compared to preventative measures for diabetes (Vehof et al., 2019). These findings are in-line with other research suggesting whether people respond to news content positively or negatively is highly dependent on the topic itself (e.g., Cabrera et al., 2018). Several studies made inferences about peoples’ attitudes towards specific issues depending on whether they responded positively or negatively to news content. For example, to understand how people feel about the European Union (Galpin and Trenz, 2019).

Additional themes within the content of user comments

While many studies focused on the incivility or sentiment in user comments, others found that commenters often discuss themes or topics that are seemingly disconnected or unrelated to the news content itself (e.g., Jaques et al., 2019). However, other research disputes this idea (Lee and McElroy, 2019).

In some research, scholars focused on how people feel towards specific policies or political issues by analyzing the content of their comments. For example, exploring how people felt about Driving Under the Influence (DUI) enforcement (Connor and Wesolowski, 2009), migration issues (Ademmer et al., 2019), beliefs about organic foods (Danner and Thøgersen, 2022), national safety and security (Bogain, 2020), celebrity suicide (Rosen et al., 2020), vaccine effectiveness (Lei et al., 2015), and trigger warnings (George and Hovey, 2020).

Several studies explored the use of narratives and facts and data within user comments—a promising area of research as past work suggests narratives can be persuasive (e.g., Oschatz and Marker, 2020) and help bridge divides between people who disagree (Kubin et al., 2023). Scholars find narratives are commonplace in comment threads (Cabrera et al., 2019) and that many commenters discussed personal opinions rather than facts or data (Lee and McElroy, 2019).

The predictors of user comments

We found that other scholars assessed the predictors of user comments and their contents (e.g., what causes people to comment, which factors shape certain types of commenting behavior). In these studies, authors examine the content of user comments (e.g., content analyses), but focus on what predicts this content.

How news shapes commenting

Some research explored how the type of content in news media can influence if and how people comment—considering the news content itself or differences across news platforms (e.g., Stroud et al., 2016a)— key considerations for understanding commenting behavior (Szabo et al., 2021; Aldous et al., 2023). Some found that news content about political and social issues are especially likely to be commented on (Tenenboim and Cohen, 2015), and that sentiment and polarizing language in news media influences commenting behaviors (Arapakis et al., 2014). Additionally, news events that have geographic proximity to readers increases commenting behavior (Maier, 2015), but there is no evidence of differences in volume of comments below hard and soft news (Ksiazek et al., 2016). However, understanding when people are likely to comment is complex as others have found that distributions of which types of news content people comment on can differ across news media sources (Ürper and Çevikel, 2016), and may be dependent on whether news reports are in-line with readers’ views (Kim et al., 2021).

In terms of the potential differing effects of news platforms (or news sources) on comments, there is less research. Some suggest news outlets with larger reach are more likely to have comment sections and are more likely to have numerous comments below their content (Stroud et al., 2016a). Multiple studies also explored differences in commenting across platforms—finding meaningful differences between news sites (Esau et al., 2017).

Not only can news content and platforms influence when people write comments, but they also shape opinions expressed in comments. Multiple studies found comments mirrored the views presented in news content (e.g., Lien, 2022)—though others find this may not be the case (e.g., Holton et al., 2014), and that comment sections may push back against media portrayals (e.g., Taylor et al., 2016).

Demographics and motives of commenters

Other studies explored who comments (and the motives for why they do it). Results indicated that while women participate less than men and tend to write more civil comments than men, women still receive similar numbers of uncivil replies as men (Küchler et al., 2023). Further, women are more likely to leave comments under news related to social issues and celebrities whereas men often leave comments under news about politics and sports (Lee and Ryu, 2019). There was evidence for differences in where people comment depending on age. People in their 30s and 40s are more likely to post comments than other age groups and teenagers are especially unlikely to comment (Lee and Ryu, 2019).

Other scholars explored how political ideology may shape user comments. For example, liberals were more willing to include cross-cutting justifications and questions within their comments than conservatives (Freelon, 2015), and moral disengagement was significantly more common in comments on right-wing (vs. left-wing) newspapers (Woods et al., 2018). The political slant of user comments in conservative media paralleled news outlets’ political leaning; yet, a similar pattern was not observed in progressive medias’ comment sections (Han et al., 2023). Taken together, these results suggest demographic considerations are important for considering who comments under which news content.

Based on analyses of comments, scholars have also pointed to peoples’ motivation for commenting as connected to their motives for information sharing, entertainment, and voicing disapproval (Otieno et al., 2021). Several scholars showed that comments can also be used as a tool to counter public opinion. For example, multiple studies found that minority groups (e.g., political movements) can make up large majorities of comments sections in leading newspapers’ comments threads—making their ideas potentially powerful in countering majority opinion (e.g., De Kraker et al., 2014; Toepfl & Piwoni, 2015).

A few studies explored the differences between frequent and infrequent commenters. Some found that frequent commenters are more civil (Coe et al., 2014), but others found frequent commenters are less likely to be civil (Blom et al., 2014). Others find when ones’ own incivility is reaffirmed by others (e.g., by up voting this content), they are more likely to continue commenting uncivilly (Shmargad et al., 2022).

Downstream consequences in comments threads

Some content analyses also explored the consequences of comments in terms of how they shaped further commenting. Results indicate populist themes (Blassnig et al., 2019), and controversial comments (Ziegele et al., 2014) all drove others to comment. On the other hand, comments that sowed doubt were less popular and were not recommended as frequently as other types of comments (Evans et al., 2023).

Correlates of user comments

As noted earlier, very few studies were survey based (i.e., correlational), meaning there are still many gaps in our understanding of what factors are correlated with news-related user comments (answering RQ2b). One study found that commenting was positively associated with political participation (Brundidge et al., 2014). Others focused on how perceptions of user comments are related to who people believe is responsible for moderating user comment content (Riedl et al., 2021). Another study found sadistic personality traits were more associated with incivility in user comments (Beckert and Ziegele, 2020).

The effects of user comments

We also qualitatively examined insights from experimental research—exploring the effect of comments to answer RQ2c.

Incivility

In experiments focusing on the consequences of user comments, many scholars focused on how uncivil comments influence others’ attitudes and perceptions—there was clear evidence that incivility was associated with negative effects. Readers were more likely to have hostile cognitions (Rösner et al., 2016), discussions in online threads were less deliberative (Lück and Nardi, 2019), caused negative emotions and aggressive intentions (Chen and Lu, 2017), and increased perceptions of attitude polarization in society (Hwang et al., 2014).

There was nearly consistent evidence that uncivil comments provoked negative perceptions of news content and journalists. Uncivil comments reduced perceptions of journalistic quality (Weber et al., 2019) and credibility (Searles et al., 2020; Masullo et al., 2023) whereas civil comments increased credibility of news outlets (Masullo et al., 2023). However, one study suggested uncivil (vs civil) comments did not influence participants’ perceptions of the credibility of the news story (Kim and Chen, 2021). Uncivil comments invoked negative attitudes towards the editorial and pushed people away from the editorial’s point of view (Liu and McLeod, 2019). Journalists can deal with uncivil comments using content moderation—however there are consequences of this as well. While uncivil comments in moderated environments are less likely to cause perceptions of news article bias (Yeo et al., 2019), some evidence suggests this is only the case when threads are moderated by machines (rather than journalists; Wang, 2021)—suggesting artificial intelligence may be a helpful tool to combat the negative effects of incivility in user comments.

Uncivil comments also shaped behavior. People were more likely to engage with uncivil comment threads (e.g., Kim and Park, 2019)—and react negatively towards them (e.g., Chen and Ng, 2017). One study found conservatives (but not liberals) were more likely to engage with uncivil comments when those comments discussed politically divisive issues (Su et al., 2021). Others found that civil comments also increased engagement in discussions (Molina and Jennings, 2018).

Thoughts about uncivil commenters tended to be negative. People believed commenters writing uncivil comments should be censored or moderated (Wang and Kim, 2020; Naab et al., 2021), and information conveyed through uncivil comments was less trustworthy (Graf et al., 2017) than other types of comments—though other research suggests people still believe these comments will be persuasive to others (Wang and Kim, 2020). Further, other research suggests people see uncivil comments from in-group members as less uncivil than uncivil comments from out-group members (Kim, 2018). Overall, there is a strong focus on research examining the effects of uncivil comments; future studies could shed further light on the effects of civil news-related user comments.

News evaluations

Beyond the many studies that suggested uncivil comments can negatively affect attitudes towards news content and journalists (Weber et al., 2019; Searles et al., 2020; Masullo et al., 2023), others have found that user comments shape evaluations of news media (e.g., Clementson, 2019). In terms of attitudes towards the news platform, results indicated it was beneficial for journalists to respond directly to commenters, as this increased positive evaluations (Masullo et al., 2022). News coverage with positive user comments was evaluated more positively than news coverage with negative comments (Dohle, 2018). Further disrespectful comments led readers to view the editorial more negatively (Liu and McLeod, 2019).

In terms of news credibility, people were more likely to question the credibility of the news source when comments suggested it was “fake news” (Jahng et al., 2021), or included negative sentiment (Waddell, 2018). Also, mixed comments (a mix of positive and negative comments) negatively affected readers’ perceived quality of a news article (von Sikorski and Hänelt, 2016). News articles were seen as less credible when a comment critiqued the news—but only when that comment was also “liked” by other readers (Naab et al., 2020). Others found no effect of news credibility based on whether comments were present or not (Marchionni, 2015).

There was little research exploring how comments affect perceived news bias. In one experiment, participants who saw like-minded comments (i.e., comments that agreed with the news story) perceived the news as less biased (Gearhart et al., 2020). People also viewed news content as less trustworthy when negative comments were present (Awobamise and Jarrar, 2021).

In terms of persuasion, comments reduced the third person effect (i.e., thinking others would be more influenced by the news article than oneself) (Chung et al., 2015). Further, those who read opinion challenging articles with comments in-line with the article’s view, perceived the article as more useful (Jahng, 2018).

Few considered over time effects regarding how comments shape perceptions of news. In one study, results indicated user comments can reduce the persuasion of a news article in the short (but not long) term (Heinbach et al., 2018). Another study found order effects and evidence for a negativity bias (i.e., interaction effect) showing that exposure to negative (vs positive) comments after (vs before) a news article decreased people’s journalistic quality perceptions (Kümpel and Unkel, 2021).

Congruent vs incongruent comments

Research indicated readers had more positive evaluations of comments that were pro-attitudinal to their beliefs (Petit et al., 2021). People also assessed in-group commenters more favorably than outgroup commenters—but only when in-group members also held a similar issue position to the participant (Kim, 2018). People with strong opinions were more likely to comment on counter-attitudinal threads (Duncan et al., 2020).

Participants perceived news-congruent user comments as more credible and better argued than comments disagreeing with news content (Kunst, 2021). Others found whether comments were congruent or not with a news story shaped perceptions of the issue discussed in that news article, especially when comments came from high-status commenters (von Sikorski, 2016).

Many studies considering (in)congruent comments focused on how these comments shape public opinion perceptions and political polarization. For example, Lee et al. (2021) showed that participants perceived both public opinion and the news tone to be more congenial to their own position when they were exposed to opinion-reinforcing (vs. opinion-challenging) news-related user comments, which (indirectly) increased opinion polarization.

Other research found those who read news related user comments not in-line with their own personal views, believed that public opinion disagreed with their own views and also held greater hostile media perceptions—but this was only true for people with higher levels of ego-involvement (Lee, 2012). Contrarily, other research informed by spiral of silence theory (Matthes et al., 2018) found incongruent user comments did not shape perceptions of public opinion, but rather made participants believe their opinion was in the minority of this particular news source (Yun et al., 2016).

Positive vs negative comments

Other research focused on the sentiment of user comments and how this impacted attitudes and perceptions. News articles about scandals including comments with negative sentiment towards the scandalized individual, were associated with readers seeing the scandalized individual as more responsible (von Sikorski and Hänelt, 2016). However, negative sentiment towards news content itself can also have implications for how people see news. Such comments can reduce the credibility of news (e.g., Waddell, 2018).

Positive and negative comments can also shape how people feel both about society (e.g., public opinion climate; Waddell, 2020), and specific issues (e.g., negative comments can reduce issue importance; Waddell, 2018) and shift peoples’ attitudes (Winter, 2019). Importantly, user comments also shape how people view others. In one study, exposure to positive comments reduced prejudice, the beneficial effects of positive comments were present even one week later—suggesting stability in this attitude change (Stylianou and Sofokleous, 2019).

Platform design, moderation, and anonymity

Fewer experiments explored how the design of the online commenting spaces shapes effects. Some research suggested that when readers were aware news sites moderate their comment threads (and that some comments are deleted), it can make readers less likely to agree with the news content (Sherrick and Hoewe, 2018)—a potential backfire effect of news platforms attempting to moderate comments. Further, others have focused on developing strategies to increase user engagement, finding that allowing users to categorize their viewpoints encourages more commenting (Peacock et al., 2019). Others studied the effect of organizations and companies responding within comment threads, finding it can increase their reputation (Spence et al., 2019), and credibility (Lin et al., 2019)—suggesting interactions within comment threads can be advantageous for organizations.

Commentor anonymity is also an important factor to consider in user comments research—though there is little research on this topic. People are more likely to share views in comment threads when they are anonymous (Wu and Atkin, 2018). However, other research found that men (but not women) are more likely to respond to comments when they are anonymous (Suh et al., 2018).

Discussion

In this systematic review, we analyzed articles assessing news-related user comments. We find a growing interest in this topic, with many articles being published after the systematic reviews of Reimer et al. (2023)—which analyzed papers published by 2016, and Naab and Sehl (2017)—which analyzed papers published by 2012. This growth in recent research emphasizes the need for our re-assessment of the literature. Additionally, previous systematic reviews focused on content analyses of user comments—we go beyond that by also considering the correlates and effects of user comments through also examining correlational and experimental research. Based on our review, we formulate several implications for research and digital journalism practice and have 4 take away messages.

Steep increase in studies, but lack of comparative and longitudinal approaches

Our analyses suggested a steep increase in research on user comments. One explanation for this increasing interest could be related to the growing use of social and digital media for news (Newman et al., 2023), and the growing pervasiveness of user comment threads in news content (e.g., Stroud et al., 2016b). We also find great diversity in the countries considered in user comment research—though more data is still needed from non-Western countries. However, few compare user-comments cross-culturally (or overtime)—making it difficult to infer whether the content, corollaries, and effects of news-related comments are universal or societally specific or have long-lasting effects. We encourage future research to take a more cross-cultural and longitudinal perspective in understanding news-related user comments.

Stronger focus on social media and video-based content

Research was narrow in terms of the type of news content (and comments) being assessed. Most studies considered text-based news content (e.g., online news articles), rather than video-based content—content frequently used on both online news platforms and social media (see Newman et al., 2023). In line with this, more than 70% of the studies examined traditional digital news. In contrast, less than 20% of studies examined news-related user comments on social media. Further emphasizing this point is that only two studies assessed news-related user comments on Instagram (Toepfl and Litvinenko, 2021; Aldous et al., 2023) and none considered TikTok—both visual-based platforms that many increasingly use to access news content (Newman et al., 2023). Future research should expand analyses to consider video-based news content (see Walther et al., 2010) and a more diverse set of social media platforms to gain a clearer understanding of the content (and effects) of user comments when presented in social media contexts.

Regarding for platform design, content moderation and artificial intelligence

Further research needs to assess the influence of platform design in comment threads, particularly regarding the role of anonymity. While some suggest anonymity in comment sections negatively influences comments (e.g., making them more uncivil; Ksiazek, 2015; see also the online disinhibition effect, Suler, 2004), other findings revealed that anonymity increased the likelihood that people share their views in a comments thread (Wu and Atkin, 2018). Many questions remain, for example, understanding the psychological mechanisms for why anonymity drives people to engage more uncivilly.

One way, journalists can deal with these uncivil comments is via content moderation. However, while some research reveals that uncivil comments in moderated environments are less likely to cause perceptions of news article bias (Yeo et al., 2019), users may feel confused or even frustrated when journalists use content moderation systems that remove content without informing people about particular reasons for content removal (Myers West, 2018), sometimes interactive or motivational moderation techniques may be more beneficial (Stockinger et al., 2023).

Further, content moderation of news-related user comments by journalists can drive negative perceptions of news media platforms, for example negatively influencing the credibility of a news story (Wang, 2021). In this context, a promising new line of research shows that artificial intelligence (AI) may be one way for journalistic organizations to deal with uncivil user comments, while simultaneously protecting news credibility. Wang (2021) showed that news bias was reduced when uncivil comments were moderated by AI (vs a human journalist). Future research should consider new possibilities offered by AI and examine these in the context of user comments. For example, understanding the psychological mechanisms for why artificial intelligence can reduce perceived news bias better than journalists.

Search for measures to reduce incivility and negative impact on digital journalism

While many explore incivility in user comments (in terms of both content and effects), there is one glaring gap in the literature—how we can motivate people to be less uncivil in their comments. The literature indicates uncivil comments have many negative consequences, for example, driving negative emotions and aggressive intentions (Chen and Lu, 2017), fostering more negative attitudes towards journalism (Liu and McLeod, 2019; Weber et al., 2019; Searles et al., 2020; Masullo et al., 2023); but also (see Kim and Chen, 2021) and promotion of (political) polarization (Hwang et al., 2014). Further, research suggests uncivil comments tend to invoke further uncivil comments by others (see Kim and Park, 2019)—potentially reducing the quality of online conversations. Based on these findings, we suggest future research should find new ways to reduce peoples’ willingness to write (or respond to) uncivil news-related comments. Finding new intervention strategies to reduce both incivility and the corresponding negative (unwanted) effects incivility has on journalism is a promising avenue for future research.

Limitations

While the review provides unique insights into current understandings of news-related user comments, it also has limitations. We focused only on peer-reviewed quantitative articles—meaning we do not consider qualitative papers nor conference or theory papers in these analyses. Focusing on quantitative papers allowed for a systematic coding of each article—however, of course, this leads to a limitation of not considering the full breadth of scholarly research on news-related user comments. Additionally, all potentially relevant papers may not have been included in our review due to divergent terminology used across papers (e.g., “posts” rather than “comments”)—a limitation we attempted to address (in part) by also collecting potentially relevant papers from other reviews (Ksiazek and Springer, 2018; Reimer et al., 2023). Finally, we view the current review as a first step in deeply understanding research on user comments. With this, we took a qualitative approach to understanding current findings. This means we do not systematically assess individual results but rather take a broad approach—examining overarching patterns and trends. We encourage scholars to further assess user comments through our open access list of included papers. These limitations should be kept in mind when interpreting results.

Conclusion

With the introduction of comment sections under news content, came normative ideas that such sections could be beneficial for public discourse and democracy (e.g., Engelke, 2020; see Habermas, 1962, 2022). However, current scholarly understandings of user comments suggest there may be many ill-effects (e.g., sowing distrust of journalism and snowballing incivility between commenters). We call on scholars to further consider (1) comparative and longitudinal approaches, (2) social media and video-based content, (3) platform design, content moderation, and artificial intelligence, and (4) search for measures to reduce incivility and negative impacts on digital journalism. Addressing these gaps will bring us one step closer to more fully understanding news-related user comments.

Author contributions

EK: Methodology, Supervision, Writing – original draft, Writing – review & editing, Investigation. PM: Methodology, Writing – review & editing, Investigation, Conceptualization, Validation. MW: Validation, Writing – review & editing, Investigation. CD: Investigation, Validation, Writing – review & editing. KG: Writing – review & editing. CS: Writing – review & editing, Conceptualization, Funding acquisition, Methodology, Supervision, Validation, Writing – original draft.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. We are grateful for support from the Schlieper Foundation to Christian von Sikorski (No. 4507-LD-28202).

Acknowledgments

We would like to thank Lisa Mai for her help with proofreading the manuscript.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fcomm.2024.1447457/full#supplementary-material

Footnotes

1. ^All papers included in this systematic review were available by the date of retrieval (i.e., April 4th 2022).

2. ^Fourteen of the 57 additional papers were included in this systematic review after being assessed by research assistants for whether they met inclusion criteria. Specific reasons for why each paper had not been found in our original Web of Science search query can be found in our Online Supplementary Materials (https://osf.io/swx76/?view_only=30aa778596aa4c469634fe84a3de63bb).

3. ^Research assistants were trained on papers that were included in the systematic review but that were not part of their own final set of assigned papers (i.e., not part of the approximately 130 papers they would code).

4. ^Research assistants also coded for whether the sample was a student sample or quota-based (representative), whether coding analyses (of content analyses) were automated or conducted manually, and whether the study considered dependent variables. However, reliability analyses indicated low reliability for these codes, therefore we do not consider these measures in our systematic review.

5. ^We conducted reliability analysis (Cohen’s Kappa; Cohen, 1960), on all codes that were quantitative in nature (i.e., ones where coders did not need to write in a free response box as these free response codes were uninterpretable with Cohen’s Kappa).

6. ^While this paper was published in 2023, it was available on the date of our paper retrieval (April 4^th, 2022) because it was published online first. This occurred with several papers. We have provided the year of publication that is most up to date at the time of writing this review.

References

Ademmer, E., Leupold, A., and Stöhr, T. (2019). Much ado about nothing? the (non) politicisation of the European Union in social media debates on migration. Eur. Union Polit. 20, 305–327. doi: 10.1177/1465116518802058