Assessing Levels and Forms of Incivility and Deliberative Quality in Online Discussions on COVID-19: A Cross-Platform Analysis

Schroll, Cornelia; Huber, Brigitte

doi:10.3389/fpos.2022.814002

ORIGINAL RESEARCH article

Front. Polit. Sci., 25 February 2022

Sec. Elections and Representation

Volume 4 - 2022 | https://doi.org/10.3389/fpos.2022.814002

This article is part of the Research Topic Negativity, Incivility, and Toxicity in Political Discussions View all 6 articles

Assessing Levels and Forms of Incivility and Deliberative Quality in Online Discussions on COVID-19: A Cross-Platform Analysis

$\nCornelia Schroll$ Cornelia Schroll¹

Brigitte Huber²^*

¹Department of Marketing, Vienna University of Economics and Business, Vienna, Austria
²Department of Communication, University of Vienna, Vienna, Austria

Studying incivility in online discussions is a crucial research endeavor, especially in light of a pandemic, as user comments have been shown to be an important source of information and basis for opinion formation. Scholars are increasingly interested in studying incivility from a comparative perspective. This study contributes to this strand of literature by investigating incivility in user comments on the news website and Facebook site of the Austrian newspaper Der Standard. A content analysis (N = 240) was employed to identify forms and levels of incivility in user comments on topics related to COVID-19. Results show Facebook comments to be significantly less uncivil than user comments on the newspaper's website. Moreover, differences regarding the prevalence of the dimensions of incivility on the two platforms were identified. Finally, results suggest that incivility does not necessarily go hand in hand with lower levels of deliberative quality. Implications for news organizations and democracy are discussed.

Introduction

In the literature, incivility is discussed controversially. On the one hand, incivility is seen as a threat to online discourse since it can discourage people from engaging in online discussions and thus hinder inclusive discussions (Frischlich et al., 2019). On the other hand, it is questioned whether “it is desirable, or even possible, for social media platforms to remove all uncivil content” (Masullo Chen et al., 2019, p. 1). Recently, scholars started exploring new ways of investigating incivility by employing cross-national and cross-platform comparison (Humprecht et al., 2020; Otto et al., 2020; Yarchi et al., 2021). In addition, recent research highlights that incivility and rationality do not necessarily exclude one another (Rossini, 2020). Our study makes an important contribution to both strands of research by investigating incivility and deliberative quality in user comments on two different platforms that are highly popular among Austrian news users - the news website and the Facebook site of the Austrian daily newspaper Der Standard. By focusing on user comments on articles dealing with the COVID-19 crisis, we not only chose a very polarizing and emotional topic, but also one that is highly relevant around the globe and will likely stay pertinent for some time. User comments have been found to be influential (Lee and Jang, 2010; Anderson et al., 2014; Hsueh et al., 2015; von Sikorski and Hänelt, 2016). In times of a health crisis, as it is currently the case, this seems to be particularly noteworthy, since in this context online discussions might not only influence opinions on health-related topics but also essential behavioral variables such as wearing face masks or vaccination intent (Kim et al., 2021; Pascual-Ferrá et al., 2021). While research already started to capture the tone of online discussions on COVID-19 by analyzing emotions in comment sections of newspapers (Fischenender et al., 2020), there is a need to investigate incivility in this regard. This study aims to close this gap by (1) investigating incivility in online discussions on COVID-19 on two different platforms - the Facebook page and the comment section on the news website of the Austrian daily newspaper derstandard.at, and (2) exploring the relationship between incivility and rationality on these platforms.

Literature Review and Research Questions

Incivility in Online Discussions

Online spaces, hoped to enable people to deliberatively discuss politics, suffer from incivility. In the literature, there is a wide range of different attempts to define incivility. Ksiazek et al. (2015) provide an overview of different definitions; they differentiate between normative definitions, which are supposed to apply in general, and contextual definitions, which state that the classification as civil or uncivil depends on the social group concerned and the social norms that apply to it, so that no general definition can be established. Recently, Bormann et al. (2021) proposed a new approach to define incivility. By linking incivility to disapprovals of norm violations, they define incivility as “the acts of communication in public political debates that participants disapprove of as severely violating communication norms of information, modality, process, relation, or context” (p. 16). When we talk about incivility in this paper, we refer only to a very specific part of this typology—the violation of relation norms (Bormann et al., 2021). While a broader conceptualization of incivility, as discussed by Bormann et al. (2021), provides interesting new perspectives for analyzing incivility, it comes with limitations when analyzing incivility by using content analysis. Bormann et al. (2021) explain that researchers would not apply the common strategy used in prior studies to define which user comments are uncivil but rather start from the participants' perspective, i.e., they argue that participants of online discussions should be the ones to judge whether a comment should be considered as civil or uncivil. Since we aim to analyze the content of user comments, we chose a definition of incivility that is often used in studies performing content analysis which is the one proposed by Coe et al. (2014). They define incivility as “features of discussion that convey an unnecessarily disrespectful tone toward the discussion forum, its participants, or its topics” (p. 660). More specifically, they mention name-calling, aspersion, lying, vulgarity, and pejorative for speech as common forms of incivility (see Methods section). Incivility is related to negativity; breaking the norms of civility can evoke negative emotions (Kingwell, 1995). It is important to distinguish between incivility and hate speech. As Porten-Chee et al. (2020) argue based on their typology of disruptive online discourse conditions, incivility does not have the extreme emotion of hate speech. Moreover, incivility does not aim to attack certain groups—a element central to the definition of hate speech (Erjavec and Kovačič, 2012): “Hate speech refers to an expression that is abusive, insulting, intimidating, harassing, and/or incites to violence, hatred, or discrimination. It is directed against people on the basis of their race, ethnic origin, religion, gender, age, physical condition, disability, sexual orientation, political conviction, and so forth” (p. 900). Prior studies show that around 20–40% of user comments on online news websites are uncivil (Coe et al., 2014; Santana, 2014; Rowe, 2015a; Su et al., 2018; Gonçalves et al., 2020; Rossini, 2020). These studies indicate that comments on articles on highly controversial topics such as same-sex marriage or immigration show higher levels of incivility compared to other articles. Since this study aims at comparing levels of incivility and deliberative quality, we decided to select a highly controversial topic in order to get as many uncivil comments as possible to make meaningful comparisons. A highly controversial topic that is heavily discussed online at the moment is COVID-19. In Austria, individual and societal dynamics amidst the COVID-19 crisis were monitored right from the beginning of the crisis by the The Austrian Corona Panel Project. For an overview of the project and information regarding COVID-19 in Austria, see Kittel et al., 2020; https://viecer.univie.ac.at/en/projects-and-cooperations/austrian-corona-panel-project/. We aim to assess levels of incivility in online discussions on COVID-19 on two different platforms by asking:

RQ1: How uncivil are user comments on COVID-19 on the website and the Facebook site of the newspaper Der Standard?

Comparative Perspectives on Incivility

In recent years, scholars increasingly engaged in studying incivility from a comparative perspective in terms of cross-national and cross-platform comparison. Humprecht et al. (2020), for instance, found lower levels of hostile emotions in comments on news organizations' Facebook sites in Germany compared to those in the United States. Research suggests that online discussions not only differ between countries but also between platforms. Hille and Bakker (2014) analyzed comments on news sites and Facebook accounts of 62 Danish newspapers. They found that users tend to write more comments on news websites than on Facebook, that the comments on news websites were more elaborated and that the discourse was also livelier on the news website compared to Facebook. Rowe (2015a) found a higher proportion of uncivil comments on the website than on the Facebook account of the Washington Post. Similarly, Rossini (2020) identified lower levels of incivility on Facebook than on news websites. Santana (2014) showed that anonymous users were more likely to post uncivil comments than non-anonymous ones. However, Rösner and Krämer (2016) did not find any significant difference regarding the number of aggressive expressions between participants posting anonymously or by name.

These studies suggest that levels of incivility in online discussions vary between different countries and platforms. Several factors have been identified that might help explain these differences. In the literature, cross-national variance in incivility is discussed in light of regulatory practices, cultural norms, journalistic cultures, media systems, and political context (Humprecht et al., 2020; Otto et al., 2020). Cross-platform differences are linked to anonymity (Halpern and Gibbs, 2013), platform affordances (Evans et al., 2017), and moderation processes (Ksiazek, 2018; Moore et al., 2020), as we will elaborate below. These studies also highlight the relevance of comparing comment sections on the website and on Facebook accounts of newspapers when investigating incivility. Several media organizations started early on to move their comments sections to Facebook (for an overview, see Hille and Bakker, 2014): The Los Angeles Times, for instance, switched to Facebook in 2011, and the same goes for the news site digitalspy.co.uk. The hope was to get comments of higher quality and less uncivil ones given that on Facebook people are using personal accounts instead of posting anonymously. “While many news organizations still maintain commenting sections on their websites, the gradual shift toward social platforms, particularly Facebook, as the mode for hosting news comments developed out of news organizations' frustration with managing thousands of comments—most of them anonymous/pseudonymous, some of them vulgar and vitriolic” (Kim et al., 2018, p. 4).

Building on theories of deindividuation (Postmes and Spears, 1998) and platform affordances (Evans et al., 2017), one would indeed expect a higher degree of incivility on news websites compared to Facebook as empirically shown by several studies discussed above. Specifically, theories on deindividuation assume that individuals act differently in anonymous settings due to reduced self-assessment - namely socially deregulated or uninhibited (Festinger et al., 1952; Zimbardo, 1969; Diener, 1980; for an overview, see Postmes and Spears, 1998). This approach is used to explain why users in anonymous settings get hostile comments (Döhring, 2008). Halpern and Gibbs (2013) argue that the level of identifiability vs. anonymity represents a media affordance that can influence the flow of online discussions. In addition, scholars argue that each platform has specific affordances (Boyd, 2010; Evans et al., 2017). Moore et al. (2020), for instance, argue that on Facebook people are aware that friends and family can see the comments and accordingly possible reactions of friends and family might be anticipated to some extent while writing comments. Hence, levels of incivility might therefore not only be a question of anonymous vs. non-anonymous but also of who is expected to read the comments.

In addition, also the type of moderation might have an impact on levels of incivility in user comments (Ksiazek, 2018; Moore et al., 2020). Online platforms differ significantly in their moderation rules, partly due to different local legal regulations, little to no regulatory bodies, or technologies (Gorwa, 2019). Due to the vast amount of content generated by users, platforms are not able to rely solely on human moderators (Gorwa et al., 2020) who are able to understand context and nuance, two very important aspects where machine learning is still in its infancy (Ruckenstein and Turunen, 2020). Rather, platforms rely increasingly on artificial intelligence to moderate the content posted by its users. Generally, the big social media platforms such as Facebook or Twitter do not give detailed insights into their moderation processes. However, all of these platforms use a mix of artificial intelligence to screen content and human moderators who make decisions in case of content that the algorithms cannot judge based on machine learning (Carlson and Rousselle, 2020; Gorwa et al., 2020). These differences in moderation systems and the speed companies are changing their algorithms (Common, 2020) also make it hard to judge how much content has been altered or removed from a platform.

In the following, we briefly describe content moderation strategies of the Der Standard website and on Facebook. The guidelines of derstandard.at contain the following aspects (Frequently Asked Questions zur Community, 2019; Community Richtlinien, 2020): Topic reference, respectful handling, factual argumentation, complying with laws, no discrimination and defamation, adequate language, no disruption of the discussion (spamming and flooding), and not disclosing advertising or personal data. Before posting the first comment or giving the first rating, users have to register with their email address and select a posting name (pseudonym or real name). The moderation takes place in several stages—first, the comments are checked and activated by a software (“Foromat”) and in the next step, reported postings are moderated manually (Frequently Asked Questions zur Community, 2019). For the moderation process on Facebook, no detailed information is available (Stjernfelt and Lauritzen, 2020). Content moderation on Facebook is basically organized along three pillars (Carlson and Rousselle, 2020): Community guidelines, artificial intelligence, and community flagging (users report inappropriate content that is manually checked by moderators). Moderators can react to the reported posts by choosing one of the following three actions: (1) deleting the content by accepting the report, (2) allowing the content to be visible on the platform by rejecting the report, or (3) forwarding the case for more detailed review (Gillespie, 2018). Based on varying platform characteristics and moderation strategies, we are interested in investigating differences between platforms. Since empirical findings on the relationship between platforms characteristics and levels of incivility are mixed, we pose a broad research question:

RQ2: To what extend do levels and forms of incivility differ on the two platforms investigated—on the news website and the Facebook site of the newspaper Der Standard?

Incivility and Deliberative Quality

As discussed above, the role of incivility in online discussions is seen ambivalent. Papacharissi (2004) pointed out early on that an excessive orientation toward civility in discussions could lead people to seek approval, only wanting to discuss safe topics and to avoid disagreements - which does not correspond to the ideals of deliberative quality in online discussions. Several different definitions for deliberative quality can be found in the literature, as we will elaborate below. While some focus, for example, on the quality of argumentation, in terms of providing sources and evidence, when studying deliberative quality in user comments (Oz et al., 2018; Jost and Köhler, 2019), others argue that cognitive complexity has to be present in order for online discussions to be deliberative (Moore et al., 2020). Our study takes a broad approach to measuring deliberative quality by building on the definition of Rowe (2015b), which consists of the following eight dimensions: topic relevance, expression of own opinions, justification for the opinion expressed, sources, narrative, alternative solutions, questions, and interaction.

When it comes to the relationship between the concepts of incivility and deliberative quality, again different views can be detected. Friess and Eilders (2015) systematically analyzed theoretical and empirical findings on online deliberation and identified the following dimensions of deliberative communication: (1) Rationality, (2) interactivity, (3) equality/inclusiveness, (4) civility, (5) common good reference, and (6) constructiveness. Also more recent studies discuss civility as a factor that facilitates constructive deliberation (Santana, 2019; Friess et al., 2021). At the same time, empirical findings are mixed. Rossini (2020), for instance, found that incivility and rational reasoning are not mutually exclusive. Contrary to Gervais (2015), who found the presence of incivility in online discussion forums to decrease the likelihood of deliberation, Jost and Köhler (2019) show that incivility and the inclusion of evidence in comments do occur together. Accordingly, there are conflicting views in the literature how the concepts of incivility and deliberative quality are related and how they can best be measured.

Several researchers are currently advocating for treating incivility and deliberative quality separately, as only one or only the other might be insufficient to determine the quality of user comments (Beckert and Ziegele, 2020; Ziegele et al., 2020). Building on this line of reasoning, we investigate the connection between incivility and deliberative quality in online discussions on COVID-19:

RQ3: Do user comments with and without incivility differ in their deliberate quality?

Methods

To answer our research questions, we conducted a quantitative content analysis. We selected the Austrian newspaper Der Standard as the largest Austrian quality newspaper in terms of reach and having the biggest online community in Austria with over 80 million user comments posted since its launch in 1999. The readership of derStandard consists of young, highly educated and mainly urban readers (Wer sind eigentlich unsere Leserinnen und Leser, 2020). Der Standard is using various social media platforms to get in touch with its readers ranging from Facebook (349,868 followers), Twitter (343,463), Instagram (326,000), YouTube (43,500) to TikTok (10,500; all numbers as of December 14th, 2021). Building on prior cross-platform research on incivility (e.g., Hille and Bakker, 2014; Rowe, 2015a; Rossini, 2020), we decided to focus on user comments on the Facebook account of Der Standard and to compare it with user comments on their news website. To do so, we selected four news stories¹ dealing with COVID-19 which were available on both platforms—the news website and the Facebook account of Der Standard—and that got at least 50 comments on each platform. We took the latest 30 comments per news story on each platform which resulted in a final sample of 240 user comments. We developed 20 categories to assess formal criteria (ID, date, platform, etc.) as well as content of user comments.

Incivility

We chose the operationalization of Coe et al. (2014) as a well-established measurement of incivility in empirical studies. Accordingly, incivility was measured by using five dimensions: (1) name-calling (“mean-spirited or disparaging words directed at a person or group of people”), (2) aspersion (“mean-spirited or disparaging words directed at an idea, plan, policy, or behavior”), (3) lying (“stating or implying that an idea, plan, or policy was disingenuous”) (4) vulgarity [“Using profanity or language that would not be considered proper (e.g., pissed, screw) in professional discourse”], and (5) pejorative for speech (“Disparaging remark about the way in which a person communicates”). If one or more of these five dimensions was coded with “yes”, then the user comment was considered to be uncivil.

Deliberative Quality

Based on Rowe (2015b), deliberative quality was assessed by using the following eight dimensions (yes/no): topic relevance (comments were coded as topically relevant when they addressed the topic at hand); opinion (we coded whether or not users expressed an opinion in the comment); justification (for all comments that express an opinion, we coded whether or not users included supporting reasons to justify their expressed opinion); sources (we coded yes when users mentioned policy documents, websites, scientific papers, etc. in the comment); narrative (we coded comments according to whether or not they include personal experiences, e.g., stories about family or friends); alternative (we coded whether or not comments offer alternative solutions or alternative ways to approach the topic at hand); question (posing questions to other participants or more generally); and interactive (whether or not comments refer to other participants or to the claims made by them). Indicators were summed up² to build the final variable of deliberative quality (M = 3.21, SD = 1.12).

Intra-Coder Reliability

One coder coded all user comments. In order to establish intra-coder reliability, 10% of the material was coded by the same coder again, 10 weeks after finishing the first coding. Intra-coder reliability scores ranged from 0.83 to 1 (Holsti); 0.79 to 1 (Krippendorff's Alpha).

Results

In total, 240 user comments on COVID-19 posted on comments section of the news website and the Facebook account of the Austrian daily newspaper Der Standard were analyzed. First, we asked how uncivil online discussions on these platforms are (RQ1). Findings in Table 1 show that 32.5% of all user comments were uncivil. Next, we were interested in the comparative perspective and wanted to check whether the two platforms analyzed differ in regard to levels of incivility (RQ2). Findings reveal significant differences: While on the news website of Der Standard 40.8% of the comments were uncivil, the comments on the same news stories on Facebook were uncivil to a lower extent (24.2%).

TABLE 1

Table 1. Incivility in user comments on news website and facebook account of derstandard.at.

Next, we were interested to see which types of incivility were used on the two platforms. When looking at all comments that contained incivility, results show that in 69.2% of these comments one type of incivility was coded, in 25.6% two types were included and in 5.1% three types of incivility were identified. Table 2 gives an overview of the different types of incivility coded in the comments. The most common type was aspersion which was coded in 42.3% of all uncivil comments, followed by accusation of lying (33.3%), name-calling (25.6%), and pejorative for speech (24.4%). Vulgarity was the least common coded type (10.3%). We found significant differences between the two platforms. Name-calling and aspersion, were used more often in comments on the news website than on the Facebook account of derstandard.at.

TABLE 2

Table 2. Types of incivility in user comments on news website and facebook account of derstandard.at.

In addition, we investigated the relationship between incivility and deliberative quality of user comments (RQ3). Before answering this research question, we provide an overview of the deliberative quality of the online discussion on COVID-19. In 6.3% of all analyzed user comments, one quality criterion was coded. Two quality criteria were coded in 21.3% of all user comments, three quality criteria in 30.8% and four criteria in 30.4 % of the cases. Five quality criteria were detected in 9.5% of the analyzed postings. The highest number of coded quality criteria was six which applied to 1.7% of all analyzed postings. Results in Table 3 reveal that the deliberative quality of user comments on COVID-19 on the two platforms is very similar. The two platforms only differ in that user discussions on the Facebook account are more interactive compared to the news website of derstandard.at. That is, users commenting on Facebook are more likely to refer to other users in their comments (79.2%) than users commenting on the same news stories on the website of derstandard.at (49.2 %). For the rest of the quality indicators, the two platforms show very similar patterns. On both platforms, almost three quarters of the comments were relevant to the topic at hand. In over 80% of the comments, users expressed own opinions. In more than half of these comments, users provided justification for their opinion. Sources were rarely mentioned by users (4.2%). However, personal experiences and alternative solutions were incorporated more often (around 8% each). Every fourth comment contained questions directed toward other users.

TABLE 3

Table 3. Deliberative quality of user comments on news website and facebook account of derstandard.at.

Next, results in Table 4 indicate that uncivil and civil comments show very similar levels of deliberative quality. Only two differences have been identified: First, users are more likely to express own opinions in uncivil comments (93.6%) compared to civil comments (82.7%). Second, civil comments are more interactive (69.1%) than uncivil ones (53.8%). Results also show that the dimensions of incivility and deliberative quality only rarely exclude one another (see Table A1): The table indicates that there are only five combinations that did never occur together: (1) Name calling & using sources; (2) aspersions & discussing alternative solutions; (3) vulgarity & using sources; (4) vulgarity & discussing alternative solutions; and (5) pejorative for speech & discussing alternative solutions.

TABLE 4

Table 4. Deliberative quality in user comments with and without incivility.

Next, results from regression analysis indicate that most variance in levels of incivility and deliberative quality is explained by the length of user comments (Tables 5, 6). The two platforms do not differ significantly regarding the length of user comments (M_Website = 33.43 words, SD = 28.63; M_Facebook = 28.10 words, SD = 29.81; t = 1.414, df = 238, p = 0.159). Moreover, results from regression analysis show that while incivility significantly differs between platforms (β = −0.180, p < 0.05), deliberative quality does not (β = 0.058, n.s.). Finally, results also indicate that dimensions of incivility are unrelated to deliberative quality (Table 5) and the dimensions of incivility are only marginally related to deliberative quality (Table 6).

TABLE 5

Table 5. OLS regression model predicting deliberative quality of user comments.

TABLE 6

Table 6. OLS regression model predicting incivility.

Discussion

Building on comparative research on incivility in online discussions, this study focused on user comments on two different types of news comment sections. More specifically, we investigated incivility in online discussions on COVID-19 on the website and the Facebook account of the Austrian daily newspaper derstandard.at. Following recent studies that challenged the common assumption that incivility goes hand in hand with low discussion quality (Beckert and Ziegele, 2020; Rossini, 2020; Ziegele et al., 2020), we explored the relationship between incivility and deliberative quality in user comments. Results of the content analysis of user comments on the website and the Facebook page of the Austrian daily newspaper Der Standard show that nearly every third of the comments analyzed contained incivility. Levels of incivility in the online discussions on COVID-19 found in this study are comparable to those found in prior studies on various other topics which ranged between 20 and 40% (Coe et al., 2014; Santana, 2014; Rowe, 2015a; Su et al., 2018; Gonçalves et al., 2020; Rossini, 2020). Despite its polarizing character, topics related to COVID-19 do not seem to generate higher levels of incivility compared to other topics. However, this also points to a limitation that applies to most studies on incivility: Since we were only able to include those comments in the sample, that were not removed by a software or moderator, the full extent of incivility was not assessable. In this regard, interviews with content moderators would be helpful to get an idea of the actual levels of incivility.

Interestingly, online discussions on COVID-19 on the news website of Der Standard were significantly more uncivil compared to discussions on their Facebook page. One possible explanation in this context could be that pseudonyms are more common on the news site, while on Facebook many are registered with real names. Anonymity increases the likelihood of expressing oneself in an uncivil manner (Santana, 2014). News media have to think about whether or not users should be motivated to use their real names when registering (Ksiazek and Springer, 2019). Research indicates that after reading uncivil comments individuals tend to estimate the quality of the whole discussion lower (Wang, 2020). Interestingly, our study shows that at least from a normative perspective, deliberative quality of user comments does not suffer from incivility; that is, we did not find significant differences in terms of the deliberative quality indicators between comments with and without incivility. Indeed, results revealed that dimensions of incivility and deliberative quality are largely unrelated. This brings important new insights into the scholarly and public discourse on incivility in online discussions. While for a long time theoretical and empirical studies on deliberativeness in online settings conceptualized civility as a dimension of deliberative quality (for an overview, see Friess and Eilders, 2015), our findings clearly point toward treating deliberativeness and (in)civility as separate concepts. This has important implications: Deleting uncivil comments might not always be the best solution since it does not necessarily contribute to higher quality of online discussions. Indeed, it can also make parts of rational contributions invisible. This poses a challenge to future content moderation systems, since deleting comments based on certain uncivil words is much more feasible than scanning comments based on quality criteria. Our results also speak to the argumentation of Rossini (2020) and Beckert and Ziegele (2020) who argue that incivility as such should not be seen as a danger to democracy, but rather a differentiated perspective is required. Similarly, Masullo Chen et al. (2019) argue that in some occasions, incivility may actually be required for marginalized groups to be heard. Incivility is often associated with increased attention and for some, incivility in user comments even is seen as a form of entertainment (Rossini, 2020). Accordingly, research on incivility should not be restricted to the negative effects of incivility, but also take into account other perspectives. It is, however, important to distinguish between incivility and hate speech in this regard—hate speech is not what we refer to here. Hate speech has to be deleted from news comment websites.

This study does not come without limitations. First, since this study focused on one country (Austria) and one media outlet (daily newspaper), readers should be careful when generalizing the results of this study. Der Standard is an Austrian quality newspaper with an audience that is rather liberal and higher educated. Hence, when investigating other (Austrian) newspapers, levels and forms of incivility as well as the quality of the discourse might differ. A German study, for example, found lower levels of argumentative quality for user comments posted on news stories of the tabloid paper bild.de compared to the quality paper Spiegel Online (Jost and Köhler, 2019). Since media landscape, political contexts and cultural norms vary from country to country, there is still a lot of research to be done in the area of comparative research on incivility (Humprecht et al., 2020; Otto et al., 2020; Yarchi et al., 2021). Similarly, we focused on only one social media account of the newspaper of interest. Hence, we were not able to make conclusions on levels and forms of incivility and deliberative quality of online discussions on other social media platforms where readers of Der Standard comment on news stories. Future studies should expand this line of research by looking at increasingly relevant platforms when it comes to news such as Instagram or TikTok (Hermida and Mellado, 2020; Vázquez-Herrero et al., 2020). Moreover, since this was an exploratory study, we had a quite small number of uncivil comments in our sample which did not allow to dig deeper into cross-platform analysis when looking at the relationship between incivility and deliberative quality. Future studies should enlarge the sample in order to be able to perform more detailed analysis. In addition, moderation also poses a limitation to this study. Since Facebook and derstandard.at have different ways of moderating its content and there is no information on how many comments were altered or deleted under a given article, different moderation could impact the results. Since the pandemic is still ongoing, more and more topics related to COVID-19 arise. Therefore, new topics such as vaccine mandates or lockdowns for the unvaccinated only, could provide additional interesting opportunities to investigate incivility in online news comment sections and on social media sites. While this certainly poses a limitation to our study, we still were able to provide important initial insights into incivility in online discussions on COVID-19. Similarly, our sampling strategy to include the latest 30 comments per news story, poses another limitation to this study since these comments are not representative for the rest of the comments. Future studies should try other sampling strategies when investigating incivility in news comment sections. Finally, our study focused on one specific type of incivility—the violation of relation norms; as a recently developed typology of incivility shows, the concept can be assessed in a much more nuanced way (Bormann et al., 2021).

Beside these limitations, this study offers important insights into the relationship between incivility and deliberative quality in online discussions on different platforms. By doing so, this study contributes to the development of a differentiated perspective and a much more nuanced understanding of incivility and deliberative quality in different online environments.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Author Contributions

CS: conceptualization, coding, data analysis, and writing. BH: conceptualization, data analysis, writing, and editing. Both authors contributed to the article and approved the submitted version.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We would like to thank the organizers Lukas P. Otto, Sanne Kruikemeier, Sophie Lecheler, and Alessandro Nai as well as the participants of the ICA preconference Comparative perspectives on negativity, incivility, and toxic talk in political discussions for helpful feedback and discussions. Moreover, we would like to thank the two Reviewers whose thoughtful comments helped to further improve and clarify this manuscript. Finally, we would like to thank the University of Vienna for providing Open Access funding.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpos.2022.814002/full#supplementary-material

Footnotes

1. ^The four articles that were selected for the content analysis:

“Kurz:Maskenpflicht in Lokalen für Personal, nicht für Gäste {Kurz: compulsory masks in bars for staff, not for guests}”, April 20th 2020. Available online at: https://www.derstandard.at/story/2000116976186/kurz-maskenpflicht-in-lokalen-fuer-personal-nicht-fuer-gaeste?utm_term=Autofeed&utm_medium=Social&utm_source=Facebook&fbclid=IwAR0hXBbQRUdNkjHnqFpy0Pf_jemXFF7qkWU7zUnVQoCgh_bedV0BlvaO1TA#Echobox=1587360604.

“Noch offen, ob Corona-Impfung für Österreicher gratis sein wird {To be determined whether or not vaccine will be for free for people in Austria}”, September 8th 2020. Available online at: https://www.derstandard.at/story/2000119852843/gesundheitsministerium-will-niederschwelligen-zugang-zur-corona-impfung?utm_term=Autofeed&utm_medium=Social&utm_source=Facebook&fbclid=IwAR1IioLdnIkVcrWfcwJFrmN64RZXNewM8wPp64kslQmu9jKe6jRhT2rwsZo#Echobox=1599559034.

“Effekt des Lockdowns laut Experten nur langsam zu erreichen” {According to experts, effects of the lockdown will only become visible slowly}, November 8th 2020. Available online at: https://www.derstandard.at/story/2000121542789/effekt-des-lockdowns-laut-experten-nur-langsam-zu-erreichen?utm_term=Autofeed&utm_medium=Social&utm_source=Facebook&fbclid=IwAR1aX2HwSapWHjTDC66JelRKVSuyl_cCP40P8QYFVq3kxP32-Hpma0D3GBI#Echobox=1604876682.

“Erstmals mehr als 200 Corona-Intensivpatienten in Wien {First time more than 200 intensive care patients due to Corona in Vienna}”, March 29th 2021. Available online at: https://www.derstandard.at/story/2000125437121/erstmals-mehr-als-200-corona-intensivpatienten-in-wien#Echobox=1617042141.

2. ^In the literature, the principles of internal consistency - that are crucial when dealing with reflective indicators - are discussed not to be applicable or suited the same way for formative ones (Döring and Bortz, 2016, S. 277; Diamantopoulos et al., 2008, S. 1215; Heiss and Matthes, 2018). The argument made in the literature is that for formative measurements, not all indicators necessarily have to correlate with one another.

References

Anderson, A. A., Brossard, D., Scheufele, D. A., Xenos, M. A., and Ladwig, P. (2014). The “NastyEffect:” Online incivility and risk perceptions of emerging technologies. J. Comp. Mediat. Commun. 19, 373–387. doi: 10.1111/jcc4.12009