Are online users influenced by what other users say? Meta-analyzing the cognitive, emotional, and behavioral impact of online comment valence

Introduction

Numerous online media such as social media and news websites provide users with the affordance of posting and reading comments. This affordance has been well-utilized by online users. Research shows that 77.9% of online users have read comments at some point (Stroud et al., 2016). Among people who read comments but do not post comments, the majority read comments at least once a week (Stroud et al., 2016).

As the act of reading online comments has become an integral part of people’s daily online media consumption, there has been a growing focus in research on understanding how these comments impact users’ cognition, emotion, and behavior. Within this body of research, one of the primary emphases has been on the influence of comment valence—distinguishing between positive comments that convey opinions that align with, support, or favor the opinions expressed in the original message, and negative comments that convey opinions that oppose, criticize, or disagree with the opinions expressed in the original message. Research has shown that online comment valence influences various psychological and behavioral outcomes such as perceived credibility of the original article (e.g., Waddell, 2018), likelihood to express opinion (e.g., Duncan & Coppini, 2019), emotion (e.g., Kuttschreuter & Hilverda, 2019), attitudes toward the focal issue (H. Kim et al., 2020), and intentions to engage in the focal behavior (e.g., Krämer et al., 2021). However, researchers have found diverse effect sizes, ranging from small (e.g., Dempsey et al., 2022) to large (e.g., Ballantine et al., 2015). In some cases, research yielded inconsistent results regarding whether there was a significant effect or not. For example, H. Kim et al. (2020) found that reading more favorable comments about the flu vaccine led to a more favorable attitudes toward the vaccine. However, Walther et al. (2010) did not find significant differences in attitudes toward marijuana use between people who read comments that supported an anti-marijuana public service announcement and those who read derisive comments.

Considering the mixed findings, conducting a meta-analytical synthesis of the existing evidence will aid in understanding the extent to which online comment valence influences people’s cognition, emotion, and behavior. Additionally, it will offer insights into the reasons for the observed variations in effect sizes. We begin by offering an overview of existing theories related to the impact of online comments, based on which we proposed a framework that predicts the cognitive, emotional, and behavioral outcomes resulting from online comment valence. Then, we used a meta-analysis approach to gauge the strengths of the predicted effects. Given the considerable variation in the effects observed in empirical studies, we also explored factors that may enhance or diminish the influence of online comment valence on these outcomes.

Theoretical Basis for the Effects of Online Comment Valence

A variety of existing theories have shed light on and guided empirical research on the influence of online comment valence, including a theory of interactive media effects (TIME; Sundar et al., 2015), exemplification theory (Zillmann, 1999), social norm influence (Cialdini et al., 1990), spiral of silence theory (Noelle-Neumann, 1993), and emotional contagion (Hatfield et al., 1993). Although to different extents, these theories all imply the power of social influence in shaping individuals’ emotional, cognitive, and/or behavioral outcomes. In other words, online comments of either positive or negative valence indicate a majority opinion and/or behavior, which modifies audiences’ emotional, cognitive, and/or behavioral outcomes to be consistent with the majority because individuals have the tendency to conform to and follow majority opinions and behaviors.

TIME

TIME (Sundar et al., 2015) posits that interactive media provide affordances, which refers to the action possibilities offered by the interactive media. For example, a news website with a comments section provides the affordance of commenting on the news articles. Affordances affect users’ perceptions, knowledge, and behaviors through two routes. First, affordances can trigger users’ actions. For example, the affordance of parallax scrolling on a product presentation website triggers user engagement, which in turn results in positive attitudes and behavioral intentions toward the product (Wang & Sundar, 2018). Second, affordances can serve as cues on the media interface. For example, the number of likes on a social media post may activate bandwagon heuristics about the nature of the post and in turn affect users’ evaluation of the post’s content and source (Sundar et al., 2015).

Within TIME, the Modality-Agency-Interactivity-Navigability (MAIN) model further specifies that four technological affordances—modality, agency, interactivity, and navigability—can cue a range of heuristics that affect users’ perceptions of media content (Sundar et al., 2015). Of these, the agency affordance is relevant to the multilayers of media content sources and its influence on users’ judgement on message quality and credibility. One of the layers of media content sources is the users themselves; in online media, users have the ability to be a source through activities such as commenting, liking, and sharing. MAIN posits that metrics representing the collective opinion of other users (e.g., number of views and likes) can cue the bandwagon heuristic and affect users’ evaluation of the original message content (Sundar, 2015). For example, an online video frequency metric that indicates a higher number of viewers may lead to more media enjoyment through influencing perceptions of audience size (Waddell & Sundar, 2017).

In a similar vein, online comments that are predominantly positive or negative can also serve as an agency cue which triggers bandwagon heuristic and influences users’ evaluation of the original message. In general, more positive (vs. negative) comments affect perceptions of majority opinions and result in more positive evaluation of the original message content (Lu & Sun, 2022; Waddell, 2018; Waddell & Sundar, 2017). Also, studies found that positive (vs. negative) comments led to more positive evaluation of companies’ Facebook pages (Y.-J. Lee et al., 2018), news articles (Ahn, 2011; von Sikorski & Hänelt, 2016), political satire shows (Möller & Boukes, 2023), and public service announcements (Shi et al., 2014; Walther et al., 2010, 2018).

Exemplification Theory and Social Norm Influence

Exemplification theory posits that people draw inferences and generalize about a larger group of people, objects, or phenomena based on a small number of examples that can represent broader phenomena of the same kind (Zillmann, 1999). Online comments can be exemplars from which people infer majority public opinions (E.-J. Lee & Tandoc, 2017; Waddell, 2019). When the comments are predominantly positive or negative, individuals may infer that the majority of the public hold a positive or negative opinion on the issue or the message content (Waddell, 2018). As predicted by the social norm literature (e.g., Cialdini et al., 1990), the perception that most people endorse an opinion or engage in a behavior (i.e., descriptive norm) may influence individuals’ judgment on the appropriate opinion and behavior for themselves. In other words, individuals who are informed of a majority norm tend to have attitudes that are consistent with the norm and subsequently stronger intentions to engage in the behavior suggested by it (Stok et al., 2014). Moreover, a meta-analysis has shown that a descriptive norm has a medium to strong association with behavioral intentions (Rivis & Sheeran, 2003). In short, predominantly more positive or negative online comments may lead to beliefs, attitudes, and behavioral intentions that are consistent with the majority opinions indicated by the comments.

Consistent with the prediction, a number of studies have shown that individuals tend to report beliefs, attitudes, and/or behavioral intentions that are consistent with the majority opinion suggested by the comments (Ahn, 2011; Ballantine et al., 2015; Cameron & Geidner, 2014; Duncan & Coppini, 2019; Hsueh et al., 2015; H. Kim et al., 2020; Kim, 2021; Kuttschreuter & Hilverda, 2019). For example, reading comments that are supportive (vs. unsupportive) of the application of nanotechnology in foods led to lower risk perception, higher benefit perception, and more positive attitude toward nanodesigned food (Kuttschreuter & Hilverda, 2019). In addition, individuals who read comments in favor of (vs. against) vaccination expressed slightly stronger intention to get vaccinated (Peter et al., 2014).

Spiral of Silence Theory

The spiral of silence theory (Noelle-Neumann, 1993) posits that individuals’ willingness to express their opinions depends on how they perceive public opinion and the extent to which their own opinions align with the perceived public opinion. The theory assumes that individuals fear that deviating from social consensus will result in isolation from society (Scheufle & Moy, 2000). Because of this fear of social isolation, individuals consistently monitor public opinions, and they are inclined to publicly express their opinions when they perceive that their viewpoint is dominant or on the rise; conversely, they tend to remain silent when they perceive that their opinion is in the minority (Scheufle & Moy, 2000).

In light of spiral of silence theory, comment valence influences individual willingness to express their opinions; individuals are more likely to post comments when their views align with the majority’s opinion, and are less likely to post comments when their views diverge from the majority. Consistent with this prediction, empirical studies have shown that individuals are more likely to express opinions consistent with the majority comments in various contexts, such as prejudice toward racial minorities (Hsueh et al., 2015), ideal body norm (Flynn, 2012), abortion (Wu & Atkin, 2018), and fake news (Colliander, 2019).

Emotional Contagion

Emotional contagion theory suggests that people catch emotions through social interactions (Hatfield et al., 1993). Emotional contagion theory originally focused on face-to-face interactions, positing that individuals pick up emotions and automatically mimic and synchronize others’ facial expressions, voices, postures, and movements during a conversation (Hatfield et al., 1993). As an extension of the original focus, Kramer et al. (2014) argued that emotional contagion can happen through interactions on social media as well: On social media, individuals can observe each other’s experiences by reading emotion-ladened content that expresses positive or negative emotions, which can trigger emotional contagion (Kramer et al., 2014; Kwon & Gruzd, 2017). Consistent with this prediction, studies on online comments have shown that online comments pass along positive (e.g., joy, glad) or negative emotions (e.g., sadness, disgust, anger) when the original messages are about organizational crisis (Zhao et al., 2013) and vaccination (Lu & Sun, 2022).

Current Study: Framework Constructed and Terms Defined

Synthesizing the relevant theories reviewed above, we constructed a framework for the effects of online comment valence (Figure 1). In summary, TIME and MAIN suggest that comment valence influences message perceptions. Exemplification theory and social norm influence suggest that comment valence influences issue-relevant beliefs, attitudes, and behavioral intentions. Spiral of silence theory suggests that comment valence influences communication intentions. And emotional contagion theory suggests that comment valence influences emotions.

Figure 1. The Framework Constructed for the Current Study.

Here we define positive comments as comments conveying opinions that align with, support, or favor the opinions expressed in the original message (e.g., pro-vaccine comments following a pro-vaccine social media post; Zhang & Wang, 2019). In the same vein, negative comments refer to comments conveying opinions that oppose, criticize, or disagree with the opinions expressed in the original message (e.g., pro-smoking comments following an anti-smoking public service announcement; Shi et al., 2014). Regarding the outcomes, message perception refers to the evaluation (e.g., credibility; Waddell, 2019) of the original message to which users respond (e.g., social media post, news articles, YouTube videos), rather than the subsequent comments. We define issue-relevant beliefs and attitudes as the beliefs or attitudes toward the issue being discussed in the original post and/or the comments (e.g., attitudes toward water conservation; Walther et al., 2018). And we define issue-relevant behaviors and behavioral intentions as the actual behaviors or the intentions to engage in behaviors that are relevant to the issue being discussed in the original post and/or the comments (e.g., intention to quit smoking; Shi et al., 2014). Communication behaviors and intentions refer to the actual behaviors or intentions to express opinions that are consistent with the comment valence (e.g., likelihood to make positive comments after reading positive versus negative comments; Colliander, 2019). Lastly, emotions refer to any affective feelings that are elicited after reading online comments (e.g., anxiety, happiness; Hilverda et al., 2018).

Given the theoretical basis and the definitions, we proposed the following hypotheses. Through a meta-analysis approach, we also assessed the sizes of the hypothesized effects (RQ1).

H1–5: Individuals exposed to predominantly positive (compared to negative) comments will exhibit the following: a more positive evaluation of the original message (H1), stronger issue-relevant beliefs and attitudes aligned with the positive comments (H2), increased likelihood of engaging in behaviors advocated by the positive comments (H3), increased likelihood to express opinions consistent with the positive comments (H4), and stronger positive emotions (H5).

RQ1: What are the sizes of the effects predicted in H1–5?

Factors Moderating the Effects of Online Comment Valence

The sizes of the proposed comment valence effects may vary depending on the features of the comments and the original messages. In particular, we discuss two comment features (i.e., number of comments and whether the comment valence is mixed or not) and one message feature (i.e., news versus non-news) in this section.

Comment Features

Exemplification theory posits that more frequently activated exemplars are more likely to create an endurable influence on individuals’ judgment about the frequency of the event (Zillmann, 2002). Because comments are exemplars of public opinion, individuals may hold a stronger perception of the majority opinion as they read more comments supporting/opposing a position, and thus adjust their judgment to align with the majority opinion. Consistent with this prediction, empirical studies found that when there were 10 exemplar comments about the eviction of violent immigrants, individuals who read favorable (vs. unfavorable) comments had a more favorable opinion about eviction, whereas this difference was not significant when they read only two exemplar comments (Zerback & Fawzi, 2017). However, the effect of comment valence seems diminished as the number of exemplars is higher than 10. In e-cigarette and genetically modified food contexts, individuals who read 10 (vs. 20) supportive or unsupportive comments did not differ in their perceptions of e-cigarette and genetically modified food (Liu, 2017).

In real-life situations, the valence of comments is often mixed. Predominantly negative comments may be accompanied by a small portion of positive or neutral comments and vice versa (e.g., Liao & Mak, 2019). Because individuals’ perceived frequency of an event is based on the relative exemplar frequencies (Zillmann, 2002), individuals exposed to comments uniformly expressing the same valence (i.e., all positive or all negative) may perceive a stronger majority opinion compared to individuals reading comments with mixed valence (i.e., mostly positive or mostly negative). Empirical studies supported this prediction. It was found that unanimously positive (negative) comments about a political candidate led to more favorable (unfavorable) opinions about the candidate, whereas mixed-valence comments had no effect in changing opinions about the candidate (Duncan & Coppini, 2019). Similarly, Kuttschreuter and Hilverda (2019) compared positive, mixed, and negative comments about nanodesigned food. They found that more positive comments led to lower risk perception, higher benefit perception and more positive attitude toward nanodesigned food than perceptions and attitudes following mixed and negative comments. Based on this evidence, it is likely that comments with mixed valence have a smaller effect size compared to comments with non-mixed valence. Therefore, we examined moderating roles of comment quantity and the mixture of valences in comments on the effects of comment valence.

RQ2–3: How are the hypothesized effects moderated by the number of comments (RQ2) and whether the comment valence is mixed or not (RQ3).

Original Message Feature

Individuals may comment on original messages involving various content types such as news, entertainment stories, public service announcements, and product advertisements. We are particularly interested in the influence of comments following news articles because such influence has significant social implications. Unlike non-news content, such as promotional material from various organizations and individual experiences or opinions, news stands out as the principal platform reporting matters of societal importance. In addition, to enhance news objectivity and credibility, news pieces often incorporate perspectives from multiple sources within society (Lee & Tandoc, 2017). As such, individuals typically acknowledge the wider societal implications of the topics under discussion while reading news (e.g., McIntyre, 2017). Comments accompanying news pieces are integral components of the discourse. Therefore, reading other people’s comments following a news article may induce a feeling of collective relevance and involvement with the news event and a more favorable perceived public opinion climate (Ksiazek & Springer, 2019; Weber, 2014). These feelings and perceptions may increase the willingness to leave a comment, which creates a greater potential for public discourse on important social issues (Ksiazek & Springer, 2019; Weber, 2014). In addition, news comment valence affects news readers’ evaluation of the news articles and journalistic quality (Ksiazek & Springer, 2019), which may influence individual trust in journalists as an information source and shape the ways individuals seek out and process news reports about important social issues in the future (Kümpel & Unkel, 2020; Trumbo & McComas, 2003; Tsfati & Cappella, 2003). Specifically, we investigated 1) if the valence of comments following a news article had significant impact on the outcomes, and 2) if the valence of comments following a news article was more or less impactful on the outcomes than the valence of comments following a non-news message.

RQ4: How are the hypothesized effects moderated by whether the original message is news or non-news?

Methods

We used a meta-analysis approach to examine the hypotheses and the research questions. In addition, we investigated how the effect sizes are influenced by the number of comments, whether the comment valence is mixed or non-mixed, and the original message feature.

Search Strategy

To systematically synthesize the existing evidence on the effects of comment valence, we aimed to conduct a comprehensive search of the literature to capture all existing empirical studies that examined the causal impact of positive (vs. negative) online comments on one or more of the outcomes listed in H1–5. Given that the effects of comment valence can be examined across different contexts, we conducted searches across 12 databases¹ relevant to the domains of communication, psychology, public health, information science, political science, and the broader social sciences. The search was conducted through EBSCOhost and ProQuest in December 2021 and August 2022. To begin broadly, we did not specify outcomes in the search string and included only keywords pertinent to online comments. The final search string is “online comment*” OR “news comment*” OR “newspaper comment*” OR ((TikTok OR YouTube OR Facebook OR Twitter OR Instagram OR “social media”) AND “comment*”). The wildcard searching technique was used to include possible variations of keywords (e.g., comment, comments, commenting). Title and abstract were searched. We limited the search to scholarly reviewed journal articles, conference papers, dissertations, and theses written in English. There was no restriction on the year of publication. The initial search resulted in 4,729 items. Figure 2 shows the searching and screening steps.

Figure 2. Process of Searching and Screening.

Next, 1,078 duplicate items were removed. The abstracts and full texts of the remaining 3,651 items were screened by two researchers. The articles are screened based on the following inclusion criterion: the article must report at least one experiment that examined the causal impact of positive (vs. negative) online comments on one or more of the following outcomes: (a) message perception (b) issue-relevant beliefs and attitudes, (c) issue-relevant behavior and behavioral intentions, (d) emotions, (e) communication intentions. After the screening, there were 42 articles remaining in the sample. Then, we searched the reference lists of the included articles. There were 15 articles that met the inclusion criteria and were added to the sample. Finally, 57 articles (64 studies) were coded and assessed for meta-analysis eligibility.

Coding Category and Intercoder Reliability

For publications that presented more than one study, each study was coded individually. For each study, we coded study context, number of participants, the number of comments, comment valence (i.e., mixed or not mixed), and media platform that comments were posted: Twitter, Facebook, News website, YouTube, Instagram, or other. We also coded the type of original post (i.e., news article or non-news article) reported by each study. Specifically, we referred news article as news reports posted on a webpage or on social media. Studies in which the original post was not a news report and studies that did not have an original post were coded as non-news article. Finally, we coded the effect size for each study.

Two coders independently coded all included studies. All discrepancies were resolved based on discussion. Krippendorff’s alpha showed good intercoder reliability, ranging from a low of .85 to a high of .97, which met the .8 criterion (Krippendorff, 2004).

Effect Size Calculation

The correlation r was used as the effect size indicator. The correlation rs were computed from other statistics including means and standard deviations, F statistics, and Chi-squares. We did the computation using an online application developed by Lenhard and Lenhard (Lenhard & Lenhard, 2016).

For most studies, only one correlation r for one dependent variable could be computed from a study. Other studies measured more than one outcome variable that can represent one of the outcomes we are interested in (i.e., message perception, beliefs and attitudes, behavior intentions, emotions, and communication intentions). That means, more than one Cohen’s ds (Cohen, 1969) could be computed from the study. In such cases, we computed the effect sizes of each outcome variable and used the average as the effect size indicator. Our treatment of the dependent variables was consistent with the practices in prior research (e.g., Shen et al., 2015).

Number of Studies for Each Outcome

A total of 101 studies examined message perception (19 studies), issue-relevant beliefs and attitudes (47 studies), issue-relevant behaviors and behavioral intentions (21 studies), communication behaviors and intentions (8 studies), and emotions (6 studies) as the outcome variables. For studies that did not provide the number of participants in positive and negative comment conditions, we assumed equal sample size across conditions to calculate the effect sizes. Also, for studies with inadequate data for effect size calculation, we contacted the authors to request the data. If the authors did not provide the data after being contacted twice, the study was excluded from the analysis. Finally, the analysis included 10, 31, 13, 5, and 4 studies for message perception, issue-relevant beliefs and attitudes, issue-relevant behaviors and behavioral intentions, communication behaviors and intentions, and emotions, respectively.

Meta-Analytic Approach

We used metafor R package (Viechtbauer, 2010) for the analysis. The effect sizes were weighted by inverse variances. Random effect models were fitted using the Hunter and Schmidt method because this method corrects for variance due to artifacts such as sampling error, reliability of measurement and range restriction (Hunter & Schmidt, 2004). The I² was used to test heterogeneity among the effect sizes as recommended by Higgins and Thompson (2002). Publication bias was examined using funnel plots and using Egger’s regression test to assess the asymmetry of the funnel plots (Egger et al., 1997). In moderation analysis, we followed the recommendation by Fu et al. (2011): to conduct subgroup meta-analysis, there should be at least 6–10 studies for a continuous study-level variable and at least 4 studies for each categorical subgroup when the sizes of the included studies are moderate or large. Therefore, for the continuous moderator (i.e., number of comments), if the total number of studies was smaller than 6, then moderation analysis was not conducted. For categorical moderators, categories that had fewer than 4 studies were excluded from the moderation analysis. If none or only one of the categories had 4 or more studies, then the moderation analysis was not conducted.

Result

Description of Studies

Included studies (k = 48) were published between 2007 to 2022, in journals across diverse areas such as communication (e.g., New Media and Society and Human Communication Research), marketing (e.g., Journal of Business Research and International Journal of Advertising), and public health (e.g., Vaccine and Drug and Alcohol Review). Studies were conducted in various contexts such as COVID-19 vaccination, political elections, water conservation, and brand marketing. Table 1 shows characteristics and effect sizes for each study by outcome type.

Table 1. Summary of Study Characteristics and Effect Sizes by Outcome Type.

Effects of Comment Valence on Emotional, Cognitive, and Behavioral Outcomes

Hypotheses 1–5 predicted that individuals exposed to predominantly positive (compared to negative) comments will have a more positive evaluation of the original message (H1), stronger issue-relevant beliefs and attitudes aligned with the positive comments (H2), increased likelihood of engaging in behaviors advocated by the positive comments (H3), increased likelihood to express opinions consistent with the positive comments (H4), and stronger positive emotions (H5).

Supporting all hypotheses, our results showed that comment valence significantly influenced all five outcomes. Answering RQ1, the inversed-variance-weighted effect sizes for each outcome were r_{message perception} = .22, 95% CI [.16, .29] (p < .001, k = 10, N = 2,179, I² = 54.09%), r_{issue-relevant beliefs and attitudes} = .29, 95% CI [.17, .39] (p < .001, k = 31, N = 7,438, I² = 93.30%), r_{issue-relevant behaviors and behavioral intentions} = .09, 95% CI [.03, .14] (p < .01, k = 13, N = 4,896, I² = 63.45%), r_{communication behaviors and intentions} = .26, 95% CI [.04, .13] (p < .001, k = 5, N = 2,883, I² = 79.99%), and r_emotions = .16, 95% CI [.06, .25] (p < .01, k = 4, N = 984, I² = 51.16%). Figures 3–7 are the forest plots for each outcome.

 Figure 3. Forest Plot—Message Perception Outcome.

Note. The forest plot shows the result of the random effect model. The weighted effect size and the effect size of each study is presented as correlational r with corresponding 95% confidence intervals. The same approach applies to all forest plots in this paper.

Figure 4. Forest Plot—Beliefs and Attitudes Outcome.

Figure 5. Forest Plot—Behaviors and Behavioral Intentions Outcome.

Figure 6. Forest Plot—Communication Behaviors and Intentions Outcome.

Figure 7. Forest Plot—Emotions Outcome.

Moderation Analysis

Number of Comments

To answer RQ2, we tested a number of comments as a moderator of comment valence impact on message perception, issue-relevant beliefs and attitudes, and issue-relevant behaviors and behavioral intentions. We did not conduct moderation analyses regarding emotions and communication behaviors and intentions outcomes because the number of studies for each outcome did not meet the moderation test requirement (Fu et al., 2011).

We found that number of comments significantly moderated the effects of comment valence on message perception (Q_M = 6.91, df = 1, p = .009), issue-relevant beliefs and attitudes (Q_M = 6.65, df = 1, p = .010), and issue-relevant behaviors and behavioral intentions (Q_M = 7.95, df = 1, p = .005). Specifically, as individuals read more comments, comment valence had a stronger effect on message perception (b = 0.06, p = .009) and weaker effects on issue-relevant beliefs and attitudes (b = −0.05, p = .010) and issue-relevant behaviors and behavioral intentions (b = −0.04, p = .005).

Mixed-Valence

To answer RQ3, we tested whether the comment valence was mixed or not as a moderator of comment valence impact on issue-relevant beliefs and attitudes and issue-relevant behaviors or behavioral intentions. We did not conduct moderation analyses regarding message perception, emotions, and communication behaviors and intentions outcomes because the number of studies for each of the mixed versus nonmixed subcategories did not meet the moderation test requirement (Fu et al., 2011).

We found that whether the comment valence was mixed or not significantly moderated the effects of comment valence on issue-relevant beliefs and attitudes (Q_M = 4.77, df = 1, p= .029) and issue-relevant behaviors or behavioral intentions (Q_M = 14.40, df = 1, p < .001). Specifically, when the comments included only positive or only negative comments, comment valence had a significant effect on issue-relevant beliefs and attitudes (r = .38, p < .001) and issue-relevant behaviors and behavioral intentions (r = .15, p < .001); when the comments were mixed-valence, comment valence did not affect issue-relevant beliefs and attitudes (r = .14, p = .127) and issue-relevant behaviors and behavioral intentions (r = .02, p = .478). Table 2 shows the moderation results.

Table 2. Moderation Analysis Results.

Original Message Feature

To investigate RQ4, we tested whether the original message was news or non-news as a moderator of comment valence impact on message perception and issues-relevant beliefs and attitudes. We did not conduct moderation analyses regarding issue-relevant behaviors and behavioral intentions, emotions, and communication behaviors and intentions outcomes because the number of studies for each of the news versus non-news subcategories did not meet the moderation test requirement (Fu et al., 2011).

We found that whether the original message is news or non-news significantly moderated the effects of comment valence on message perception (Q_M = 5.17, df = 1, p = .023). Specifically, when the original message was a news article, comment valence had a smaller effect on message perception (r = .17, p < .001) than when the original message was not a news article (r = .29, p < .001). Whether the original message is news or non-news did not significantly moderate the effect of comment valence on issue-relevant beliefs and attitudes (Q_M = 0.32, df = 1, p = .572).

Publication Bias

Publication bias refers to the phenomenon that a study is more likely to be published if the findings are statistically significant (Sedgwick, 2015). Figures A1–A5 show the funnel plots for each outcome. Although visual examination of the funnel plots suggest asymmetry, the Egger’s tests indicated that there was no publication bias regarding message perception (z = −1.15, p = .250), issue-relevant beliefs and attitudes (z = −0.61, p = .545), and issue-relevant behaviors and behavioral intentions (z = −0.92, p = .359). However, there was publication bias regarding communication behaviors and intentions (z = 2.01, p = .045) and emotions (z = −2.80, p = .005).

Discussion

Result Summary

The results show that compared to negative comments, positive comments led to more positive message perception (r = .22), stronger issue-relevant beliefs and attitudes that align the positive comments (r = .29), higher likelihood of engaging in behaviors advocated by the positive comments (r = .09), higher likelihood to express opinions consistent with the positive comments (r = .26), and stronger positive emotions (r = .16).

This meta-analysis also reveals moderators for the effects of comment valence on different outcomes. First, the number of online comments moderated the effects of comment valence on 1) message perception, 2) issue-relevant belief and attitudes, and 3) issue-relevant behaviors and behavioral intentions. The effect of comment valence on message perception was stronger when the number of comments was greater, whereas the effects of comment valence on both issue-relevant beliefs and attitudes and issue-relevant behaviors and behavioral intentions were weaker when the number of comments was fewer. Second, whether comment valence was mixed or not significantly moderated the effects of comment valence on issue-relevant behaviors or behavioral intentions and issue-relevant beliefs and attitudes, such that comment valence effects were larger when comments were univalent compared to mixed-valent. Comments that were all positive vs. comments that were all negative led to stronger issue-relevant behaviors or behavioral intentions and issue-relevant beliefs and attitudes. However, comments where the majority were positive vs. those where the majority were negative showed no impacts on these two outcomes. Lastly, type of original post (i.e., news article vs. non-news posts) significantly moderated the effects of comment valence on message perception, although such a moderation effect was not found for the outcome of issues-relevant beliefs and attitudes. Comment valence exerted greater effects on message perceptions when the original post was a non-news post (vs. news article).

Impact of Online Comment Valence on Psychological and Behavioral Outcomes

Message Perception

The result that positive (vs. negative) comments lead to more positive perception toward the media content supports the TIME model, which postulates that those online comments, as heuristic cues, shape individuals’ evaluation of the message. The results showed that comment valence has a stronger effect on message perception when individuals read more comments. This is not surprising given that the number of comments itself may serve as a heuristic cue that allows audiences to gauge the reliability of the comment valence in representing the public opinions on the message (Lim & Van Der Heide, 2015). Compared to a few comments, a considerable number of comments can be perceived as a stronger exemplar of public opinion and thus has stronger effects of comment valence on message perceptions. Results also showed that comment valence has a smaller impact on message perception when the original message is news (vs. non-news). It is likely because individuals have relatively stable pre-existing perceptions of news articles given the characteristics of the affiliated news agency (e.g., party affiliation; Landreville & Niles, 2019) and relatively stable levels of trust in news media in general (Fletcher & Park, 2017). Therefore, individuals may rely less on other people’s opinions when reading news articles compared to other types of media content.

Issue-Relevant Beliefs and Attitudes

The results suggest that positive (vs. negative) comments lead to issue-relevant beliefs and attitudes that more align with the stances taken by the positive comments. This shows that online comments, serving as exemplar for majority public opinions or cues for social norms, are powerful to push individuals to opinion conformity (Duncan et al., 2020). The results also showed two moderators of the effects of comment valence on issue-relevant beliefs and attitudes. First, the moderating role of the number of comments was intriguing: comment valence has a stronger effect on issue-relevant beliefs and attitudes when fewer comments are presented. This is possibly because compared to a fewer number of comments, a larger number of comments are often loaded with more information, which can be harder for readers to process and gauge the opinion climate, especially during an issue evaluation process which requires intensive cognitive efforts (Cacioppo et al., 1986).

In addition, whether or not the comments are mixed-valence moderated the effects of comment valence on issue-relevant beliefs and attitudes. First, comment valence significantly affects issue-relevant beliefs and attitudes when the comment was univalent. This is probably because unanimous valence, compared to mixed valence, demonstrates a higher level of consistency of the commenters’ opinions, which yields stronger normative influence in changing issue-relevant beliefs and attitudes (Andrighetto & Vriens, 2022). On the other hand, comment valence has no effect on issue-relevant beliefs and attitudes when the comments are mixed-valanced. This is probably because the small portion of comments with certain valence opposite to the majority comments stand out to the readers as a divergent opinion cluster (Kohout et al., 2023), which may attenuate, offset, and even outweigh the effects of majority comments (Gellerstedt & Arvemo, 2019). The finding suggests a need for future researchers to consider the potential impacts of operationalizations of comment valence (e.g., proportions of comments with varying valences) when investigating and interpreting the effects of comment valence.

The non-significant moderation effect of original message type (i.e., news vs. non-news) suggests that online comment valence influences online users’ issue-relevant beliefs and attitudes, regardless of the original message type. As previous research suggested, overt message cues may diminish the role of message source (Wise & McLaughlin, 2016). In our context, user comments with predominant valence may serve as strong message cues, which may be of paramount importance in the online users’ issue evaluation process.

Behaviors and Behavioral Intentions

In line with predictions from the exemplification theory and existing literature on social norm influence, our results show that positive (vs. negative) comments are effective in promoting behaviors and intentions to perform the behaviors advocated by the positive comments. It should be noted that the effect size of comment valence on behavior and behavioral intentions was much smaller than the effect size for issue-relevant beliefs and attitudes. This may be due to the fact that changes in behaviors and behavioral intentions depend on changes in antecedent variables such as attitudes, subjective norms, and perceived behavioral control (Rivis & Sheeran, 2003), which may render behaviors and behavioral intentions more challenging to change. However, for individuals who habitually read online comments daily, the small effect size on behaviors and behavioral intentions may grow stronger due to repeated exposure to similar online comments over time (Chen et al., 2022).

Communication Behaviors and Intentions

In line with the spiral of silence theory, our finding shows that predominantly positive (vs. negative) comments lead to increased likelihood that the users will express opinions consistent with the positive comments. However, it should be noted that the topics investigated in these studies—such as fake news, body image, and abortion—may be frequently discussed by online users. For instance, individuals often share personal experiences or opinions on issues like abortion or body image to seek or offer social support within online communities (e.g., Lands et al., 2023). Furthermore, people tend to correct misinformation online, driven by the perception that others might be vulnerable to misinformation (e.g., Koo et al., 2021). Thus, more research efforts are needed to test the effects of comment valence on a wider range of topics for more accurate conclusions.

Emotions

The significant effect of positive (vs. negative) comments on emotions supports that emotion contagion could happen in cyberspace through user-generated comments (Kwon & Gruzd, 2017). Studies found that emotions aroused by online media messages may further affect cognition such as perceived health risk (Oh et al., 2021) and motivate behaviors such as engaging in social movement (Ahmed et al., 2017). In this sense, emotion change may be considered a mediator when theorizing the effect of online comments on beliefs, attitudes, and behaviors in future research.

Theoretical Implications

This meta-analysis yields important theoretical implications. First, are online users influenced by what other users say? The evidence we meta-analyzed suggests that they are. Drawing from TIME, exemplification theory, social norm influence, spiral of silence theory, and emotional contagion theory, this meta-analysis provides cohesive support for the proposition that online comment valence is powerful in shaping cognitive, emotional, and behavioral responses.

Second, this analysis demonstrates the efficacy of existing theories in predicting the effects of online comment valence on various outcomes among online users. Specifically, our findings support TIME and MAIN’s postulates that online comment valence, as a heuristic cue, affect users’ perceptions of media content. In line with exemplification theory and the concept of social norm, the results confirm that online comment valence, as exemplars or cues of prevailing opinions, steer online users towards adopting the majority’s issue-relevant beliefs, attitudes, and behaviors. Additionally, the findings align with the fundamental premises of spiral of silence theory, showing that online comment valence prompts individuals to express opinions aligned with the majority’s opinions in online environments. Lastly, the finding that online comments valence evokes corresponding emotions supports emotion contagion theory by expanding its purview from face-to-face interactions to digital spaces. 

Moving forward, this meta-analysis suggests avenues for theoretical refinement. Firstly, instead of focusing on singular psychological or behavioral outcomes evoked by the interactive media (e.g., comment valence), theoretical models should include a wider array of psychological and behavioral outcomes based on empirical evidence. For example, the TIME model may extend its propositions, delineating how bandwagon cues influence not just message perceptions but also issue-related outcomes and subsequent behaviors. Move beyond merely identifying effects, theories should delve into the mechanisms behind these effects across various outcomes. For instance, research might explore if there is a sequence in these outcomes when people are exposed to comment valence. It could be that comment valence initially alters how messages are perceived, which then triggers changes on issue-relevant perceptions that ultimately result in behavior changes. In addition, research could explain how different outcomes interact to produce other effects. For example, message-related perceptions and emotions might interact, leading to shifts in issue-related perceptions. Answering these questions is beyond the scope of our meta-analysis; more research efforts are needed to explore the possibilities presented here.

With respect to findings regarding moderators of comment valence effects, our results showed that the types of original message, number of comments, and compositions of comment valence (i.e., mixed-valence or univalence) should be considered when predicting the effects of online comments. It should be noted that our meta-analysis may only scratch the surface regarding the moderators of comment valence effects. For further theoretical refinement, future meta-analyses and systematic reviews should examine the moderating roles of a broader range of media and message characteristics, such as the type of comments (e.g., real-time vs. retrospective), bandwagon metrics (e.g., emoji reactions), and the comment source (e.g., human vs. machine).

Practical Implications

The current meta-analysis showed that online comments can be a double-edged sword with potential positive and negative impacts on individuals who read the comments. Comments expressing positive opinions toward socially desirable phenomena or expressing negative opinions on socially undesirable phenomena hold promise in fostering desirable social impacts, such as promoting health outcomes (Lu & Sun, 2022) and reducing prejudice toward disadvantaged social groups (Stylianou & Sofokleous, 2019). Conversely, comments conveying positive opinions on socially undesirable phenomena or conveying negative opinions on socially desirable phenomena may exacerbate undesirable social outcomes, such as promoting undesirable health behaviors (Lu & Sun, 2022) and violence and prejudice against minorities (Stylianou & Sofokleous, 2019). 

The finding that comment valence influences cognitive, emotional, and behavioral outcomes highlights the importance of early and continuous monitoring of comments as an integral part of media campaigns. Media practitioners need to recognize that the valence of the comments on their communication material holds substantial power to shape message perceptions, issue-related beliefs, attitudes, behavioral intentions, and emotions. While the presence of only positive comments may imply effective communication efforts, predominantly negative comments may signal a need to adjust communication strategies before the negative comment valence influences more viewers. Therefore, effective media campaigns aimed at public welfare, like those addressing public health concerns, empowering minoritized groups, or advocating for positive public policies, should vigilantly track the online comments’ valence and adjust their communication strategies accordingly for optimal communication effects. In addition, to empower online users in generating and consuming online comments, government agencies, educational organizations, and the media platforms can implement media literacy education and interventions (Cho et al., 2024). As such, online users are expected to be aware of a) the influences of online opinion climate on themselves as media consumers, and b) the impacts of opinion they expressed through online comments on other individuals.

The results of the moderators of comment valence effects also have implications for media practitioners. The insights about the moderating role of the number of comments indicate that, to alter message perceptions, media practitioners can amplify the impact of positive comments by encouraging online comments through compelling and interactive communication methods. When the objective is to influence issue-relevant beliefs, attitudes, behavioral intentions, and behaviors, practitioners may control the number of comments by, for example, limiting the volume of comments or implementing moderation/filtering strategies. The moderating effect of mixed valence suggests that fostering a consistent comment valence that aligns with campaign objectives (through approaches like active user engagement) should be beneficial for changing issue-relevant beliefs, attitudes, behaviors, and behavioral intentions. Lastly, the moderating role of original message frames (news vs. non-news) demonstrates that those disseminating non-news content ought to be particularly vigilant regarding comment valence. Unlike news writers, who experience a comparatively diminished impact of comment valence on message perception, practitioners sharing non-news content should actively monitor comment sentiments and make strategic adjustments in response to comment valence to maintain or alter the desired perception among audiences.

Limitations and Future Research

The study’s limitations offer some insights for interpreting the results and guiding future research efforts. First, because we did not receive replies from some authors after we requested data, we had to exclude a number of studies from analysis. This may lead to less precise and possibly biased effect size estimates of online comments on the outcomes. We would like to call for collective efforts of future researchers, whether actively requesting or responsively providing necessary data, to ensure precise meta-analysis results. The list of articles that were excluded due to the lack of data to calculate effect sizes can be requested from the corresponding author.

Second, the identification of publication bias in communication behaviors and intentions and emotions poses a limitation to our study. This bias, potentially inflating effect sizes, demands caution when interpreting and generalizing the findings regarding these two outcomes. To mitigate bias, future research should report comprehensive data regardless of significance of the effects. Continuous synthesis of updated evidence is crucial for a more accurate understanding of how comment valence impacts communication behaviors, intentions, and emotions. In addition, the high heterogeneity issue was remains unaddressed by our moderation test. Future studies might explore alternative moderators to account for this ongoing high heterogeneity.

Third, we calculated average effect sizes for studies that reported multiple outcome variables under a single outcome category. For example, the effect sizes of brand trust and brand attitudes were averaged to represent the effects on beliefs and attitudes (Y.-I. Lee et al., 2020). Although this method is an effective approach to synthesize studies that measured different outcome variables, the method may overlook valuable information about the effect sizes of specific outcome variables. When there are enough studies examining each specific outcome variable in the future, we would recommend conducting meta-analyses for each outcome variable separately.

Fourth, a number of studies in this review examined potential moderating roles of individual characteristics, such as issue familiarity (Shi et al., 2014) and pre-existing attitudes toward the issue (Lu & Sun, 2022), in comment valence effects. However, only partial studies included in this review investigated the moderation effects of individual characteristics. Also, the wide range of topics (e.g., smoking, fake news, child corporal punishment) investigated across the included studies may result in varying levels of individual attributes, such as issue familiarity, issue involvement, and strengths of pre-existing attitudes, which cannot be objectively coded. Thus, it was not feasible to analyze the extent to which individual traits moderate the overall effect sizes of comment valence across all included studies.

Lastly, despite our comprehensive investigation into the effects of comment valence on cognitive, emotional, and behavioral outcomes, an unexplored facet remains pertinent to the landscape of online comments—the interplay between comment valence and comment civility/incivility. Prior research has convincingly demonstrated that the civility or incivility of comments significantly influences a wide range of cognitive and emotional outcomes (Chen & Lu, 2017; Hwang et al., 2014; Rösner et al., 2016). Thus, it is possible that uncivil comments, relative to civil comments, attenuate the effects of positive vs. negative comments on issue-related attitudes and intentions due to stronger negative emotions (Chen & Lu, 2017), stronger perceived intergroup difference in issue positions (Hwang et al., 2014), and stronger aggressive feelings and hostile cognitions toward the commenters (Rösner et al., 2016). Given little empirical research in this field, continue research efforts are needed to explore the potential interactions between comment valence and comment civility/incivility on various outcomes.

Footnotes

¹ The 12 databases included Academic Search Ultimate, Communication & Mass Media Complete, APA PsycInfo, SocINDEX with Full Text, MEDLINE, Library & Information Science Source, Psychology and Behavioral Sciences

Conflict of Interest

The authors have no conflicts of interest to declare.

Acknowledgement

The data and R code can be found on OSF via the link: https://osf.io/34krz/?view_only=7f2ff649d4e743179de1f15fd963a20d.

Appendix

Figure A1. Funnel Plot—Message Perception Outcome.

Note. The dots represent the studies included in the meta-analysis. The white region corresponds to p-values greater than .10. The red region corresponds to p-values between .10 and .05. The orange region corresponds to p-values between .05 and .01. The grey region corresponds to p-values smaller than .01. The same illustration approach applies for all funnel plots in this paper.

Figure A2. Funnel Plot—Beliefs and Attitudes Outcome.

Figure A3. Funnel Plot—Behaviors and Behavioral Intentions Outcome.

Figure A4. Funnel Plot—Communication Behaviors and Intentions Outcome.

Figure A5. Funnel Plot—Emotions Outcome.