The psychological impacts of content moderation on content moderators: A qualitative study

Introduction

Social media and the internet have become ubiquitous. It is estimated that nearly three billion people use Facebook each day (Facebook, 2022b), while Instagram has 2 billion active users (Statista, 2022). However, with the freedom to share information comes the ability to upload and distribute more troubling content. In the second quarter of 2022 alone, Facebook actioned 20.4 million pieces of child sexual exploitation (Facebook, 2022a). The increasing amount of user generated uploaded content means that companies need to hire more and more content moderators (CMs), these are the individuals who evaluate and remove material that is flagged by users as harmful or in violation of platform policy.

There is a wealth of evidence that exposure to other people’s trauma at work affects wellbeing and mental health (e.g., Léonard et al., 2020; Ralph, 2020) and research focusing on CMs suggest they suffer from impacts similar to those found in other professionals (Steiger et al., 2021). However, despite a growing number of businesses employing moderators (Fact.MR, 2022), research attempting to evaluate the impact on CMs has largely focused on volunteer moderators (Cook et al., 2022; Schöpke-Gonzalez et al., 2022). It is likely that volunteer and commercial CMs have distinct working conditions, likely leading to experiencing the role differently, in that it may be associated with different expectations, training, and levels of support (Burns et al., 2008).

The present study was designed to qualitatively explore any psychological impacts in commercial CMs. This was to illustrate the potential difficulties experienced within their roles, while also providing information for more effective work practice in this field. The use of in-depth semi-structured interviews allows participants to express themselves freely and articulate a diversity of experiences. The results can be used to inform organisations about how the work may affect employees and therefore what types of support should be put in place. Indeed, without a sound understanding of any specific psychological effects on CMs, attempts to ameliorate or counter any impacts could be misguided or even harmful (Wessely et al., 2000). This would impede efficiencies within the role and could adversely affect staff retention and recruitment. In the remainder of this introduction, a review of the different work environments populated by CMs and how the work CMs undertake might give rise to psychological difficulties is considered.

The Work of a Content Moderator

CMs enforce the online rules which tell users how to behave and what content is acceptable on a particular internet site (Gerrard, 2022). Some sites use a “community reliant approach” where communities define their own standards that are then put into effect by volunteer moderators, whilst other sites use an industrial approach, where workers are employed to enforce a set of standardised rules (Caplan, 2018). As such, the ways in which CMs are employed vary. Whilst some are volunteers, paid moderators can range from in-house workers employed directly by the company needing moderation; to boutique firms specialising in content moderation for other companies; to outsourced third party vendors and microlabour platforms (Roberts, 2019). This means CMs can experience a range of working conditions where some have less workplace protection as technically, they are not employees - thus the company requiring moderation work has a level of “plausible deniability” to the harm faced by these CMs (Barrett, 2020; Roberts, 2019).

Nevertheless, CMs play a critical role in protecting firms’ reputations, ensuring online platforms do not become unusable, hostile spaces that are unprofitable for these business (Drootin, 2021). Furthermore, they shape what is seen online as they make hundreds of daily decisions, and for some this can be with limited oversight (Parks, 2019). In 2014 Chen estimated there were over 100,000 paid CMs worldwide. It is likely that in the intervening years this number has increased. In 2021 Messenger and Simmons reported Facebook and Instagram had expanded their moderation workforce to 15,000 people, and companies continue to hire more with a proliferation of businesses that specialise in brand protection and moderation (Fact.MR, 2022). These figures do not include the numerous unpaid CMs used by sites like Reddit (Witynski, 2022).

As part of their role, material deemed offensive or harmful is sorted into queues and sent to CMs for review where they must make decisions about suitability in line with the set of policies they are currently enforcing (Ahmad & Krzywdzinski, 2022). CMs can be required to view depictions of child sexual abuse and violence, as well as less graphic but no less noxious content that displays cruelty, humiliation and discrimination. Thus, moderators engage in emotional labour as they are expected to regulate their own emotions when exposed to this material, as well as during interactions with users as they work to represent the platform (Steiger et al., 2021). The content they view can range from text, to still photos to explicit video and the amount of time spent viewing material varies; it can however be substantial in terms of hours every working day (Dwoskin, 2019). In addition, CMs have to maintain high accuracy levels and throughput quotas to meet acceptable job performance standards, which contribute to stress above that which might be experienced through viewing the content alone (Dwoskin et al., 2019; Wohn, 2019). Dwoskin et al. (2019) described how moderators had their performance regularly audited and were expected to maintain an accuracy score of over 95 percent when identifying and correctly removing harmful content in accordance with company standards.

There is a body of evidence highlighting the consequences of chronic job-related stress or exposure to difficult or traumatic content at work being serious and long-lasting (Vasconcelos et al., 2021). Research with front-line professionals such as first responders has revealed that exposure to other people’s traumatic experiences can lead to post-traumatic stress disorder (PTSD), a disorder that can develop after exposure to actual or threatened death, serious injury, or sexual violence either directly, through experiencing or witnessing the event, or experiencing repeated or extreme exposure to aversive details of the traumatic event(s). It is marked by intrusion, avoidance, negative symptoms and hyperarousal (American Psychiatric Association, 2013). Professionals in caring professions such as social workers and therapists who are exposed to second-hand accounts of trauma can develop syndromes such as secondary traumatic stress (STS), vicarious trauma (VT) and burnout (Craig & Sprang, 2010; Greinacher et al., 2019; Lee et al., 2018). These have been referred to as the “cost of caring” and relate to the stress reaction experienced by someone in response to another’s distress (Figley, 1995). STS has similar symptoms to PTSD such as intrusive thoughts, hyperarousal and hypervigilance, as does VT, although it is also associated with a transformation of self-image and worldview (Foley & Massey, 2021; Krause, 2009). Burnout is characterised by emotional exhaustion, cynicism and reduced feelings of accomplishment (Brady, 2017; O’Malley et al., 2019).

This is also true for professions not traditionally considered front-line staff. For example, journalists exposed to traumatic events while working with difficult content such as child abuse, child cruelty, war and aviation accidents, report higher levels of PTSD and the internalization of guilty cognitions (Browne et al., 2012). Drone pilots involved in military operations have reported higher levels of burnout, and an increased likelihood of developing anxiety, depression and PTSD (Asaro, 2013). Judges and lawyers report sleep disturbances, feeling isolated, concentration problems and interpersonal issues (Jaffe et al., 2003; Léonard et al., 2020). Lastly, police officers who specialise in internet child exploitation and are repeatedly exposed to images and videos of child abuse as part of their investigations show raised rates of STS, mental health difficulties and absenteeism (Bourke & Craun, 2014; Ralph, 2020). Furthermore, disorders such as PTSD, STS and burnout are often comorbid with other mental health problems such as depression and anxiety and in some cases their onsets are linked (e.g., Hakanen et al., 2008; Schindel-Allon et al., 2010).

Anecdotal accounts from investigative journalists suggest the work of CMs places them at risk for a range of psychopathological difficulties. CMs report anxiety, depression, nightmares, fatigue and panic attacks, with their relationships and physical health also affected (Newton, 2019, 2020b). Both Microsoft and Facebook have been sued for substantial sums by CMs who developed PTSD on the job and held the companies responsible (Ghoshal, 2017; Newton, 2020a). Similarly, academic research highlights high stress levels and fatigue (Cook et al., 2022) and symptoms of insomnia, anxiety, low mood and experiencing intrusive images (Benjelloun & Otheman, 2020) and work by Roberts (2019) found that although CMs would report not being negatively impacted by the work they would demonstrate via anecdotes that the work was affecting them and their relationships.

More broadly, moderators have reported feeling burnout and emotionally exhausted (Dosono & Semaan, 2019; Riedl et al., 2020; Schöpke-Gonzalez et al., 2022), apathy (Lo, 2018) and underappreciated (Wohn, 2019), as well as being harassed by users or feeling guilty for enforcing rules or taking actions they do not necessarily agree with (Dosono & Semaan, 2019; Wohn, 2019). Moral injury occurs when someone acts, fails to prevent or bears witness to acts that transgress their moral beliefs (Litz et al., 2009) and therefore for some, the work of CMs could manifest as moral injury. Indeed, a mismatch between the community and moderator’s values is associated with quitting in volunteer moderators (Schöpke-Gonzalez et al., 2022).

Nevertheless, some individuals inevitably are more susceptible to trauma than others and there is individual variability to the degree of negative impact as a result of the role (Powell et al., 2015). People exposed to trauma can experience post-traumatic growth, whereby their coping leads to a greater sense of purpose and life meaning (Steiger, 2020). A survey of internet child exploitation police officers found they had higher wellbeing than a normative adult sample (Tomyn et al., 2015), and similarly, a study of volunteer moderators found low levels of STS (Schöpke-Gonzalez et al., 2022). Organisational support can also raise feelings of compassion satisfaction where the focus rests on the benefits rather than costs of the job, and positive feelings are experienced when helping someone who has been traumatised (Brady, 2017; Hunsaker et al., 2015). This can act as a buffer against compassion fatigue, STS and burnout (Brady, 2017; Conrad & Kellar-Guenther, 2006). Thus, the role of CM may produce significant stress for some, but not all.

The evidence suggests that content moderation is associated with a variety of negative impacts, although it is possible for some individuals the work will be associated with positive effects. To date, there have been few academic studies that explore what impacts the demands of the job might engender and many that have been conducted have focused on the experience of volunteer CMs (e.g., Dosono & Semaan, 2019; Lo, 2018; Wohn, 2019). However, volunteer moderators may experience different types of psychological distress due to their distinct motivational factors and the range of activities they engage in which commercial CMs do not (Matias, 2019). The current study aims to qualitatively explore the impact of the job on commercial CMs, with the objective of using these results to highlight areas where problems may arise. Specifically, the study set out to investigate a) what, if any, behavioural changes moderators may experience due to their job and b) how, if at all, the work affects their relationships with others. The findings can act as a first step to guide larger scale studies on the impacts.

Methods

Participants

Eleven CMs with experience of moderating CSAM were recruited. Initial recruitment was through online advertising on social media platforms and utilising existing networks, with further recruitment relying on a snowball technique where interviewees provided the research team’s details to potential participants. Participation was voluntary and CMs were not remunerated for their time. Moderators who wanted to participate were given an information sheet which explained the study and outlined that they had to have experience of moderating CSAM. All participants were given the opportunity to ask questions before, during and after involvement. There was no minimum requirement with regards to how much exposure a participant had as the study was interested in exploring various CM experiences. All participants spoke English but for the majority it was not their native language. Specific employment-related details such as the participant’s employer or specific sites they may have moderated were not requested, however at the beginning of each interview participants were asked to outline what their job entailed and confirm they were exposed to CSAM as part of their work. All participants worked full-time as CMs either for commercial companies or CSAM specific hotlines. The sample included six females and five males, from eight countries: the Netherlands, UK, Ireland, Germany, Brazil, Colombia, Malaysia and India.

Procedure

All participants took part in an online interview, and could participate with their camera on or off. Interviews were administered by two of the research team and were conducted from April to June 2022. They ranged in duration from 47 to 78 minutes, with average interview time of an hour. The interviews all followed a similar structure with participants asked about their role, how the material affected them, how they coped, other occupational stressors, how the job was perceived by others and the impact of COVID on their working conditions. This paper focuses on data relating to impacts of the work. Interviewers asked open-ended questions and follow up probes were used as needed to elicit further information and clarify answers. Informed consent was recorded verbally before the interview commenced. Ethical approval was granted for the study by the university’s psychology research ethics committee (ref: 20888). The interviewers did not specifically ask about non-disclosure agreements (NDAs; contracts CMs may be required to sign that prohibit them from talking about their work) but anonymity and confidentiality were guaranteed through the use of participant codes, and participants personal details were not stored. All data was secured on password protected drives in line with Data Protection and UK GDPR regulations and the interview recordings were deleted after transcription.

Data Analysis

All interviews were audio-recorded, transcribed verbatim and anonymised through removal of identifiable information. A Framework approach was used to analyse the data (Gale et al., 2013). This enables in-depth exploration of the data whilst making the process of data analysis transparent and is particularly suited to the analysis of cross-sectional descriptive data (Ritchie & Lewis, 2003). It involved six stages, Familiarisation: the transcripts are read repeatedly to enable familiarisation with the data and identify key themes. Coding: descriptive or conceptual labels are systematically applied to interesting or notable excepts of raw data. The coding framework is then discussed and agreed by the research team. Developing an analytical framework: codes are grouped into categories representing interrelated ideas or concepts. Applying the analytical framework: the codes and categories are applied across all the transcripts. Entering the data into the framework matrix: a spreadsheet is used to generate a matrix. In the matrix, each column represents a key category and each row represents an individual participant. The summarised data from each transcript are entered into the matrix so each cell represents a category by participant. Interpreting the data: the final analytic stage involves working through the data in the matrix, drawing out the range of experiences and views expressed during the interviews, identifying similarities and differences across participants, and interrogating the data to seek to explain emergent patterns and findings.

Results

The CMs reported a number of different impacts across a variety of domains, particularly emotional and cognitive. They also discussed factors that exacerbated work stress aside from the content. The identified themes and sub-themes are presented in Table 1.

Table 1. Themes Identified Through Framework Analysis.

Impact Awareness

Regardless of if CMs felt they were personally affected by the work, there was an awareness amongst all the moderators that the content they were being exposed to was harmful and had the potential to affect them long-term.

“I think the turnover’s always higher in the more problematic content. We didn’t see that much turnover in terms of bullying or election misinformation or adult nudity because those policies are, that kind of content doesn’t really traumatise you as much as the other ones.” (CM8)

Nevertheless, some CMs did assert they were unaffected by the job. However, further probing always elicited either current or past examples of difficulties. The reluctance to talk about impacts was notable and possibly reflects attitudes towards the work, such as not wanting to talk about the difficulties, because they agreed to take on the job and accepted that exposure to content was part of their profession. Additionally, many effects may come on gradually and may be hard for CMs to recognise, quantify and put into words. CMs are aware that something has changed and can talk about being affected in general, but find it difficult to identify specifics. “It has an effect on you detrimentally when you’re on your own. It affects your imagination in ways that you can’t quantify, but it still affects you.” (CM3)

Therefore, although all CMs gave examples of how they had been negatively affected by their role, the level of distress caused and insight about effects varied. A couple reported they could not identify any impacts within themselves and viewing traumatic content was “just their job” but noticed impacts in others, or gave examples of impacts that occurred in the past, possibly reflecting that any effects or symptoms associated with the work change over time for some CMs.

Detachment

Associated with change in effects over time was the ability of CMs to detach from the material. It seemed after the initial shock of viewing content, a common way to cope with the feelings evoked was to shut down emotionally. This ability seemed to develop over the first few months in the job and enabled CMs to look at content without experiencing any adverse reactions. “They kind of get used to see those tickets and to see that kind of content and it’s like they transform into robots and suppressing their emotions and stuff like that.” (CM7)

In particular, CMs tried to keep an emotional distance from the victims. This both involved remaining analytical, but also a conscious effort not to think about the children as people or think about what had happened to those victims both before and after that particular piece of content was created.

“I’m just trying to keep my mind to analysing the material, and it’s been working so far... It helps me because then I don’t remind myself that it’s actual people and actual stuff that happened. It helps me to just think about it as something like a TV series that they’re analysing or whatever”. (CM1)

However, others struggled with maintaining that boundary and could not remain detached from the victims. This resulted in them subjecting themselves to the content more than they had to as a result.

“That child, when on his own, showed his trauma and as a reviewer I kind of think this is my responsibility to make sure that, even if I do it years after the event happened, it’s my responsibility to make sure that the child in some way wasn’t on his own going through this trauma.” (CM9)

At the extreme end the emotional blunting could turn into desensitisation, where CMs became anesthetised to what they were witnessing. Although perhaps an adaptive response overall, it made a couple of CMs question themselves and if their ability to do the work meant there was something wrong with them. However, it may be that those who were unable to turn off their emotional response in this way did not last long in the job.

“If you see a child being sexually abused how do you look at this in a numb way or can you ever look at it? It leads to a lot of burnout. A lot of people in the child sexual exploitation or terrorism spheres particularly have a quick turnaround in terms of the work that they do because it’s traumatic for them. It’s not something that people want to view.” (CM8)

Desensitisation did not occur across all content. Some CMs had personal sensitivities and reported certain types of content continued to be upsetting, even after they had adjusted to what they were being exposed to overall. This could be both in terms what was in the content but also across different types of content, e.g., text rather than images was more disturbing for some. “My trigger is bestiality. I cannot work on the bestiality content, but I had to do it at some point, and it was really tough for me because I have my dog.” (CM5)

Negative Cognitive Changes

CMs experienced intrusive thoughts and discussed how images of CSAM remained in their minds after initial exposure. The intrusive thoughts were generally associated with feelings of psychological distress or unease.

“So that was quite disturbing for that moment. Yes, I did take about two or three days off from work after that... I was not physically ill. It just was immediate that I couldn’t take off my head from that.” (CM6)

CMs reported that intrusive thoughts were particularly prevalent during, and indeed triggered by, sexual relations with their partners. This often resulted in them having to stop and/or deliberately distract themselves from thoughts of CSAM. This could leave them questioning their sexuality and reduced feelings of intimacy within the relationship.

“Same with my own sexuality, sometimes I say to my boyfriend, just let me have a minute for myself and then I’ll try to bring my thoughts to nice thoughts and then it’s okay, but that wasn’t there before. That’s new since I started the job.” (CM2)

The presence of children also appeared to trigger thoughts of CSAM and worry about child welfare. In a couple of cases this even led to CMs trying to avoid children, especially situations where a child might become upset as this was particularly distressing and triggered or intensified intrusive thoughts of CSAM. CMs who had children particularly struggled with the content and it was reported that staff often left the job when they were either planning on or having children of their own, demonstrating a recognition that having children would impact their ability to do the work. CMs in general demonstrated heightened concerns about child safety and feeling more protective of children.

“When my aunt had her child… I tried to explain to her what’s happening, what’s actually happening… and that you need to be more attentive as a parent when you’re leaving your child even to a simple game. It’s not just a game anymore.” (CM7)

More general changes in cognition were also evident, with the exposure to disturbing content leading to the development of a more negative world view. CMs talked about having lived in a bubble or having been sheltered but that their job had revealed another darker side of the world. This was also associated with the development of a cynical view of people and their behaviour. This general distrust of others experienced by CMs could result in difficulties with interpersonal relationships. In particular, a couple of female CMs expressed concerns about men in general as they developed suspicions of their possible involvement in perpetrating or watching CSAM. As a result, some described experiencing a more general loss of hope and fewer positive emotions.

“But sometimes I feel like my joy has been taken away from life. It’s not in a very serious way but, I don’t know, I just see the world a little bit less colourful. Yes, it’s a bit difficult to explain, but I guess I know now how dark the world can be.” (CM1)

Emotional Toll

The work also took an emotional toll on CMs. Many talked about feelings of anger or frustration and this was discussed generally in the context of the content, feeling anger when they thought about what they had seen or the people that were involved. However, a couple were more explicit in saying their anger was more generalised than that. CMs also spoke about physical affects experienced at work, with one even describing having panic attacks. The descriptions offered varied but all were indicative of experiencing a “fight or flight” response to the content.

“Sometimes I feel not exactly nervous but like my body is very awake but not in the sense of not being tired, more like being a little, maybe shaking a little but not shaking on the inside. More like nervous but without a reason when I see very much content or content that is very disturbing.” (CM2)

In general though, CMs did not articulate the exact emotional affects. They tended to talk in generalities about mental health issues, getting “sick” due to the content and experiencing emotional stress. This suggested the job might be associated with a general malaise rather than specific problems.

“He just said, ‘I knew that there would be some really gross content, but I really didn’t expect this.’ Honestly, I didn’t have really words to tell him. I mean, literally, I just go and tell him that that’s the part of work that you’re feeling.” (CM6)

They also discussed feeling fatigue at work and being drained or exhausted by the work. This might be a slightly different way of expressing the malady experienced by workers in this role. This weariness could be linked to the emotional effects of being exposed to content, or a business need in meeting quotas. “When I started here, I actually was off sick for 60 per cent, because I had no energy and I wasn‘t feeling good. I couldn‘t even stay awake when I was here.” (CM1)

CMs reported sleep disturbances, such as nightmares, which often were connected to the violence they had witnessed during work. There was also difficulty falling or staying asleep, which may have contributed to day time tiredness. This was due to replaying images as they were trying to get to sleep or waking up for fear of being asleep.

“I started to have sleeping problems, I don’t know it was really irrational, everything comes to me and I had problems sleeping, I was afraid to sleep for a week. I wanted to sleep, but I didn’t sleep well, that week I would sleep 2 or 3 hours. I would wake up and be like no no, I wasn’t sleeping, no I have to be awake.” (CM4)

CMs also mentioned that self-care suffered as a result of the work. This was attributed to stress, whether caused by the content itself or having to hit specific targets and quotas, individuals developed ways to comfort themselves at work.

“I know of many people who say that they do eat a lot of chocolate while working. Chocolate is one; coffee is another… Thirdly, smokers who smoke, they tend to take a lot more smoking breaks… One is to help them cope with what they see. Secondly, the stress of meeting their daily numbers.” (CM6)

CMs felt unable to talk to friends and family about their work. They felt that others did not want to know or that they would be traumatised if they did know. However, for some this was associated with a feeling of disconnection because not only were they shielding others from what they had seen, but also they were hiding the affects the job could have on them. For some this contributed to a sense of isolation.

“For my grandparents, I’m a researcher. That’s all that they know about me, they don’t know more. My parents know that I’m working on CSAM but I don’t tell them anything else. Same thing for my partner, but because of this, when you’re having a bad day, when you have your emotions all over the place, you still have to pretend that you’re okay, that everything is fine.” (CM9)

Stressful Work Environment

The emotional toll involved in the work could be exacerbated or alleviated by the response of management, and this seemed to hinge on whether managers respected the role that CMs played within the company. This was often more likely if leaders had direct experience of the work and could empathise with what CMs experienced, both in terms of content but also the targets that CMs had to hit.

“I got affected by the management when they started pressuring you. Lots of things, and you didn’t even realise that how much we are affected by the content as well. It’s really disturbing content what we’re working in, specifically in a minor content, but still, the managers didn’t care about that. They were like, ‘Now, oh, your performance this way. We need to extend something, or…’” (CM5)

Quotas were also mentioned by many of the CMs as something which added to a feeling of pressure or stress on top of what they were witnessing. This was both the levels of accuracy that had to be maintained and throughput. This too, was often linked to the approach of management with a feeling that their focus was on reaching the quotas with no appreciation of the wider context of the job.

“’You’re not getting your quota.’ Bring you into the office, ‘[CM’s name] you need to improve on this, this, and this,’ and it’s like, ‘Woah, hold on a second, you should watch this stuff.’ There’s a bit of a divorce going on there, a disconnect.” (CM3)

Lastly, CMs felt that the tools they had to work with often were not designed with content moderation and their role in mind. This added to the frustration and stress associated with their job because they felt it prevented them from being able to do it effectively.

Discussion

Exposure to other people’s trauma at work can be detrimental to employees (Greinacher et al., 2019; Lee et al., 2018) and to the efficient fulfilment of their work. Existing literature suggests that this is also the case for CMs (Cook et al., 2022; Schöpke-Gonzalez et al., 2022). Research to date examining the psychological effects of this work is sparse and tends to focus on the experience of volunteer moderators who may have a different set of motivations and job-related tasks (Matias, 2019). The present study adds to this literature by qualitatively exploring the psychological effects of content moderation in commercial CMs. The results show both negative cognitive and emotional changes take place and that these are associated with behavioural changes such as avoidance and hypervigilance. Overall, the effects are consistent with commercial CMs experiencing a trauma response in relation to their work, and provides evidence demonstrating why organisations employing CMs need to provide trauma-informed care for their employees.

Similar to previous research, although CMs knew the content was difficult and that people struggled, there was a reluctance to talk openly about the personal impacts of the job (Roberts, 2019). There are several possible reasons for this, such as: a lack of awareness around how they are being impacted or the inability to put the impacts into words. Syndromes such as VT and burnout are cumulative and gradual, this can make it difficult for individuals to recognise any changes within themselves or others (Krause, 2009; Ledingham et al., 2019). Alternatively, CMs may not want to admit to impacts either because they agreed to the job or because as Roberts (2019) argued, adverse effects may suggest they are unable to perform the job adequately. Equally, the participants all worked full-time as content moderators and therefore some may have signed NDAs, which can be associated with a fear of speaking out about the job (Schiffer, 2021).

Nevertheless, some CMs did appear to be coping well. Again, there are several possible reasons for this: some CMs reported they were more sensitive to particular types of content. It could be that certain people, for example those who do not interact with children in their daily lives, are more resilient to working with CSAM material. Similarly, people who stay in in this profession for a long time, may be the ones who are affected less, cope relatively well and experience little in the way of impairment, or even undergo a period of post-traumatic growth. There appeared to be an initial period at the beginning of the job whereby people either tended to adjust to the role or leave. Therefore, CM resilience may be dependent on the content they are exposed to, or may change over time. However, the adjustment seemed to be associated with the ability to suppress emotions, which may have longer-term consequences for social and mental wellbeing (Chervonsky & Hunt, 2017; Hu et al., 2014). Future research should investigate what predicts resiliency in CMs, but also the potential long-term effects both in terms of ongoing resiliency and any possible deleterious consequences of this.

Many of the impacts described by CMs are in line with the symptoms described by volunteer moderators (Dosono & Semaan, 2019; Schöpke-Gonzalez et al., 2022). The symptoms are also reminiscent of those occurring after repeated or extreme exposure to aversive details of trauma as described by PTSD or STS, and the majority of themes could be grouped in this way (see Table 2). When CMs gave examples of how they were affected, they talked about intrusive thoughts and images of CSAM including that these were triggered by children and their own sexual activity. Additionally, some described avoiding children and there were a number of negative cognitive and emotional affects, including hypervigilance around children, anger and increased distrust of others. There were also symptoms of hyperarousal, presenting as sleep disturbance and bodily sensations of anxiety. However, the interviewers asked open ended questions regarding possible impacts. Specific questions that could be used to assess or diagnose different disorders were not utilised. More research is needed to investigate whether CMs reach the threshold for any potential diagnoses, to what extent symptoms reflect one or many different potential syndromes such as burnout, PTSD or VT and to estimate their potential prevalence in this population.

Nevertheless, this study suggests the psychological impacts experienced by commercial CMs are comparable to those experienced by professionals such as first responders and police tasked with analysing child exploitation material (Greinacher et al., 2019; Bourke & Craun, 2014). In these professions, these risks are more widely recognised, for the most part, contributing to support being implemented and facilitating more understanding from managers and the community (Burns et al., 2008; Krause, 2009). In contrast, despite their centrality to the business model of many online platforms (Barrett, 2020) moderators remain an undervalued, often hidden profession (Gillespie, 2018), frequently located in the global south and commonly not afforded the same benefits and provisions as offered to other professionals (Roberts, 2019; Jereza, 2021). However, the increasing amount of literature demonstrating the potential negative consequences of this work (e.g., Benjelloun & Otheman, 2020; Roberts, 2019) illustrates that companies need to take more responsibility for the welfare of these employees and volunteers and provide better protection against the possible deleterious psychological effects.

Understanding the effects this work has on CMs from a psychological perspective allows the symptoms to be positioned in a framework that can draw on the large psychological literature around trauma and its sequalae. It can link the experiences and emotional responses direct from the voices of CMs to clinical practice and empirically ground potential recommendations. This includes creating interventions based on “what works” and for whom, as well as better designing systems to protect CMs. Many organisations provide technological solutions such as blurring to minimise the impact (Das et al., 2020). However, these are not necessarily considered useful by moderators (Spence et al., 2023) and a lack of support has been associated with quitting in volunteer moderators (Schöpke-Gonzalez et al., 2022). Nevertheless, there are potentially processes which could be put in place to alleviate emotional distress, for instance playing Tetris after trauma exposure has shown some early promise in reducing the occurrence of intrusive memories (Iyadurai et al., 2018). Regardless, these results suggest that companies should provide trauma-informed care at work, which is guided by the principles of safety, trustworthiness, peer support collaboration and mutuality, empowerment, and cultural, historical and gender issues (Substance Abuse and Mental Health Services Administration, 2014) and is shown to reduce STS and burnout amongst staff (Handran, 2015; Schepers & Young, 2022).

Table 2. Themes Rearranged Into PTSD Symptom Domains.

Although trauma responses such as intrusive thoughts might be disturbing, what may be more insidious is the slow change in perceptions of the world and those in our communities, which could be more pervasive and affect all spheres of the CMs life. Research with volunteer moderators found they experience cognitive changes like increased apathy and guilt (Dosono & Semaan, 2019; Lo, 2018). In this study, commercial CMs described having a darker, more cynical view of the world and other people, and these views may not change once they leave their job (O’Malley et al., 2019). Although the specific ages of the CMs were unknown, they tended to be young and, generally, most CMs are younger than thirty (Mukhopadhyay, 2020). Although the evidence is inconsistent, younger age has been associated with greater distress after experiencing trauma and more symptoms (Acierno et al., 2006; Adams & Boscarino, 2006), as well as higher levels of burnout (Brady, 2017). This may be because when compared to older workers, they lack a wide array of life experiences to draw on, making them less equipped to deal with stressful situations (Lerias & Byrne, 2003). Older workers are also more likely to use positive emotion regulation strategies and possess higher emotional competencies to stave off negative consequences (Steiger, 2020). Therefore because of their age, CMs may be a group that are particularly vulnerable to repeatedly working with traumatic material.

There is currently limited research that investigates the psychological impacts of the role. However, there is some evidence that suggests the harmful impacts of moderation are not necessarily restricted to those viewing overtly disturbing material such as CSAM. In work by Jereza (2022, p. 12) a moderator commented “working with hate speech so much is it would get into my head. Like these images and words would flash across my brain like with no warning, and I didn’t want them there. It was—it was awful. I was like, I would kind of torturing myself, um, and so I do think, you know, it can get into your head easier than you would think perhaps? Or when I was out and bout like if I ever saw a white guy with a tattoo, I would immediately start scanning his body for hate symbols. And every white guy has a tattoo in [City] . . . I was like hating myself for it. They’re not supposed to be there. I’m not this person” (emphasis added). This suggests that intrusive thoughts and hypervigilance may also be associated with working with hate speech. This content is the most prevalent type that CMs have to analyse on social media platforms (Facebook, 2022a; Twitter, 2021) therefore, research into the effects of working with this content specifically is needed. In particular, if it is associated with VT, where the world views of moderators may be changed, possibly even in agreement with what they being exposed to. Indeed, it is possible that many of the same impacts might be present across different content types but the substance of the impacts could vary in line with the material the CM is exposed to.

Limitations

The current study was a cross-sectional qualitative design therefore was not able to capture changes over time. Additionally, time in post was not reported, therefore it cannot be ascertained if certain impacts were more likely to be described by those earlier or later in their career as a CM. Similarly, to encourage participation CMs were not asked about details of their role such as if they were employed directly by the company or were outsourced labour, although all participants were in paid positions and were able to conduct an interview in English. There are numerous volunteer CMs used by sites like Reddit (Caplan, 2018) and moderation is a global phenomenon. Thus, this study was unable to fully capture the full range of different work environments CMs will be exposed to, not only in terms of support provided but also working conditions, which are likely impact CM resilience (Burns et al., 2008; Substance Abuse and Mental Health Services Administration, 2014). However, this study opens up future avenues for research, including larger scale studies that can quantify potential impacts and explore potential factors that increase CM resilience.

Conclusions

Many of the CMs described a range of symptoms that were associated with experiencing trauma at work, including intrusive thoughts, avoidance and hypervigilance around children. This suggests it is crucial for companies to provide psychoeducation, intervention and trauma informed care. Of particular concern were the cognitive biases that highlighted a cynical and suspicious framework as these may be harder for CMs to recognise and be resistant to change, even after leaving the role. Larger scale quantitative and longitudinal research are both needed to quantify the possible effects and their long-term consequences.

Conflict of Interest

The authors have no conflict of interest to declare.

Acknowledgement

This work was conducted with the help of Amy Harrison.