Texting and the brain: The time-course of social brain activation in face-to-face versus text-based computer-mediated-communication

Introduction

The advancement of technology has created new ways for people to communicate. A paradigm shift is occurring whereby the usage of text-based computer-mediated communication (CMC) is now among the most frequent ways to communicate with others, and even surpasses face-to-face (FtF) communication in some reports (Aaron, 2015; OFCOM, 2020). Here, we use text-based CMC to refer to any form of text-based instant messaging communication. This shift toward text-based CMC raises a number of interesting questions. For instance, to what extent are our brains equipped to process text-based CMC exchanges? From an evolutionary perspective, anthropoid primates’ brains have evolved primarily to deal with the interactions arising in social groups, with FtF communication the main tool to create and maintain bonds between members of the community (Dunbar, 1998). Likewise, the development of interpersonal social skills is adapted for FtF interaction (Kock, 2004, 2005) providing the foundation for our development and attunement of social cognitive skills. Building on such proposals, the current study addresses the fundamental question of how neural activation differs between FtF and text-based CMC during a conversation. To our knowledge, this is the first time that this foundational question has been empirically addressed. Here, we synthesise a model of communication (“media naturalness theory”; Kock, 2004, 2005, 2011) with neuro-imaging data from key structures involved in social cognition. Our rationale is that this approach allows us to explore how different components of social cognition might be affected by communication medium, using a tool (functional near-infrared spectroscopy; fNIRS), that can gather data over the time-course of a conversation without interrupting the interaction. Understanding differences in neural activation as a conversation unfolds may allow us to generate questions and hypotheses about the key challenges and limitations involved in social exchanges that take place via text-based CMC.

Comparing Face-to-Face and Computer Mediated Communication

CMC in the broader sense allows for multiple modalities of communication methods with a variety of features, including instant text messaging, audio messages and video calls. Each mode permits the transmission of information in a different manner. Text-based CMC lacks vocal and facial signals and is therefore believed to be a “leaner” communication method than FtF due to the lack of social cues offered during the communication act (Daft & Lengel, 1986). For instance, the lack of facial expression, tone of voice, and synchronicity tend to diminish the quality of the experience and even the effectiveness of communication (Mehrabian & Ferris, 1967; Walther, 2011). Media naturalness theory (Kock, 2011) further proposes that human communication systems have been shaped by evolutionary forces to communicate with other individuals in a co-located environment using facial expressions, body language, and speech synchronously (Kock, 2011). Mediums that afford co-location, synchronicity, the ability to convey facial expressions, body language, and speech will provide a higher level of naturalness than a medium which does not support those features. A medium that comprises one or more features to a larger degree than another will have a higher degree of naturalness. Kock (2011) argues that reduced media naturalness increases both the ambiguity and cognitive effort involved in interaction. Specifically, removing social signals, or altering the communication environment in ways that are unnatural for communicative processing, would lead to a greater cognitive load (C. D. Frith, 2008; Kock, 2011; Zaki, 2013). In addition, FtF communication is predicted to elicit more physiological arousal due to the presence of evoking stimuli such as tone of voice and facial expression (Kock, 2011). It is therefore plausible that such effects will be evident in patterns of neural activity between communication channels that differ in media naturalness (FtF vs Text based CMC). Therefore, looking into the neural structures involved in social cognitive processes is of particular relevance to understand how individuals process these communication exchanges.

Social Cognition

Social cognition refers to the multiple psychological processes that provide an individual with the information and skills necessary to process and manage information from actors within the social environment (C. D. Frith, 2008). One of the main cognitive tasks facing a social actor is the correct reading, processing, and manipulation of social cues. Information from social cues provides a fundamental way to obtain social knowledge from a context and from an individual. These cues include facial expressions, vocal tone, gestures, and body language. Their function is to clarify people’s meanings and intentions (Zaki, 2013). Facial expressions are formed from complex muscle arrangements and provide information about emotional states, some of which are considered to be universal (Centelles et al., 2013; Ekman, 1993; Vuilleumier & Pourtois, 2007). The information conveyed by the expression of human faces seems to be attuned to facilitate understanding. Gestures and movements with the hands, arms or even head can help to communicate a message further. These help not only the sender to communicate the message but also the receiver (Langton & Bruce, 2000). A study looking into the processing of social cues and speech accompanied by gestures suggested that non-verbal social cues may play an important role in language comprehension during FtF interactions (Straube et al., 2010). When we see someone turn their gaze, we can infer where someone focuses their attention, which can help to synchronize attention between social actors. Watching someone’s body movements and facial expressions can help us to build inferences of what that person is thinking, feeling and what they may intend to do (C. D. Frith & Frith, 2006). Referred to as “Mentalising”, or theory of mind (ToM) ability, these processes set the basis for multiple social-emotional responses such as empathy, cooperation, reciprocity, and deception. Being able to use these skills adequately can help to build cohesion within a social environment. This process of mentalisation also allows us to predict the behavior of others within a particular context (U. Frith & Frith, 2003; Premack & Woodruff, 1978). When removing salient social cognitive signals from the communication act, as in text-based CMC, there is increased scope for ambiguity, a likely heightened cognitive load, and reduced emotional information during communication.

The Neuroscience of Social Cognition

Since the infamous case of railway worker, Phineas Gage, it has become increasingly clear that the frontal lobes are crucially connected to social conduct and in detecting and interpreting stimuli of social and emotional significance (e.g., Adolphs, 1999; Eslinger & Damasio, 1985). It is now understood that multiple systems and sub-structures are involved in the processing of social information (Adolphs, 2003; Cacioppo et al., 2007; C. D. Frith & Frith, 2007; Lieberman, 2007). This study is focused on two neural structures widely associated with social cognitive processes, the dorsomedial prefrontal cortex (dmPFC) and the ventromedial prefrontal cortex (vmPFC; Adolphs, 2009; Forbes & Grafman, 2010).

The dmPFC is located at the front of the frontal lobes, above the orbitofrontal cortex and medial frontal cortex. This part of the social brain has been mainly associated with self-knowledge, person knowledge, and mentalising (Amodio & Frith, 2006). Previous studies have shown that, when a series of adjectives were displayed before names of people or inanimate objects, the dmPFC displayed activity when the adjective was shown before the names of people (Lieberman & Pfeifer, 2005; Mitchell et al., 2002). Multiple studies utilizing functional magnetic resonance imaging (fMRI) have reported that dmPFC is a central component in the mentalizing network (Amodio & Frith, 2006; Gallagher & Frith, 2003). In addition, clinical studies have also shown that patients with dmPFC lesions can present severe ToM impairments (Happé et al., 1999; Rowe et al., 2001; Stuss et al., 2001).

The ventromedial prefrontal cortex (vmPFC) is located along the inferior sub-region of the medial prefrontal wall of the frontal lobes, roughly above the eyes. Deficits in this area have been reported to lead to major interpersonal problems and a deficit of empathy (Eslinger & Damasio 1985; Shamay-Tsoory et al., 2003). The vmPFC has been identified as a key mechanism for human social and affective functions (Fellows, 2011; Quirk & Beer, 2006) and observed to play a critical role in emotional facial recognition (Heberlein et al., 2008). Patients with damage in the vmPFC have shown deficiencies interpreting non-verbal social information such as facial expressions, gestures, or body language even though they maintain their understanding and knowledge of social norms and moral norms (Adolphs, 2009).

When speculating on the contrasting functions of the dmPFC and vmPFC, there are somewhat different emphases in different accounts. For instance, Arioli et al. (2021) have considered the dmPFC to have more involvement in cognitive aspects of mentalising and the vmPFC to have more involvement in affective aspects of mentalising. Other authors emphasise a role of the dmPFC in top-down control of self-other information, whilst the vmPFC is more involved in processing of bottom-up sensory information. For instance, Bzdok et al. (2013) proposed that the dmPFC is involved in more complex, top-down, meta-cognitive social processes, including perspective taking and episodic memory retrieval, which are independent of sensory input. The authors suggest that the vmPFC, in contrast, is involved in bottom-up processing associated with affective and motivational information of both the self and the communication partner.

Synthesising these proposals with media naturalness theory (Kock, 2011), we might speculate that the dmPFC is more impacted by the cognitive effort imposed by communication activities whereby lean communication channels, such as text-based CMC, involve increased ambiguity. Cognitive effort might therefore be introduced when it is hard to adopt the perspective of the communication partner and we would therefore expect increasing dmPFC involvement for mediums which challenge these activities. In contrast, the vmPFC would be more implicated in monitoring ongoing emotional information generated from the exchange and we would therefore predict more involvement of the vmPFC in the FtF condition due to the availability of more salient bottom-up signals conferring emotional state.

Within this view, the dmPFC and vmPFC are seen to be continuously processing cognitive and emotional information from the communication environment, integrating bottom-up sensory information with prior knowledge and information. Galotti et al. (2017) emphasise that reciprocal interactions, involving the mutual exchange of information, unfold over time, and argue for the value of modelling the time-course of neural activity of interacting partners. Therefore, an additional element of our study seeks to capture the time-course of neural activation during CMC and FtF conversation. We divided the conversation into smaller time windows allowing us to measure the dynamic temporal variation in neural response across the duration of the interaction. Averaging the neural activity across time would remove useful information about the time-course of these neural processes.

Various mechanisms have been evoked to describe the dynamic nature of information exchange during social interaction. Brain-to-brain coupling, or interpersonal neural synchronisation, refers to an emergent process that occurs when the listener’s brain becomes tuned to the visual and auditory information provided by the speaker (Hasson et al., 2012) and has been suggested as an indicator for successful communication (Stephens et al., 2010). Similarly, “interactive alignment” (Garrord & Pickering, 2004) occurs as conversational partners begin to mirror each other’s linguistic phrases and grammatical structures which helps to converge on a shared mental model of the topic. This is seen to provide a cognitive benefit through development of a shared “situation model” which supports mutual understanding and resolves conversational ambiguity. In the current study, we might expect that the neural activity within the dmPFC would increase over time as social signals are integrated by the conversational partners to perform mentalising tasks and generate mutual understanding. If the dmPFC is more implicated in top-down and metacognitive processing (Bzdok et al., 2013), we might expect the differential availability of bottom-up sensory information to have no impact on time-course of activation. In contrast, should interactional alignment occur across the conversation which in turn assists in resolving conversational ambiguity, we might see a reduction in activity based on reduced cognitive effort. We therefore took an exploratory approach to the question of the time-course of dmPFC activation. Rather, our key interest was how time-course may differ between CMC and FtF conditions.

Other authors have implicated forms of conversational synchrony, such as synchronising the length of turn-taking, in social bonding processes (Gordon et al., 2014) and social closeness and connection (Kinreich et al., 2017). Synchrony can also occur in non-verbal aspects of communication, such as body posture and movements, which has been implicated in increased liking and empathy (Chartrand & Bargh, 1999). These studies focus on the availability of bottom-up sensory and affective processing and may therefore suggest a role for vmPFC activity in forms on interactional synchrony. Indeed, Yun et al. (2012) found temporal alignment in finger tapping behaviours, and this synchronisation correlated with vmPFC activity. Gordon et al. (2014) compared the number of words per conversational turn at the beginning and end of a natural conversation, finding convergence in conversational turn length between non-brain damaged individuals, but not in patients with vmPFC damage. We therefore speculate that neural activity in regions associated with vmPFC would increase as additional bottom-up sensory and affective information becomes available to support communication, and this increase will be greater according to the relatively richer medium of FtF interaction. The CMC condition should demonstrate a relatively smaller effect of time-course since the main cues involved in social emotional processing are absent.

Near Infrared Spectroscopy for Social Interaction

This study employed functional near-infrared spectroscopy (fNIRS), which has been recognized as a suitable tool for recording frontal neural activity during real-time social interaction (Burns & Lieberman, 2019; Pinti et al., 2020). One benefit of fNIRS is that the neural activity associated with an ongoing interaction can be recorded as it unfolds, which allows analysis without interrupting the interaction with self-report measures. Like fMRI, fNIRS relies on the BOLD to collect neuroactivity—the blood oxygen level-dependent signal that occurs when localized populations of neurons fire, consume oxygen, and therefore require more oxygen to be pumped to the area to continue operating. Both fMRI and fNIRS can detect changes in hemoglobin concentrations caused by neuronal firing, due to physical differences in the oxygenated and deoxygenated hemoglobin molecules (HbO and Hb, respectively). fNIRS allows indoor and outdoor scenarios, and its portability features, safe setups, and tolerability to environmental noise and body movements make it a versatile tool for social interaction conditions. Importantly, fNIRS allows participants to engage in FtF communication that is not constrained by the same requirements as fMRI scanning technology. fNIRS, compared to fMRI, has increased sensitivity to temporal factors, allowing exploration of changes in activation over time. This versatility increases the ecological viability for experiments when compared to fMRI, creating a more naturalistic scenario (Pinti et al., 2020). As such it has been employed to explore neural activation in several studies. For instance, Jiang et al. (2012) found neural synchronization during FtF communication but not during back-to-back communication (even though still listening to each other). Tang et al. (2019) used fNRIS to look into behavioral and neural differences in cortical regions in a deception game task. However, whilst these have demonstrated the ability of fNIRS to detect patterns of activation in social cognitive neural structures, to our knowledge no studies have investigated differences between FtF and text-based CMC.

The Current Study

The current study aimed to simulate a human communication setup that could happen during an unconstrained conversation. Our main questions concerned (RQ1) what difference in neural activation exist between regions of the PFC associated with social cognition (vmPFC and dmPFC) when humans communicate FtF or via text-based CMC, and (RQ2) how activation may change over time as the conversation develops.

Methods

Design

A repeated measures design was employed with two conditions: Face to Face (FtF) and Computer Mediated Communication (CMC). Participants were asked to engage in 10-minute conversations within each condition. During the interaction, for one of the participants in each pair, the brain activity in the dmPFC and vmPFC was measured in both hemispheres. The principle of the design was to compare neural activity in these key brain areas involved in social cognition between the FtF and CMC conditions. The conversation conditions were counterbalanced. In addition, we were interested in how those brain differences might evolve during the process of the conversation. Analysis of the data was therefore based on a 2 [communication condition: FtF, CMC] * 4 [time quartile: (Q1, Q2, Q3, Q4)] repeated measures design for each brain region (dmPFC and vmPFC) separately.

Participants

A total of 20 pairs of participants were recruited from undergraduate and postgraduate programs at the University of Salford, UK. The pairs consisted of 12 males and 28 females, aged between 18 and 40 years old (M_age = 27.2 years old, SD = 5.8). Three pairs of participants were discarded from the analysis due to technical issues encountered when collecting the data, leaving 17 pairs of data. The pairs varied in terms of their prior familiarity with each other but this was not controlled in the current study. All of the pairs were at least acquaintances from the same University course, except for 1 pair of participants who did not know each other. Written consent was acquired from both participants, once they read the Participant Information sheet and the procedural information about the experiment that was provided. Ethical approval was acquired from the College of Health and Social Care Ethics Panel at the University of Salford. Participants were paid £7 in compensation for their time. Data was collected between August 2017 and June 2019.

Materials

In order to measure the neural activity (oxy-Hb) on the selected regions of the prefrontal cortex, Biopac fNIRS100B multichannel system (Figure 1) was utilized to collect the data from one of the participants from each pair. The system provides a temporal resolution of 500ms (2Hz) and is formed of a headband comprising 4 light sources and 10 light detectors resulting in 16 measurement channels. The probe was placed on the forehead and aligned with the frontal polar (Fp) 1 and Fp2 of the international 10–20 system (Jasper, 1958). Figure 3 shows the approximate location of optodes on the fNIRS headband (©BIOPAC Systems, Inc). Only relevant optodes were utilized for the data analysis, with regions of interest named according to anatomical congruence. Optodes 5, 7, 9, & 11 were ascribed to the dmPFC, and optodes 6, 8, 10, & 12 were ascribed to the vmPFC.

Figure 1. The fNRIS Biopac fNIR100B Model.

Note. This figure shows the hardware utilised to collect the level of HbO on the PFC of the subject. The box was connected to a computer loaded with Cognitive Optical Brain Imaging studio (COBI) software. Photo Courtesy of ©BIOPAC Systems, Inc.

Cognitive Optical Brain Imaging studio (COBI) software was used to record data from the fNIRS device. “fNIRS-Soft” software was utilized for the post-process of the data recorded via COBI Studio software. Additionally, for the CMC condition, one iPhone5s with “WhatsApp” (Room A) and one laptop with the WhatsApp application (Room B) were utilized for the computer mediated communication task. During the process a video camera recorded the participant wearing the fNIRS band in both conditions, screen capture was used to record the “WhatsApp chat” conversation of the participants. These recordings were intended to monitor the participants’ behavior throughout the experiment in case it was needed for later data analysis purposes.

Procedure

Once the participants arrived, the participant information sheet and consent form were handed out and returned completed. The participant information sheet requested that the participants provide demographic data and some basic information about their daily usage of CMC, to make sure that all participants were fairly familiar with the communication method employed in the experiment. Participants were seated approximately 1.5 m away from each other. A coin was flipped to choose who would be wearing the fNIRS band (Participant A) to collect her/his neural activity (oxy-Hb).

The participants were required to talk about a standard pre-selected topic. This topic was “jobs automation and whether they thought their job would be replaced by a machine”. This topic was proposed, since it could potentially allow any student to provide a personal opinion and to develop their own views, enabling both partners to engage in what we could define as “regular conversation”.

Once the participants were informed of the conversation topic, the band with the sensors was placed on the forehead of “participant A” and secured with a Velcro elastic band. The 16-channel probe was placed on the forehead and the center line of the sensor band was aligned visually to roughly the fp1-fp2 line of the international 10–20 system. Software was initialized and then the baseline was acquired. This required the participant to relax and empty their mind during the time device and software acquire the corresponding information, this took around 30sec to be completed. To help with the process it was suggested to look at the white wall. After the baseline process was completed, the experiment began. In the FtF condition, both participants were left uninterrupted to develop the conversation until the time was up. They were given a 5-minute break and the fNIRS’ sensors were removed from the participant’s forehead, for comfort.

In the CMC condition, Participant B was placed into another room next door (Room B). A laptop was set up with WhatsApp to enable to them to carry on the conversation with Participant A in room A. Additionally, in room B, an A4 sheet was provided with a few cues and questions to assist and keep the chat rolling, in case the conversation came to a halt due to lack of ideas or contribution. This was proposed and explained to Participant B and only Participant B in room B was aware of this additional aid. Participant A was never aware of the set-up in the room B. This set-up in room B was meant to facilitate and aid the condition and no neural data was collected from Participant B. Although we did not formally code the extent to which conversation was facilitated by the prompts, the researcher did not observe any lack of conversational activity or interaction while Participant A was engaging in the communication act and CMC conditions unfolded without any discontinuance. After another ten minutes of text-based communication the experiment was concluded, and the data collection finalized. The order of the conditions was counterbalanced.

Figure 2. Flexible Sensor Band Fit on Subject’s Forehead.

Note. This image shows; left side: Participant A during the CMC condition. Right side: Participant A during the FtF communication.

Data Analysis

To analyze the data provided by the fNIRS sensors, we looked at the changes in HbO.

The first step required converting the raw fNIRS data delivered by the (COBI) Studio software (Biopac System Inc.). fNIRS data pre-processing was carried out in fNIR Soft. Raw light intensity data was smoothed with a low-pass FIR filter with an order of 20, before sliding window motion artefact rejection (SMAR) was applied (see Ayaz, 2010; Ayaz et al., 2010) which involved the automatic identification and rejection of motion artefacts. This process also helped with potential noise and interferences introduced by the hair, helping to refine the data. Changes in the concentration of haemoglobin (from each baseline to the associated experimental condition) were then calculated, for both face-to-face and computer-mediated communication, using the modified Beer-Lambert Law. We focused on HbO following arguments that changes in HbO are the most reliable indicators of cerebral blood flow (and consequently, neural activity; Malonek et al., 1997; Strangman et al., 2002), and is the primary focus in many areas of cognitive neuroscience (e.g., Bendall et al., 2016). The HbO measure is expressed in arbitrary units (a. u.) meaning it is a relative measure of the strength of activation rather than an absolute unit of concentration. This is a common approach employed in fNIRS research (see Herold et al., 2018, for a review). Average HbO changes from the baseline were calculated for three time periods. Once this process was completed the final resulting information was organized for analysis in SPSS.

Since the vmPFC and dmPFC have been implicated in social cognition (Adolphs, 2003, 2009) analyses were focused on these regions of interest. Four central channels were selected to represent each of the vmPFC and dmPFC areas, located below and above the midline of the forehead sensor respectively (see Figure 3). The rest of the data from other optodes was discarded.

Figure 3. Channels Area Where the fNIRS Sensor Records HbO Measures.

Note. This figure displays the fNIRS probe location, showing the approximate area where the 16 optodes are positioned and the 8 optodes that we will be focusing on this research. Brain Photo Courtesy of fNIRSoft®.

For the ten minutes of sample collection for each condition, we separated four equal segments of duration for each optode to assess how HbO signals develop over the course of the interaction. For each quartile of the conversation (Q1, Q2, Q3, Q4), we took 300 samples of data which is comparable to 2 minutes 30 seconds of conversation, making a total of 10 minutes. We calculated the mean for each one of the time-blocks, for each optode and condition. This resulted in eight optodes (4 dmPFC, 4 vmPFC) times four blocks of data. The results from those calculations were fed onto SPSS for the analysis.

Results

Participant A exchanged an average of 26.2 messages (max = 45, min = 18, SD = 12.3) and Participant B exchanged an average of 25.8 messages (SD = 11.3). Emoticons were used very infrequently within the text-based CMC and by some participants not at all, therefore messages were almost exclusively text-based and were therefore not delineated in analysis by the presence of emoticons. As we were not making predictions regarding brain hemisphere, the data from optodes for corresponding brain regions was collapsed across hemisphere. Table 1, below, shows means and SDs across conditions. Figure 4, below, shows overall differences between conditions in both brain areas.

Table 1. Mean HbO Levels by Brain Region, Communication Condition, and Time Quartile.

Figure 4. Mean Activity Measures for HbO Levels by Condition and Across Brain Region.

Note. The mean activity for both brain regions across conditions (* = p < .05). The scaling of HbO parameters is in arbitrary units. Error bars: + / − 1 standard error of the mean (SEM).

Analyses were run separately on the dmPFC and vmPFC by conducting a 2 (communication condition: CMC vs FtF) * 4 (time quartile: Q1, Q2, Q3, Q4) repeated measures ANOVA on each brain region. Greenhouse-Geisser corrections (GG-corr) are provided when Mauchly’s test of revealed violations of sphericity.

dmPFC Analysis

Shapiro Wilkes tests revealed that data for the dmPFC analysis violated the assumption of normality of the residuals. This was driven by a single participant whose means in the four levels of the time quartile variable in the FtF condition were between 2.78 and 3.36 SDs from the condition means. Removal of this participant meant that data met the assumptions of normality of residuals (all Shapiro-Wilkes tests revealed p values >.05) and the data analysis reported below is therefore based on the 16 remaining participants (although it should be noted that the pattern of effects was the same under analysis of the full data set). Analysis of the data from dmPFC revealed a significant effect of condition, F(1, 15) = 7.110, MSE = 2.054, p = .018, η_p² = .322, due to greater activation during the CMC condition (see Figure 4, above). No interactions were observed between condition or time (see Figure 5), but a significant main effect of time was observed, F(1.377, 20.660) = 8.776, MSE = 0.691, p = .004, η_p² = .369; GG-cor. Analysis of contrasts for the time variable revealed a significant linear trend, F(1, 15) = 10.262, MSE = 0.767, p = .006, η_p² = .406, indicating increasing neural activity throughout the conversation (see Figure 6). However, the quadratic trend also approached significance, F(1, 15) = 4.495, MSE = 0.106, p = .051, η_p² = .231. Analysis of the time variable alone (collapsed across communication conditions) suggests the near quadratic effect was due to smaller increases in activity at successive quartiles. Bonferroni pairwise comparisons revealed differences between quartile 1 and both quartiles 3 and 4 (p = .039 and p = .034 respectively), with no other significant differences between quartiles.

Figure 5. Mean HbO Levels in Dorsomedial Prefrontal Cortex by Condition and Across Time.

Note. This figure shows the changes in HbO across time in areas associated with dmPFC. CMC is consistently higher than FtF. The scaling of HbO parameters is in arbitrary units. Error bars: + / − 1 SEM.

Figure 6. Mean HbO Levels in Dorsomedial Prefrontal Cortex Across Time.

Note. This figure shows the main effect of changes in HbO across time in areas associated with dmPFC, collapsed across CMC and FtF conditions. The scaling of HbO parameters is in arbitrary units. Error bars: + / − 1 SEM. Significant pairwise comparisons (Bonferroni) are indicated by * (p < .05).

vmPFC Analysis

The second ANOVA assessed activation in the vmPFC (optodes 6–8–10–12). The data met the assumptions of normality of residuals in the analysis of the vmPFC (Shapiro-Wilkes revealed all p values > .05), and therefore analysis was conducted on the full data set. Analysis revealed no main effect of condition, F(1, 16) = 0.097, MSE = 8.407, p = .760, η_p² = .006. As before, a significant effect of time was found, F(1.272, 20.344) = 8.225, MSE = 1.475, p = .006, η_p² = .340; GG-cor, which also interacted with condition, F(1.952, 31.236) = 4.863, MSE = 0.439, p = .015, η_p² = .233; GG-cor. To further investigate this interaction, presented in Figure 7, ANOVAs were conducted on the time variable for the FtF and CMC conditions separately. A significant effect of time was observed in the FtF condition, F(1.25, 19.997) = 9.742, MSE = 1.457, p = .003, η_p² = .378; GG-cor, and contrast analysis revealed that the effect most resembled a linear trend, F(1, 16) = 10.779, MSE = 1.605, p = .005, η_p² = .403, with greater activity at each successive time quartile. However, a significant quadratic trend was also found, F(1, 16) = 4.723, MSE = 0.094, p = .045, η_p² = .228, capturing the fact that the amount of increase levelled off statistically following from quartile 2. Bonferroni post-hoc tests revealed significant differences between quartile 1 and each successive quartile (Q1 vs Q2 = .002; Q1 vs Q3 = .005; Q1 vs Q4 = .028), but no other significant differences were found. In contrast to the FtF condition, no effect of time was found in the CMC condition, F(1.795, 28.716) = 2.03, MSE = 0.507, p = .154, η_p² = .113; GG-cor. Neither the linear (p = .100) or quadratic contrasts (p = .980) were significant for the time variable in the CMC condition, and Bonferroni pairwise tests revealed no differences between any quartiles. However, lack of an effect in this condition should be interpreted with extreme caution as the observed power for this test was low (.364) demonstrating a high chance of obtaining effects over time with a larger sample.

Figure 7. Mean HbO Levels in the Ventro-Medial Prefrontal Cortex by Condition and Across Time.

Note. This figure shows the evolution on the area associated vmPFC activity across time. Levels of HbO grew across the experiment during FtF, but not during CMC which showed a non-significant trend. The scaling of HbO parameters is in arbitrary units. Error bars: + / − 1 SEM. Significant pairwise comparisons (Bonferroni) existed only in the FtF condition and are indicated by * (p < .05).

Discussion

The current study examined neural activity in vmPFC and dmPFC during face-to-face communication compared to text-based computer mediated communication. The initial predictions presented for the study proposed that, since FtF was a much richer communication system and most of the social cues developed through evolution for social interaction are removed during CMC, the brain regions involved in social cognition would respond differently to the different mediums of text-based CMC and FtF communication. Specifically, the dMPFC was expected to show greater activity in the CMC condition, in line with the additional cognitive effort involved in disambiguating the message and taking the perspective of the conversation partner. The vmPFC was expected to show greater activity in the FtF condition due to the greater availability of emotional information. Two key results emerged from our research. First, in line with our expectations, greater activation was observed during CMC communication within the dmPFC. Second, and in contrast to activation in the dmPFC, communication medium did not affect overall levels of activation in the vmPFC. However, an interaction was observed between condition and time, whereby a significant linear increase in activation was observed for FtF but not the CMC condition.

The first finding showed greater dmPFC activation for CMC which did not interact with time. One interpretation of the increase in dmPFC activation during texting is that more cognitive resources were recruited during text-based CMC than during FtF. If this interpretation is correct, it indicates that the action of text-based chatting via smartphone generates more cognitive effort on the dmPFC. One of the constructs of media naturalness theory proposes that removing elements from communication that are considered “evolutionary intrinsic” would create a higher cognitive burden on the brain. This burden would generate a more unnatural communication and consequently require extra effort, suggesting an increase in neural activation (Kock, 2011). This theory can help to explain some of the results in the present study. As media naturalness theory proposes, some communication situations may require more resources (Kock, 2005). dmPFC is associated with performing tasks concerning theory of mind, self-knowledge, and person knowledge (Amodio & Frith, 2006). Therefore, one interpretation for the results obtained could be that the removal of social cues creates an additional challenge to build inferences about the other person’s state of mind based only on the limited social information being broadcasted. This suggests an additional effort must be made by the brain in order to run internal processes (Kock, 2005).

A similar perspective evoking the concept of cognitive load highlights differences in the ability to process information about communication partners. Social cognition theories have proposed that two processes can happen during social interaction: automatic and controlled (Adolphs, 2009; C. D. Frith & Frith, 2007; Lieberman, 2007), with controlled processes linked to dmPFC activity (Lieberman, 2007). Automatic processes are considered to be faster and reflexive, developed early in evolution and shared with other animals (e.g., interpreting emotional facial expression, body language), and requiring a lower cognitive load than social cognitive controlled processes. In contrast, controlled processes are associated with conscious awareness, and slower, effortful processing. Explicit thoughts about someone else’s mind have also been associated within the domain of controlled processed (Lieberman, 2007). We believe that text-based CMC is best characterised as a more controlled process, requiring increased conscious reflection to support the interpretation of the mental states of a communication partner.

The parallel dmPFC increase in activation across time in both the CMC and FtF conditions provide no evidence that cognitive effort was reduced over the course of the conversation, or differentially according to medium. Rather, our data are most consistent with an interpretation that the dynamic involvement of the dmPFC involves the ongoing coordination of top-down and cognitive aspects of mentalising, which appears to unfold in a similar way irrespective of medium. This could suggest that the time-course effects observed in both conditions reflect top-down, meta-cognitive social processes that are independent of sensory input (Bzdok et al., 2013).

We expected that the vmPFC would show increased activity in the FtF condition due to the evidence of vmPFC involvement in processing bottom-up emotional input during social interaction (Arioli et al., 2021; Bzdok et al., 2013). However, the measures obtained from the vmPFC did not reflect any global difference across conditions, although differences were observed when assessing the time-course of activation. Significant increases in activation were identified across time, but only for FtF communication. However, we offer a tentative interpretation of that result here due to very low power in the analysis of the CMC effect (.364). The results indicate that activity in the vmPFC was initially lower during the first quartile of the conversation via FtF, but then increased, eventually surpassing CMC activity (Figure 7). The vmPFC has been associated with social rewards and emotional assignation to social stimuli (Fellows, 2011; Koenigs et al., 2007; Quirk & Beer, 2006). One interpretation of our results is therefore that the availability of bottom-up socio-emotional stimuli increases as a FtF conversation unfolds, reflected in increasing activity in the vmPFC. It is important to also consider whether such information may also become increasingly available in text-based CMC, albeit at a lesser rate, as indicated by a non-significant (and underpowered) trend in vmPFC activation in this condition. A similar interpretation could be that connectedness between participants can increase as the time of FtF interaction increases. This is consistent with a study where bonding has been associated with the social reward system managed by vmPFC, amygdala and other regions of the medial frontal cortex (Atzil et al., 2017). Indeed, a study measuring bonding during FtF and CMC conditions reported that the FtF condition showed the highest rated level of bonding and text messaging the lowest (Sherman et al., 2013).

In the CMC condition vmPFC measures indicated that the level of activation did not show a significant time-course effect, displaying less increase over time than the FtF condition (Figure 7). The lack of a significant linear trend in the CMC condition, whilst being underpowered, could tentatively be taken to suggest that conversational responses and social cognitive processing of the conversation was similar at the beginning and the end of the task. This is consistent with an interpretation suggesting that the social signals in the environment remained relatively unchanged during the 10 minutes the task lasted. We are therefore directed to speculate whether the relatively steeper increase in activation in the FtF condition indicates the participants gradually attuning to the greater number of communication cues provided by the conversational partner. Potentially relevant here are the phenomena of conversational synchrony, whereby conversational partners begin to adopt similar verbal and non-verbal behaviours, and conversational adaptation, whereby communication is adapted to the particular needs of a conversation partner. Both have been implicated by damage to the vmPFC. For example, Gordon et al. (2014) compared the conversational patterns of patients with damage to the vmPFC to healthy control participants. They found that the controls began to synchronise the length of their conversational turns whilst this effect was absent in those with damaged vmPFC. Stolk et al. (2015) found that patients with vmPFC lesions failed to dynamically adjust their communication to the presumed abilities of a (hidden) conversational partner who they believed to be either a child or an adult. This finding suggests the importance of the vmPFC for using knowledge of the conversational partner for adaption of communication style. It is reasonable to question whether differences in the time-course in vmPFC activity between text-based CMC and FTF conversational conditions reflect the differential access to social cues which underpin dynamic aspects of communication adjustment such as synchrony and adaptation.

Conclusions, Limitations & Further Research

The current study marks a first attempt to understand neural differences in communication depending on FtF and text-based CMC. Our results indicate differences may exist across time and in different brain regions implicated in social cognition. We tentatively propose that such differences may reflect cognitive effort and how socio-emotional information and bonding develops over time and is processed by different aspects of the social cognition network. However, future research is needed to confirm these proposals.

As described in the introduction, media naturalness theory (Kock, 2011) describes how ambiguity, cognitive effort and arousal are affected by naturalness of communication. We suggest that cognitive effort may be reflected in the observed neural differences in the dmPFC and arousal reflected in the processing of bottom-up emotional information in the vmPFC. However, a limitation of the current data concerns the lack of control measures for arousal and ambiguity. It would be insightful to use measures of physiological arousal such as galvanic skin response to further assess the impact of communication medium. Future research could seek to synchronise measures of arousal with neural indicators of cognitive effort and changes in level of message ambiguity. Indeed, it is a reasonable prediction that FtF communication, particularly following processes of bonding and synchrony, might assist in resolving conversational ambiguity and may therefore be less impacted by the presence of ambiguity.

One of the benefits of using fNIRS is the ability to understand online engagement during conversations as they unfold, without interrupting them to gather explicit feedback from participants. The successful use of fNIRS in our study provides a means for future research to probe deeper into the differences that exist for different communication modes. Establishing a model for neural activity during communication allows us to generate more specific questions. Future research could systematically vary different affordances in the communication context and assess how these impact neural responses. For instance, what features in text-based CMC can assist in perspective taking and mentalising, to reduce cognitive effort reflected in the dmPFC? The possibility that time-course effects reflect social bonding could be assessed further by comparing conversational partners who know each other with those who are strangers (Anders et al., 2015). Likewise, how might different communication affordances support socio-emotional aspects of communication differently for familiar versus unfamiliar partners? The use of a hyper-scanning setup—a method to study social interactions where neural data is collected simultaneously from both participants—could also provide data that would allow the comparison of different stimuli at the same moment during the interaction act (Hamilton, 2020). Additionally, a focus only on adolescent groups could be employed—a span of socioemotional maturing and an age at which the development of the brain and of social cognitive skills are relatively susceptible and quickly adapting to environmental changes and new experiences. The relative novelty of this field of investigation still presents many questions to answer—paving a fresh road for communication scholars to tread.

Conflict of Interest

The authors have no conflict of interests to declare.