Texting and the brain: The time-course of social brain activation in face-to-face versus text-based computer-mediated-communication

Vol.17,No.4(2023)

Abstract

Through evolution, humans have adapted their interactions to face-to-face communication, supported by a network of neural systems which facilitate the transmission and interpretation of social signals for successful communication. However, emerging methods of mediated communication are rapidly shifting our communication habits. For instance, text messaging has become a dominant mode of communication, surpassing face-to-face interaction in some contexts. This study explored differences in neural activation between face-to-face (FtF) and text-based computer-mediated communication (CMC) during a conversation between two communication partners. Seventeen pairs of participants were recruited and each pair undertook a ten-minute conversation in each communication condition. Functional near-infrared spectroscopy was utilized to measure neural activity in two relevant neural structures involved in social cognition: the ventro-medial prefrontal cortex (vmPFC), and the dorsomedial prefrontal cortex (dmPFC). The results indicated that text-based CMC generated more activity in dmPFC relative to FtF. No overall differences were observed between text-based CMC and FtF conditions in the vmPFC, although a linear trend existed across time showing increasing activation through the conversation in the FtF condition only. The results suggest there are differences in neural activations in key brain regions involved in social cognition and highlight the utility of brain imaging to reveal the way neural systems are engaged in different communication contexts. Understanding these differences in neural activation can provide insight into how the brain processes different communication methods and guide us to build tools that will aid text-based communication to provide more naturalistic experiences.


Keywords:
social brain; social cognition; computer-mediated communication; texting; brain imaging
Author biographies

Raul Sacristan

Directorate of Psychology and Sport, University of Salford, Salford, UK

Raul Sacristan, MSc, is a graduate of the Media Psychology Masters programme from the University of Salford, and a documentary film editor for current affairs, science and history documentaries. He researches computer mediated communication and social neuroscience in collaboration with the University of Salford.

Sam Royle

Directorate of Psychology and Sport, University of Salford, Salford, UK

Sam Royle is a specialist technician and lecturer at the University of Salford, UK. His research interests include the effects of alcohol and alcohol hangover, applications of technology in psychology research, and cognitive restoration.

Adam Galpin

Directorate of Psychology and Sport, University of Salford, Salford, UK

Adam Galpin, PhD, is senior lecturer in cognitive and media psychology at the University of Salford, UK. His research interests include human-technology interaction, prosthesis use, human experience design and child and family media use. He developed and teaches the MSc Media Psychology programme at the University of Salford and regularly provides consultancy to the media industries.

References

Aaron, S. (2015). US smartphone use in 2015. Pew Research Center. https://www.pewresearch.org/internet/2015/04/01/us-smartphone-use-in-2015/

Adolphs, R. (1999). Social cognition and the human brain. Trends in Cognitive Sciences, 3(12), 469–479. https://doi.org/10.1016/s1364-6613(99)01399-6

Adolphs, R. (2003). Cognitive neuroscience of human social behaviour. Nature Reviews Neuroscience, 4, 165–178. https://doi.org/10.1038/nrn1056

Adolphs, R. (2009). The social brain: Neural basis of social knowledge. Annual Review of Psychology, 60, 693–716. https://doi.org/10.1146/annurev.psych.60.110707.163514

Amodio, D. M., & Frith, C. D. (2006). Meeting of minds: The medial frontal cortex and social cognition. Nature Reviews Neuroscience, 7(4), 268–77. https://doi.org/10.1038/nrn1884

Anders, S., Heussen, Y., Sprenger, A., Haynes, J.-D., & Ethofer, T. (2015). Social gating of sensory information during ongoing communication. NeuroImage, 104, 189–198. https://doi.org/10.1016/j.neuroimage.2014.10.007

Arioli, M., Cattaneo, Z., Ricciardi, E., & Canessa, N. (2021). Overlapping and specific neural correlates for empathizing, affective mentalizing, and cognitive mentalizing: A coordinate‐based meta‐analytic study. Human Brain Mapping, 42(14), 4777–4804. https://doi.org/10.1002/hbm.25570

Atzil, S., Touroutoglou, A., Rudy, T., Salcedo, S., Feldman, R., Hooker, J. M., Dickerson, B. D., Catana, C., & Barrett, L. F. (2017). Dopamine in the medial amygdala network mediates human bonding. Proceedings of the National Academy of Sciences, 114(9), 2361–2366. https://doi.org/10.1073/pnas.1612233114

Ayaz, H. (2010). Functional near infrared spectroscopy based brain computer interface [Doctoral dissertation, Drexel University]. Google Patents. https://patents.google.com/patent/US9946344B2/de

Ayaz, H., Izzetoglu, P. A., Shewokis, & B. Onaral. (2010). Sliding-window motion artifact rejection for functional near-infrared spectroscopy. 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology, 6567–6570. https://doi.org/10.1109/IEMBS.2010.5627113

Bendall, R. C., Eachus, P., & Thompson, C. (2016). A brief review of research using near-infrared spectroscopy to measure activation of the prefrontal cortex during emotional processing: The importance of experimental design. Frontiers in Human Neuroscience, 10, Article 529. https://doi.org/10.3389/fnhum.2016.00529

Burns, S., & Lieberman, M. D. (2019, July 16). The use of fNIRS for unique contributions to social and affective neuroscience. PsyArXiv. https://doi.org/10.31234/osf.io/kygbm

Bzdok, D., Langner, R., Schilbach, L., Engemann, D. A., Laird, A. R., Fox, P. T., & Eickhoff, S. B. (2013). Segregation of the human medial prefrontal cortex in social cognition. Frontiers in Human Neuroscience, 7, Article 232. https://doi.org/10.3389/fnhum.2013.00232

Cacioppo, J. T., Amaral, D. G., Blanchard, J. J., Cameron, J. L., Carter, C. S., Crews, D., Fiske, S., Heatherton, T., Johnson, M. K., Kozak, M. J., Levenson, R. W., Lord, C., Miller, E. K., Ochsner, K., Raichle, M. E., Shea, M. T., Taylor, S. E., Young, L. J., & Quinn, K. J. (2007). Social neuroscience: Progress and implications for mental health. Perspectives on Psychological Science: A Journal of the Association for Psychological Science, 2(2), 99–123. https://doi.org/10.1111/j.1745-6916.2007.00032.x

Centelles, L., Assaiante, C., Etchegoyhen, K., Bouvard, M., & Schmitz, C. (2013). From action to interaction: Exploring the contribution of body motion cues to social understanding in typical development and in autism spectrum disorders. Journal of Autism and Developmental Disorders, 43(5), 1140–1150. https://doi.org/10.1007/s10803-012-1655-0

Chartrand, T. L., & Bargh, J. A. (1999). The chameleon effect: The perception–behavior link and social interaction. Journal of Personality and Social Psychology, 76(6), 893–910. https://doi.org/10.1037/0022-3514.76.6.893

Daft, R. L., & Lengel, R. H. (1986). Organizational information requirements, media richness and structural design. Management Science, 32(5), 554–571. http://www.jstor.org/stable/2631846

Dunbar, R. I. M. (1998). The social brain hypothesis. Evolutionary Anthropology, 6(5), 178–190. https://doi.org/10.1002/(SICI)1520-6505(1998)6:5<178::AID-EVAN5>3.0.CO;2-8

Ekman, P. (1993). Facial expression and emotion. American Psychologist, 48(4), 384–392. https://doi.org/10.1037/0003-066X.48.4.384

Eslinger, P. J., & Damasio, A. R. (1985). Severe disturbance of higher cognition after bilateral frontal lobe ablation: Patient EVR. Neurology, 35(12), 1731–1741. https://doi.org/10.1212/wnl.35.12.1731

Fellows, L. K. (2011). Orbitofrontal contributions to value-based decision making: Evidence from humans with frontal lobe damage. Annals of the New York Academy of Sciences, 1239(1), 51–58. https://doi.org/10.1111/j.1749-6632.2011.06229.x

Forbes, C. E., & Grafman, J. (2010). The role of the human prefrontal cortex in social cognition and moral judgment. Annual Review of Neuroscience, 33, 299–324. https://doi.org/10.1146/annurev-neuro-060909-153230

Frith, C. D. (2008). Social cognition. Philosophical Transactions of the Royal Society B, 363(1499), 2033–2039. http://doi.org/10.1098/rstb.2008.0005

Frith, C. D., & Frith, U. (2006). How we predict what other people are going to do. Brain Research, 1079(1), 36–46. https://doi.org/10.1016/j.brainres.2005.12.126

Frith, C. D., & Frith, U. (2007). Social cognition in humans. Current Biology, 17(16), R724–R732. https://doi.org/10.1016/j.cub.2007.05.068

Frith, U., & Frith, C. D. (2003). Development and neurophysiology of mentalizing. Philosophical Transactions of the Royal Society B, 358(1431), 459–473. https://doi.org/10.1098/rstb.2002.1218

Gallagher, H. L., & Frith, C. D. (2003). Functional imaging of ‘theory of mind’. Trends in Cognitive Sciences, 7(2), 77–83. https://doi.org/10.1016/S1364-6613(02)00025-6

Gallotti, M., Fairhurst, M. T., & Frith, C. D. (2017). Alignment in social interactions. Consciousness and Cognition, 48, 253–261. https://doi.org/10.1016/j.concog.2016.12.002

Garrod, S., & Pickering, M. J. (2004). Why is conversation so easy? Trends in Cognitive Sciences, 8(1), 8–11. https://doi.org/10.1016/j.tics.2003.10.016

Gordon, R. G., Tranel, D., & Duff, M. C. (2014). The physiological basis of synchronizing conversational rhythms: The role of the ventromedial prefrontal cortex. Neuropsychology, 28(4), 624–630. https://doi.org/10.1037/neu0000073

Hamilton, A. (2020). Hype, hyperscanning and embodied social neuroscience. PsyArXiv. https://doi.org/10.31234/osf.io/rc9wp

Happé, F., Brownell, H., & Winner, E. (1999). Acquired theory of mind impairments following stroke. Cognition, 70(3), 211–240. https://doi.org/10.1016/S0010-0277(99)00005-0

Hasson, U., Ghazanfar, A. A., Galantucci, B., Garrod, S., & Keysers, C. (2012). Brain-to-brain coupling: A mechanism for creating and sharing a social world. Trends in Cognitive Sciences, 16(2), 114–121. https://doi.org/10.1016/j.tics.2011.12.007

Heberlein, A. S., Padon, A. A., Gillihan, S. J., Farah, M. J., & Fellows, L. K. (2008). Ventromedial frontal lobe plays a critical role in facial emotion recognition. Journal of Cognitive Neuroscience, 20(4), 721–733. https://doi.org/10.1162/jocn.2008.20049

Herold, F., Wiegel, P., Scholkmann, F., & Müller, N. G. (2018). Applications of functional near-infrared spectroscopy (fNIRS) neuroimaging in exercise–cognition science: A systematic, methodology-focused review. Journal of Clinical Medicine, 7(12), Article 466. https://doi.org/10.3390/jcm7120466

Jasper, H. H. (1958). The ten-twenty electrode system of the International Federation. Electroencephalography and Clinical Neurophysiology, 10, 367–380.

Jiang, J., Dai, B., Peng, D., Zhu, C., Liu, L., & Lu, C. (2012). Neural synchronization during face-to-face communication. Journal of Neuroscience, 32(45), 16064–16069. https://doi.org/10.1523/JNEUROSCI.2926-12.2012

Kinreich, S., Djalovski, A., Kraus, L., Louzoun, Y., & Feldman, R. (2017). Brain-to-brain synchrony during naturalistic social interactions. Scientific Reports, 7(1), Article 17060. https://doi.org/10.1038/s41598-017-17339-5

Kock, N. (2004). The psychobiological model: Towards a new theory of computer-mediated communication based on Darwinian evolution. Organization Science, 15(3), 327–348. https://doi.org/10.1287/orsc.1040.0071

Kock, N. (2005). Media richness or media naturalness? The evolution of our biological communication apparatus and its influence on our behavior toward e-communication tools. IEEE Transactions on Professional Communication, 48(2), 117–130. https://doi.org/10.1109/TPC.2005.849649

Kock, N. (2011). Media naturalness theory: Human evolution and behaviour towards electronic communication technologies. In S. C. Roberts (Ed.), Applied evolutionary psychology (pp. 381–398). Oxford University Press. https://doi.org/10.1093/acprof:oso/9780199586073.003.0023

Koenigs, M., Young, L., Adolphs, R., Tranel, D., Cushman, F., Hauser, M., & Damasio, A. (2007). Damage to the prefrontal cortex increases utilitarian moral judgements. Nature, 446(7138), 908–911. https://doi.org/10.1038/nature05631

Langton, S. R. H., & Bruce, V. (2000). You must see the point: Automatic processing of cues to the direction of social attention. Journal of Experimental Psychology: Human Perception and Performance, 26(2), 747–757. https://doi.org/10.1037/0096-1523.26.2.747

Lieberman, M. D. (2007). Social cognitive neuroscience: A review of core processes. Annual Review of Psychology, 58, 259–289. https://doi.org/10.1146/annurev.psych.58.110405.085654

Lieberman, M. D., & Pfeifer, J. H. (2005). The self and social perception: Three kinds of questions in social cognitive neuroscience. In A. Easton & N. Emery (Eds.), The cognitive neuroscience of social behaviour (pp. 195–235). Psychology Press. https://doi.org/10.4324/9780203311875_chapter_7

Malonek, D., Dirnagl, U., Lindauer, K., Yamada, K., Kanno, I., & Grinvald, A. (1997). Vascular imprints of neuronal activity: Relationships between the dynamics cortical blood flow, oxygenation, and volume changes following sensory stimulation. PNAS, 94(26), 14826–14831. https://doi.org/10.1073/pnas.94.26.14826

Mehrabian, A., & Ferris, S. R. (1967). Inference of attitudes from nonverbal communication in two channels. Journal of Consulting Psychology, 31(3), 248–252. https://doi.org/10.1037/h0024648

Mitchell, J. P., Heatherton, T. F., & Macrae, C. N. (2002). Distinct neural systems subserve person and object knowledge. Proceedings of the National Academy of Sciences, 99(23), 15238–15243. https://doi.org/10.1073/pnas.232395699

OFCOM. (2020). The communications market report 2020. GOV.UK. https://www.gov.uk/find-digital-market-research/communications-market-report-2020-ofcom#citation

Pinti, P., Tachtsidis, I., Hamilton, A., Hirsch, J., Aichelburg, C., Gilbert, S., & Burgess, P. W. (2020). The present and future use of functional near-infrared spectroscopy (fNIRS) for cognitive neuroscience. Annals of the New York Academy of Sciences, 1464(1), 5–29. https://doi.org/10.1111/nyas.13948

Premack, D., & Woodruff, G. (1978). Chimpanzee problem-solving: A test for comprehension. Science, 202(4367), 532–535. https://doi.org/10.1126/science.705342

Quirk, G. J., & Beer, J. S. (2006). Prefrontal involvement in the regulation of emotion: Convergence of rat and human studies. Current Opinion in Neurobiology, 16(6), 723–727. https://doi.org/10.1016/j.conb.2006.07.004

Rowe, A. D., Bullock, P. R., Polkey, C. E., & Morris, R. G. (2001). ‘Theory of mind’ impairments and their relationship to executive functioning following frontal lobe excisions. Brain, 124(3), 600–616. https://doi.org/10.1093/brain/124.3.600

Shamay-Tsoory, S. G., Tomer, R., Berger, B. D., & Aharon-Peretz, J. (2003). Characterization of empathy deficits following prefrontal brain damage: The role of the right ventromedial prefrontal cortex. Journal of Cognitive Neuroscience, 15(3), 324–337. https://doi.org/10.1162/089892903321593063

Sherman, L. E., Michikyan, M., & Greenfield, P. M. (2013). The effects of text, audio, video, and in-person communication on bonding between friends. Cyberpsychology: Journal of Psychosocial Research on Cyberspace, 7(2), Article 3. https://doi.org/10.5817/CP2013-2-3

Stephens, G. J., Silbert, L. J., & Hasson, U. (2010). Speaker–listener neural coupling underlies successful communication. Proceedings of the National Academy of Sciences, 107(32), 14425–14430. https://doi.org/10.1073/pnas.1008662107

Stolk, A., D’Imperio, D., di Pellegrino, G., & Toni, I. (2015). Altered communicative decisions following ventromedial prefrontal lesions. Current Biology, 25(11), 1469–1474. http://dx.doi.org/10.1016/j.cub.2015.03.057

Strangman, G., Culver, J. P., Thompson, J. H., and Boas, D. A. (2002). A quantitative comparison of simultaneous BOLD fMRI and NIRS recordings during functional brain activation. NeuroImage, 17(2), 719–731. https://doi.org/10.1006/nimg.2002.1227

Straube, B., Green, A., Jansen, A., Chatterjee, A., & Kircher, T. (2010). Social cues, mentalizing and the neural processing of speech accompanied by gestures. Neuropsychologia, 48(2), 382–393. https://doi.org/10.1016/j.neuropsychologia.2009.09.025

Stuss, D. T., Gallup Jr, G. G., & Alexander, M. P. (2001). The frontal lobes are necessary for ‘theory of mind’. Brain, 124(2), 279–286. https://doi.org/10.1093/brain/124.2.279

Tang, H., Zhang, S., Jin, T., Wu, H., Su, S., & Liu, C. (2019). Brain activation and adaptation of deception processing during dyadic face-to-face interaction. Cortex, 120, 326–339. https://doi.org/10.1016/j.cortex.2019.07.004

Vuilleumier, P., & Pourtois, G. (2007). Distributed and interactive brain mechanisms during emotion face perception: Evidence from functional neuroimaging. Neuropsychologia, 45(1), 174–194. https://doi.org/10.1016/j.neuropsychologia.2006.06.003

Walther, J. B. (2011). Theories of computer-mediated communication and interpersonal relations. In M. L. Knapp & J. A. Daly (Eds.), The SAGE handbook of interpersonal communication (4th ed., pp. 443–479). SAGE Publications. https://www.researchgate.net/publication/285323169_Theories_of_computer-mediated_communication_and_interpersonal_relations

Yun, K., Watanabe, K., & Shimojo, S. (2012). Interpersonal body and neural synchronization as a marker of implicit social interaction. Scientific Reports, 2(1), Article 959. https://doi.org/10.1038/srep00959

Zaki, J. (2013). Cue integration: A common framework for social cognition and physical perception. Perspectives on Psychological Science, 8(3), 296–312. https://doi.org/10.1177/1745691613475454

Additional information

Authors’ Contribution

Raul Sacristan: conceptualization, methodology, data curation, formal analysis, investigation, project administration, visualization, writing—original draft, writing—review & editing. Sam Royle: software, resources, data curation, formal analysis, methodology, writing—review & editing. Adam Galpin: supervision, conceptualization, methodology, formal analysis, visualization, writing—original draft, writing—review & editing.

 

Editorial Record

First submission received:
October 22, 2021

Revisions received:
May 5, 2022
December 21, 2022
May 12, 2023

Accepted for publication:
June 13, 2023

Editor in charge:
Lenka Dedkova

Full text

Introduction

The advancement of technology has created new ways for people to communicate. A paradigm shift is occurring whereby the usage of text-based computer-mediated communication (CMC) is now among the most frequent ways to communicate with others, and even surpasses face-to-face (FtF) communication in some reports (Aaron, 2015; OFCOM, 2020). Here, we use text-based CMC to refer to any form of text-based instant messaging communication. This shift toward text-based CMC raises a number of interesting questions. For instance, to what extent are our brains equipped to process text-based CMC exchanges? From an evolutionary perspective, anthropoid primates’ brains have evolved primarily to deal with the interactions arising in social groups, with FtF communication the main tool to create and maintain bonds between members of the community (Dunbar, 1998). Likewise, the development of interpersonal social skills is adapted for FtF interaction (Kock, 2004, 2005) providing the foundation for our development and attunement of social cognitive skills. Building on such proposals, the current study addresses the fundamental question of how neural activation differs between FtF and text-based CMC during a conversation. To our knowledge, this is the first time that this foundational question has been empirically addressed. Here, we synthesise a model of communication (“media naturalness theory”; Kock, 2004, 2005, 2011) with neuro-imaging data from key structures involved in social cognition. Our rationale is that this approach allows us to explore how different components of social cognition might be affected by communication medium, using a tool (functional near-infrared spectroscopy; fNIRS), that can gather data over the time-course of a conversation without interrupting the interaction. Understanding differences in neural activation as a conversation unfolds may allow us to generate questions and hypotheses about the key challenges and limitations involved in social exchanges that take place via text-based CMC.

Comparing Face-to-Face and Computer Mediated Communication

CMC in the broader sense allows for multiple modalities of communication methods with a variety of features, including instant text messaging, audio messages and video calls. Each mode permits the transmission of information in a different manner. Text-based CMC lacks vocal and facial signals and is therefore believed to be a “leaner” communication method than FtF due to the lack of social cues offered during the communication act (Daft & Lengel, 1986). For instance, the lack of facial expression, tone of voice, and synchronicity tend to diminish the quality of the experience and even the effectiveness of communication (Mehrabian & Ferris, 1967; Walther, 2011). Media naturalness theory (Kock, 2011) further proposes that human communication systems have been shaped by evolutionary forces to communicate with other individuals in a co-located environment using facial expressions, body language, and speech synchronously (Kock, 2011). Mediums that afford co-location, synchronicity, the ability to convey facial expressions, body language, and speech will provide a higher level of naturalness than a medium which does not support those features. A medium that comprises one or more features to a larger degree than another will have a higher degree of naturalness. Kock (2011) argues that reduced media naturalness increases both the ambiguity and cognitive effort involved in interaction. Specifically, removing social signals, or altering the communication environment in ways that are unnatural for communicative processing, would lead to a greater cognitive load (C. D. Frith, 2008; Kock, 2011; Zaki, 2013). In addition, FtF communication is predicted to elicit more physiological arousal due to the presence of evoking stimuli such as tone of voice and facial expression (Kock, 2011). It is therefore plausible that such effects will be evident in patterns of neural activity between communication channels that differ in media naturalness (FtF vs Text based CMC). Therefore, looking into the neural structures involved in social cognitive processes is of particular relevance to understand how individuals process these communication exchanges.

Social Cognition

Social cognition refers to the multiple psychological processes that provide an individual with the information and skills necessary to process and manage information from actors within the social environment (C. D. Frith, 2008). One of the main cognitive tasks facing a social actor is the correct reading, processing, and manipulation of social cues. Information from social cues provides a fundamental way to obtain social knowledge from a context and from an individual. These cues include facial expressions, vocal tone, gestures, and body language. Their function is to clarify people’s meanings and intentions (Zaki, 2013). Facial expressions are formed from complex muscle arrangements and provide information about emotional states, some of which are considered to be universal (Centelles et al., 2013; Ekman, 1993; Vuilleumier & Pourtois, 2007). The information conveyed by the expression of human faces seems to be attuned to facilitate understanding. Gestures and movements with the hands, arms or even head can help to communicate a message further. These help not only the sender to communicate the message but also the receiver (Langton & Bruce, 2000). A study looking into the processing of social cues and speech accompanied by gestures suggested that non-verbal social cues may play an important role in language comprehension during FtF interactions (Straube et al., 2010). When we see someone turn their gaze, we can infer where someone focuses their attention, which can help to synchronize attention between social actors. Watching someone’s body movements and facial expressions can help us to build inferences of what that person is thinking, feeling and what they may intend to do (C. D. Frith & Frith, 2006). Referred to as “Mentalising”, or theory of mind (ToM) ability, these processes set the basis for multiple social-emotional responses such as empathy, cooperation, reciprocity, and deception. Being able to use these skills adequately can help to build cohesion within a social environment. This process of mentalisation also allows us to predict the behavior of others within a particular context (U. Frith & Frith, 2003; Premack & Woodruff, 1978). When removing salient social cognitive signals from the communication act, as in text-based CMC, there is increased scope for ambiguity, a likely heightened cognitive load, and reduced emotional information during communication.

The Neuroscience of Social Cognition

Since the infamous case of railway worker, Phineas Gage, it has become increasingly clear that the frontal lobes are crucially connected to social conduct and in detecting and interpreting stimuli of social and emotional significance (e.g., Adolphs, 1999; Eslinger & Damasio, 1985). It is now understood that multiple systems and sub-structures are involved in the processing of social information (Adolphs, 2003; Cacioppo et al., 2007; C. D. Frith & Frith, 2007; Lieberman, 2007). This study is focused on two neural structures widely associated with social cognitive processes, the dorsomedial prefrontal cortex (dmPFC) and the ventromedial prefrontal cortex (vmPFC; Adolphs, 2009; Forbes & Grafman, 2010).

The dmPFC is located at the front of the frontal lobes, above the orbitofrontal cortex and medial frontal cortex. This part of the social brain has been mainly associated with self-knowledge, person knowledge, and mentalising (Amodio & Frith, 2006). Previous studies have shown that, when a series of adjectives were displayed before names of people or inanimate objects, the dmPFC displayed activity when the adjective was shown before the names of people (Lieberman & Pfeifer, 2005; Mitchell et al., 2002). Multiple studies utilizing functional magnetic resonance imaging (fMRI) have reported that dmPFC is a central component in the mentalizing network (Amodio & Frith, 2006; Gallagher & Frith, 2003). In addition, clinical studies have also shown that patients with dmPFC lesions can present severe ToM impairments (Happé et al., 1999; Rowe et al., 2001; Stuss et al., 2001).

The ventromedial prefrontal cortex (vmPFC) is located along the inferior sub-region of the medial prefrontal wall of the frontal lobes, roughly above the eyes. Deficits in this area have been reported to lead to major interpersonal problems and a deficit of empathy (Eslinger & Damasio 1985; Shamay-Tsoory et al., 2003). The vmPFC has been identified as a key mechanism for human social and affective functions (Fellows, 2011; Quirk & Beer, 2006) and observed to play a critical role in emotional facial recognition (Heberlein et al., 2008). Patients with damage in the vmPFC have shown deficiencies interpreting non-verbal social information such as facial expressions, gestures, or body language even though they maintain their understanding and knowledge of social norms and moral norms (Adolphs, 2009).

When speculating on the contrasting functions of the dmPFC and vmPFC, there are somewhat different emphases in different accounts. For instance, Arioli et al. (2021) have considered the dmPFC to have more involvement in cognitive aspects of mentalising and the vmPFC to have more involvement in affective aspects of mentalising. Other authors emphasise a role of the dmPFC in top-down control of self-other information, whilst the vmPFC is more involved in processing of bottom-up sensory information. For instance, Bzdok et al. (2013) proposed that the dmPFC is involved in more complex, top-down, meta-cognitive social processes, including perspective taking and episodic memory retrieval, which are independent of sensory input. The authors suggest that the vmPFC, in contrast, is involved in bottom-up processing associated with affective and motivational information of both the self and the communication partner.

Synthesising these proposals with media naturalness theory (Kock, 2011), we might speculate that the dmPFC is more impacted by the cognitive effort imposed by communication activities whereby lean communication channels, such as text-based CMC, involve increased ambiguity. Cognitive effort might therefore be introduced when it is hard to adopt the perspective of the communication partner and we would therefore expect increasing dmPFC involvement for mediums which challenge these activities. In contrast, the vmPFC would be more implicated in monitoring ongoing emotional information generated from the exchange and we would therefore predict more involvement of the vmPFC in the FtF condition due to the availability of more salient bottom-up signals conferring emotional state.

Within this view, the dmPFC and vmPFC are seen to be continuously processing cognitive and emotional information from the communication environment, integrating bottom-up sensory information with prior knowledge and information. Galotti et al. (2017) emphasise that reciprocal interactions, involving the mutual exchange of information, unfold over time, and argue for the value of modelling the time-course of neural activity of interacting partners. Therefore, an additional element of our study seeks to capture the time-course of neural activation during CMC and FtF conversation. We divided the conversation into smaller time windows allowing us to measure the dynamic temporal variation in neural response across the duration of the interaction. Averaging the neural activity across time would remove useful information about the time-course of these neural processes.

Various mechanisms have been evoked to describe the dynamic nature of information exchange during social interaction. Brain-to-brain coupling, or interpersonal neural synchronisation, refers to an emergent process that occurs when the listener’s brain becomes tuned to the visual and auditory information provided by the speaker (Hasson et al., 2012) and has been suggested as an indicator for successful communication (Stephens et al., 2010). Similarly, “interactive alignment” (Garrord & Pickering, 2004) occurs as conversational partners begin to mirror each other’s linguistic phrases and grammatical structures which helps to converge on a shared mental model of the topic. This is seen to provide a cognitive benefit through development of a shared “situation model” which supports mutual understanding and resolves conversational ambiguity. In the current study, we might expect that the neural activity within the dmPFC would increase over time as social signals are integrated by the conversational partners to perform mentalising tasks and generate mutual understanding. If the dmPFC is more implicated in top-down and metacognitive processing (Bzdok et al., 2013), we might expect the differential availability of bottom-up sensory information to have no impact on time-course of activation. In contrast, should interactional alignment occur across the conversation which in turn assists in resolving conversational ambiguity, we might see a reduction in activity based on reduced cognitive effort. We therefore took an exploratory approach to the question of the time-course of dmPFC activation. Rather, our key interest was how time-course may differ between CMC and FtF conditions.

Other authors have implicated forms of conversational synchrony, such as synchronising the length of turn-taking, in social bonding processes (Gordon et al., 2014) and social closeness and connection (Kinreich et al., 2017). Synchrony can also occur in non-verbal aspects of communication, such as body posture and movements, which has been implicated in increased liking and empathy (Chartrand & Bargh, 1999). These studies focus on the availability of bottom-up sensory and affective processing and may therefore suggest a role for vmPFC activity in forms on interactional synchrony. Indeed, Yun et al. (2012) found temporal alignment in finger tapping behaviours, and this synchronisation correlated with vmPFC activity. Gordon et al. (2014) compared the number of words per conversational turn at the beginning and end of a natural conversation, finding convergence in conversational turn length between non-brain damaged individuals, but not in patients with vmPFC damage. We therefore speculate that neural activity in regions associated with vmPFC would increase as additional bottom-up sensory and affective information becomes available to support communication, and this increase will be greater according to the relatively richer medium of FtF interaction. The CMC condition should demonstrate a relatively smaller effect of time-course since the main cues involved in social emotional processing are absent.

Near Infrared Spectroscopy for Social Interaction

This study employed functional near-infrared spectroscopy (fNIRS), which has been recognized as a suitable tool for recording frontal neural activity during real-time social interaction (Burns & Lieberman, 2019; Pinti et al., 2020). One benefit of fNIRS is that the neural activity associated with an ongoing interaction can be recorded as it unfolds, which allows analysis without interrupting the interaction with self-report measures. Like fMRI, fNIRS relies on the BOLD to collect neuroactivity—the blood oxygen level-dependent signal that occurs when localized populations of neurons fire, consume oxygen, and therefore require more oxygen to be pumped to the area to continue operating. Both fMRI and fNIRS can detect changes in hemoglobin concentrations caused by neuronal firing, due to physical differences in the oxygenated and deoxygenated hemoglobin molecules (HbO and Hb, respectively). fNIRS allows indoor and outdoor scenarios, and its portability features, safe setups, and tolerability to environmental noise and body movements make it a versatile tool for social interaction conditions. Importantly, fNIRS allows participants to engage in FtF communication that is not constrained by the same requirements as fMRI scanning technology. fNIRS, compared to fMRI, has increased sensitivity to temporal factors, allowing exploration of changes in activation over time. This versatility increases the ecological viability for experiments when compared to fMRI, creating a more naturalistic scenario (Pinti et al., 2020). As such it has been employed to explore neural activation in several studies. For instance, Jiang et al. (2012) found neural synchronization during FtF communication but not during back-to-back communication (even though still listening to each other). Tang et al. (2019) used fNRIS to look into behavioral and neural differences in cortical regions in a deception game task. However, whilst these have demonstrated the ability of fNIRS to detect patterns of activation in social cognitive neural structures, to our knowledge no studies have investigated differences between FtF and text-based CMC.

The Current Study

The current study aimed to simulate a human communication setup that could happen during an unconstrained conversation. Our main questions concerned (RQ1) what difference in neural activation exist between regions of the PFC associated with social cognition (vmPFC and dmPFC) when humans communicate FtF or via text-based CMC, and (RQ2) how activation may change over time as the conversation develops.

Methods

Design

A repeated measures design was employed with two conditions: Face to Face (FtF) and Computer Mediated Communication (CMC). Participants were asked to engage in 10-minute conversations within each condition. During the interaction, for one of the participants in each pair, the brain activity in the dmPFC and vmPFC was measured in both hemispheres. The principle of the design was to compare neural activity in these key brain areas involved in social cognition between the FtF and CMC conditions. The conversation conditions were counterbalanced. In addition, we were interested in how those brain differences might evolve during the process of the conversation. Analysis of the data was therefore based on a 2 [communication condition: FtF, CMC] * 4 [time quartile: (Q1, Q2, Q3, Q4)] repeated measures design for each brain region (dmPFC and vmPFC) separately.

Participants

A total of 20 pairs of participants were recruited from undergraduate and postgraduate programs at the University of Salford, UK. The pairs consisted of 12 males and 28 females, aged between 18 and 40 years old (Mage = 27.2 years old, SD = 5.8). Three pairs of participants were discarded from the analysis due to technical issues encountered when collecting the data, leaving 17 pairs of data. The pairs varied in terms of their prior familiarity with each other but this was not controlled in the current study. All of the pairs were at least acquaintances from the same University course, except for 1 pair of participants who did not know each other. Written consent was acquired from both participants, once they read the Participant Information sheet and the procedural information about the experiment that was provided. Ethical approval was acquired from the College of Health and Social Care Ethics Panel at the University of Salford. Participants were paid £7 in compensation for their time. Data was collected between August 2017 and June 2019.

Materials

In order to measure the neural activity (oxy-Hb) on the selected regions of the prefrontal cortex, Biopac fNIRS100B multichannel system (Figure 1) was utilized to collect the data from one of the participants from each pair. The system provides a temporal resolution of 500ms (2Hz) and is formed of a headband comprising 4 light sources and 10 light detectors resulting in 16 measurement channels. The probe was placed on the forehead and aligned with the frontal polar (Fp) 1 and Fp2 of the international 10–20 system (Jasper, 1958). Figure 3 shows the approximate location of optodes on the fNIRS headband (©BIOPAC Systems, Inc). Only relevant optodes were utilized for the data analysis, with regions of interest named according to anatomical congruence. Optodes 5, 7, 9, & 11 were ascribed to the dmPFC, and optodes 6, 8, 10, & 12 were ascribed to the vmPFC.

 

Figure 1. The fNRIS Biopac fNIR100B Model.

Note. This figure shows the hardware utilised to collect the level of HbO on the PFC of the subject. The box was connected to a computer loaded with Cognitive Optical Brain Imaging studio (COBI) software. Photo Courtesy of ©BIOPAC Systems, Inc.

 

Cognitive Optical Brain Imaging studio (COBI) software was used to record data from the fNIRS device. “fNIRS-Soft” software was utilized for the post-process of the data recorded via COBI Studio software. Additionally, for the CMC condition, one iPhone5s with “WhatsApp” (Room A) and one laptop with the WhatsApp application (Room B) were utilized for the computer mediated communication task. During the process a video camera recorded the participant wearing the fNIRS band in both conditions, screen capture was used to record the “WhatsApp chat” conversation of the participants. These recordings were intended to monitor the participants’ behavior throughout the experiment in case it was needed for later data analysis purposes.

Procedure

Once the participants arrived, the participant information sheet and consent form were handed out and returned completed. The participant information sheet requested that the participants provide demographic data and some basic information about their daily usage of CMC, to make sure that all participants were fairly familiar with the communication method employed in the experiment. Participants were seated approximately 1.5 m away from each other. A coin was flipped to choose who would be wearing the fNIRS band (Participant A) to collect her/his neural activity (oxy-Hb).

The participants were required to talk about a standard pre-selected topic. This topic was “jobs automation and whether they thought their job would be replaced by a machine”. This topic was proposed, since it could potentially allow any student to provide a personal opinion and to develop their own views, enabling both partners to engage in what we could define as “regular conversation”.

Once the participants were informed of the conversation topic, the band with the sensors was placed on the forehead of “participant A” and secured with a Velcro elastic band. The 16-channel probe was placed on the forehead and the center line of the sensor band was aligned visually to roughly the fp1-fp2 line of the international 10–20 system. Software was initialized and then the baseline was acquired. This required the participant to relax and empty their mind during the time device and software acquire the corresponding information, this took around 30sec to be completed. To help with the process it was suggested to look at the white wall. After the baseline process was completed, the experiment began. In the FtF condition, both participants were left uninterrupted to develop the conversation until the time was up. They were given a 5-minute break and the fNIRS’ sensors were removed from the participant’s forehead, for comfort.

In the CMC condition, Participant B was placed into another room next door (Room B). A laptop was set up with WhatsApp to enable to them to carry on the conversation with Participant A in room A. Additionally, in room B, an A4 sheet was provided with a few cues and questions to assist and keep the chat rolling, in case the conversation came to a halt due to lack of ideas or contribution. This was proposed and explained to Participant B and only Participant B in room B was aware of this additional aid. Participant A was never aware of the set-up in the room B. This set-up in room B was meant to facilitate and aid the condition and no neural data was collected from Participant B. Although we did not formally code the extent to which conversation was facilitated by the prompts, the researcher did not observe any lack of conversational activity or interaction while Participant A was engaging in the communication act and CMC conditions unfolded without any discontinuance. After another ten minutes of text-based communication the experiment was concluded, and the data collection finalized. The order of the conditions was counterbalanced.

Figure 2. Flexible Sensor Band Fit on Subject’s Forehead.

Note. This image shows; left side: Participant A during the CMC condition. Right side: Participant A during the FtF communication.

 

Data Analysis

To analyze the data provided by the fNIRS sensors, we looked at the changes in HbO.

The first step required converting the raw fNIRS data delivered by the (COBI) Studio software (Biopac System Inc.). fNIRS data pre-processing was carried out in fNIR Soft. Raw light intensity data was smoothed with a low-pass FIR filter with an order of 20, before sliding window motion artefact rejection (SMAR) was applied (see Ayaz, 2010; Ayaz et al., 2010) which involved the automatic identification and rejection of motion artefacts. This process also helped with potential noise and interferences introduced by the hair, helping to refine the data. Changes in the concentration of haemoglobin (from each baseline to the associated experimental condition) were then calculated, for both face-to-face and computer-mediated communication, using the modified Beer-Lambert Law. We focused on HbO following arguments that changes in HbO are the most reliable indicators of cerebral blood flow (and consequently, neural activity; Malonek et al., 1997; Strangman et al., 2002), and is the primary focus in many areas of cognitive neuroscience (e.g., Bendall et al., 2016). The HbO measure is expressed in arbitrary units (a. u.) meaning it is a relative measure of the strength of activation rather than an absolute unit of concentration. This is a common approach employed in fNIRS research (see Herold et al., 2018, for a review). Average HbO changes from the baseline were calculated for three time periods. Once this process was completed the final resulting information was organized for analysis in SPSS.

Since the vmPFC and dmPFC have been implicated in social cognition (Adolphs, 2003, 2009) analyses were focused on these regions of interest. Four central channels were selected to represent each of the vmPFC and dmPFC areas, located below and above the midline of the forehead sensor respectively (see Figure 3). The rest of the data from other optodes was discarded.

 

Figure 3. Channels Area Where the fNIRS Sensor Records HbO Measures.

 

Note. This figure displays the fNIRS probe location, showing the approximate area where the 16 optodes are positioned and the 8 optodes that we will be focusing on this research. Brain Photo Courtesy of fNIRSoft®.

 

For the ten minutes of sample collection for each condition, we separated four equal segments of duration for each optode to assess how HbO signals develop over the course of the interaction. For each quartile of the conversation (Q1, Q2, Q3, Q4), we took 300 samples of data which is comparable to 2 minutes 30 seconds of conversation, making a total of 10 minutes. We calculated the mean for each one of the time-blocks, for each optode and condition. This resulted in eight optodes (4 dmPFC, 4 vmPFC) times four blocks of data. The results from those calculations were fed onto SPSS for the analysis.

Results

Participant A exchanged an average of 26.2 messages (max = 45, min = 18, SD = 12.3) and Participant B exchanged an average of 25.8 messages (SD = 11.3). Emoticons were used very infrequently within the text-based CMC and by some participants not at all, therefore messages were almost exclusively text-based and were therefore not delineated in analysis by the presence of emoticons. As we were not making predictions regarding brain hemisphere, the data from optodes for corresponding brain regions was collapsed across hemisphere. Table 1, below, shows means and SDs across conditions. Figure 4, below, shows overall differences between conditions in both brain areas.

 

Table 1. Mean HbO Levels by Brain Region, Communication Condition, and Time Quartile.

Time quartile

dmPFC

vmPFC

FtF

CMC

FtF

CMC

 

M

SD

M

SD

M

SD

M

SD

1

0.06

1.56

1.11

1.06

0.65

1.68

1.35

1.09

2

0.38

1.75

1.38

1.33

1.26

1.81

1.46

1.39

3

0.69

1.83

1.49

1.34

1.72

2.00

1.66

1.51

4

0.86

1.88

1.54

1.39

2.00

2.39

1.77

1.50

Overall

0.50

1.71

1.38

1.21

1.41

1.87

1.56

1.30

Note. Mean (M) and standard deviation (SD) HbO levels are presented in arbitrary units. Abbreviations refer to dmPFC (dorsomedial prefrontal cortex), vmPFC (ventromedial prefrontal cortex), FtF (face to face condition), CMC (computer mediation condition).

 

Figure 4. Mean Activity Measures for HbO Levels by Condition and Across Brain Region.

Note. The mean activity for both brain regions across conditions (* = p < .05). The scaling of HbO parameters is in arbitrary units. Error bars: + / − 1 standard error of the mean (SEM).

 

Analyses were run separately on the dmPFC and vmPFC by conducting a 2 (communication condition: CMC vs FtF) * 4 (time quartile: Q1, Q2, Q3, Q4) repeated measures ANOVA on each brain region. Greenhouse-Geisser corrections (GG-corr) are provided when Mauchly’s test of revealed violations of sphericity.

dmPFC Analysis

Shapiro Wilkes tests revealed that data for the dmPFC analysis violated the assumption of normality of the residuals. This was driven by a single participant whose means in the four levels of the time quartile variable in the FtF condition were between 2.78 and 3.36 SDs from the condition means. Removal of this participant meant that data met the assumptions of normality of residuals (all Shapiro-Wilkes tests revealed p values >.05) and the data analysis reported below is therefore based on the 16 remaining participants (although it should be noted that the pattern of effects was the same under analysis of the full data set). Analysis of the data from dmPFC revealed a significant effect of condition, F(1, 15) = 7.110, MSE = 2.054, p = .018, ηp2 = .322, due to greater activation during the CMC condition (see Figure 4, above). No interactions were observed between condition or time (see Figure 5), but a significant main effect of time was observed, F(1.377, 20.660) = 8.776, MSE = 0.691, p = .004, ηp= .369; GG-cor. Analysis of contrasts for the time variable revealed a significant linear trend, F(1, 15) = 10.262, MSE = 0.767, p = .006, ηp2 = .406, indicating increasing neural activity throughout the conversation (see Figure 6). However, the quadratic trend also approached significance, F(1, 15) = 4.495, MSE = 0.106, p = .051, ηp2 = .231. Analysis of the time variable alone (collapsed across communication conditions) suggests the near quadratic effect was due to smaller increases in activity at successive quartiles. Bonferroni pairwise comparisons revealed differences between quartile 1 and both quartiles 3 and 4 (p = .039 and p = .034 respectively), with no other significant differences between quartiles.

 

Figure 5. Mean HbO Levels in Dorsomedial Prefrontal Cortex by Condition and Across Time.

Note. This figure shows the changes in HbO across time in areas associated with dmPFC. CMC is consistently higher than FtF. The scaling of HbO parameters is in arbitrary units. Error bars: + / − 1 SEM.

 

 

Figure 6. Mean HbO Levels in Dorsomedial Prefrontal Cortex Across Time.

Note. This figure shows the main effect of changes in HbO across time in areas associated with dmPFC, collapsed across CMC and FtF conditions. The scaling of HbO parameters is in arbitrary units. Error bars: + / − 1 SEM. Significant pairwise comparisons (Bonferroni) are indicated by * (p < .05).

 

vmPFC Analysis

The second ANOVA assessed activation in the vmPFC (optodes 6–8–10–12). The data met the assumptions of normality of residuals in the analysis of the vmPFC (Shapiro-Wilkes revealed all p values > .05), and therefore analysis was conducted on the full data set. Analysis revealed no main effect of condition, F(1, 16) = 0.097, MSE = 8.407, p = .760, ηp2 = .006. As before, a significant effect of time was found, F(1.272, 20.344) = 8.225, MSE = 1.475, p = .006, ηp2 = .340; GG-cor, which also interacted with condition, F(1.952, 31.236) = 4.863, MSE = 0.439, p = .015, ηp2 = .233; GG-cor. To further investigate this interaction, presented in Figure 7, ANOVAs were conducted on the time variable for the FtF and CMC conditions separately. A significant effect of time was observed in the FtF condition, F(1.25, 19.997) = 9.742, MSE = 1.457, p = .003, ηp2 = .378; GG-cor, and contrast analysis revealed that the effect most resembled a linear trend, F(1, 16) = 10.779, MSE = 1.605, p = .005, ηp2 = .403, with greater activity at each successive time quartile. However, a significant quadratic trend was also found, F(1, 16) = 4.723, MSE = 0.094, p = .045, ηp2 = .228, capturing the fact that the amount of increase levelled off statistically following from quartile 2. Bonferroni post-hoc tests revealed significant differences between quartile 1 and each successive quartile (Q1 vs Q2 = .002; Q1 vs Q3 = .005; Q1 vs Q4 = .028), but no other significant differences were found. In contrast to the FtF condition, no effect of time was found in the CMC condition, F(1.795, 28.716) = 2.03, MSE = 0.507, p = .154, ηp2 = .113; GG-cor. Neither the linear (p = .100) or quadratic contrasts (p = .980) were significant for the time variable in the CMC condition, and Bonferroni pairwise tests revealed no differences between any quartiles. However, lack of an effect in this condition should be interpreted with extreme caution as the observed power for this test was low (.364) demonstrating a high chance of obtaining effects over time with a larger sample.

 

Figure 7. Mean HbO Levels in the Ventro-Medial Prefrontal Cortex by Condition and Across Time.

Note. This figure shows the evolution on the area associated vmPFC activity across time. Levels of HbO grew across the experiment during FtF, but not during CMC which showed a non-significant trend. The scaling of HbO parameters is in arbitrary units. Error bars: + / − 1 SEM. Significant pairwise comparisons (Bonferroni) existed only in the FtF condition and are indicated by * (p < .05).

 

Discussion

The current study examined neural activity in vmPFC and dmPFC during face-to-face communication compared to text-based computer mediated communication. The initial predictions presented for the study proposed that, since FtF was a much richer communication system and most of the social cues developed through evolution for social interaction are removed during CMC, the brain regions involved in social cognition would respond differently to the different mediums of text-based CMC and FtF communication. Specifically, the dMPFC was expected to show greater activity in the CMC condition, in line with the additional cognitive effort involved in disambiguating the message and taking the perspective of the conversation partner. The vmPFC was expected to show greater activity in the FtF condition due to the greater availability of emotional information. Two key results emerged from our research. First, in line with our expectations, greater activation was observed during CMC communication within the dmPFC. Second, and in contrast to activation in the dmPFC, communication medium did not affect overall levels of activation in the vmPFC. However, an interaction was observed between condition and time, whereby a significant linear increase in activation was observed for FtF but not the CMC condition.

The first finding showed greater dmPFC activation for CMC which did not interact with time. One interpretation of the increase in dmPFC activation during texting is that more cognitive resources were recruited during text-based CMC than during FtF. If this interpretation is correct, it indicates that the action of text-based chatting via smartphone generates more cognitive effort on the dmPFC. One of the constructs of media naturalness theory proposes that removing elements from communication that are considered “evolutionary intrinsic” would create a higher cognitive burden on the brain. This burden would generate a more unnatural communication and consequently require extra effort, suggesting an increase in neural activation (Kock, 2011). This theory can help to explain some of the results in the present study. As media naturalness theory proposes, some communication situations may require more resources (Kock, 2005). dmPFC is associated with performing tasks concerning theory of mind, self-knowledge, and person knowledge (Amodio & Frith, 2006). Therefore, one interpretation for the results obtained could be that the removal of social cues creates an additional challenge to build inferences about the other person’s state of mind based only on the limited social information being broadcasted. This suggests an additional effort must be made by the brain in order to run internal processes (Kock, 2005).

A similar perspective evoking the concept of cognitive load highlights differences in the ability to process information about communication partners. Social cognition theories have proposed that two processes can happen during social interaction: automatic and controlled (Adolphs, 2009; C. D. Frith & Frith, 2007; Lieberman, 2007), with controlled processes linked to dmPFC activity (Lieberman, 2007). Automatic processes are considered to be faster and reflexive, developed early in evolution and shared with other animals (e.g., interpreting emotional facial expression, body language), and requiring a lower cognitive load than social cognitive controlled processes. In contrast, controlled processes are associated with conscious awareness, and slower, effortful processing. Explicit thoughts about someone else’s mind have also been associated within the domain of controlled processed (Lieberman, 2007). We believe that text-based CMC is best characterised as a more controlled process, requiring increased conscious reflection to support the interpretation of the mental states of a communication partner.

The parallel dmPFC increase in activation across time in both the CMC and FtF conditions provide no evidence that cognitive effort was reduced over the course of the conversation, or differentially according to medium. Rather, our data are most consistent with an interpretation that the dynamic involvement of the dmPFC involves the ongoing coordination of top-down and cognitive aspects of mentalising, which appears to unfold in a similar way irrespective of medium. This could suggest that the time-course effects observed in both conditions reflect top-down, meta-cognitive social processes that are independent of sensory input (Bzdok et al., 2013).

We expected that the vmPFC would show increased activity in the FtF condition due to the evidence of vmPFC involvement in processing bottom-up emotional input during social interaction (Arioli et al., 2021; Bzdok et al., 2013). However, the measures obtained from the vmPFC did not reflect any global difference across conditions, although differences were observed when assessing the time-course of activation. Significant increases in activation were identified across time, but only for FtF communication. However, we offer a tentative interpretation of that result here due to very low power in the analysis of the CMC effect (.364). The results indicate that activity in the vmPFC was initially lower during the first quartile of the conversation via FtF, but then increased, eventually surpassing CMC activity (Figure 7). The vmPFC has been associated with social rewards and emotional assignation to social stimuli (Fellows, 2011; Koenigs et al., 2007; Quirk & Beer, 2006). One interpretation of our results is therefore that the availability of bottom-up socio-emotional stimuli increases as a FtF conversation unfolds, reflected in increasing activity in the vmPFC. It is important to also consider whether such information may also become increasingly available in text-based CMC, albeit at a lesser rate, as indicated by a non-significant (and underpowered) trend in vmPFC activation in this condition. A similar interpretation could be that connectedness between participants can increase as the time of FtF interaction increases. This is consistent with a study where bonding has been associated with the social reward system managed by vmPFC, amygdala and other regions of the medial frontal cortex (Atzil et al., 2017). Indeed, a study measuring bonding during FtF and CMC conditions reported that the FtF condition showed the highest rated level of bonding and text messaging the lowest (Sherman et al., 2013).

In the CMC condition vmPFC measures indicated that the level of activation did not show a significant time-course effect, displaying less increase over time than the FtF condition (Figure 7). The lack of a significant linear trend in the CMC condition, whilst being underpowered, could tentatively be taken to suggest that conversational responses and social cognitive processing of the conversation was similar at the beginning and the end of the task. This is consistent with an interpretation suggesting that the social signals in the environment remained relatively unchanged during the 10 minutes the task lasted. We are therefore directed to speculate whether the relatively steeper increase in activation in the FtF condition indicates the participants gradually attuning to the greater number of communication cues provided by the conversational partner. Potentially relevant here are the phenomena of conversational synchrony, whereby conversational partners begin to adopt similar verbal and non-verbal behaviours, and conversational adaptation, whereby communication is adapted to the particular needs of a conversation partner. Both have been implicated by damage to the vmPFC. For example, Gordon et al. (2014) compared the conversational patterns of patients with damage to the vmPFC to healthy control participants. They found that the controls began to synchronise the length of their conversational turns whilst this effect was absent in those with damaged vmPFC. Stolk et al. (2015) found that patients with vmPFC lesions failed to dynamically adjust their communication to the presumed abilities of a (hidden) conversational partner who they believed to be either a child or an adult. This finding suggests the importance of the vmPFC for using knowledge of the conversational partner for adaption of communication style. It is reasonable to question whether differences in the time-course in vmPFC activity between text-based CMC and FTF conversational conditions reflect the differential access to social cues which underpin dynamic aspects of communication adjustment such as synchrony and adaptation.

Conclusions, Limitations & Further Research

The current study marks a first attempt to understand neural differences in communication depending on FtF and text-based CMC. Our results indicate differences may exist across time and in different brain regions implicated in social cognition. We tentatively propose that such differences may reflect cognitive effort and how socio-emotional information and bonding develops over time and is processed by different aspects of the social cognition network. However, future research is needed to confirm these proposals.

As described in the introduction, media naturalness theory (Kock, 2011) describes how ambiguity, cognitive effort and arousal are affected by naturalness of communication. We suggest that cognitive effort may be reflected in the observed neural differences in the dmPFC and arousal reflected in the processing of bottom-up emotional information in the vmPFC. However, a limitation of the current data concerns the lack of control measures for arousal and ambiguity. It would be insightful to use measures of physiological arousal such as galvanic skin response to further assess the impact of communication medium. Future research could seek to synchronise measures of arousal with neural indicators of cognitive effort and changes in level of message ambiguity. Indeed, it is a reasonable prediction that FtF communication, particularly following processes of bonding and synchrony, might assist in resolving conversational ambiguity and may therefore be less impacted by the presence of ambiguity.

One of the benefits of using fNIRS is the ability to understand online engagement during conversations as they unfold, without interrupting them to gather explicit feedback from participants. The successful use of fNIRS in our study provides a means for future research to probe deeper into the differences that exist for different communication modes. Establishing a model for neural activity during communication allows us to generate more specific questions. Future research could systematically vary different affordances in the communication context and assess how these impact neural responses. For instance, what features in text-based CMC can assist in perspective taking and mentalising, to reduce cognitive effort reflected in the dmPFC? The possibility that time-course effects reflect social bonding could be assessed further by comparing conversational partners who know each other with those who are strangers (Anders et al., 2015). Likewise, how might different communication affordances support socio-emotional aspects of communication differently for familiar versus unfamiliar partners? The use of a hyper-scanning setup—a method to study social interactions where neural data is collected simultaneously from both participants—could also provide data that would allow the comparison of different stimuli at the same moment during the interaction act (Hamilton, 2020). Additionally, a focus only on adolescent groups could be employed—a span of socioemotional maturing and an age at which the development of the brain and of social cognitive skills are relatively susceptible and quickly adapting to environmental changes and new experiences. The relative novelty of this field of investigation still presents many questions to answer—paving a fresh road for communication scholars to tread.

Conflict of Interest

The authors have no conflict of interests to declare.

Metrics

0

Crossref logo

0


607

Views

55

PDF views