Analysis of cognitive behavioural therapy apps for generalised anxiety disorder: Evidence-based content and user experience

Introduction

Mental health substantially contributes to the global burden of disease and can affect daily functioning, employment, financial stability, education, and relationships (Doran & Kinchin, 2019; Drissi et al., 2020). Anxiety disorders, such as generalised anxiety disorder (GAD), are prevalent, with a global 12-month rate of 1.8% and a higher rate of 3.6% in the Australian population (Bandelow et al., 2017; Ruscio et al., 2017). The core feature of GAD is disproportionate and excessive worry and anxiety about a range of life circumstances that is difficult for a person to control (American Psychiatric Association, 2013).

Cognitive behavioural therapy (CBT) is the recommended first line of psychological treatment for GAD according to UK and US clinical guidelines (Ballenger et al., 2001; Hunot et al., 2007; National Institute for Health Care Excellence; 2019). Hirsch et al. (2019) noted that meta-analyses of randomised control trials on CBT for GAD, confirmed its superiority as a treatment modality. CBT for GAD often includes worry postponement, targeting of intolerance of uncertainty, and modification of beliefs about worry (Dugas & Robichaud, 2007; Wells, 1999), which are not typically targeted in other anxiety disorders. Given the established strength of CBT for GAD, the quality of mobile app CBT treatments for GAD should be considered.

CBT interventions can be successfully delivered through various platforms including mobile applications (Howell et al., 2019; Kessler et al., 2009; Wootton et al., 2018). Self-help and digital health interventions have been found to be effective in obsessive compulsive disorder (OCD), depression, and anxiety disorders and can increase access to support, reduce symptoms, and are cost efficient (for example Axelsson et al., 2018; Miller et al., 2021; Nordgreen et al., 2012; Popa & Porumbu, 2017; Wright et al., 2019). Apps that target anxiety can significantly reduce symptoms compared to controls (Firth, Torous, Nicholas, Carney, Pratap, et al., 2017; Firth, Torous, Nicholas, Carney, Rosenbaum, & Sarris, 2017; Wang et al., 2018). CBT delivered this way can significantly ameliorate anxiety symptoms, including for GAD (Miller et al., 2021; Watts et al., 2013).

However, several studies have found that the content of mental health apps were not strongly linked to evidence-based interventions (e.g., CBT; Alyami et al., 2017; Kertz et al., 2017; Lui et al., 2017; Shen et al., 2015). For example, a study on apps for social anxiety found that most of the apps did not state the source of their content or the apps’ efficacy (Alyami et al., 2017). A review of child and adolescent anxiety apps found evidence-based content was scarce and despite claiming to be evidence-based, apps included features that were not evidence-based (Bry et al., 2018).

Further, an apps’ empirical basis and clinical usefulness may be unrelated to its popularity and consumer ratings (Kertz et al., 2017; Neary & Schueller, 2018; Wasil et al., 2019). A review of top-rated apps for depression found that only 48% of apps claimed to be based on psychological interventions with only 7% providing evidence of the app’s efficacy in reducing depressive symptoms, and less than 41% including input from a mental health professional (Qu et al., 2020).

Mental health apps that have engaging user experiences can encourage users to engage with an app, through the inclusion of design features such as gamification, the ability to use apps offline, and action reminders (Chandrashekar, 2018). Amongst user reviews for bipolar disorder apps (N = 2,173) and mental health apps generally (N = 88,125), positive reviews focused on helpfulness and ease of use, whilst negative reviews focused on functionality and useability issues, lack of features, or lack of content (Nicholas et al., 2017; Oyebode et al., 2020). Research on app users’ reviews provides insight into users’ needs, and the strengths, advantages, and disadvantages of mental health apps (e.g., Baumel et al., 2020; Calderia et al., 2017; Stawarz et al., 2014, 2018). Balaskas et al. (2021, p. 2) stated, “…we need to understand the content in mental health apps from the viewpoint of not only evidence-based strategies but also of engagement elements and other features relevant to digital delivery.”

Therefore, understanding the status of evidence-based apps for GAD and users’ experience of such apps, may assist in future development of effective apps for the treatment of anxiety, specifically GAD. The current study aimed to explore the therapeutic content and user reviews of apps that are commercially available in English and target GAD and worry, and which purport to be CBT-based. The apps’ content were assessed for their fidelity to CBT principles in general and for GAD specifically (as outlined as best practice, e.g., in the National Institute for Health Care Excellence, 2019; guidelines). The apps’ user experience was assessed firstly by the features that would increase user engagement (e.g., visual design of the app, notifications and reminders, ability to customise) and secondly, by thematic analysis of the publicly available user reviews on the apps.

Methods

Research Design Overview

Following ethic’s approval by the Univeristy of the Sunshine Coast Human Ethics' Committee (S201480), analyses of commercially available apps for anxiety that employed CBT were undertaken in three steps, consistent with Stawarz et al. (2018). First, identification of the apps (e.g., using keyword searches, identifying CBT in the app) suitable for inclusion was undertaken. Second, the analysis of therapeutic features (based on clinical guidelines for CBT) and engagement features (i.e., what encourages continued use of the app) was undertaken, and third, a qualitative analysis of users’ experiences (based on user reviews of the apps) was conducted.

Identification of Apps

Phase 1: Keyword Search and Data Clean

Searches of Google Play and iTunes App Store (as commercial providers of apps) were conducted with the defined keywords: ‘Anxiety’, ‘generalised anxiety disorder’, ‘generalized anxiety disorder’, ‘GAD’ and ‘worry.’ Searches were manually undertaken for each key word separately, by two researchers in December 2020. As shown in Figure 1, 1,888 apps were identified in the two app stores using the search terms, with 985 being excluded as not being suitable (i.e., not relevant conditions). The names and links of remaining apps (n = 903) were recorded for the next phase (712 from Google Play, 191 from iTunes).

Phase 2: Identify Potentially Relevant Apps

A manual review of the apps’ titles and short descriptions was conducted collaboratively by two researchers to ensure that the apps were CBT-based and designed for GAD and/or worry. Duplicate apps and apps that were identified to be not relevant to the research question were excluded. The exclusion criteria were: (1) Apps that were not relevant to the research such as mobile phone themes, games, podcasts and music; (2) Apps that stated therapeutic theories or frameworks other than CBT; (3) General single purpose apps that only offer one therapeutic strategy/tool; (4) Apps that targeted mental health and/or anxiety generally (i.e., not specific to GAD) or included comorbidities such as depression, mental health conditions other than GAD and worry; (5) Apps that had limited availability (e.g., accessed through an employer or insurance); (6) Apps that were originally identified in the first search, though not found in the app store in the second phase of the review; or (7) Apps not available in English. As shown in Figure 1, the list of 903 apps was reduced to 26 potentially relevant apps that were identified for further study (16 Google Play, 10 iTunes).

Phase 3: Identify CBT Apps Targeting GAD and/or Worry

The 26 apps included in the final step were assessed in detail for the presence of CBT content in the apps (Table 1). This step was completed by two researchers with graduate training in CBT for GAD (Monique Jones and Lee Kannis-Dymand). In this stage, only 8 of the 26 apps were identified as using CBT and stating to assist GAD and/or worry. Four apps self-identified as CBT apps (2 from Google and 2 from iTunes). Another 4 apps (2 from Google Play and 2 from iTunes) that clearly depicted CBT strategies and/or tools in their descriptions or screen shot displays, were identified, and included in the study. As shown in Figure 1, in the final sample, 6 apps were included in the current study for the analysis, as one app would not function when downloaded, and one was found to be a duplicate, in further review stages of analysis.

Figure 1. Data Search and Exclusion Procedure.

Note. Separate searches were initially conducted for “generalised anxiety disorder” and “generalized anxiety disorder”.

Analysis of the Therapeutic and Engagement Features of the Apps

Design of the App

Functionality of the apps was assessed by downloading and viewing the app and rated by (1) the therapeutic value of the app (i.e., has best practice or recommended CBT features, has strategies to reduce anxiety symptoms). This included CBT therapeutic features and other therapeutic features that were identified but not congruent to CBT principles. Then, (2) features that would engage the user and increase the likelihood of future use of the app (purposeful functionality that may increase the appeal of the app content or encourages engagement in the app and/or the therapy within the app).

Presence of Best Practice CBT Content Within the Apps

Best practice, evidence-based CBT criteria, to review the apps against, were drawn from the United Kingdom’s National Institute for Health Care Excellence (NICE; 2019) guidelines and its associated resources (National Collaborating Centre for Mental Health; 2011) and references (Barlow et al., 1992; Borkovec, 2006; Borkovec & Ruscio, 2001; Borkovec & Whisman, 1996; Borkovec et al., 2002; Craske et al., 2006; Dugas, 2002; Dugas & Robichaud, 2007; Robichaud & Dugas, 2006, 2019; Wells, 2010; Wells et al., 2010). The criteria included 26 CBT items that address aspects of GAD, for example, self-assessment, instruction of strategy use, setting user goals, psychoeducation about GAD/anxiety/worry, and information about CBT. The full list of therapeutic items is shown in Table 1. Other CBT criteria, not specific to GAD, were also recorded (and shown in Table 2), for example, general CBT information and distraction.

Two researchers with extensive training and practitioner experience of CBT for GAD conducted the next phase to determine if the proposed CBT focus within the apps were congruent with best practice CBT guidelines for GAD. A clinical psychologist (Lee Kannis-Dymand) and a mental health nurse (Monique Jones); both with postgraduate qualifications in CBT and clinical experience working at a specialised anxiety disorders service in New Zealand. There were two apps removed from the analysis at this stage. One app (Beat Anxiety from Google Play) malfunctioned when downloaded and was inaccessible for analysis. The other app was found in both Google play and iTunes stores with minimal difference between their interface, content, and function, as such, this was analysed as one app. The researchers jointly reviewed and discussed each app (i.e., thoroughly went through the app’s menu and strategies) to determine the therapeutic features offered and how these matched with best practice CBT criteria for GAD, and if there had been CBT expert involvement in the development of the app.

Identification of Engagement Functions of Apps

Based on the engagement features defined in the Stawarz et al. (2018) study and additional relevant research (Muroff & Robinson, 2020; see Table 4), a manual review of the CBT apps was conducted by a researcher (Teresa Munteanu). This was conducted by a researcher who was not an expert in user experience to ensure that engagement features were easily identifiable and would be discoverable by a user/consumer from the public, rather than someone who was an expert in user experience or user design.

Users’ Experiences of the Apps

The final step of the analyses was to qualitatively assess publicly available user reviews of the apps. Explorative research of publicly available app reviews has been previously conducted to gain an understanding of users’ opinions of apps and their features (Caldeira et al., 2017; Shen et al., 2015; Stawarz et al., 2014, 2018). Similar methodology was utilised in this study to explore users’ opinions and attitudes of the CBT apps. Reviews were manually retrieved from each app store with the date, star rating and review comment recorded. Where apps were available on both Google and iTunes, reviews were retrieved from both stores. Visible iTunes reviews on mobile devices were limited and only one iTunes app had a substantial number of reviews. An explanation for this is the iTunes policy that removes reviews from previous versions of an app, unless otherwise requested by the developer. There were 129 comment reviews in total (77 from Google and 52 from iTunes).

Data Analysis

Data analysis followed the study’s main aims, being identification of apps’ therapeutic and engagement functions. The frequencies of the criteria for CBT that apply specifically to GAD, non-GAD-specific CBT, and non-CBT strategies in each app were tallied, with definitions for each criterion. Qualitative analysis of the users’ comments used Braun and Clark’s (2006) stages of thematic analysis. Microsoft Excel was used to organise data and all comments were reviewed independently by two researchers (Lee Kannis-Dymand, Teresa Munteanu). After the initial coding and development of themes, the authors began a collaborative review and discussion of coding, thematic definitions, and discrepancies between the initial findings. This review process enabled greater data familiarity and reduced potential for bias due to high levels of agreement between the coders. Themes were generated and driven by the explicit meaning of the users’ words and the features to which the comments applied (e.g., usefulness of CBT features, ease of app’s technology and user interface); correspondingly, themes and what they encompassed was informed by prior research that identified core themes in user experience of mental health apps (e.g., Stawarz et al., 2018). Themes were generated, reviewed, refined, and defined in relation to the overall data by the two researchers, until clear themes were developed and identified (Lee Kannis-Dymand, Teresa Munteanu).

Results

Identification of the Apps

Following the three phases to identify apps for GAD and worry with a CBT basis (Figure 1), six apps were included for the functional analysis. These apps were, in alphabetic order, from Google (G) and/or from Apple iTunes (A): (1) FearTools (G and A), (2) HappiMe for young people (A), (3) Shrinkinabox (A), (4) WorryTime by ReachOut (A) and ReachOut Worrytime (G) which are the same app with different names, (5) WorryTree (G and A), and (6) Worrywatch habit tracker (A).

Design of Apps and Their Functionality

The analysis of identified CBT apps for GAD and worry found 39 therapeutic features and 13 engagement features, with Fear Tools having the most evidence-based features (10/30) and most non-CBT features (6). The therapeutic features and their prevalence in the apps are detailed in Tables 1 to 4.

Therapeutic Features of CBT Apps for GAD and Worry

As shown in Table 1, of the 26 GAD specific items within the guidelines, 9 were not found within any app. All identified apps included some of the 30 evidence-based features (see Tables 1 & 2), ranging between 2 and 10 features (M = 6.8, Md = 7.5, Mo = 8). All apps were found to include non-CBT therapeutic features ranging from 1 to 6 features (mean = 2.65, median = 1.5, mode = 1; see Table 3). The most frequently identified therapeutic features found across the apps were instructions for a CBT strategy (5/6, 83%), identifying negative or unhelpful cognitions (4/6, 67%), and addressing negative automatic thoughts (4/6, 67%). Fear Tools had the most evidence-based features (10/30), along with the greatest number of non-CBT features (6). Overall, the data uncovered that the apps contained a limited number of CBT strategies, and no app had a complement of strategies consistent with a full GAD treatment protocol (see Tables 1–3 detailing which apps had which strategies).

CBT features were also considered in comparison to app user ratings, although one app did not have a star rating (Shrinkinabox). The average of user star ratings (5-point scale) of apps ranged between 4 and 4.8. For those apps with star ratings, the app (Worry Watch) that had fewest number of CBT features (6) had an average star rating of 4.6 and the app (Fear Tools) with the most CBT features (10) rated 4.2 on average.

Table 1. Therapeutic Features: GAD-Specific CBT Features Within the Apps, With the Number and Names of Apps in Which the Features Appear.

 Table 2. Therapeutic Features: Non-GAD-Specific CBT Features Within the Apps, With the Number and Names of Apps in Which the Features Appear.

 Table 3. Therapeutic Features: Non-CBT Techniques With the Number and Names of Apps in Which the Features Appear.

Experts and App Development

Each app was investigated for the involvement of one or more CBT experts in the development of an app’s content (i.e., a person publicly listed on that app was a mental health professional or noted as having CBT training or experience). The apps involving CBT experts (4/6, 67%) were found to hold between 2 and 10 CBT features (N = 4, M = 6.75, Md = 7.5), whilst apps without CBT expert involvement ranged from 6 to 7 CBT features (N = 2, M = 6.5). The app with the least CBT features (Shrinkinabox) and the app with the most CBT features (Fear Tools) both included experts in their development, although exact numbers of experts involved in each app were unclear.

Table 4. Engagement Features Within Identified CBT Apps, With Numbers of Apps in Which the Features Appeared.

Engagement Features

As shown in Table 4, features that were considered to increase, sustain and/or encourage engagement within the app itself or the therapy offered within, were recorded. All apps offered some aspect of customisation, and most contained a degree of features available offline. Notifications and reminders were found in the majority of apps, while additional support such as weblinks, data shareability, security features, and video and/or other informational content were present in 50% of the apps. A minimal number of apps were found to have the ability to add user content, employed gamification, or contained audio content, graphs or statistics of changes or progress, or additional text content such as blogs.

Analysis of User Reviews of CBT Apps for GAD and Worry

When applicable, user reviews were retrieved for both Google Play and Apple iTunes versions of the identified apps (N = 120), as shown in Table 5. Three Apple iTunes apps did not have written reviews (Worry Tree, Shrinkinabox, Happime). Eight comments that were either of a general nature (Thank u; Great app love it so much), unclear if referring to functionality or therapeutic aspects (Does not work), or not specific to the use of the app (Feeling ungood at being fine) were excluded. Due to some apps being available on both platforms the resulting sample was seven apps (4 Google play, 3 iTunes) with 112 reviews for analysis. Of the reviews, 69.64% (78/112) were positive, 14.29% (16/112) were negative, 3.57% (4/112) were neutral, and 12.5% (14/112) were mixed. Three key themes of user experience were identified across all the user comments: technological experience, global experience, and therapeutic experience. The components of the themes of user experience are shown in Figure 2 and are explored in more detail below. The examples comments given relate to the predominant theme of the user’s comment, however, it is acknowledged that comments could relate to several themes.

Table 5. Numbers of Comments and Ratings for Each Individual App.

 Figure 2. Thematic Map of User Reviews.

Technological Experience

Throughout the theme of technological experience, there were 2 sub-themes uncovered: App operation and App features.

App Operation

A small group of reviews focused on the operation of an app overall. Comments tended to be negative as the use of the app was perceived to be due to development issues: “New interface is unusable. The new design is so confusing and cluttered that you can’t even operate the app. And you can’t come up with a logical interface, definitely don’t fill half the screen with upbeat mottos! A mess.” (Worry Watch)

“Where there was overall dysfunction of the app, several users indicated the negative impact this may have on their mental health: “Great way to burn $2.49. Crashes on my iPad touch. Never got it open. Actually increased my anxiety as a result. Such a shame.” (Worry Watch)

“Instructions? All I get is a blank page. Not sure this soothes my anxiety.” (Worry Watch)

App Features

Most comments relating only to a user’s technological experience mentioned app features. These comments focused on feature requests, such as: “It’s ok for when you [are] at home but why not have the meditation downloadable so you can use while you are out without data or wifi.” (Fear Tools: Anxiety Aid)

Additionally, customisation requests of features were common: “My worries usually pop up during night time and I find it very unpleasant to my eyes. Please add dark mode Edit: Still no dark mode.” (Worry Tree)

Among these customisation comments there were some that specifically mentioned security and privacy features as important: “I like the fact you can add a PIN. The UI [user interface] is also amazing (Worry Tree), and …I like that you can’t create an account so they’re not storing your data anywhere (hopefully) and it’s securely…” (Worry Watch)

Global Experience

This theme captured comments that described a mix of features and experiences and was found to span the spectrum of positive to negative sentiments. Comments described the importance of the combination of technological and therapeutic experience: “Love this app!! Simple. Effective. Easy to use and making entries is quick. Has definitely helped me put life things in perspective! Interface is clean and professional. Has not crashed. Highly recommended.” (Worry Watch)

Another comment highlighted that apps could assist in accessibility to therapy and therefore necessitate reliability:

“Ok so I have no idea what to do because every time I try to type in the Situation in the Thought Diary, it glitches then kicks me out of the app! My counselor recommended this app because I can’t go there often because of COVID-19 and it’s super hard for me to talk about my anxiety so please fix that.” (Fear Tools)

A single comment suggesting the therapeutic experience was not useful, also commented regarding the technological experience: “Star for graphic layout and for idea. No stars for delivery and execution. not actually assisting at all. Too basic.” (Worry Tree)

Even when users indicated that the app was therapeutically useful, engagement and therapeutic feature suggestions were common:

“Awesome with room to grow! I have used this app for more than two weeks and it has been very helpful in helping me identify and get in control of my anxiety… This app is not cloud-based so I’m concerned that my saved information will be deleted if I need to uninstall and reinstall this app.” (Worry Watch)

Therapeutic Experience

A large proportion of comments focused solely on the therapeutic experience of the identified apps; comments were almost exclusively positive. The sub-themes identified within this theme were: General comments of app helpfulness, app usefulness specific to anxiety and worry, and specific therapeutic strategies.

General Comments of App Helpfulness

General comments regarding the therapeutic helpfulness of an app mentioned that the app helped them feel active in their mental health: “Definitely helps reading through this APP! It relates to my circumstances really well and helps me handle the smaller things in life and be able to tackle anxiety on my own! Great app!” (Beat Anxiety)

Or assisted the user to enact change for their mental health: “Super simple yet super helpful. It helps me stay calm and realize that everything will be okay if I just take it slowly. It really deserves more reviews than it has, it’s an amazing app.” (Worry Tree)

Usefulness Specific to Anxiety and Worry

Many specifically mentioned a positive impact on their experience of anxiety and/or worry through the use of the app: “Super helpful!. This app really helps me sort out my worries and how I’m feeling. It’s really helped me manage my anxiety. I highly recommend downloading it!” (Worry Watch)

Specific Therapeutic Strategies

Whilst there were limited CBT features within the apps, many users stated their appreciation of CBT-specific strategies, such as thought diaries (first quote) and postponing worry (second quote): “I think the thought diary thing is an excellent thing...I just thought I was going to be writing my bad day down but it actually really helped.” (Fear Tools: Anxiety Aid); “Helpful. Very helpful to unload worries onto this list / app and then review the outcomes later. Helps put things in perspective.” (Worry Watch)

Some related these strategies to other therapeutic experiences in the user’s life:

“Went alongside my therapy perfectly, GAD tests very similar to what I went through with my therapist, and the app is a fantastic reminder of helpful and unhelpful thinking. Brilliant! Great tool for anyone going through or has gone through therapy.” (Fear Tools)

Discussion

The current study evaluated GAD apps, for their use of CBT, evidence-based content, and the nature of user reviews. Despite the large number of available apps (N = 1,888) proposing to assist with GAD and worry, the current study found only a small proportion of these targeted GAD (n = 26), and only 6 apps had CBT elements. This is not dissimilar to a study on apps for anxiety (Balaskas et al., 2021) that found few evidence-based CBT apps for anxiety. In the present research, the most common CBT features within the apps included identifying negative or unhelpful cognitions, addressing negative automatic thoughts, as well as instructions for using the various CBT strategies within the app. Although experts were involved with 4 of the 6 apps, the apps still varied greatly in how many CBT features were included. This highlights the difficulties the public may have in finding an app specific to GAD that provides quality, evidence-based, therapeutic strategies and supports previous research that has found that few mental health apps have a strong evidence base, including apps for GAD (Kertz et al., 2017; Marshall et al., 2020; Sucala et al., 2017; Wasil et al., 2019). Analysis of the engagement features found a limited range of features, with customisation and offline availability being the most common. The themes of ‘technological experience,’ ‘global experience’, and ‘therapeutic experience’ showed that both technological experience and therapeutic content were valuable to users’ overall experience, coherent with prior mental health app research (Alqahtani & Orji, 2019; Nicholas et al., 2017; Oyebode et al., 2020; Thach, 2018). Thus, apps that are grounded in CBT and have a ‘good’ user experience may be more likely to be seen as beneficial by users.

Design of the Apps

All apps in this study included CBT features; albeit few. The apps also included non-CBT based features, consistent with previous findings (Stawarz et al., 2018). While non-CBT features may include content that is engaging (games), non-CBT features fall outside of best practice guidelines and may or may not be consistent with CBT strategies for GAD. For example, there is insufficient evidence supporting the emotion freedom technique for GAD or indeed anxiety, yet it is being employed in apps. The lack of evidence-based therapeutic features within the apps of this study is consistent with prior, abundant research raising this concern (Alyami et al., 2017; Firth, Torous, Nicholas, Carney, Pratap, et al., 2017; Lui et al., 2017; Rathbone et al., 2017; Shen et al., 2015; Stawarz et al., 2018; Wang et al., 2018). This is disconcerting as the aim of quality self-help CBT apps is to convey core CBT principles that are consistent with therapist guided CBT such as, psychoeducation, self-monitoring, cognitive and behaviour change, problem solving, communication skills, and relapse prevention (Newby et al., 2021). Within the apps espousing they were CBT driven, few CBT strategies were offered, and often presented in isolation or lacked a comprehensive framework. That is, strategies lacked a coherent structure wherein one strategy should follow on from another in a logical, systematic, and therapeutically beneficial manner, as you would experience within a CBT treatment protocol for GAD (Borkovec et al., 2002). For example, no app offered a version of a CBT model or formulation for GAD. Formulation of a GAD CBT model is essential in therapy (Fenn & Byrne, 2013; Zivor et al., 2013), to outline maintaining factors of worry/GAD and orientate and socialise the user to the importance of the suggested CBT strategies. Other effective core interventions such as behavioural experiments, behavioural activation, metacognitive beliefs about worry, worry postponement, and problem solving were startlingly minimal or absent.

The limited inclusion or absence of CBT features may potentially be linked to contributions of CBT trained experts to the development of these apps. It was difficult to find information on who the relevant experts were and their training, which reflects findings of previous research (Oyebode et al., 2020; Stawarz et al., 2018; Sucala et al., 2017). This highlights assertions from earlier researchers that app developers should adhere to evidence-based guidelines, involve mental health professionals and mental health consumers in their development, and the necessity for regulation for standardised, evidence-based evaluations of the apps and their efficacy (Alyami et al., 2017; Oyebode et al., 2020; Sundram et al., 2017; Van Ameringen et al., 2017). Currently, consumers are left unable to determine which apps are based on evidence-based interventions to offer them any therapeutic benefit (Sucala et al., 2017). This was reiterated by Neary and Schueller (2018, p. 532) who stated, “With a bewildering abundance of apps available, clinicians and consumers will continue to face challenges in attempting to discern which are most effective, usable, engaging, or safest; how do we separate the good from the bad?”

Despite the minimal number of CBT strategies with the apps, a large proportion of the user reviews indicated these strategies were positive and beneficial, consistent with previous CBT app research (Stawarz et al., 2018; Thach, 2018). The value of including CBT, self-directed activities in the apps was noted within the current sample; this is important as CBT principles have been shown to empower individuals to understand their challenges and build skills to cope and effect change (Newby et al., 2021; Newman et al., 2020).

Given the user reviews regarding a sense of control gained through the use of strategies within the apps, it may be reasonable to consider that many users experience a sense of empowerment with just one or a few strategies. However, the longevity of gained benefits needs to be considered, as some research suggests that improvements gained through CBT apps are not maintained long term and may decline with the cessation of app use (Rathbone et al., 2017). The apps within this study had low therapy protocol adherence; consequently, lasting therapeutic gain may be questionable. Future research should seek to resolve the seeming contradiction between the limited CBT content and higher user ratings in apps, by using longitudinal designs to assess how and by how much users improve whilst using the apps for longer periods.

Mental health apps can be employed in many situations such as preparation for, or as an adjunct to, therapy, relapse prevention, as well as a standalone self-help therapy (Newby et al., 2021). Similar to Thach (2018) and Starwaz et al.’s (2018) finding that apps can augment therapy, another explanation for the perceived benefit of the apps included in this study is that some users utilised the apps as part of, or as an extension to therapy, as some indicated the similarities of features to therapeutic experiences. The tools within the apps may serve to support, remind, or enhance skills that have been acquired or are developing in therapy, therefore, the apps are not expected to be a full therapeutic intervention.

Irrespective of the number of CBT features, all apps were found to have a high user rating, mirroring previous research (Kertz et al., 2017; Stawarz et al., 2018; Wasil, 2019). Notably, Kertz’ study on anxiety and worry apps revealed that 50% of the apps had no CBT content but higher user ratings. Commonly, users consult star ratings and reviews when deciding to download apps (Mudambi & Schuff, 2010), therefore, the high star ratings for all apps in this study may be problematic for individuals looking for evidence-based apps. Star ratings are generally considered to represent the quality of the product; however, these rely upon a large number of reviews to be a meaningful communication (Tay et al., 2020). Star ratings influence the perceived credibility of a product, with high star ratings more often trusted when there are also positive written reviews (Hong & Pittman, 2020). Most of the apps in this study had small review numbers, a high proportion of positive comment reviews, and low therapeutic evidence-based features, these high star ratings may extend the difficulties of identifying apps to provide quality therapeutic support. This may place the users of these apps, who are a potentially vulnerable population, at a disadvantage as they may believe they are receiving a high quality, comprehensive CBT intervention.

The analysis of engagement features found a variable, sometimes limited number of features, similar to previous research regarding CBT apps for anxiety and depression (Balaskas et al., 2021; Stawarz et al., 2018). Interactive and tailored strategies, along with supportive (guided self-help) and social (community and/or peer connection) strategies have been identified as important in the role of engagement in online mental health interventions (Balaskas et al., 2021; Doherty, 2012; Nicholas et al., 2017). Whilst many apps contained information on how to connect to therapists or support services, no supportive or social strategies, such as peer connection forums, were identified within the apps. This may be appropriate as these apps are designed as self-help interventions and would require supervision of mental health professionals if including peer interaction.

Self-management features, such as monitoring/tracking symptoms or activity, and graph or chart features that demonstrate progress or changes, allow for interactive engagement in an app and have been found to be one of the most positively viewed features in other mental health apps (Muroff & Robertson, 2020; Thach, 2018). Balaskas and colleagues (2021) found that apps for anxiety with these self-management features were more frequently used, however, we found these features were limited within the apps of this study. Additionally, notifications, reminders, and gamification are considered forms of customisation that can substantially increase motivation and engagement in app use. Customisability has been positively reviewed and noted as a requested feature in previous mental health app research, as well as multiple customisations being found to be beneficial (Chandrashekar, 2018; Muroff & Robertson, 2020; Nicholas et al., 2017; Oyebode et al., 2020). While, customisations were the most frequent features found, multiple customisations within an app were found in just 2 apps and only one app employed gamification.

User Experience

User experience is known to impact the efficacy of an app (Baumel & Kane, 2018; Rubanovich et al., 2017; Doherty et al., 2012). Coherent with existing research, user reviews illustrated that operational dysfunction of apps can negatively impact user experience and individual’s mental health, and that app features including security and privacy, and the customisation of features are valued (Alqahtani & Orji, 2019; Nicholas et al., 2017; Oyebode et al., 2020; Thach et al., 2018). Therapeutic features were identified to assist users in feeling active and in control of managing their mental health generally, and some with their anxiety and worry in particular. CBT strategies were often highlighted as benefits of the app, with users relating them to therapy experiences, such as strategies used with a therapist previously. The largely positive therapeutic and global experience comment reviews emphasise that many perceived a therapeutic benefit through the use of an app. Even though evidence-based features were minimal, the perceived benefit from the strategies that were present may explain the high user review ratings reported here and in previous studies (Kertz et al., 2017; Stawarz et al., 2018; Wasil, 2019). Whilst minimal engagement features were identified within the apps, users highlighted these features were wanted and important to their experience. Concordant to research on apps for various mental health conditions (Alqahtani & Orji, 2019; Nicholas et al., 2017; Oyebode et al., 2020; Stawarz et al., 2018; Thach et al., 2018), we found that the user reviews for CBT GAD apps identified that the functioning of both therapeutic and engagement features impact on user experience, including access to support and their engagement in the use of the app. Considering therapeutic benefits may not be retained after discontinuation of app use (Rathbone et al., 2017), and engagement features encourage, and motivate app use (Chandrashekar, 2018), the inclusion of quality therapeutic and engagement features may be crucial to the effectual long-term use of an app for mental health support.

Limitations

A number of limitations need to be considered when interpreting our findings. First, the small number of apps suitable for inclusion limited the breadth of analysis and reflects the limited availability of CBT-based apps for GAD and worry. In addition, other therapeutic interventions, such as mindfulness, were not included in this review due to not being included in best practice guidelines for GAD. Second, the diagnoses of the users whose data was captured in this study was unknown; that is, while this research focused on CBT apps for GAD, the number of users who experience GAD in this sample was not identified. Further, few apps had substantial user reviews available, which may limit the generalisability of the resultant themes. Further, it is important to note that despite two researchers coding themes, that qualitative analyses of user reviews are limited by potential subjective biases. Assessing reviews of apps and their CBT content within the context of the apps’ costs (i.e., free versus requiring payment) was also beyond the scope of the current paper. Evaluation of engagement features was conducted by a researcher who was not a user experience researcher, arguably this may have influenced the identification of some of the apps’ engagement features. Finally, this study relied on publicly available information on app developers and anonymous user reviews rather than interviews with users, and such information and reviews may be biased or fictitious. Whilst analysis of app store reviews can expand the understanding of mental health app use, and the users’ needs and user experience (Baumel et al., 2020; Calderia et al., 2017; Stawarz et al., 2014), interviews of app users may yield greater knowledge of these aspects through purposeful and defined questioning.

Conclusions and Future Directions

The current study found there are few CBT-based apps commercially available to support GAD and worry, with limited features that promote engagement. However, users identified that therapeutic and engagement elements were appreciable to their experience of an app. The future development of apps for GAD should consider evidence-based content in the context of engaging, easy-to-use, and reliable design features to ensure a quality user experience. Additionally, user experience including an app’s design and interface needs to be prioritised. Notably, Alqahtani et al. (2019) and Oyebode et al. (2020) provide excellent guidance on improving mental health apps’ content and useability such as participatory design, high quality evidence-based content, reminders to complete therapeutic tasks, simplicity and ease of use, allowing users to customise (e.g., font size and colour, layout), ability to back-up data, data privacy, and guides on how to use the app, and regular app updates to ensure quality, up-to-date content and technology. Another area for consideration is whether apps that are advertising as CBT-based need to be regulated to ensure these claims can be substantiated and are not misleading. Further, as highlighted by Balaskas et al. (2021), an app’s consistency with evidence-based strategies does not ensure the efficacy and effectiveness of intervention elements delivered within an app. Thus, future research should consider randomised control trials to investigate the overall user experience, including long-term efficacy of symptom reduction and/or mental health maintenance, as well as comparing functionality elements to evaluate user engagement and the effect on user evaluations of an app.

Research should establish what percentage of CBT strategies makes an app compliant with best practice and what level of use is effective in reducing symptoms (i.e., like treatment adherence in clinical practice). Further, given a meta-analysis found that transdiagnostic CBT demonstrated effectiveness similar to disorder-specific CBT in the treatment of anxiety disorders (Pearl & Norton, 2017), it is worth considering if apps that are non-disorder specific to GAD (i.e., transdiagnostic) are effective for GAD. Likewise, if apps that target stress or anxiety, regardless of diagnosis, are effective for people experiencing a range of anxiety conditions, depression, or stress. Finally, the ways in which apps are used, being self-directed, an adjunct to therapy, or post therapy, are important for future research, along with the technological experience that enhances the efficacy of these apps. The ubiquitous nature of smart phones and the easy access to apps means that therapeutic assistance can be just a click away. Therefore, apps that are evidence-based and offer a high-quality user experience are much more likely to be able to offer assistance in a timely manner and contribute to better mental health and well-being. 

Conflict of Interest

The authors have no conflict of interest to declare.