Appropriate controls for digital therapeutic clinical trials: A narrative review of control conditions in clinical trials of digital therapeutics (DTx) deploying psychosocial, cognitive, or behavioral content

Digital therapeutics (DTx) are software programs that treat a disease or condition. Increasingly, DTx are part of medical care, and in the US healthcare system they are regulated by the FDA as Software as a Medical Device (SaMD). Randomized controlled trials (RCT) remain a key evidence generation step for most DTx. However, developing a unified approach to the design of appropriate control conditions has been a challenge for two main reasons: (1) inheriting control condition definitions from pharmacotherapy and medical device RCT that may not directly apply, and (2) challenges in establishing control conditions for psychosocial interventions that build the core of many DTx. In our critical review we summarize different approaches to control conditions and patient blinding in RCT evaluating DTx with psychosocial, cognitive or behavioral content. We identify control condition choices, ranging from very minimal digital controls to more complex and stringent digital applications that contain aspects of “fake” therapy, general wellness content or games. Our review of RCTs reveals room for improvement in describing and naming control conditions more consistently. We further discuss challenges in defining placebo controls for DTx and ways in which control choices may have a therapeutic effect. While no one-size-fits-all control conditions and study designs will apply to all DTx, we propose points to consider for defining appropriate digital control conditions. At the same time, given the rapid iterative development and optimization of DTx, treatments with low risk profile may be evaluated with minimal digital controls followed by extensive real-world effectiveness trials.

Introduction

Digital therapeutics (DTx), a class of software-based (internet- and/or app based) technologies, aim to directly prevent, manage or treat health conditions (1). DTx often translate face-to-face treatment modalities, such as cognitive behavioral therapy (CBT), into mobile application or web-based interventions. Self-guided internet-based cognitive behavioral therapy programs (iCBT) have been studied extensively (2), but many DTx technologies contain treatment principles beyond iCBT, such as cognitive training components or symptom tracking/medication adherence. DTx have the potential to revolutionize modern medicine by improving access to evidence-based, personalized treatments that may for some indications even evolve into first-line therapy. As such, DTx - whether as standalone therapy or adjunctive to medications, devices, or other therapies - have shown promise for a broad spectrum of medical conditions ranging from mental health and behavioral health conditions (3) (e.g., depression (4), insomnia (5), post-traumatic stress disorder (PTSD) (6), attention deficit hyperactivity disorder (ADHD) (7), and substance use disorders (8)) to managing conditions such as diabetes (9), irritable bowel syndrome (10) or post-stroke aphasia (11).

At the same time, the relatively novel field of DTx is still in search of common standards for how to generate evidence necessary to support medical claims and regulatory clearance (12). Here, we will discuss the concept of control conditions in randomized controlled trials (RCT) in the context of pharmacological and psychotherapeutic trials, and within the FDA regulatory framework for Software as a Medical Device (SaMD) that DTx fall under. We will then review and discuss typical control conditions that DTx companies have used in their RCTs to date.

Randomized controlled trials and control conditions

RCTs are considered the “gold standard” to evaluate therapeutic interventions (13). In RCTs, participants are either allocated to the active treatment or a control condition. Different control conditions can be distinguished based on their stringency, what biases they control for. The control condition a treatment is compared to is therefore a crucial choice as it directly impacts the measurable treatment effect, with less stringent control conditions leading to larger effect sizes.

In the early clinical development phase, control conditions are often less stringent. For example, waitlist or no intervention control groups have been used. These will control for effects related to the natural progression of the illness, regression to the mean effects (14), effects from regular social interaction with study trial personnel, or effects from other study procedures such as being assessed repeatedly or simply from feeling observed during a trial (i.e., Hawthorne effect) (15, 16). However, studies employing these control groups are not blinded, and thus do not control for the expectation of benefit.

 In later clinical development phases, more stringent placebo control conditions are typically used. In pharmacological trials, placebo pills look exactly like the active treatment, but do not contain any active ingredients. Participants are told during the informed consent process that they will be randomized to either the treatment or placebo condition, and if blinding to the assignment was successful, the placebo pill condition controls for any bias due to any non-specific mechanisms beyond the physiological effects that the active drug ingredients have on the individual (17). Thus, the placebo pill will also control for expectation of change during a trial (see Table 1 for definitions).

TABLE 1

Table 1. Definitions.

Control conditions for psychological interventions are more challenging to design for a few reasons. First, while CBT and related psychosocial therapies are regarded as generally efficacious (18), the exact mechanisms that produce therapeutic effects in cognitive and behavioral therapies are not fully understood (19). This makes it challenging to define which treatment components should be included into the design of a control condition that is not intended to produce a therapeutic effect, similar to a placebo (sometimes referred to as attention control in psychotherapy research). Similarly, the relative contribution of factors shared across therapies, referred to as common factors (20) (e.g., therapeutic support, positive expectations of treatment benefit, hope, structure provided by the treatment (21)) is relatively large across different psychological interventions compared to specific treatment effects on disease symptoms (22, 23). In fact, a supportive therapist-client relationship would be considered part of the placebo control (often referred to as attention placebo control in this literature) in the context of a pharmacological trial, but the therapeutic alliance has been suggested as a common efficacy factor in psychotherapy¹². Thus, a control condition which establishes structured and supportive therapeutic alliance already contains major efficacy factors of psychotherapy and will therefore not be inactive. On the flip side, trying to design placebo therapies with low or no active ingredients often result in therapies with low face-validity and the risk to unblind all or at least some participants to study assignment (22).

Because DTx often translate face-to-face treatment modalities, such as CBT, into mobile application or web-based interventions, they inherit the described challenges in defining appropriate controls of psychological interventions in general. In addition to disease specific therapeutic content, many DTx integrate disease management features (e.g., disease/symptom tracking, behavior tracking, goal setting and tracking, community support, disease related psychoeducation or general lifestyle and wellness content), which have been studied individually and shown positive effects for some indications (24–26). DTx may also deploy cognitive remediation and cognitive training principles to strengthen aspects of cognition, such as cognitive control or emotion regulation (27, 28). See also Table 1 in the Supplementary Material for a description of typical DTx treatment components. DTx that contain several interacting components as described above – i.e. disease specific therapeutic content, additional disease management tools, cognitive interventions and even coaching and text messages – often present complex interventions, and the particular challenge of studying complex interventions is well described and links back to a good understanding of the treatment mechanisms under investigation (29).

Some authors argue that designing a placebo control for DTx is easier compared to traditional face-to-face therapy since there is no human therapist interaction that could confound trial results (14). However, there may be digital placebo mechanisms beyond human interaction, whose impacts are not fully understood, such as beliefs about technology or the feeling of being connected to coaches or a therapist through using a digital health app (30). In addition, beyond typical placebo effects, there is currently a lack of understanding of efficacy related to the digital mode of delivery, such as the intervention structure, engagement support, or regular, focused time spent in the app (31). Most DTx recommend regular, often daily use (27, 32, 33), and deliver engaging experiences (e.g. gamification (27, 34) or social support (33)) to support adherence. However, such elements may share considerable overlap with disease specific therapeutic techniques. For example, the regular interaction with an engaging DTx may share elements with behavioral activation, a therapeutic activity with the goal of exposing patients to pleasurable activities that has been shown to be an effective treatment across a number of diseases (e.g. depression (35, 36), chronic pain, personality disorders, and schizophrenia (37)). Another example is the regular, scheduled interaction with a DTx, which overlaps with structuring the environment, a component of treatment for children with ADHD (38). Finally, if the DTx deploys game elements, additional therapeutic mechanisms could lie in distraction from negative affect or induction of positive affect through mastery and flow (39, 40). In summary, providing structure and creating positive engaging experiences and rewards may improve symptoms in addition to the specific therapeutic content. And while it is in the interest of patients to optimize and fully harness such mechanisms, this means that digital control conditions, which employ structurally equivalent protocols and engagement features may not be truly inert but have some therapeutic efficacy.

Additional control considerations related to US/FDA regulations

DTx seeking FDA clearance fall into the category of SaMD and undergo testing and clinical validation like traditional medical devices. Several DTx have been authorized or cleared by the FDA as SaMD (e.g. reSET (8, 45), reSET-O (46), Somryst (5), EndeavorRx (27), Parallel (47)) and several digital health companies have communicated pursuing FDA clearance (e.g. Click Therapeutics (48), Kao Health (49), Posit Science (34), Wise Mind (50)).

The FDA provides guidance on suitable control conditions and blinding for evidence generation. The FDA describes a “sham”, the medical device equivalent of a placebo control in pharmacological trials, as being “an ineffective device (or simulated procedure or possibly a drug or biological product) used under conditions designed to resemble the conditions of use under investigation as far as possible” (42). In the case of SaMD, where treatment is based on software only, creating the equivalent of a placebo control poses particular challenges, a challenge even acknowledged by the FDA, which states that “it may be challenging to construct a placebo control that appears to function like the investigational device but delivers no therapy” (51). Specifically, it may be hard to keep appropriate face-validity to ensure patient blinding and comparable engagement with a placebo control while creating something that is “ineffective” (i.e., delivers “no therapy”). The FDA also outlines several other types of control groups, such as active controls (“an effective regimen of therapy may be used for comparison”. However, proving statistically that a treatment performs similar to a standard treatment (non-inferiority analysis) is challenging in practice because they rely themselves on strong historical placebo controlled RCT data to inform the non-inferiority margin and require larger sample sizes (52, 53). Taken together, the FDA guidelines applicable for SaMD follow typical medical device settings, which usually contain a hardware component, without concrete recommendations for control conditions in software only SaMD interventions. We expect additional guidance specific to SaMD may be issued by FDA and other regulatory bodies over time as best practices and standards emerge from the DTx industry, as the published literature evolves on the topic, and as further regulatory precedents are established through additional SaMD marketing clearances.

In summary, challenges of selecting appropriate control conditions for DTx, stem from the breadth of underlying treatment principles and mechanisms, sometimes interacting in a complex nature, known challenges in designing placebo controls for psychological treatments, and the fact that no physical device can aid in the blinding of participants. Here we review control condition choices in this nascent field of DTx and how DTx companies define appropriate controls for software based medical devices.

Methods

In this narrative review, we explore control conditions in DTx RCTs deploying cognitive and/or behavioral therapeutic activities to manage or treat diseases. We reviewed the DTx Alliance product list (https://dtxalliance.org/understanding-dtx/product-library, accessed Jan 2022) to identify DTx that adhered to the DTx definition and core principles (eg. safety, efficacy, privacy, patient centricity). From this starting point, we focused on DTx which are purely software-based (internet or app) and do not contain additional physical devices for their functioning (e.g., wearables, inhalers, sensors), and where the primary activity was a cognitive or behavioral intervention. There were 10 products that fulfilled this criterion. From these, we explored phase 2 and 3 RCTs including those used to provide the evidence for FDA registration. In the case of several RCTs for a specific DTx, we focused on the most recent trial. To gain a broader picture of the fast-moving field, the authors added additional trials from DTx companies who planned and/or conducted phase 2 and 3 trials to support regulatory submission, based on their knowledge in the field of DTx and the companies press-releases (including a few planned, halted, or failed trials). For a full overview of reviewed DTx characteristics see Table 1, Supplementary Material).

Results

General summary

Fourteen RCTs were reviewed (Table 1 in the Supplementary Material). Sample sizes ranged from 80 to 1149. Most RCTs deployed 2 arms, but two studies ran 3 arms. Indications covered several DSM (54) categories (insomnia, schizophrenia, compulsive disorders, substance related disorders, depression, anxiety), neurodevelopmental disorders (ADHD) and physical disorders (diabetes, irritable bowel syndrome). A full overview of study characteristics is available in Table 1, Supplementary Material.

Summary of control conditions

Several control strategies have been employed. We found about half of the trials used unblinded waitlist or treatment-as-usual (TAU) control groups. The other half deployed different forms of sham controls. Only one trial deployed an active comparator and only as part of a 3-arm RCT including a TAU control arm for the main comparison.

Waitlisted RCT

We identified three RCTs in Depression, Generalized Anxiety Disorder, and Alcohol related disorder deploying a waitlist control and one planned study for body dysmorphic disorder (55), but none of these products have been FDA-cleared to date.

Treatment as usual RCT

TAU was chosen as the comparator in three studies. TAU was the main comparator arm in a 3-arm RCT of iCBT for Irritable Bowel Syndrome (IBS), the data from which was used to support the FDA clearance of Parallel (10). Regarding the control choice, the authors Everitt et al. (47) note that “blinding is not possible for psychotherapy studies” In addition to the active intervention and TAU conditions, the study included an active comparator group, which received weekly telephone-delivered CBT sessions (compared to the active intervention group, which received only minimal telephone-delivered therapist support in addition to the iCBT program). The primary outcome was a comparison between the active intervention and the TAU comparator group. While this comparison does not control for effects related to expectation of benefit, the design does allow comparing CBT for IBS across different forms of delivery. Similarly, two DTx that received FDA clearance as SaMD for the adjunctive treatment of opioid use and substance use disorder (8, 46) used TAU or reduced TAU as comparators (see Table 1 for a description of the reduced TAU versus full TAU condition) (8).

Sham control RCT

About half of the reviewed studies deployed a form of sham control. It is notable that different terms were used by the authors, from digital control, placebo, sham, and attention-matched placebo control reflect the different frames of reference for evidence generation – from regulatory to pharmaceutical or psychotherapeutic trials - discussed in the introduction.

Different approaches were used to design sham content: Sham controls either replaced core treatment activities with “fake, but plausible therapies” (33) or they delivered general, disease agnostic wellbeing tips (32, 56). Another strategy was to “disarm” certain content, which means removing key aspects hypothesized to drive efficacy. For example, psychoeducational content included in the DTx may be retained in the sham control, but related quizzes or concrete skill training based on the educational content were specific to the therapeutic (32). Finally, certain content and/or features were removed in the control condition, mainly disease specific therapeutic content (e.g. CBT (32, 56)) or additional disease management features (community features (33)).

It is interesting to note that by analyzing which aspects of digital interventions were changed or removed in a digital control, we can draw conclusions about which features were considered to have the highest likelihood for being efficacious in the DTx. For example, in a trial examining SHUTi, a DTx for insomnia, the researchers did not include a sleep window suggestion in their “attention control” (56). Therefore the investigational treatment and control condition differ on at least these two aspects, and thus the difference between them might be based on core CBT content, the concrete sleep time suggestions, or both (56). Similarly, Sleepio, another DTx for the treatment of insomnia, created a digital control app that contained no social community feature compared to the active intervention (33). The authors described the social community features as mainly targeting engagement, but don't include it in the control, highlighting the fact that appropriate engagement, i.e., ensuring the patients use the DTx and experience other treatment components is relevant for treatment efficacy. The above examples highlight the complexity of designing control conditions for DTx as well as the challenge of designing inactive control conditions while keeping enough credible content for participants blinding.

Control conditions for DTx delivered cognitive interventions

It is noteworthy that all reviewed DTx containing gamified cognitive training or remediation interventions chose a form of sham control, with some sham controls looking similar to consumer grade apps or video games. For example, Endeavor, a video game-based treatment to improve cognition in children with ADHD was compared to a digital control “word game” (letter-connecting to spell words) designed to not engage cognitive targets associated with the primary outcome (27). The study found that the treatment separated from the control on the primary, objective measure of attention, but not on clinical measures related to ADHD subjective symptoms (27). A similar example comes from Posit Science that compared their gamified cognitive training DTx targeting specific impairments in patients with schizophrenia against off-the-shelf computer games (e.g., solitaire, checkers) in a registration trial. The authors stated that the controls were “matched to the experimental treatment program in intensity and duration”, while “plausibly engaging cognitive systems” (34). While the trial was designed as a superiority trial, the authors use the term “active control” for the off-the-shelf games. The use of active control here is likely applying a more academic nomenclature of active controls which is not in line with the FDA definition where active controls are interventions with known therapeutic effects (42). In this trial, the treatment did not statistically separate on the primary outcome of cognitive functioning, but surprisingly the authors did not discuss whether their “active control” could indeed have improved cognitive functioning and may have been better described as a lower dose cognitive training (akin to lower dose control conditions in pharmacological trials). The above examples of game-based digital cognitive treatment studies point to the fact that regular, focused engagement using games as controls may not be truly inert controls as they can produce therapeutic effects.

Blinding

Surprisingly, none of the trials report blinding checks, which are recommended by trial reporting standards (Consolidated Standards Of Reporting Trials, CONSORT (57)). This may be related to many trials being blind-to-hypothesis instead of blind-to-assignment. Nevertheless, study conditions could be compared regarding the respective expectation of benefit before a trial (see Akili's Endeavor trial for an example of expectation matching (27)) or perceived credibility of the intervention after the trial. In general, low rates of reported blinding checks are a known issue in nonpharmacologic treatments which DTx (58).

One trial for a DTx targeting schizophrenia symptoms deployed a minimal control condition during a Phase 2 study that consisted of an app displaying a count-down timer to indicate the remaining study duration (59). The study protocol offers limited information on the exact instruction for the control condition or patient's perception of being allocated to this minimal digital engagement control, it is notable that the digital control improved some of the secondary outcomes and did not perform significantly worse than the investigational treatment (59). This effect may be due to at least partially ineffective blinding in the group receiving the minimal control condition or they could be related to regular engagement with a structured digital experience, which resembles specific components discussed in psychotherapy, such as behavioral activation, discussed above. But without data on the patient experience in such a minimal, potentially unblinded digital control condition these remain hypotheses.

Discussion

Summary of the findings and challenges

This review highlights some key challenges in the design and description of control choices for DTx RCTs. Overall, the reviewed studies show a range of control strategies ranging from unblinded waitlist and TAU designs, to placebo-controlled trials which replace hypothesized core treatment components and often additional disease management and engagement related tools.

Control condition nomenclature across studies was inconsistent, ranging from sham or placebo control to digital control, attention-matched placebo control or active control. This may relate to different definitions of these terms in the literature, for example in the behavioral research literature versus FDA guidance (see Definitions Table 1). Thus, the names used for control conditions may not accurately describe the actual control designs, which limits the comparability of results across studies and even across fields. Similarly, a clear description of the control intervention is missing in most cases, and standardization of approach to control condition descriptions will be critical moving forward (for example following a standardized framework for intervention descriptions, such as that proposed by Tidier (60)).

Authors did not regularly discuss control design choices such as why certain features were removed or disarmed. In addition, the potential efficacy of similar control conditions is described differently across studies. For example, a wellbeing program (The Health and Well-Being Program) included as a control condition for obsessive compulsive disorder was described as an “active intervention” (61), while similar “general health and well-being educational content” was called a placebo control for a RCT for diabetes mellitus, type 2 (32). This may exemplify the “known unknowns” of mechanisms and efficacy factors in DTx and how individual authors judge the efficacy of different features differently or at the very least point to inconsistencies of describing control conditions.

Lastly, the exact instructions researchers provided to participants with regard to study conditions and blinding was often unclear. It is not often described whether participants were blind-to-assignment (i.e., expecting to receive either an active treatment or a placebo), the standard for pharmacological trials, or blind-to-hypothesis (i.e., expecting to receive one of two potential treatments for their diagnosis, with debriefing after study participation). Based on results from research into the placebo effect, such trial specifications may bias RCTs in different ways (62) and affect comparability between trials.

Recommendations for choosing and reporting control conditions in DTx

Given the breadth of DTx content and applications, it is unlikely that there will ever be a one-size-fits-all approach to control condition design. This is in line with Blease's comment that (14) placebos in RCTs should be thought of as “moving targets designed to mimic specific interventions, rather than “as a particular kind of thing”. In line with this thought, the author recommends calling placebo interventions “control interventions” (14). For control conditions that are using the same, digital format, digital control may be a meaningful term for the field. Further, based on our review and the realization that no ‘one-size-fits-all’ control intervention exists for DTx, we conclude with recommendations for choosing and describing control conditions.

Choosing control conditions

• Define the appropriate degree of stringency for digital control conditions:

The necessary stringency of digital control conditions should reflect the risk profile and novelty of the DTx and whether it is designed to treat versus manage or prevent disease. For low-risk devices/indications that are managing versus treating diseases, less stringent control conditions may be suitable (see Figure 1 for a schematic representation of control condition choices and corresponding stringency). In addition, relatively more novel interventions may require more stringent controls compared to digital translations of known psychosocial behavioral interventions, such as CBT (12).

• Minimal control level:

All digital control conditions should, at the very least, control for incidental effects of being in a trial (e.g., Hawthorne effect), or bias related to being assessed repeatedly (15)), disease progression over time, and regression to the mean effects. This applies to drug trials as much as to DTx. In DTx, control condition design should take into consideration additional instructions on how to use the application and time with the application should be matched as closely as possible in the digital control. We recommend that in studies which opt for a minimal control condition, data on actual engagement with the control should be part of the primary publication describing the RCT results.

• Inactive digital control conditions:

To truly design an inactive digital control condition, DTx features and components designed with the intent to deliver disease specific therapeutic content should not be part of the control condition. In addition, lifestyle and disease management features (e.g., psychoeducation, pharmacological tracking, chatbots and interactions around digital working alliance, a form of working alliance that is effective in face-to-face therapy and that is actively investigated to enhance DTx engagement and efficacy (63)), motivational interviewing or features related to motivated and regular engagement should be carefully assessed as to whether they share aspects with known treatment activities and affect the primary outcome. This may include a careful literature review of known efficacy features and activities in face to face and digital interventions in a specific indication. The results of such a review and rationale for control design choices should be stated in the trial publication. At the same time, if current knowledge is limited, feasibility testing of controls may be a way to de-risk control choices before a larger RCT is conducted. The potential choices in designing a digital control condition for DTx are exemplified in Figure 2.

• Three-arm studies:

Adding a third arm (e.g., TAU or Waitlist), could be helpful to add to a RCT when it is likely that a digital control condition may not be fully inert, as seen in the trial by Mahana Therapeutics (47). This will help elucidate real-world effectiveness of DTx while also providing the potential to compare to a more stringent or even active control condition.

FIGURE 1

Figure 1. Control conditions. Control condition stringency can be distinguished with regards to what aspects are controlled for. For example, waitlist control conditions will control for aspects of natural disease progression and being regularly assessed in a trial. More stringent sham or placebo controls will also contain aspects of a digital control, such as engagement tools, digital disease management tools, such as wellness content or trackers. DTx may also try and establish a working alliance (common factors). Thus, stringent digital sham conditions may not be fully inactive.

FIGURE 2

Figure 2. Potential choices in designing a digital control condition for DTx.

Providing details on control conditions and other relevant trial information:

• Describing digital control conditions in detail:

Given the active research into mechanisms of action in face-to-face and DTx interventions, both the intervention and control condition should be described in detail following already established standardized frameworks (e.g. Tidier (60)). Exploratory endpoints related to potential mechanisms of digital controls may help elucidate the underlying principles and may even inform more potent digital treatment designs down the line.

• Describing additional design characteristics related to expectation setting (informed consent) and blinding:

If blinding to study condition was attempted for participants, it should be backed up with expectancy or blinding-check data in the published RCT. Further, authors should clearly state how expectations around study allocation were presented to participants. Whether participants expected two potentially active treatments versus a treatment and placebo control can influence the size of nonspecific effects of the two arms and may affect the number needed to enroll to show a significant difference between the conditions.

Limitations

This review is based on DTx deploying cognitive, behavioral and psychosocial interventions that that fulfill the DTx definition, may have attempted to or are attempting FDA clearance, and are presented by the DTx alliance and similar trials based on the authors' awareness of the field. The review therefore does not constitute a systematic review and may not contain all control condition choices in the field. Nevertheless, by reviewing exemplary SaMD phase 2 and registration trials, the authors believe that the review represents a meaningful basis to foster discussion of a current key challenges in the field of DTx. Future systematic reviews should further elucidate DTx trial practices and provide guidance to the nascent field.

Further, the regulatory considerations provided and majority of trials in this review are US-centered, and therefore our assessment does not represent or discuss potential differences in regulatory practices beyond the US. Finally, while DTx risk profiles will in most cases be lower compared to pharmacological interventions, there are still risks to be considered and studied in DTx, ranging from technical risks, to a risk of a non-adequate treatment selection and the related risk of more general loss of confidence in treatments, or the risks of a DTx failing to detect serious symptoms such as suicidality. Future DTx clinical trials should adequately test these aspects.

Final remarks and conclusions

RCTs remain a key aspect in evidence generation in DTx, especially those therapeutics seeking regulatory approval. However, the stringency of digital control conditions may vary based on factors such as risk profile and novelty of the intervention. Given the general low risk profile and potential of DTx to increase access to personalized care, many DTx may choose a less stringent minimal control or even waitlist control. Alternatively, minimal digital controls in the form of engagement with regular, general digital content (digital diversions) may be appropriate for efficacy studies conducted under controlled, artificial settings.

While waitlist controls may overestimate treatment effects, they may be a viable option as a control condition in large-scale real-world studies. Indeed, in light of the iterative nature of software development, there has been a call for more innovative real-world data approaches to evidence generation for DTx beyond classical RCTs (64). DTx are uniquely positioned to collect real-world data, engagement patterns, user reported outcome data, and/or clinically relevant digital phenotypes directly through their software application to assess their real-world engagement and effectiveness. Indeed, real-world and pragmatic study approaches are gaining popularity, and have been recommended to support regulatory decisions (e.g. The 21st Century Cures Act) (65).

Acknowledgments

Writing assistance, under the direction of the authors, was provided by The Med Writers, Willington, FL, with funding by Click Therapeutics Inc., in accordance with Good Publication Practice guidelines.

Author contributions

JL - Made a substantial contribution to the concept of the work, by reviewing and interpreting the literature to form recommendations, and drafted the article. EO - Made a substantial contribution to the concept of the work by reviewing and interpreting the literature to form recommendations and critically revised the article. LT - Made a substantial contribution to the concept of the work by reviewing and interpreting the literature to form recommendations and critically revised the article. SL - Made a substantial contribution to the concept of the work by reviewing and interpreting the literature to form recommendations and critically revised the article. TC - Made a substantial contribution to the concept and design of the work by reviewing and interpreting the literature to form recommendations and critically revised the article. All authors contributed to the article and approved the submitted version.

Conflict of interest

All authors have equity interest and are employed by Click Therapeutics, Inc., which sponsored the writing of this manuscript, except for EO who is currently employed by Lumos Medical Labs.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fdgth.2022.823977/full#supplementary-material.

References

1. Digital Therepeutics Alliance. Digital Therapeutics Definition and Core Principles. (2020). Available at: https://dtxalliance.org/wp-content/uploads/2021/01/DTA_DTx-Definition-and-Core-Principles.pdf

2. Karyotaki E, Riper H, Twisk J, Hoogendoorn A, Kleiboer A, Mira A, et al. Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms: a meta-analysis of individual participant data. JAMA Psychiatry. (2017) 74(4):351–9. doi: 10.1001/jamapsychiatry.2017.0044

PubMed Abstract | CrossRef Full Text | Google Scholar

3. ICD-10 Version (2019). Available at: https://icd.who.int/browse10/2019/en (cited 2022 Jul 17).

4. Beevers CG, Pearson R, Hoffman JS, Foulser AA, Shumake J, Meyer B. Effectiveness of an internet intervention (deprexis) for depression in a United States adult sample: a parallel-group pragmatic randomized controlled trial. J Consult Clin Psychol. (2017) 85: 367–80. doi: 10.1037/ccp0000171

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Morin CM. Profile of somryst prescription digital therapeutic for chronic insomnia: overview of safety and efficacy. Expert Rev Med Devices. (2020) 17(12):1239–48. doi: 10.1080/17434440.2020.1852929

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Malgaroli M, Hull TD, Schultebraucks K. Digital health and artificial intelligence for PTSD: improving treatment delivery through personalization. Psychiatr Ann. (2021) 51(1):21–6. doi: 10.3928/00485713-20201203-01

CrossRef Full Text | Google Scholar

7. Kollins SH, DeLoss DJ, Cañadas E, Lutz J, Findling RL, Keefe RSE, et al. A novel digital intervention for actively reducing severity of paediatric ADHD (STARS-ADHD): a randomised controlled trial. The Lancet Digital Health. (2020) 2(4):e168–e78. doi: 10.1016/S2589-7500(20)30017-0

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Campbell ANC, Nunes EV, Matthews AG, Stitzer M, Miele GM, Polsky D, et al. Internet-delivered treatment for substance abuse: a multisite randomized controlled trial. Am J Psychiatry. (2014) 171(6):683–90. doi: 10.1176/appi.ajp.2014.13081055

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Berman MA, Guthrie NL, Edwards KL, Appelbaum KJ, Njike VY, Eisenberg DM, et al. Change in glycemic control with use of a digital therapeutic in adults with type 2 diabetes: cohort study. JMIR diabetes. (2018) 3(1):e9591. doi: 10.2196/diabetes.9591

CrossRef Full Text | Google Scholar

10. Everitt H, Landau S, Little P, Bishop FL, McCrone P, O’Reilly G, et al. Assessing cognitive behavioural therapy in irritable bowel (ACTIB): protocol for a randomised controlled trial of clinical-effectiveness and cost-effectiveness of therapist delivered cognitive behavioural therapy and web-based self-management in irritable. BMJ Open. (2015) 5(7):e008622. doi: 10.1136/bmjopen-2015-008622

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Braley M, Pierce JS, Saxena S, De Oliveira E, Taraboanta L, Anantha V, et al. A virtual, randomized, control trial of a digital therapeutic for speech, language, and cognitive intervention in post-stroke persons with aphasia. Front Neurol. (2021) 12:626780. doi: 10.3389/fneur.2021.626780

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Mohr DC, Azocar F, Bertagnolli A, Choudhury T, Chrisp P, Frank R, et al. Banbury forum consensus statement on the path forward for digital mental health treatment. Psychiatr Serv. (2021) 72:e168–e78. appi.ps.2020005. doi: 10.1176/appi.ps.202000561

CrossRef Full Text | Google Scholar

13. D’Agostino RB, Kwan H. Measuring effectiveness: what to expect without a randomized control group. Med Care. (1995) 33:AS95–105.

Google Scholar

14. Blease C. The placebo effect and psychotherapy: implications for theory, research, and practice. (2016) 3:105–7. doi: 10.1037/cns0000094

CrossRef Full Text | Google Scholar

15. McCambridge J, Witton J, Elbourne DR. Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects. J Clin Epidemiol. (2014) 67:267–77. doi: 10.1016/j.jclinepi.2013.08.015

PubMed Abstract | CrossRef Full Text | Google Scholar

16. De Amici D, Klersy C, Ramajoli F, Brustia L, Politi P. Impact of the Hawthorne effect in a longitudinal clinical study: the case of anesthesia. Control Clin Trials. (2000) 21(2):103–14. doi: 10.1016/S0197-2456(99)00054-9

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Julien RM. A primer of drug action: A concise nontechnical guide to the actions, uses, and side effects of psychoactive drugs, revised and updated. New York: Holt Paperbacks (2013).

18. Stiles WB, Barkham M, Mellor-Clark J, Connell J. Effectiveness of cognitive-behavioural, person-centred, and psychodynamic therapies in UK primary-care routine practice: replication in a larger sample. Psychol Med. (2008) 38(5):677–88. doi: 10.1017/S0033291707001511

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Kazdin AE. Mediators and mechanisms of change in psychotherapy research. (2007). Available from: http://clinpsy.annualreviews.org (cited 2021 Apr 2).

20. Asay TP, Lambert MJ. The empirical case for the common factors in therapy: quantitative findings. In: Hubble MA, Duncan BL, Miller SD, editors. The heart and soul of change: What works in therapy. American Psychological Association (2006). p. 23–55. Available from: /record/1999-02137-001 (cited 2022 Feb 11).

21. Mulder R, Murray G, Rucklidge J. Common versus specific factors in psychotherapy: opening the black box [Internet]. The Lancet Psychiatry. (2017) 4:953–62. doi: 10.1016/S2215-0366(17)30100-1

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Enck P, Zipfel S. Placebo effects in psychotherapy: a framework. Front Psychiatry. (2019) 10:456 [cited 2021 Apr 2]. doi: 10.3389/fpsyt.2019.00456

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Wampold BE. How important are the common factors in psychotherapy? An update. World Psychiatry. (2015) 14(3):270–7. doi: 10.1002/wps.20238

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Lukens EP, Mcfarlane WR. Psychoeducation as evidence-based practice: considerations for practice, research, and policy. Br Treat Cris Interv. (2004) 4(3):205–25. doi: 10.1093/brief-treatment/mhh019

CrossRef Full Text | Google Scholar

25. Zhang R, Nicholas J, Knapp AA, Graham AK, Gray E, Kwasny MJ, et al. Clinically meaningful use of mental health apps and its effects on depression: mixed methods study. J Med Internet Res. (2019) 21(12):e15644. doi: 10.2196/15644

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Gould CE, Kok BC, Ma VK, Marie Zapata AL, Owen JE, Kuhn E, et al. Veterans affairs and the department of defense mental health apps: a systematic literature review. Psychol Serv. 16:196–207 [cited 2021 May 16]. doi: 10.1037/ser0000289

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Kollins SH, DeLoss DJ, Cañadas E, Lutz J, Findling RL, Keefe RSE, et al. A novel digital intervention for actively reducing severity of paediatric ADHD (STARS-ADHD): a randomised controlled trial. Lancet Digit Heal. (2020) 2(4):e168–78. doi: 10.1016/S2589-7500(20)30017-0

CrossRef Full Text | Google Scholar

28. Iacoviello BM, Wu G, Alvarez E, Huryk K, Collins KA, Murrough JW, et al. Cognitive-emotional training as an intervention for major depressive disorder. Depress Anxiety. (2014) 31(8):699–706. doi: 10.1002/da.22266

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Craig P, Dieppe P, Macintyre S, Mitchie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. Br Med J. (2008) 337(7676):979–83. doi: 10.1136/bmj.a1655

CrossRef Full Text | Google Scholar

30. Torous J, Firth J. The digital placebo effect: mobile mental health meets clinical psychiatry. Lancet Psychiatry. (2016) 3:100–2. doi: 10.1016/S2215-0366(15)00565-9

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Dombrowski SU, O’Carroll RE, Williams B. Form of delivery as a key “active ingredient” in behaviour change interventions. Br J Health Psychol. (2016) 21:733–40. doi: 10.1111/bjhp.12203

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Boucher E, Moskowitz JT, Kackloudis GM, Stafford JL, Kwok I, Parks AC. Immediate and long-term effects of an 8-week digital mental health intervention on adults with poorly managed type 2 diabetes: protocol for a randomized controlled trial. JMIR Res Protoc. (2020) 9(8):e18578. doi: 10.2196/18578

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Espie CA, Kyle SD, Williams C, Ong JC, Douglas NJ, Hames P, et al. A randomized, placebo-controlled trial of online cognitive behavioral therapy for chronic insomnia disorder delivered via an automated Media-rich web application. Sleep. (2012) 35(6):769–81. doi: 10.5665/sleep.1872

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Mahncke HW, Kim S-J, Rose A, Stasio C, Buckley P, Caroff S, et al. Evaluation of a plasticity-based cognitive training program in schizophrenia: results from the eCaesar trial. Schizophr Res. (2019) 208:182–9. doi: 10.1016/j.schres.2019.03.006

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Jacobson NS, Martell CR, Dimidjian S. Behavioral activation treatment for depression: returning to contextual roots. Clin Psychol Sci Pract. (2001) 8(3):255–70. doi: 10.1093/clipsy.8.3.255

CrossRef Full Text | Google Scholar

36. Lewinsohn PM, Graf M. Pleasant activities and depression. J Consult Clin Psychol. (1973) 41(2):261–8. doi: 10.1037/h0035142

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Dimaggio G, Shahar G. Behavioral activation as a common mechanism of change across different orientations and disorders. Psychotherapy. (2017) 54(3):221–4. doi: 10.1037/pst0000117

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Hoofdakker BJVD, Veen-Mulders LVD, Sytema S, Emmelkamp PMG, Minderaa RB, Nauta MH. Effectiveness of behavioral parent training for children with ADHD in routine clinical practice: a randomized controlled study. J Am Acad Child Adolesc Psychiatry. (2007) 46(10):1263–71. doi: 10.1097/chi.0b013e3181354bc2

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Fish MT, Russoniello CV, O’Brien K. The efficacy of prescribed casual videogame play in reducing symptoms of anxiety: a randomized controlled study. Games Health J. (2014) 3(5):291–5. doi: 10.1089/g4h.2013.0092

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Russoniello CV, Fish M, O’Brien K. The efficacy of casual videogame play in reducing clinical depression: a randomized controlled study. Games Health J. (2013) 2(6):341–6. doi: 10.1089/g4h.2013.0010

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Evers AWM, Colloca L, Blease C, Annoni M, Atlas LY, Benedetti F, et al. Implications of placebo and nocebo effects for clinical practice: expert consensus. Psychother Psychosom. (2018) 87(4):204–10. doi: 10.1159/000490354

PubMed Abstract | CrossRef Full Text | Google Scholar

42. U.S. Department of Health and Human Services. Design considerations for pivotal clinical investigations for medical devices. 2013. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/design-considerations-pivotal-clinical-investigations-medical-devices

43. Boot WR, Simons DJ, Stothart C, Stutts C. The pervasive problem with placebos in psychology: why active control groups are not sufficient to rule out placebo effects. Perspect Psychol Sci. (2013) 8(4):445–54. doi: 10.1177/1745691613491271

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Aycock DM, Hayat MJ, Helvig A, Dunbar SB, Clark PC. Essential considerations in developing attention control groups in behavioral research. Res Nurs Health. (2018) 41(3):320–8. doi: 10.1002/nur.21870

PubMed Abstract | CrossRef Full Text | Google Scholar

45. De Novo Summary reSET [Internet]. De novo summary (DEN160018). FDA (2016). Available at: https://www.accessdata.fda.gov/cdrh_docs/reviews/DEN160018.pdf (cited 2021 Apr 5).

46. Christensen DR, Landes RD, Jackson L, Marsch LA, Mancino MJ, Chopra MP, et al. Adding an internet-delivered treatment to an efficacious treatment package for opioid dependence. J Consult Clin Psychol. 2014 82(6):964–72. doi: 10.1037/a0037496

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Everitt HA, Landau S, O’Reilly G, Sibelli A, Hughes S, Windgassen S, et al. Assessing telephone-delivered cognitive-behavioural therapy (CBT) and web-delivered CBT versus treatment as usual in irritable bowel syndrome (ACTIB): a multicentre randomised trial. Gut. (2019) 68(9):1613–23. doi: 10.1136/gutjnl-2018-317805

PubMed Abstract | CrossRef Full Text | Google Scholar

48. mobihealth news. Otsuka, Click Therapeutics Kick off Decentralized Pivotal Trial for Depression Digital Therapeutics | MobiHealthNews. (2021). Available from: https://www.mobihealthnews.com/news/otsuka-click-therapeutics-kick-decentralized-pivotal-trial-depression-digital-therapeutics [cited 2021 Jun 14].

49. Digital Health Interventions for Obsessive Compulsive Disorder (OCD) - Full Text View - ClinicalTrials.gov. Available from: https://clinicaltrials.gov/ct2/show/NCT04136626 [cited 2021 Apr 20].

50. Wise Therapeutics. Wise Therapeutics Pursues FDA Clearance for Gamified Prescription Digital Therapeutics Targeting Three Initial Indications. (2021). Available from: https://www.prnewswire.com/news-releases/wise-therapeutics-pursues-fda-clearance-for-gamified-prescription-digital-therapeutics-targeting-three-initial-indications-301258871.html (cited 2021 Apr 4).

51. U.S. Department of Health and Human Services, Administration F and D, Health C for D and R, Research C for BE and. Design Considerations for Pivotal Clinical Investigations for Medical Devices Guidance for Industry, Clinical Investigators. (2013). Available from: http://www.fda.gov/RegulatoryInformation/Guidances/ucm373750.htm

52. Hung HMJ, Wang S-J, Tsong Y, Lawrence J, O’neil RT. Some fundamental issues with non-inferiority testing in active controlled trials. Stat Med Stat Med. (2003) 22:213–25. doi: 10.1002/sim.1315

PubMed Abstract | CrossRef Full Text | Google Scholar

53. D’Agostino RB, Massaro JM, Sullivan LM. Non-inferiority trials: design concepts and issues - the encounters of academic consultants in statistics. Stat Med. (2003) 22(2):169–86. doi: 10.1002/sim.1425

CrossRef Full Text | Google Scholar

54. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5th ed. Arlington, VA (2013).

55. Waitlist-Control Trial of Smartphone CBT for Body Dysmorphic Disorder (BDD) - Full Text View - ClinicalTrials.gov [Internet]. Available from: https://www.clinicaltrials.gov/ct2/show/NCT04034693?term=koa+health+body&draw=2&rank=1 (cited 2021 Apr 20).

56. Christensen H, Batterham PJ, Gosling JA, Ritterband LM, Griffiths KM, Thorndike FP, et al. Effectiveness of an online insomnia program (SHUTi) for prevention of depressive episodes (the GoodNight Study): a randomised controlled trial. The Lancet Psychiatry. (2016) 3(4):333–41. doi: 10.1016/S2215-0366(15)00536-2

PubMed Abstract | CrossRef Full Text | Google Scholar

57. Altman D, Egger M, Elbourne D. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Artic Ann Intern Med. (2001) 348:110–16. Available at: https://www.researchgate.net/publication/12030511 (cited 2021 May 17). doi: 10.7326/0003-4819-134-8-200104170-00012

CrossRef Full Text | Google Scholar

58. Boutron I, Tubach F, Giraudeau B, Ravaud P. Blinding was judged more difficult to achieve and maintain in nonpharmacologic than pharmacologic trials. J Clin Epidemiol. (2004) 57(6):543–50. doi: 10.1016/j.jclinepi.2003.12.010

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Study of Efficacy of PEAR-004 in Schizophrenia - Full Text View - ClinicalTrials.gov. Available from: https://www.clinicaltrials.gov/ct2/show/NCT03751280?term=NCT03751280&draw=2&rank=1 [cited 2021 Apr 6].

60. Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. Br Med J. (2014) 348. Available at: https://pubmed.ncbi.nlm.nih.gov/24609605/ (cited 2021 Apr 21). doi: 10.1136/bmj.g1687

CrossRef Full Text | Google Scholar

61. Koa Health spins out from Telefónica Moonshot Alpha, scores $16.5M in initial funding | FierceHealthcare. Available from: https://www.fiercehealthcare.com/tech/koa-health-spins-out-from-telefonica-moonshot-alpha-scores-16-5m-initial-funding [cited 2021 Apr 20].

62. Kaptchuk TJ. The double-blind, randomized, placebo-controlled trial: gold standard or golden calf? J Clin Epidemiol. (2001) 54:541–9. doi: 10.1016/S0895-4356(00)00347-4

PubMed Abstract | CrossRef Full Text | Google Scholar

63. Henson P, Wisniewski H, Hollis C, Keshavan M, Torous J. Digital mental health apps and the therapeutic alliance: initial review. BJPsych Open. (2019) 5(1):e15. doi: 10.1192/bjo.2018.86

PubMed Abstract | CrossRef Full Text | Google Scholar

64. Guo C, Ashrafian H, Ghafur S, Fontana G, Gardner C, Prime M. Challenges for the evaluation of digital health solutions—a call for innovative evidence generation approaches. npj Digit Med. (2020) 3(1):1–14. doi: 10.1038/s41746-019-0211-0

PubMed Abstract | CrossRef Full Text | Google Scholar

65. 21st Century Cures Act | FDA.Available from: https://www.fda.gov/regulatory-information/selected-amendments-fdc-act/21st-century-cures-act [cited 2021 May 5].