Usability, Feasibility, and Effect of a Biocueing Intervention in Addition to a Moderated Digital Social Therapy-Platform in Young People With Emerging Mental Health Problems: A Mixed-Method Approach

Introduction: To optimize treatment, it is of utmost importance to take into account the myriad of biological, social, and psychological changes that young people go through during adolescence which make them more vulnerable for developing mental health problems. Biocueing, a non-invasive method to transform physiological parameters into an observable signal, could strengthen stress- and emotion regulation by cueing physiologically unusual values in daily life. The aim of this study is to investigate the usability, feasibility, and exploratory effect of biocueing in addition to ENgage YOung people earlY (ENYOY), a moderated digital social therapy-platform, in young people with emerging mental health complaints.

Methods: A user-centered mixed-method design was used. A focus group was conducted to optimize the ENYOY-platform and biocueing intervention. Biocueing was operationalized by a smartwatch and the Sense-IT app. A within-subjects design was used; 10 days for all participants ‘biofeedback off' (control), followed by 10 days ‘biofeedback on' (experimental). Emotional awareness and perceived stress were measured using ecological momentary assessment. Eight individuals participated. User-friendliness, usability, and acceptance were assessed using a qualitative design.

Results: Findings from the focus group resulted in several adaptations of the biocueing intervention to the ENYOY-platform and vice versa. The average measurement compliance rate was 78.8%. Level-one findings showed different individual effects on perceived stress and emotional awareness. Level-two analyses showed no overall effects on perceived stress (B = −0.020, p = 0.562) and overall positive effects on emotional awareness (B = 0.030, p = 0.048) with small effect sizes (Improvement Rate Difference = 0.05–0.35). The intervention was found to be acceptable and showed moderate usability. Participants indicated they experienced improvements in reflection on feelings and changes in behavior, such as pausing and evaluating the situation.

Conclusion: These preliminary results show that biocueing could be a promising addition to digital treatment platforms and help young people become more emotionally aware. Improvements should be made regarding the usability and acceptability of the smartwatch, as well as more extensive integration of the biocueing intervention with a digital treatment platform. It would be relevant to gain a better understanding of which individuals would benefit most from an additional biocueing intervention.

Introduction

Approximately three-quarters of all mental health problems start before the age of 25 (1–3), and half of all episodes of mental health disorders start at the age of 14 (4, 5). Mental disorders make up for the number one burden of disease in young people (6). Untreated mental health disorders could lead to chronicity, comorbidity, lost potential, a lower social and occupational health, increased risk of suicide, and a lower quality of life for the individuals concerned (7, 8). On a broader scale, mental health disorders are associated with considerable economic consequences (9). Mental health costs are the single highest source of global economic burden in the world (10) and the Netherlands is estimated to spend around 5.7 billion euro annually on curative mental healthcare (11). Therefore, it is very important that early prevention and appropriate interventions are designed for young people (12) at the time when they are developing strategies to cope with stress and negative emotions (13). A very successful Australian initiative has implemented a comprehensive, evidence based youth focused indicative prevention program, which combines care in a stepped-care manner and blends peer-to-peer support with online counseling and therapy (Moderated Online Social Therapy (MOST+); (14–17).

Despite the success of MOST+ it is unknown whether this initiative can be translated to different countries and cultures. Currently, a study to implement the MOST+ platform in the Netherlands is being carried out under the name ENgage YOung people earlY (ENYOY) (18). In addition, most activities in the MOST+ platform require participants to have the ability to note fluctuating levels of stress and physiological arousal. Unfortunately, the ability to sense and feel stress and physiological changes (i.e., interoceptive sensitivity), might be especially impaired in the vulnerable adolescent target group (19–21). Therefore, additional tools that help young people note when physiological arousal is high and help them to apply methods to deal with this (such as behavioral experiments, mindfulness or muscle relaxation) would offer a potential further benefit to a wider group of adolescents. For this purpose biocueing might be an interesting addition. Biocueing is a non-invasive method to transform physiological parameters with a small electronic device into a visible, audible, or tactile signal, cued to the wearer (22). It automatically engages the user by prompts in daily stressful situations (23). The aim of the present study is to exploratory investigate the effects of biocueing as an addition to the moderated online social therapy-platform ENYOY for young people with emerging mental health problems.

During adolescence, the transitional period between child- and adulthood, young people go through a myriad of physical, social, and psychological changes (24–26). Brain areas involved in arousal and risk-reward perception rapidly develop, resulting in heightened stress responsivity and emotional reactivity (27). Fortunately, strategies to cope with stress and negative emotions also begin to form (28), with emotional awareness, (the ability to identify and label internal emotional experiences (29, 30), being an important building block in guiding individuals toward adaptive emotion regulation (19–21). Nonetheless, the functional change in arousal and risk-reward perception seems to precede the development of functional coping strategies, which creates a situation where, as well put by Steinberg (27), ‘an engine is started without yet having a skilled driver behind the wheel'. This discrepancy renders adolescents more vulnerable for negative emotions and psychological stress, and consequently heightens the risk of developing mental health problems (13). Indeed, research shows that there is a relation between stress and mental disorders (31). Less stress reactivity has been shown to predict symptom improvement in, for example, children with anxiety disorders (32) and adolescents with depressive symptoms (33). Moreover, different patterns in parasympathetic and sympathetic nervous system activation have been found to respond to different kinds of treatments (34).

It is of importance to take into account the multiple biological processes that make young people more vulnerable for psychological stress and further optimize treatment accordingly. In an attempt to reduce mental health problems among young people and reach them for indicative prevention mental health treatment, an Australian initiative has been successfully implemented (12). In more than 100 Headspace centers young people can find help for general health and education problems, drug use and (emerging) mental health complaints. Evidence-based interventions are provided in a stepped- care manner (17).

In addition to face-to-face interventions, Moderated Online Social Therapy (MOST+) has been developed to provide young people with comprehensive, adaptive, and integrated digital support. MOST+ merges peer-to-peer social networking; theory-driven and evidence-informed therapeutic interventions; expert clinician and vocational support; and peer support and moderation (14, 17, 35, 36). Findings from several pilot studies with young people diagnosed with (ultra-high risk for) psychosis, depression, social anxiety, and suicidal ideation have consistently shown high levels of feasibility, acceptability, engagement, and safety; and indicated effectiveness in a wide range of clinical and social outcomes [for a comprehensive overview, see: (14, 16, 17, 35)]. Additionally, MOST+ was found to be effective in improving vocational recovery and reducing utilization of emergency services in young people with a first episode of psychosis (16). Currently, a study to implement the MOST+ platform in the Netherlands is being carried out under the name ENgage YOung people earlY (ENYOY) (18). The MOST+ platform has been translated and adapted in cooperation with Dutch experts to fit the needs of young Dutch people, and will be continuously adapted to the wants and needs of participants over the course of the study.

With the goal of all young people benefitting fully from the platform, it would be of interest to look for adaptations or additions to the platform in the Netherlands. Although high levels of feasibility and acceptability were found for the use of the platform in Australia, not all users concluded that MOST+ was relevant to their needs (17), and engagement rates with the platform in young people with psychosis were found to be good, but not for all (80% engaged for the first 3 months, and 47% at 9 months) (16).

A factor that could play a role is the largely verbal content on the platform - though the platform also provides visual and audible exercises such as audio tracks and comics -, exercises are explained and information is given in text and metaphors, and the community is largely made up of verbal posts. However, not all individuals are equally verbally strong (e.g., (37, 38), and thus might not be able to understand its contents. Moreover, some of the exercises of the evidence-based therapies offered on the platform rely on interoception. For example, noting one's own thoughts and emotions (39) is required in the “reflective actions” on the platform. Possibly, some young people may not benefit because they have problems with interoception or, for some, interoceptive sensitivity might not have been fully developed yet (40). Another possible explanation for lower engagement rates could be that the intervention requires a certain amount of self-reliance of the individual, although monitored by a clinical moderator, namely to actively go online themselves. This might be challenging for some, for example for individuals with attention-deficit/hyperactivity disorder who have problems with, for instance planning [e.g., (41)], or individuals with relatively mild mental health complaints (42). Lastly, adherence rates for online interventions are found to be highly varying [e.g., 27.9 and 98% in (43)], for which a medium that could send reminders might contribute (44).

For individuals who are verbally less strong, have problems with interoception, and/or have problems with adherence, an intervention that addresses these individual factors may be helpful. Biocueing could be a suitable extra tool to support the development of coping strategies, as well as interoception, thereby strengthening an individual in stress- and emotion regulation (23).

Biocueing is a non-invasive method to transform physiological parameters, e.g., heartbeat or skin conductance, with a small electronic device - such as a smartwatch - into a visible, audible, or tactile signal, cued to the wearer (22). It automatically engages the user by prompts in everyday life stressful situations. These wearable devices can be helpful in supporting antecedent emotional regulation strategies in response to emotion evoking events and have been found to decrease self-reported stress-levels (23). De Looff and colleagues (45) found that in the 20 minutes leading up to an incident of aggression in individuals with borderline personality disorder, skin conductance level and the number of nonspecific skin conductance responses per minute rose significantly. This means that it is possible to detect certain interoceptive signals, thereby heightening emotional awareness and supporting reappraisal of emotions (20, 46), a process which is essential to mental health (46).

The aim of this study is to exploratory investigate the effects of biocueing as an addition to the moderated online social therapy-platform ENYOY for young people with emerging mental health problems. The objectives are three-fold: (1) to qualitatively explore whether biocueing could be of additional value to ENYOY; (2) to assess the effect of biocueing in combination with ENYOY on perceived stress and emotional awareness; (3) to evaluate the attitude of users concerning the usability, acceptance, and user-friendliness of the biocueing application and smartwatch in combination with the ENYOY-platform as an intervention.

Methods

Study Context

The present study took place within the context of the ENYOY-project [ENgage YOung people earlY; for a comprehensive summary, see (18)], where young people between the ages of 16 and 25 with emerging mental health complaints use the Dutch Moderated Online Social Therapy (MOST+) platform for 6 months. The goal of ENYOY is to offer young people a self-efficacious way of reducing their mental health complaints with digital interactive psychological interventions focused on using and developing personal strengths in combination with online counseling meetings with a psychologist and/or peer worker (a young person with lived experience with mental health complaints).

The ENYOY-project has received ethical approval from the Medical Ethics Review Committee (MERC) at Amsterdam University Medical Centers (AMC), the Netherlands (reference: NL66345.018.18), and was registered in the Netherlands Trial Register (ID NL8966). Written informed consent was obtained from all participants before inclusion to the study.

Study Design

A user-centered (47) mixed method design was used consisting of three phases, see Figure 1. In the first phase (Phase 1–Understanding and specifying end user needs regarding the interventions), a qualitative design using a focus group, consisting of peer workers of the ENYOY-platform, was administered in which the requirements of end users in relation to the interventions were mapped and used to optimize the usage of the ENYOY-platform with the device and app used for biocueing (see materials > Sense-IT). In the experimental period (Phase 2–Testing the intervention with end users), emotional awareness and perceived stress-levels were measured in real time using ecological momentary assessment (EMA). By sampling stress and emotional awareness in real time, EMA aims to minimize recall bias and maximize ecological validity (48). After the experimental period (Phase 3–Evaluating the intervention with end users), user friendliness, usability, and acceptance of the interventions using a qualitative design. The Statement on Reporting of Evaluation Studies in health informatics framework will be followed in reporting this study to create a better understanding of the study flow with the different phases (49).

FIGURE 1

Figure 1. Study design. N, amount and type of participants; ENYOY, ENgage YOung people earlY project; S-DERS, State Difficulties in Emotion Regulation Scale; EMA, ecological momentary assessment; SUS, System Usability Scale; Health-ITUES, Health Information Technology Usability Evaluation Scale.

Phase 1–Understanding and Specifying End User Needs Regarding the Interventions

Participants

Three peer workers of the ENYOY-platform with lived experience with similar mental health complaints as the end users participated. Being involved with the ENYOY-platform for over 4 years, the peer workers are experts in using the platform. They have been moderating the community of ENYOY and are in contact with participants since launch. The members of the focus group were aged between 22 and 29 years (M = 25.6, SD = 3.3), two were female and one male, and were all native Dutch speaking.

Materials

A semi-structured interview was conducted with questions regarding barriers for use, ideas for exercises on the platform, and ideas for integration of the biocueing intervention (see materials > Sense-IT) with the ENYOY-platform.

Procedure

Three out of four peer workers of the ENYOY-platform were able to participate in the focus group. In a focus group the informative source is a group of individuals and the heuristic value of this technique lies in the kind of interaction that emerges during the debate they have (50). End user needs were analyzed following the user-centered design framework (47), aiming to increase user acceptance and satisfaction, and decrease user errors and drop-out. The central question of the focus group was: What is the optimal way to combine the usage of biocueing via the Sense-IT/smartwatch with the ENYOY-platform? A researcher with experience and knowledge of the ENYOY-platform and the biocueing intervention led the focus group using open-ended questions such as: ‘How do you think the ENYOY-platform should be changed or adjusted to work well with a smartwatch?”, which was followed by a discussion in the focus group. Participants were invited to build on each other's answers. Afterwards, participants were sent a summary of their answers and were given the opportunity to make changes. The intervention used in Phase 2 was adjusted based on the feedback received from the peer workers (see Results, Phase 1).

Data Analysis

The qualitative data was analyzed using deductive thematic analysis (51). Predetermined themes, as assessed in the semi-structured interview, were formed into categories. Data that did not fit these predetermined categories were further coded openly using inductive coding (51).

Phase 2–Testing the Intervention With End Users

Participants

A total of eight young people from the population of ENYOY users participated. The sample size was chosen with care, as using the biocueing intervention could be burdensome for some individuals when used for extended periods of time, and since the intervention has not yet been tested in combination with a treatment platform. Therefore, a relatively small sample size was used for this feasibility study to not unnecessarily burden participants. EMA was used with multiple measures per day per individual in order to maintain sufficient power, based on previous biocueing research (52). No additional inclusion criteria were administered (18). The exclusion criterion was the inability to wear the smartwatch during daytime (e.g., because of a job in healthcare or the catering industry). All participants were female, with a mean age of 22.38 years (SD = 2.13), and all had a formal education, split evenly between higher vocational education and university. Mental health problems were categorized by the clinical staging model (see (53) for the clinical staging model, and (18) for the operationalization in the ENYOY-study). Participants were included if the mental health complaints were found to be stage 1a (help-seeking individuals with mild symptoms and mild functional impairment) or 1b (people with attenuated syndromes with partial specificity, often with mixed or ambiguous symptoms and moderate functional impairment), and their occurrence was evenly divided over both categories. The time that a participant was on the platform ranged from 35 to 175 days [M(SD) = 112.38(52.03)].

Materials

Intervention

Sense-IT. The Sense-IT application (54) is designed to help an individual become aware of physiological parameters, such as heartbeat, as measured by the smartwatch. It notifies the user of changes in heartbeat by giving notifications and vibrations through a smartwatch (TicWatch Pro 3 GPS) and smartphone (Moto G30). The app was tailored to the needs of the user by changing the frequency and the message of the notification. The smartwatch ran on Android OS 11 and was connected to the smartphone by Bluetooth. The data on the smartphone was saved offline for privacy concerns. Through the smartphone, the EMA items were administered (see questionnaires).

ENYOY. Participants continued to have access to the ENYOY-platform. Young adults could use the plethora of options the platform has to offer: (1) therapeutic exercises based on positive psychology, acceptance and commitment therapy, and cognitive behavioral therapy; (2) biweekly online contact with a clinical moderator and/or peer worker that coaches the young adult to work on their mental health problems; (3) a community newsfeed with peers and peer workers.

Questionnaires

Emotional Awareness. To measure emotional awareness, the Dutch version of the State Difficulties in Emotion Regulation Scale [S-DERS, (55)] was used. This 21-item questionnaire, scored on a 5-point Likert scale (1 = “Not at all” to 5 = “Completely”), was adjusted to EMA for this study, meaning the questions were adapted for answering on a smartphone. The Awareness and Clarity subscales were used: These scales relate to emotional awareness and the other constructs of the questionnaire are beyond the scope of this study. Examples of these subscales, respectively, are: “I am paying attention to how I feel” (α = 0.79) and “I am confused about how I feel” (α = 0.65). Subscale scores were calculated by summing the scores and averaging: A high score on the aggregated subscale indicates high awareness of emotion.

Perceived Stress. To measure perceived stress, the Dutch EMA survey item “I feel stressed” was used, scored on a 0 to 5-point scale (“Not at all” to “Very much”). This item measures negative affect with high arousal and was extracted from the EMA repository of Kirtley and colleagues (56). A higher score on this scale indicates higher perceived stress-levels.

Procedure

All participants of the ENYOY-study that were online at the time of this study were asked if they were interested in wearing a smartwatch to cope with their stress-levels in addition to their participation on the ENYOY-platform. Out of 40 participants, eight had the time and interest to participate. A total of four appointments with the research assistant took place (a set-up meeting, two calls to check whether the Sense-IT was working properly, and a meeting to evaluate the usability of the intervention). Participants were sent a package by mail containing the hard- and preinstalled software.

During the set-up meeting, participants were asked to choose two moments every day, one in the morning and one in the evening, to prompt the EMA questions. The baseline heart rate of the participant was determined using the protocol used by ter Harmsel and colleagues (57) and participants were asked what they would do when experiencing an above average heartbeat. The ENYOY-platform was mentioned by the researcher to cope with a high heart rate. Specific interventions of the platform that could help reduce stress according to the participant were added to the “toolkit,” so that exercises could be accessed easily. The condition alternated over the course of 20 days, in the first 10 days for all participants “biofeedback off” (control), and in the last 10 days “biofeedback on” (experimental). Participants were prompted twice daily to answer the EMA items regarding perceived stress-levels and emotional awareness during both conditions.

Data Analysis

Data from the subscales Emotional Awareness and Emotional Clarity were aggregated per participant, after which a multi-single case experimental design (multiSCED) was used for measuring perceived stress-levels and emotional awareness per participant (one-level) and overall (two-level). This statistical test is produced using the MultiSCED-tool by Declercq et al. (58). For each participant, condition and time were used to predict stress and emotional awareness by means of the following regression formula:

$\begin{array}{ll}\begin{array}{c}{Y}_i=\beta 0(Intercep{t}_i)+\beta 1Time{}_i+\beta 2Condition{}_i\\ +\beta 3({Time\times Condition}_i)+e_i\hfill \end{array}& (1)\end{array}$

The subscript i denotes the measurement nested within a case. To determine the overall effect of bio cueing, condition and time were used to predict stress and emotional awareness by means of the following regression formula,

$\begin{array}{ll}\begin{array}{c}{Y}_{ij}=\beta 0(Intercep{t}_i)+\beta 1{Time}_{ij}+\beta 2Condition{}_{ij}\\ +\beta 3({Time\times Condition}_{ij})+e_{ij}\backslash n\hfill \end{array}& (2)\end{array}$

The Improvement Rate Difference (IRD; (59) was used to calculate the effect sizes. The IRD is defined as the improvement rate of the treatment phase minus the improvement rate of the control phase.

Phase 3–Evaluating the Intervention With End Users

Participants

See Phase 2.

Materials

Evaluative Interview

User-Friendliness. To measure user-friendliness, semi-structured interviews took place in which questions were asked regarding the participants' experience with the hard- and software. These questions were based on earlier research by Derks and colleagues (52).

Usability and Acceptance. To measure the usability and acceptance of the Sense-IT, two questionnaires were administered. The Dutch version of the System Usability Scale [SUS; (60)] is a 10-item questionnaire scored on a 5-point Likert scale (1 = “Strongly disagree” to 5 = “Strongly agree”), measuring the general usability and acceptance of an intervention. An example of an item is “I think that I would need the support of a technical person to be able to use this system.” The composite measure ranges between 0 and 100, with higher scores indicating higher usability. Scores below 50 indicate non-acceptance, while scores above 50 indicate acceptance (61).

The Health Information Technology Usability Evaluation Scale [Health-ITUES; (62)] is a 20-item questionnaire, scored on a 5-point Likert scale (1 = “Strongly disagree” to 5 = “Strongly disagree”), which focuses on usability of mobile health technology, and has four subscales: Impact, Perceived Usefulness, Perceived Ease of Use, and User Control. An example item of Impact is: “I think ENYOY combined with the Sense-IT would be a positive addition for persons living with beginning mental health problems” (α = 0.85). A high score on User Control and Perceived Ease of Use captures the user-system interaction, whereas Perceived Usefulness evaluates task accomplishments through system use. Impact refers to the system's impact on daily life. A high overall score indicates a higher perceived usability of the technology.

Procedure

After the 20 day experimental period, a meeting took place between the research assistant and the participants of phase 2 to evaluate the intervention by means of the semi-structured interview, the SUS, and Health-ITUES. The interviews were conducted through video call in a secured Microsoft Teams environment by a research assistant with experience and knowledge of the ENYOY-platform and the biocueing intervention.

Data Analysis

The qualitative data was analyzed using deductive thematic analysis (51). Predetermined themes, as assessed in the semi-structured interviews, were formed into categories. Data that did not fit these predetermined categories were further coded openly using inductive coding (51). Finally, selective coding was used to exploratory determine possible relations and connections between the data and categories (63). Considering the SUS and Health-ITUES averages and ranges were calculated and interpreted.

Results

Phase 1–Understanding and Specifying End User Needs Regarding the Interventions

The following adaptations were implemented as a result of the focus group for the optimization of the smartwatch to the ENYOY-platform and vice versa. For a full overview of the results of the thematic analyses, see Supplementary Appendix A.

Firstly, in the manual of the Sense-IT the benefits of the intervention were specified, as well as steps to personalize the smartwatch to the issues the young person struggles with. Secondly, access to the ENYOY-platform was made easier by (a) making exercises available offline, (b) helping young people fill their ENYOY toolkit with useful exercises beforehand, and (c) a link was added from the Sense-IT to the ENYOY toolkit. Third, ENYOY was adjusted to the smartwatch and vice versa to help young people slow down and reflect by (i) creating an exercise that helps differentiate between emotions, (ii) questions were prompted about young people's needs and experienced emotions on the smartwatch, and (iii) it was individually addressed whether or not the participant also wanted to be signaled when their heartbeat is low. Finally, in order to deal with obstacles for use, the following steps were taken: (1) young people were helped with deciding whether a smartwatch fits their lifestyle and were helped creating a new habit by linking the intervention to old habits during the introductory meeting; (2) it was addressed in the introductory meeting that hypervigilance could increase stress and that there is no correct or false way of experiencing stress, and (3) participants were contacted 2 days into the experiment to ask about the functioning of the smartwatch and Sense-IT application.

Phase 2–Testing the Intervention With End Users

Compliance, as defined by the ratio of the number of measurement occasions that participants completed in relation to the maximum number (64), was on average 78.8%, ranging from 59.4% to 100%. An exact McNemar's chi square test determined that there was a significant difference in compliance between the control condition (86.1%) and the experimental condition (70.8%), p < 0.001. In order to improve the quality and reliability, missing data (N = 64) was imputed using the Expectation-Maximization (EM) method, following Chen, Feng, Wu, and Peng (65).

The assumption of normality was tested by depicting a histogram of standardized residuals, which indicated that the data contained approximately normally distributed errors, as did the normal probability plot (P-P plot of) standardized residuals, which showed data points followed a straight line.

Table 1 displays an overview of the results from the level-one ordinary least squares regression analysis. Regarding perceived stress-levels, in the control condition (biocueing off) no significant effect was found for six out of eight participants (see Figure 2; participant (P) 1: B = −0.001, SD = 0.04, p = 0.980; P 2: B = 0.005, SD = 0.03, p = 0.872; P 3: B = 0.038, SD = 0.03, p = 0.178; P 5: B = 0.021, SD = 0.03, p = 0.480; P 6: B = 0.007, SD = 0.03, p = 0.802; P 8: B = 0.034, SD = 0.04, p = 0.450). One participant showed an increase (P 7: B = 0.079, SD = 0.03, p = 0.009) and one a decrease (P 4: B = −0.065, SD = 0.03, p = 0.047). In the experimental condition (biocueing on), no significant effect of the intervention on stress-levels was found for six out of eight participants (P 1: B = 0.007, SD = 0.06, p = 0.903; P 2: B = −0.068, SD = 0.04, p = 0.129; P 3: B = −0.018, SD = 0.04, p = 0.177; P 5: B = 0.008, SD = 0.05, p = 0.868; P 6: B = 0.025, SD = 0.04, p = 0.511; P 8: B = 0.017, SD = 0.11, p = 0.870). One participant showed an increase (P 4: B = 0.094, SD = 0.04, p = 0.03) and one participant showed a decrease (P 7: B = −0.154, SD = 0.06, p = 0.01). Improvement rate difference (IRD) effect sizes ranged from 0.16 to 0.35, which is indicative of a small effect size of the combination of biocueing and ENYOY on perceived stress-levels. Results from the level-two ordinary least squares regression analysis show that, across cases, there was no significant effect of the intervention on perceived stress-levels, (B = −0.020, SD = 0.03, t(6.39) = –0.61, p = 0.562). These findings indicate that biocueing in combination with ENYOY does not decrease perceived stress-levels among participants.

TABLE 1

Table 1. Summary of study results for dependent variables per participant and per condition.

FIGURE 2

Figure 2. Perceived stress-levels (left) and emotional awareness (right) as a function of condition and measuring time per participant.

No significant effect for emotional awareness in the control condition was found for five out of eight participants, see Table 1 and Figure 2 (P 1: B = −0.027, SD = 0.02, p = 0.17; P 2: B = 0.009, SD = 0.02, p = 0.644; P 3: B = 0.016, SD = 0.02, p = 0.487; P 4: B = −0.002, SD = 0.01, p = 0.891; P 8: B = 0.018, SD = 0.02, p = 0.456). Three participants showed a decrease in levels of emotional awareness (P 5: B = −0.027, SD = 0.01, p = 0.012; P 6: B = −0.071, SD = 0.02, p < 0.001; P 7, B = −0.049, SD = 0.02, p = 0.004). In the experimental condition, no significant effect was found for six out of eight participants (P 1: B = 0.044, SD = 0.03, p = 0.111; P 2: B = 0.034, SD = 0.03, p = 0.211; P 3: B = 0.002, SD = 0.03, p = 0.941; P 4: B = 0.013, SD = 0.02, p = 0.484; P 5: B = 0.008, SD = 0.02, p = 0.612; P 8: B = 0.002, SD = 0.06, p = 0.966). Two participants showed an increase in levels of emotional awareness (P 6: B = 0.061, SD = 0.02, p = 0.008; P 7: B = 0.039, SD = 0.03, p = 0.007). The IRD for significant results ranged from 0.05 to 0.31, which is indicative of a small effect size. Results from the level-two ordinary least squares regression analysis showed that, across cases, there was a significant fixed interaction effect of time and condition on emotional awareness (B = 0.030, SD = 0.01, t (6.71) = 2.42, p = 0.048), indicating that in the experimental condition, emotional awareness increased significantly over time. These findings indicate that the combination of ENYOY and biocueing could increase emotional awareness.

Phase 3–Evaluating the Intervention With End Users

Thematic analysis from the evaluative interviews yielded the following themes: usability of the Sense-IT, integration of the Sense-IT and the ENYOY-platform, and ideas for improvement of the Sense-IT and the ENYOY-platform.

The overall usability of the Sense-IT could be rated as moderate with reported usability issues. On the one hand, positive experiences were reported. All participants mentioned that they reflected more on how they were feeling and three noted that this helped them understand their emotions better. All participants mentioned making an effort to change something in their behavior because of the biofeedback, such as pausing and evaluating the situation or doing a breathing exercise. Scores on the Health-ITUES indicated that participants judged the impact and usefulness of the intervention to be moderate (M = 3.93, SD = 0.43; M = 3.71, SD = 0.42, see Table 2). These scores indicate that the intervention impacted the participant's daily life moderately and participants were moderately able to accomplish tasks through system use.

TABLE 2

Table 2. Descriptive statistics for subscales of the Health ITUES subscales.

P5: “I liked using it. For me personally, the watch was quite big, as I have small wrists. I liked the notifications, like: “Hey, watch out, your heartbeat is quite high, is something the matter?” [...] having the device tell me that something might be happening [within me] helped me to be mindful of that. I liked that very much.” ¹

On the other hand, participants mentioned several technical problems. Most participants [5] noted difficulties with the interface, resulting from problems with making notes in the app [2], not receiving text notifications [1], the watch display changing to default settings [1], and difficulties in figuring out how to know when the biocueing intervention was turned on [1]. Furthermore, most participants [5] expressed doubts about the accuracy of the heartbeat measurements, with participants receiving too much [4] or too little biofeedback [1], and three participants mentioned that their baseline measurement failed multiple times.

P6: “Sometimes I got a notification while I was sitting still, it said that my heartbeat was too high and I thought ‘That's weird', because I wasn't doing anything stressful. There was also a situation in which I was stressed but I didn't see that on the watch”

Lastly, half of the participants mentioned that the usage of the intervention was disrupted due to limited battery life of the smartwatch. Half found the appearance of the smartwatch acceptable, others noted that the smartwatch was too big [2], did not fit their style [1], or found the strap uncomfortable [1]. The average system usability score (SUS) of the group was above the cut-off score (M = 63.78, SD = 10.96), meaning the intervention was permissible in terms of acceptance, though indicating that there are usability issues that are cause for concern. Results from the Health-ITUES yielded a moderate usability score (M = 3.69, SD = 0.30, ranging from 2 to 4.33; also see Table 2 for average scores per subscale).

Regarding the integration of the Sense-IT and the ENYOY-platform, the breathing exercises from the ENYOY-platform seemed the most suitable choice when stressed and were done by most participants [7]. Half of the participants did not use the ENYOY-platform during the study, reasons being not having thought about it or not having time [2], finding it difficult to use ENYOY due to double verification which is used for logging in [1], or already having used the ENYOY-platform extensively in the past [1]. Scores on the Health-ITUES were reflective of this fact, as user control and ease of use was judged to be moderate (M = 3.63, SD = 0.46, see Table 2), indicating that the user-system interaction was not optimal.

P2: “Well, I haven't thought about it [using the ENYOY-platform], and often when it [the watch] vibrated, I wasn't near my laptop. I like it better to use it on my laptop than to use it on my phone. I didn't open my laptop the past week, so that must've played a role.”

Of the participants that had used the platform, most noted that they used the platform on their laptop [3] and that they planned time in their agenda to go on the platform [2]. The most used online tools were the toolkit [2] and the explore function [2]. Participants expressed their need for exercises that are short [2], are adapted to their needs [1], are practical [1], or include psycho-education [1]. Half of the participants mentioned that they did not always carry the research smartphone with them, which hindered them from using the intervention fully. They mentioned that they would use the Sense-IT more often if the research smartphone was directly connected to the ENYOY-platform or if they could use their own smartphone [both of which were not possible in this study because the research smartphone was an ‘offline' device for privacy concerns (see methods)].

P5: “I used it [the ENYOY-platform] once or twice, because this phone wasn't connected to the internet, which made it more of an effort, because then I had to see when I receive a notification and ask myself “why, when?” and then I have to use the platform here [in the bedroom] on my laptop. So for me, it's a bit of an obstacle, because there are more actions involved.”

Some participants thought they would be more inclined to use ENYOY if there were an app [3], if logging in was made easier [1], and if reminders for using the platform were given [1].

Participants gave many suggestions for improvement of the Sense-IT and the ENYOY-platform to further integrate the two interventions and to increase user experience (see Table 3). Important suggestions referred to the accessibility of the ENYOY-platform, the integration between the platform and the Sense-IT, and bugs in the Sense-IT.

TABLE 3

Table 3. Overview of bugs, problems and proposed solutions from the semi-structured evaluative interviews.

Discussion

The focus of this study was to investigate the usability, feasibility, and effects of biocueing in addition to the moderated online social therapy-platform ENYOY in young people with emerging mental health problems, following a user-centered (47) mixed-method design framework. The current study had three objectives: 1) to qualitatively explore whether biocueing could be of additional value to ENYOY; 2) to assess the exploratory effect of biocueing in combination with ENYOY on perceived stress and emotional awareness; 3) to evaluate the attitude of users concerning the usability, acceptance, and user-friendliness of biocueing in combination with ENYOY as an intervention.

The findings from the phase 1 (Understanding and specifying end user needs regarding the interventions) focus group indicated an additional value of a biocueing intervention to ENYOY and resulted in several suggestions regarding briefing, access, and integration for adaptation of the smartwatch to the ENYOY-platform and vice versa, which were implemented for phase 2 (Testing the intervention with end users). The compliance rate of phase 2 was on average 78.8%, which is similar to compliance rates in patient populations (66), and slightly higher then compliance rates for digital therapy platforms for youths [63,81% on average, (43)]. Exploratory findings of phase 2 showed different individual effects of biocueing on perceived stress and emotional awareness. Overall, no effects were found on perceived stress-levels, suggesting that the current combination of ENYOY and biocueing does not decrease perceived stress in young people. Positive exploratory effects were found on emotional awareness, suggesting that the combination of ENYOY and biocueing could increase emotional awareness and thereby possibly poses a relevant addition to a digital treatment platform. Nonetheless, some caution is advised because of the small sample size of this explorative feasibility study, and since Improvement Rate Difference [IRD; (59)] showed small effect sizes of the intervention.

Results from phase 3 (Evaluating the intervention with end users) showed improvements on reflection on feelings and positive changes in behavior - such as pausing and evaluating the situation, doing an exercise on ENYOY, and having a rest - following the biocueing intervention combined with the use of ENYOY. Participants preferred exercises on the ENYOY-platform that were short, adapted to their needs, practical, or included psycho-education. The most common tools used were breathing exercises, the toolkit (a library where the young person can save his/hers favorite exercises), and the explore function (a quick search function to show exercises per category, e.g., “rumination” or “stress”). Not all participants used the platform during the experimental phase, a reason being that the integration of the two interventions was very rudimentary due to the early implementation. These kinds of new intervention products are also known as Minimum Viable Products (MVP), which are first versions of a product or service which are being delivered to a target group as early as possible. The most common issues for MVPs are the non-optimal technical support and lack of resources (67), as was the case in this study. The intervention was found to be permissible in terms of acceptance and showed a moderate usability, with reported usability issues. The most common issues that were reported concerned the use of the smartwatch (e.g., technical issues, difficulties with the interface, doubts about the accuracy of the heart rate measurements, and limited battery life), and design of the smartwatch (appearance not fitting their style and lacking wearing comfort), as well as the non-optimal integration of the smartwatch and the ENYOY-platform (not being available on the same device). Main suggestions for improvement were related to the accessibility (e.g., the option to use both interventions on one device, within one app, and add reminders), engagement of users (e.g., adding gamification elements, create positive feedback for the platform, and make use of fun elements such as a quiz), and the integration (e.g., show heartbeat measurements of the smartwatch on the ENYOY dashboard).

As indicated by the results, the intervention significantly increased emotional awareness, and did not significantly decrease perceived stress. This means that the biocueing intervention in addition to ENYOY seems to make participants more aware of their (negative) emotional states and does not seem to play a part in decreasing perceived stress. The last is contrary to previous research where decreases in self-reported stress-levels were found (23). There are several factors that could play a role in this finding. First, stress was measured using one survey item, asking participants to indicate their stress-levels on Likert scale. It is questionable whether a single survey item, prompted twice a day, can accurately encapsulate a multifaceted concept like stress (68). Even though subjective measures have their obvious limitations, their merit lies in the ability to measure beliefs, thoughts, emotions, and attitudes about stress (69). Stress is not just the simple product of a perceived threat; according to the Transaction Model, stress is a product of the antecedent personal resources and external stressors, mediated by coping, followed by both short- and long-term effects (70). Studying the person-environment interaction and coping may provide additional valuable insight on the way these factors play a role in stress-reduction through biocueing. Furthermore, participants participating in the intervention had relatively low levels of stress, as indicated by the low levels of stress in the control condition. Therefore, stress had a low mean score and had little room for change. This little room for change was emphasized by the short period of time of the experiment: two weeks may not be enough to observably change stress in participants. Additionally, there were no fixed time frames for the measurements. This means that participants were free to answer the stress-related survey item any time after the question was prompted. Presumably, participants who experience heightened stress may not be able to direct their time toward answering the survey question and therefore answer the question later when heightened stress has subsided.

Even though emotional awareness plays a role in stress reduction, being more emotionally aware alone may not automatically cause someone to experience less stress without active attempts to reduce it (71). A biocueing wearable functions as an indicative system that provides the user with feedback on symptoms associated with stress, such as heightened heart rate through skin conductance. Participants that experience stress, but do not make an effort to reduce it through meaningful effort, may therefore become more aware of aversive feelings, but not experience relief from it. To emphasize, only half of all participants used ENYOY during the experimental phase, which raises the question how many participants made an effort to actively reduce stress if any at all. Participants felt that the integration of Sense-IT and ENYOY was lacking. Participants could not use their own phone and carried a research phone that was not connected to the internet or offering direct access to the ENYOY-platform for privacy concerns (an ‘online' device is more prone to data leaks and this was found too big a risk for the target population and the nature of their complaints), which was seen as a limiting factor in terms of usability. Participants mentioned they were more likely to use ENYOY if these issues were addressed and participants did not have to switch devices to access ENYOY. Furthermore, increased awareness of bodily sensations may adversely increase anxiety if they are interpreted catastrophically (72). Some young people might stop wearing their smart watches because the watches disturb their life by making them too 'aware' of their stress and situation. Finally, in the field of clinical psychology experiencing negative emotions is seen more and more as being a normal part of life. A stance more toward accepting these and learning how to deal with them might provide individuals with a more fulfilling life than striving to get rid of them [e.g., (73)]. Moreover, one of the keystones of ENYOY is normalizing difficult emotions (18). In this way, one could wonder whether the goal of biocueing and ENYOY should be to reduce stress, or to provide ways in how to deal with the stress that a person has in his or her life. For future research it would be of interest to not only measure stress-levels, but, more importantly, coping with stress.

The individual differences in effects of the intervention could have multiple reasons. First of all, in individual cases it is uncertain whether results are due to progress rather than time [(74); also see Limitations]. Moreover, individual differences in baseline stress-levels and emotional awareness [e.g., (75, 76)] could mean that for some there was less room for improvement (77) and for others that 10 days of biocueing was insufficient. Interestingly, three out of eight participants showed a decrease in emotional awareness in the control condition, of which two subsequently showed an increase in the experimental condition. This reinforces the idea that the intervention itself changed outcomes rather than time. Moreover, biocueing might not be a suitable intervention for all. If an individual already has a hypersensitivity toward one's internal state and/or experiences panic because of bodily sensation [e.g., for individuals who suffer from illness anxiety disorder, see (78); or individuals who are diagnosed with bipolar disorder, see (79)], biocueing could exacerbate this focus and panic even more. Another example regards individuals with problems with interoceptive sensibility (the capacity to focus on internal sensations and take them into consideration), such as individuals with autism and/or alexithymia, where difficulties are found in discriminating among interoceptive signals, which might result in difficulties understanding one's bodily states (80). For these individuals, a biocueing intervention could be especially challenging since it focuses mainly on interoceptive accuracy (a higher heartbeat is identified) and not on interoceptive sensibility (what does this mean to me?). Extra treatment regarding how to interpret, distinguish, and deal with these sensations and emotions would be required (80). Although ENYOY offers some exercises along these lines, this might be insufficient for some. All in all, the individual differences in effects indicate that the biocueing intervention might not be suitable for every individual, which should be addressed before providing individuals with a smartwatch.

A main point that might have countered the true added effects of biocueing to ENYOY is the usability of the biocueing intervention. The overall usability of the intervention was found to be moderate, with reported usability issues. Even though participants reported that the intervention helped them reflect more on how they were feeling or helped them in understanding their emotions, several issues were reported that impacted the usability of the smartwatch. Half of all participants found the smartwatch either too large, not fitting their style, or too uncomfortable. These factors, “aesthetics,” play an important part in deciding to either “use or lose” the smartwatch (44, 81, 82), even more so in adolescents who highly value the aesthetics of wearables (83). Therefore, it is important that the smartwatch fits a participant's sense of style to increase usability, which can either be achieved through customizable options or a choice of different styles. The Sense-IT application which was used in the study works on any Android OS watch, however for the purpose of this study all participants received the same smartwatch. One could imagine that a flashy self-chosen smartwatch linked seamlessly to a digital platform would significantly improve the link between signaling and outcomes.

Limitations and Strengths

Several strengths and limitations of this study should be highlighted. A limitation of this study is that not all suggestions opted by the phase 1 focus group could be implemented for the intervention due to missing technical support and resources. This could have contributed to the lower usability and acceptance rates and lower use of ENYOY, and is a missing requirement that is necessary for a user-centered design (47). However, we hope that our findings could provide useful information for future research and implementation. Another limitation of this study are the compliance rates. Over 20% of measurements were missing and significantly more data was missing in the experimental condition. This was dealt with using Expectation-Maximization; a method that is found to keep the power sensitivity stable in case of missing data, even up to 40% (65). Additionally, an AB (A control phase, B experimental phase) approach was opted, which means it cannot be said with certainty that individual effects in phase B are due to progress rather than time. For this, at least three phase changes (ABAB) are necessary. This was compensated by using the multiple single-case design which eliminates the need to return to baseline (74), and enables to make conclusions about the intervention at group-level. Moreover, this study had a relatively small sample size. This was deemed fit for the type of research (an explorative feasibility study), and was chosen to not unnecessarily burden participants. Nonetheless, the small sample size could have affected the reliability of the results. The contrasting direction of the individual responses found in the level-one analysis of phase 2 of this study are also indicative of a greater sampling variability. Lastly, all participants were female and had followed higher vocational education or university. This could have influenced the representativeness of the user group in relation to the overall population.

A strength of this study is its user-centered mixed-method design. The end users played a central role in every iteration of the study and were involved in the developmental process of the integration of the ENYOY-platform and the smartwatch (co-creation). This provided a deeper understanding of psychological, social, and ergonomic factors related to the used technology. Though a pitfall of this could be that the end product would be too specific for general use and less transferable to other clients. Moreover, this method has been found to improve effectiveness, efficiency, and safety of technological products (47). The use of mixed-methods also provided us with rich, comprehensive data by integrating qualitative and quantitative data (84). The use of ecological momentary assessment further allowed us to measure in real-time, which minimizes recall bias and maximizes ecological validity (48).

Considerable efforts were made to increase and ensure quality control. The Sense-IT app has already been used in previous studies after being developed with well-recognized research methods (52, 57) which decreases the risk of statistical anomalies due to hard- or software factors. In addition, the Sense-IT was built in accordance with the Medical Devices Directive (MDD) and Active Implantable Medical Devices Directive (AIMDD) ensuring that Sense-IT has to meet certain quality standards (54). Additionally, the SUS (60) and S-DERS (55) have shown adequate validity and reliability making it useful tools for this study. Furthermore, the Health-ITUES (62) which had no original Dutch version, was carefully translated back-and-forth (English-Dutch, Dutch-English, and English-Dutch) between four independent translators to limit diminished validity and reliability through mistakes in translation.

Future Research

For future research, foremost, usability studies are recommended to improve the smartwatch and integration with a digital platform. This study highlighted the need for research with the goal of full optimization, preferably in co-creation with end-users [see for example (52, 85, 86)]. It would be of further relevance to gain a better understanding of which individuals would, and which would not, benefit from a biocueing intervention (in addition to a digital treatment platform). In the future, it would be valuable to connect the ENYOY-platform to the smartwatch in such a manner that the activities of young people on the platform could be followed to see which part(s) of the platform is/are most effective to relieve stress. Moreover, multidimensional assessments of both stress and emotional awareness could provide valuable new insights [also see (69)]. It would also be of interest to investigate whether an optimized detection of stress and corresponding interventions lead to greater effects; and to evaluate via rigorous designs whether biocueing improves outcomes. Lastly, a positive psychology framework using ecological momentary assessment could be opted to see the effects on e.g., psychological wellbeing, since not all stress and ‘negative' emotions are necessarily a bad thing.

Recommendations

All in all, biocueing could be a promising intervention to add to a digital treatment platform to help young people become more aware of their emotions. Be that as it may, notable improvements have to be made regarding the usability and acceptability of the smartwatch as well as more extensive integration of the smartwatch to a digital treatment platform before further implementing such an approach. Moreover, it is of importance to tailor the need to the individual of such an added intervention since large individual differences exist. The intervention might not work for all. It is of relevance to discover more about the contraindications before using a smartwatch. Finally, young people indicated that adding elements of gamification could improve learning and the use of the interventions.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by METC, AMC: NL66345.018.18. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

MD, DN, TA, AP, MN, and MJ completed the initial study design. AM was involved with the execution of all phases of the study. DN, TA, MJ, AP, MN, and MA-J provided an expert assessment and feedback. The manuscript was written by MD, AM, and LN. All authors read and approved the final manuscript.

Funding

Financial funding by The Netherlands Organisation for Health Research and Development for personnel, materials, and implementation (File number 60-63600-98-319).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We would like to thank the organizations involved in the development of Sense-IT for use in this study project: University of Twente, Scelta/GGNet, Amsterdam University Medical Centers, Inforsa/Arkin, and Pluryn.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2022.871813/full#supplementary-material

Footnotes

1. ^All quotes were translated from Dutch to English.

References

1. CBS. Mentale gezondheid in eerste helft 2021 op dieptepunt [Mental Health in the First Half of 2021 at All Time Low] (2021). Available online at: https://www.cbs.nl/nl-nl/nieuws/2021/35/mentale-gezondheid-in-eerste-helft-2021-op-dieptepunt (accessed December 10, 2021).

2. Jacobi F, Höfler M, Siegert J, Mack S, Gerschler A, Scholl L, et al. Twelve-month prevalence, comorbidity and correlates of mental disorders in Germany: the mental health module of the german health interview and examination survey for adults (DEGS1-MH). Int J Methods Psychiatr Res. (2014) 23:304–19. doi: 10.1002/mpr.1439

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Wagner G, Zeiler M, Waldherr K, Philipp J, Truttmann S, Dür W, et al. Mental health problems in Austrian adolescents: a nationwide, two-stage epidemiological study applying DSM-5 criteria. Eur Child Adolesc Psychiatry. (2017) 26:1483–99. doi: 10.1007/s00787-017-0999-6

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Kessler RC, Chiu WT, Demler O, Merikangas KR, Walters EE. Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. (2005) 62:617–27. doi: 10.1001/archpsyc.62.6.617

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Galván A. Adolescence, brain maturation and mental health. Nat Neurosci. (2017) 20:503–4. doi: 10.1038/nn.4530

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Reeves WC, Strine TW, Pratt LA, Thompson W, Ahluwalia IB, Dhingra SS, et al. Mental illness surveillance among adults in the United States. Morb Mortal Wkly Rep. (2011) 60:1–29. Available online at: https://europepmc.org/article/med/21881550

PubMed Abstract | Google Scholar

7. Rüsch N, Corrigan P, Todd A, Bodenhausen G. Implicit self-stigma in people with mental illness. J Nerv Ment Dis. (2010) 198:150–3. doi: 10.1097/NMD.0b013e3181cc43b5

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Green AC, Hunt C, Stain HJ. The delay between symptom onset and seeking professional treatment for anxiety and depressive disorders in a rural Australian sample. Soc Psychiatry Psychiatr Epidemiol. (2012) 47:1475–87. doi: 10.1007/s00127-011-0453-x

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Evans-Lacko S, Knapp M. Global patterns of workplace productivity for people with depression: absenteeism and presenteeism costs across eight diverse countries. Soc Psychiatry Psychiatr Epidemiol. (2016) 51:1525–37. doi: 10.1007/s00127-016-1278-4

PubMed Abstract | CrossRef Full Text | Google Scholar

10. WHO. Economic Aspects of Mental Health in Children and Adolescents. Geneva: Author; (2007). Available online at: https://www.euro.who.int/en/health-topics/Life-stages/child-and-adolescent-health/publications/2007/economic-aspects-of-mental-health-in-children-and-adolescents (accessed December 10, 2021).

11. VWS. Vaststelling begroting ministerie van volksgezondheid, welzijn en sport [Declaration Budget Ministry of Public Health, Welfare and Sport]. (2019). Available online at: https://archief.rijksbegroting.nl/2020/kamerstukken,2019/10/21/kst266935.html (accessed December 10, 2021).

12. McGorry PD, Tanti C, Stokes R, Hickie IB, Carnell K, Littlefield LK, et al. Headspace: Australia's national youth mental health foundation-where young minds come first. Med J Aust. (2007) 187:S68–70. doi: 10.5694/j.1326-5377.2007.tb01342.x

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Zimmermann P, Iwanski A. Emotion regulation from early adolescence to emerging adulthood and middle adulthood: age differences, gender differences, and emotion-specific developmental variations. Int J Behav Dev. (2014) 38:182–94. doi: 10.1177/0165025413515405

CrossRef Full Text | Google Scholar

14. Alvarez-Jimenez M, Gleeson JF, Bendall S, Penn DL, Yung AR, Ryan RM, et al. Enhancing social functioning in young people at Ultra High Risk (UHR) for psychosis: a pilot study of a novel strengths and mindfulness-based online social therapy. Schizophr Res. (2018) 202:369–77. doi: 10.1016/j.schres.2018.07.022

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Alvarez-Jimenez M, Bendall S, Koval P, Rice S, Cagliarini D, Valentine L, et al. HORYZONS trial: protocol for a randomised controlled trial of a moderated online social therapy to maintain treatment effects from first-episode psychosis services. BMJ Open. (2019) 9:e024104. doi: 10.1136/bmjopen-2018-024104

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Alvarez-Jimenez M, Koval P, Schmaal L, Bendall S, O'Sullivan S, Cagliarini D, et al. The Horyzons project: a randomized controlled trial of a novel online social therapy to maintain treatment effects from specialist first-episode psychosis services. World Psychiatry. (2021) 20:233–43. doi: 10.1002/wps.20858

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Alvarez-Jimenez M, Rice S, D'Alfonso S, Leicester S, Bendall S, Pryor I, et al. A novel Multi-modal Online Service (MOST+) for help-seeking young people experiencing mental ill-health: a pilot evaluation within a national youth e-mental health service. J Med Intern Res. (2020) 22:e17155. doi: 10.2196/17155

PubMed Abstract | CrossRef Full Text | Google Scholar

18. van Doorn M, Popma A, van Amelsvoort T, McEnery C, Gleeson JF, Ory FG, et al. ENgage YOung people earlY (ENYOY): a mixed-method study design for a digital transdiagnostic clinical – and peer- moderated treatment platform for youth with beginning mental health complaints in the Netherlands. BMC Psychiatry. (2021) 21:368. doi: 10.1186/s12888-021-03315-x

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Van Beveren ML, Goossens L, Volkaert B, Grassmann C, Wante L, Vandeweghe L, et al. How do I feel right now? Emotional awareness, emotion regulation, and depressive symptoms in youth. Eur Child Adolesc Psychiatry. (2019) 28:389–98. doi: 10.1007/s00787-018-1203-3

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Gross JJ. The emerging field of emotion regulation: an integrative review. Rev Gen Psychol. (1998) 2:271–99. doi: 10.1037/1089-2680.2.3.271

CrossRef Full Text | Google Scholar

21. Sendzik L, Ö Schäfer J, C Samson A, Naumann E, Tuschen-Caffier B. Emotional awareness in depressive and anxiety symptoms in youth: a meta-analytic review. J Youth Adolesc. (2017) 46:687–700. doi: 10.1007/s10964-017-0629-0

PubMed Abstract | CrossRef Full Text | Google Scholar

22. McKee MG. Biofeedback: an overview in the context of heart-brain medicine. Cleve Clin J Med. (2008) 75 Suppl 2:S31–4. doi: 10.3949/ccjm.75.Suppl_2.S31

PubMed Abstract | CrossRef Full Text | Google Scholar

23. ter Harmsel JF, Noordzij ML, Goudriaan AE, Dekker JJM, Swinkels LTA, van der Pol M, et al. Biocueing and ambulatory biofeedback to enhance emotion regulation: a review of studies investigating non-psychiatric and psychiatric populations. Int J Psychophysiol. (2021) 159:94–106. doi: 10.1016/j.ijpsycho.2020.11.009

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Blakemore SJ. Adolescence and mental health. Lancet. (2019) 393:2030–1. doi: 10.1016/S0140-6736(19)31013-X

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Pringle J, Mills K, McAteer J, Jepson R, Hogg E, Anand N, et al. A systematic review of adolescent physiological development and its relationship with health-related behaviour: a protocol. Syst Rev. (2016) 5:3. doi: 10.1186/s13643-015-0173-5

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Spear LP. Heightened stress responsivity and emotional reactivity during pubertal maturation: Implications for psychopathology. Dev Psychopathol. (2009) 21:87–97. doi: 10.1017/S0954579409000066

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Steinberg L. Cognitive and affective development in adolescence. Trends Cogn Sci. (2005) 9:69–74. doi: 10.1016/j.tics.2004.12.005

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Eschenbeck H, Schmid S, Schröder I, Wasserfall N, Kohlmann C-W. Development of coping strategies from childhood to adolescence: cross-sectional and longitudinal trends. Eur J Health Psychol. (2018) 25:18–30. doi: 10.1027/2512-8442/a000005

CrossRef Full Text | Google Scholar

29. Lane RD, Schwartz GE. Levels of emotional awareness: a cognitive-developmental theory and its application to psychopathology. Am J Psychiatry. (1987) 144:133–43. doi: 10.1176/ajp.144.2.133

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Penza-Clyve S, Zeman J. Initial validation of the Emotion Expression Scale for Children (EESC). J Clin Child Adolesc Psychol. (2002) 31:540–7. doi: 10.1207/S15374424JCCP3104_12

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Maddock C, Pariante CM. How does stress affect you? An overview of stress, immunity, depression and disease. Epidemiol Psichiatr Soc. (2001) 10:153–62. doi: 10.1017/S1121189X00005285

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Dieleman GC, Huizink AC, Tulen JH, Utens EM, Tiemeier H. Stress reactivity predicts symptom improvement in children with anxiety disorders. J Affect Disord. (2016) 196:190–9. doi: 10.1016/j.jad.2016.02.022

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Charbonneau A, Mezulis A, Hyde J. Stress and emotional reactivity as explanations for gender differences in adolescents' depressive symptoms. J Youth Adolesc. (2009) 38:1050–8. doi: 10.1007/s10964-009-9398-8

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Walker AJ, Kim Y, Price JB, Kale RP, McGillivray JA, Berk M, et al. Stress, inflammation, and cellular vulnerability during early stages of affective disorders: biomarker strategies and opportunities for prevention and intervention. Front Psychiatry. (2014) 5:34. doi: 10.3389/fpsyt.2014.00034

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Rice S, Gleeson J, Davey C, Hetrick S, Parker A, Lederman R, et al. Moderated online social therapy for depression relapse prevention in young people: pilot study of a ‘next generation' online intervention. Early Interv Psychiatry. (2018) 12:613–25. doi: 10.1111/eip.12354

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Rice SM, Goodall J, Hetrick SE, Parker AG, Gilbertson T, Amminger GP, et al. Online and social networking interventions for the treatment of depression in young people: a systematic review. J Med Internet Res. (2014) 16:e206. doi: 10.2196/jmir.3304

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Lehalle H. Cognitive development in adolescence: thinking freed from concrete constraints. In: Handbook of Adolescent Development. 1st ed. New York: Psychology Press (2020). p. 71–89. doi: 10.4324/9780203969861-5

CrossRef Full Text | Google Scholar

38. Spencer S, Clegg J, Stackhouse J. Language and disadvantage: a comparison of the language abilities of adolescents from two different socioeconomic areas. Int J Lang Commun Disord. (2012) 47:274–84. doi: 10.1111/j.1460-6984.2011.00104.x

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Gibson J. Mindfulness, interoception, and the body: a contemporary perspective. Front Psychol. (2019) 10:2012. doi: 10.3389/fpsyg.2019.02012

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Li D, Zucker NL, Kragel PA, Covington VE, LaBar KS. Adolescent development of insula-dependent interoceptive regulation. Dev Sci. (2017) 20:1–16. doi: 10.1111/desc.12438

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Nordby ES, Kenter RMF, Lundervold AJ, Nordgreen T. A self-guided Internet-delivered intervention for adults with ADHD: a feasibility study. Internet Interv. (2021) 25:100416. doi: 10.1016/j.invent.2021.100416

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Achilles MR, Anderson M, Li SH, Subotic-Kerry M, Parker B, O'Dea B. Adherence to e-mental health among youth: considerations for intervention development and research design. Digit Health. (2020) 6:1–9. doi: 10.1177/2055207620926064

PubMed Abstract | CrossRef Full Text | Google Scholar

43. van Doorn M, Nijhuis LA, Egeler MD, Daams JG, Popma A, van Amelsvoort T, et al. Online indicated prevention mental health interventions for youth: a scoping review. Front Psychiatry. (2021) 12:1–19. doi: 10.3389/fpsyt.2021.580843

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Nelson EC, Verhagen T, Noordzij ML. Health empowerment through activity trackers: an empirical smart wristband study. Comput Hum Behav. (2016) 62:364–74. doi: 10.1016/j.chb.2016.03.065

CrossRef Full Text | Google Scholar

45. de Looff P, Noordzij ML, Moerbeek M, Nijman H, Didden R, Embregts P. Changes in heart rate and skin conductance in the 30 min preceding aggressive behavior. Psychophysiology. (2019) 56:e13420. doi: 10.1111/psyp.13420

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Füstös J, Gramann K, Herbert BM, Pollatos O. On the embodiment of emotion regulation: interoceptive awareness facilitates reappraisal. Soc Cogn Affect Neurosci. (2013) 8:911–7. doi: 10.1093/scan/nss089

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Abras C, Maloney-Krichmar D, Preece J. User-centered design. In: Brainbridge WS, editor. Berkshire Encyclopedia of Human-Computer Interaction. 2nd ed. Great Barrington, MA Berkshire (2004). p. 763–8.

48. Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol. (2008) 4:1–32. doi: 10.1146/annurev.clinpsy.3.022806.091415

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Brender J, Talmon J, de Keizer N, Nykänen P, Rigby M, Ammenwerth E, et al. Statement on reporting of evaluation studies in health informatics: explanation and elaboration. Appl Clin Inform. (2013) 4:331–58. doi: 10.4338/ACI-2013-04-RA-0024

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Acocella I. The focus groups in social research: advantages and disadvantages. Qual Quant. (2012) 46:1125–36. doi: 10.1007/s11135-011-9600-4

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Bennett D, Barrett A, Helmich E. How to analyse qualitative data in different ways. Clin Teach. (2019) 16:7–12. doi: 10.1111/tct.12973

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Derks YP, Klaassen R, Westerhof GJ, Bohlmeijer ET, Noordzij ML. Development of an ambulatory biofeedback app to enhance emotional awareness in patients with borderline personality disorder: multicycle usability testing study. JMIR Mhealth Uhealth. (2019) 7:e13479-e. doi: 10.2196/13479

PubMed Abstract | CrossRef Full Text | Google Scholar

53. McGorry PD, Hickie IB, Yung AR, Pantelis C, Jackson HJ. Clinical staging of psychiatric disorders: a heuristic framework for choosing earlier, safer and more effective interventions. Aust N Z J Psychiatry. (2006) 40:616–22. doi: 10.1080/j.1440-1614.2006.01860.x

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Noordzij ML, ter Harmsel JF, Nijhof KS. Investigational Medical Device Dossier. SenseIT app, version 3. Twente: University of Twente (2018).

55. Lavender JM, Tull MT, DiLillo D, Messman-Moore T, Gratz KL. Development and validation of a state-based measure of emotion dysregulation. Assessment. (2017) 24:197–209. doi: 10.1177/1073191115601218

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Kirtley OJ, Hiekkaranta AP, Kunkels YK, Verhoeven D, Van Nierop M, Myin-Germeys I. The Experience Sampling Method (ESM) Item Repository. (2019). Available online at: osf.io/kg376 (accessed December 30, 2021).

57. ter Harmsel A, van der Pol T, Swinkels L, Goudriaan AE, Popma A, Noordzij ML. Development of a wearable biocueing app (Sense-IT) among forensic psychiatric outpatients with aggressive behavior: design and evaluation study. JMIR Form Res. (2021) 5:e29267. doi: 10.2196/29267

PubMed Abstract | CrossRef Full Text | Google Scholar

58. Declercq L, Cools W, Beretvas SN, Moeyaert M, Ferron JM, Van den Noortgate W. MultiSCED: a tool for (meta-)analyzing single-case experimental data with multilevel modeling. Behav Res Methods. (2020) 52:177–92. doi: 10.3758/s13428-019-01216-2

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Parker RI, Vannest KJ, Brown L. The improvement rate difference for single-case research. Except Child. (2009) 75:135–50. doi: 10.1177/001440290907500201

CrossRef Full Text | Google Scholar

60. Brooke J. SUS-A quick and dirty usability scale. In: Brooke J, editor. Usability Evalulation in Industry. London: Taylor & Francis (1996). p. 4–7.

61. Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum Comput Interact. (2008) 24:574–94. doi: 10.1080/10447310802205776

CrossRef Full Text | Google Scholar

62. Schnall R, Cho H, Liu J. Health Information Technology Usability Evaluation Scale (Health-ITUES) for usability assessment of mobile health rechnology: validation study. JMIR Mhealth Uhealth. (2018) 6:e4. doi: 10.2196/mhealth.8851

PubMed Abstract | CrossRef Full Text | Google Scholar

63. Scott C, Medaugh M. Axial Coding. New Jersey: John Wiley & Sons, Inc (2017). Available online at: https://onlinelibrary.wiley.com/doi/abs/10.1002/9781118901731.iecrm0012 (accessed March 30, 2021).

64. Vachon H, Viechtbauer W, Rintala A, Myin-Germeys I. Compliance and retention with the experience sampling method over the continuum of severe mental disorders: meta-analysis and recommendations. J Med Internet Res. (2019) 21:e14475. doi: 10.2196/14475

PubMed Abstract | CrossRef Full Text | Google Scholar

65. Chen LT, Feng Y, Wu PJ, Peng CJ. Dealing with missing data by EM in single-case studies. Behav Res Methods. (2020) 52:131–50. doi: 10.3758/s13428-019-01210-8

PubMed Abstract | CrossRef Full Text | Google Scholar

66. Myin-Germeys I, Kuppens P. The Open Handbook of Experience Sampling Methodology: A Step-by-step Guide to Designing, Conducting, and Analyzing ESM Studies. Leuven: Center for Research on Experience Sampling and Ambulatory Methods (2021).

67. Tripathi N, Oivo M, Liukkunen K, Markkula J. Startup ecosystem effect on minimum viable product development in software startups. Inform Softw Technol. (2019) 114:77–91. doi: 10.1016/j.infsof.2019.06.008

CrossRef Full Text | Google Scholar

68. Crosswell AD, Lockwood KG. Best practices for stress measurement: how to measure psychological stress in health research. Health Psychol Open. (2020) 7:1–12. doi: 10.1177/2055102920933072

PubMed Abstract | CrossRef Full Text | Google Scholar

69. Bornemann B, Herbert BM, Mehling WE, Singer T. Differential changes in self-reported aspects of interoceptive awareness through 3 months of contemplative training. Front Psychol. (2015) 5:1504. doi: 10.3389/fpsyg.2014.01504

PubMed Abstract | CrossRef Full Text | Google Scholar

70. Obbarius N, Fischer F, Liegl G, Obbarius A, Rose M. A modified version of the transactional stress concept according to Lazarus and Folkman was confirmed in a psychosomatic inpatient sample. Front Psychol. (2021) 12:584333. doi: 10.3389/fpsyg.2021.584333

PubMed Abstract | CrossRef Full Text | Google Scholar

71. McRae K, Misra S, Prasad AK, Pereira SC, Gross JJ. Bottom-up and top-down emotion generation: implications for emotion regulation. Soc Cogn Affect Neurosci. (2012) 7:253–62. doi: 10.1093/scan/nsq103

PubMed Abstract | CrossRef Full Text | Google Scholar

72. Feinstein JS, Khalsa SS, Yeh H, Al Zoubi O, Arevian AC, Wohlrab C, et al. The elicitation of relaxation and interoceptive awareness using floatation therapy in individuals with high anxiety sensitivity. Biol Psychiatry Cogn Neurosci Neuroimaging. (2018) 3:555–62. doi: 10.1016/j.bpsc.2018.02.005

PubMed Abstract | CrossRef Full Text | Google Scholar

73. Hayes SC. Acceptance and commitment therapy, relational frame theory, and the third wave of behavioral and cognitive therapies. Behav Ther. (2016) 47:869–85. doi: 10.1016/j.beth.2016.11.006

PubMed Abstract | CrossRef Full Text | Google Scholar

74. Krasny-Pacini A, Evans J. Single-case experimental designs to assess intervention effectiveness in rehabilitation: a practical guide. Ann Phys Rehabil Med. (2018) 61:164–79. doi: 10.1016/j.rehab.2017.12.002

PubMed Abstract | CrossRef Full Text | Google Scholar

75. Cameron OG. Interoception: the inside story — a model for psychosomatic processes. Psychosom Med. (2001) 63:697–710. doi: 10.1097/00006842-200109000-00001

PubMed Abstract | CrossRef Full Text | Google Scholar

76. Hankin BL, Badanes LS, Smolen A, Young JF. Cortisol reactivity to stress among youth: stability over time and genetic variants for stress sensitivity. J Abnorm Psychol. (2015) 124:54–67. doi: 10.1037/abn0000030

PubMed Abstract | CrossRef Full Text | Google Scholar

77. McBee M. Modeling outcomes with floor or ceiling effects: an introduction to the Tobit Model. Gift Child Q. (2010) 54:314–20. doi: 10.1177/0016986210379095

CrossRef Full Text | Google Scholar

78. Scarella TM, Boland RJ, Barsky AJ. Illness anxiety disorder: psychopathology, epidemiology, clinical characteristics, and treatment. Psychosom Med. (2019) 81:398–407. doi: 10.1097/PSY.0000000000000691

PubMed Abstract | CrossRef Full Text | Google Scholar

79. Faurholt-Jepsen M, Frost M, Christensen EM, Bardram JE, Vinberg M, Kessing LV. The effect of smartphone-based monitoring on illness activity in bipolar disorder: the MONARCA II randomized controlled single-blinded trial. Psychol Med. (2020) 50:838–48. doi: 10.1017/S0033291719000710

PubMed Abstract | CrossRef Full Text | Google Scholar

80. Poquérusse J, Pastore L, Dellantonio S, Esposito G. Alexithymia and autism spectrum disorder: a complex relationship. Front Psychol. (2018) 9:1196. doi: 10.3389/fpsyg.2018.01196

PubMed Abstract | CrossRef Full Text | Google Scholar

81. Harrison D, Marshall P, Bianchi-Berthouze N, Bird J. Activity tracking: barriers, workarounds and customisation. In: ACM International Joint Conference on Pervasive and Ubiquitous Computing. New Jersey: John Wiley & Sons Ltd; Osaka Association for Computing Machinery (2015). p. 617–21.

82. Hermsen S, Moons J, Kerkhof P, Wiekens C, De Groot M. Determinants for sustained use of an activity tracker: observational study. JMIR Mhealth Uhealth. (2017) 5:e164. doi: 10.2196/mhealth.7311

PubMed Abstract | CrossRef Full Text | Google Scholar

83. Muller C, Klerk ND. Influence of design aesthetics and brand name on beneration Y students' intention to use wearable activity-tracking devices. Int J eBus eGov Stud. (2020) 12:107–21. doi: 10.34111/ijebeg.202012202

CrossRef Full Text | Google Scholar

84. Wisdom J, Creswell JW. Mixed Methods: Integrating Quantitative and Qualitative Data Collection and Analysis While Studying Patient-Centered Medical Home Models. Rockville, MD: Agency for Healthcare Research Quality (2013). Available online at: https://www.ahrq.gov/sites/default/files/wysiwyg/ncepcr/tools/PCMH/mixed-methods.pdf (accessed March 30, 2021).

85. Feijt MA, de Kort YA, Bongers IM, IJsselsteijn WA. Perceived drivers and barriers to the adoption of eMental Health by psychologists: the construction of the levels of adoption of eMental Health Model. J Med Internet Res. (2018) 20:e153. doi: 10.2196/jmir.9485

PubMed Abstract | CrossRef Full Text | Google Scholar

86. Titzler I, Saruhanjan K, Berking M, Riper H, Ebert DD. Barriers and facilitators for the implementation of blended psychotherapy for depression: a qualitative pilot study of therapists' perspective. Internet Interv. (2018) 12:150–64. doi: 10.1016/j.invent.2018.01.002

PubMed Abstract | CrossRef Full Text | Google Scholar