In Case of an Emergency: The Development and Effects of a Digital Intervention for Coping With Distress in Norway During the COVID-19 Pandemic

Background: The COVID-19 pandemic and its consequences has been found to negatively affect the general population’s psychological well-being.

Objective: The objectives of this paper are to report on the development and clinical effects of a self-guided Internet-delivered intervention for adults in Norway who suffer from mild to moderate psychological distress during the COVID-19 pandemic.

Methods: The participants, recruited between April and December 2020, were randomized to receive a new treatment module either every third or every fifth day. The clinical outcomes were self-reported depressive and anxiety symptoms and change in positive and negative emotions.

Results: A total of 1256 individuals accessed the pre-screening survey, 407 were eligible and 92 provided contact information, where 82 were included in the study, n = 44 in the 3-day group and n = 38 in the 5-day group. Overall, the statistical analyses showed a significant decrease in depressive and anxiety symptoms and an increase in positive emotions, with small and moderate within group effect sizes. No significant differences between the groups were identified in clinical outcomes or adherence.

Conclusion: These findings indicate that psychological distress in the general population during the COVID-19 pandemic may be reduced through the use of a scalable self-guided Internet-delivered intervention. Furthermore, the lack of significant differences between the 5-day and 3-day group may indicate that the intervention can be delivered at a more intensive pace without negatively affecting treatment outcomes. The results need to be interpreted with caution as the sample was self-selected, as well as the lack of passive control group. Hence the results may be attributed to external factors.

Introduction

The COVID-19 pandemic and its consequences has been found to negatively impact mental well-being (Wang et al., 2020). Since the start of the pandemic, there has been an increase in symptoms of depression, anxiety, and stress in the general population (Wang et al., 2020). As such, there is a need for effective and scalable psychological interventions for the general population with the aim to increase coping and reduce the risk for anxiety and depression (Holmes et al., 2020). The negative psychological effects of the COVID-19 pandemic may be linked to three psychological dimensions: (1) Increased distress, worry, and rumination (Limcaoco et al., 2020); (2) Consequences of quarantine and social isolation (Nitschke et al., 2021); (3) Reduction of positive experiences and emotions (Moroń and Biolik-Moroń, 2021).

Distress, worry, and rumination are core characteristics in the majority of mental illnesses and, more specifically, core features in generalized anxiety disorder and depression (Ehring and Watkins, 2008). Studies also suggest that stressful life events increase rumination, which further increases the risk of developing anxiety and depression (Michl et al., 2013). Treatment approaches like Cognitive Behavioral Therapy (CBT) and Metacognitive Therapy have been found to have a positive effect on both the treatment and prevention of worry and rumination (Topper et al., 2017).

Quarantine and social isolation have various consequences on individuals in different life situations. Negative psychological effects of quarantine include fear, fatigue, anger, irritability, grief, numbness, and sleep problems (Brooks et al., 2020). The psychological consequences of quarantine are affected by several components, such as duration of quarantine, no clear time limit of the quarantine period, fear of infection, frustration and boredom, lack of access to essential goods, and lack of information. Quarantine periods longer than reasonable based on the knowledge on incubation periods would increase the negative psychological effects (Brooks et al., 2020). One study found that participants who were in quarantine for longer than 10 days had more post-traumatic stress symptoms than those who were in quarantine for less than 10 days (Hawryluck et al., 2004). A large Norwegian epidemiological study including 10,084 adult participants also found that social distancing strategies during the COVID-19 pandemic were associated with increased self-reported symptoms of anxiety and depression (Ebrahimi et al., 2020). During the COVID-19 pandemic, the prevalence rate in the general population was estimated to 30.78% for depression and 27.57% for anxiety, whereas the prevalence rate of the same population before the pandemic in 2015 was 10.24% for depression and 14.70% for anxiety (Ebrahimi et al., 2020). For comparison, the prevalence rate in the general population in other countries in a non-pandemic period ranged from 6.10 to 10.80% for depression (Johansson et al., 2013; Maske et al., 2016; Kauffman et al., 2020) and 8.20 to 14.70% for anxiety (Johansson et al., 2013; Maideen et al., 2015; Hinz et al., 2017). Due to the negative psychological effects of long-lasting quarantine, social isolation, and social distancing, it is recommended that the general population receive help to maintain meaningful activity during pandemics (Brooks et al., 2020).

The reduction of positive experiences and emotions plays a vital role in the development and persistence of common mental health disorders (Moroń and Biolik-Moroń, 2021). Psychological prevention- and treatment programs have to a large degree focused on reducing negative emotions and functional impairments (Craske et al., 2016). At the same time, previous research shows that a lack of interest and pleasure from daily activities, which are the core symptoms of depression, are difficult to treat with the current treatment approaches (Craske et al., 2016). This might be due to the small or non-existent sense of reward that depressed individuals receive from their daily activities (Terry et al., 2020). Interventions with the aim to increase the attention to and expectancy of positive emotions have been shown to result in an increase in positive emotions (Craske et al., 2019). Moreover, behavioral activation, a psychotherapeutic intervention focused on increasing engagement in activities which increase coping and positive emotions, has also been found to be effective in reducing core symptoms of depression (Cuijpers et al., 2007).

Digital technology is a widespread and viable distributor of effective and scalable psychological interventions. Research and healthcare service innovations over the past 20 years have shown that digital technologies are effective methods for reaching more individuals with evidence-based psychological interventions (Andersson and Titov, 2014; Titov et al., 2018). Within mental health care, there are now hundreds of studies that document the effect of both therapist-guided and self-guided Internet-delivered interventions for mental illnesses like anxiety disorders, depression, and sleep disorders (Ebert et al., 2018; Karyotaki et al., 2018; Titov et al., 2018; Karekla et al., 2019). In light of the COVID-19 pandemic, self-guided Internet-delivered interventions may be a feasible way to provide psychological interventions, as they do not require face-to-face meetings, traveling, or any other form of physical contact (Karyotaki et al., 2017; Wind et al., 2020). In general, therapist-guided Internet-delivered interventions are associated with better treatment outcomes than self-guided Internet-delivered interventions, however, many people still benefit from a self-guided format. For instance, individuals with milder depressive symptoms experience similar benefits from therapist-guided interventions and self-guided interventions (Karyotaki et al., 2021). Another challenge with self-guided interventions for anxiety and depression is lack of sustained adherence, where the usage gradually declines with time, with between 21 and 88% of participants using the intervention in the beginning and between 0.5 and 28.7% completing the intervention (Fleming et al., 2018). Therefore, it is plausible that reducing time spent in treatment and offering a more concentrated treatment could help to improve completion rates. It is also important to note that there are some barriers when it comes to Internet-delivered interventions, as many elderly have lower technical iteracy than the younger generations (Kruse et al., 2020). Internet delivered interventions can also impose difficulties for vulnerable populations, such as those with impaired eyesight. However, as the COVID-19 pandemic adds more demand and strain on the healthcare system, self-guided interventions could be of extra benefit as they require less resources than other formats of delivery within mental healthcare (Karyotaki et al., 2017).

In this paper, we will first present the development of a self-guided psychological Internet-delivered intervention for adults experiencing increased distress during the first 9 months of the COVID-19 pandemic in Norway. Secondly, we will present between-group differences in clinical outcomes and adherence among participants receiving intervention modules at a 3-day interval and participants receiving intervention modules at a 5-day interval, as well as preliminary clinical effects for the total sample.

Materials and Methods

Study Design

The current study was a randomized-controlled trial (RCT, ClinicalTrials.gov: 126376). Using a randomizing generator through the website randomization.com, eligible participants were randomized to either (1) a new module every third day (3-day group, a total of 28 days) or (2) a new module every fifth day (5-day group, a total of 40 days). In this study, we present results from pre, post and follow-up measurements from the total sample of 82 participants.

Ethical Approval

The study received ethical approval from the Norwegian Regional Ethical Committee (REK 126376) in an one week fast-track approval process as a response to COVID-19 related studies. Electronic informed consent was obtained from all participants.

Setting

The intervention was developed in a cross-disciplinary and cross-sectorial team within the framework of the research project Introducing mental health through adaptive technology (INTROMAT, 259293), an ICT Lighthouse research project funded by the Norwegian Research Council, hosted by a University Hospital in Norway. In the INTROMAT-project, four Internet-delivered interventions, both therapist-guided and self-guided, were developed before the onset of the pandemic, and are currently being evaluated in multiple clinical trials. The developmental team included clinical psychologists, researchers in clinical psychology, Human-Computer Interaction, machine learning and computer modeling, and one industry partner.

Development of the Intervention

The person-based approach (PBA; Yardley et al., 2015), a user-centered method for developing Internet-delivered interventions, served as a framework for the development of the current COVID-19 intervention. The PBA integrates knowledge and expertise from the research literature, theoretical approaches, and the end-users needs and preferences, applying both quantitative and qualitative methods in the planning, development, and evaluating phases of a new intervention (Yardley et al., 2015). In the present study we consulted the literature for possible effective intervention elements (Topper et al., 2017; Craske et al., 2019). The PBA recommends involving end-users to form guiding principles of the intervention development. However, due to the urgent need for an internet-delivered intervention, we opted to not include end-users in the development phase during a nationwide lockdown. Therefore, we revisited qualitative interviews conducted with individuals suffering from mental disorders in previous interventions in order to understand the potential needs and challenges of the target users. This formed the basis of the COVID-19 intervention’s guiding principles (Table 1), which state the design objectives and key features that address these. The guiding principles were used as a map to guide the development and adaptation of content, such as text and design features (Table 1).

TABLE 1

Table 1. Guiding principles.

Based on the guiding principles, the majority of the content in the digital COVID-19 intervention was adapted from four previously developed digital interventions, all including systematic user-involvement and user-testing: (1) MyADHD; (2) RestDep; (3) Gynea, and (4) YoungSpotlight. MyADHD targets adults with ADHD with an aim to increase coping with everyday challenges related to ADHD. The intervention is self-guided, consists of seven modules, and builds upon principles from cognitive rehabilitation and mindfulness- and acceptance-based approaches. Preliminary results indicate improvement in self-reported symptoms of inattention and stress, and quality of life (Nordby et al., 2021; ClinicalTrials.gov NCT04511169). RestDep targets adults with cognitive residual symptoms after depression aiming to reduce the negative consequences, such as depression relapse. The intervention is guided, consists of ten modules, and is based on psychoeducation, attention training, and compensatory strategies. Preliminary results indicate reduction in residual cognitive symptoms and rumination (Myklebost et al., 2021). Gynea targets women who have had gynecological cancer with the aim to increase quality of life and adjusting to life after cancer treatment. The intervention is guided and consists of eight modules and includes psychoeducation, and mindfulness and self-compassion exercises (Sekse et al., 2021). The intervention is currently being evaluated for clinical effects (ClinicalTrials.gov NCT04414436). YoungSpotlight targets adolescents’ fear of public speaking with the short-term aim to reduce public speaking anxiety symptoms in the classroom and the long-term aim to prevent the onset of generalized social anxiety disorder. The intervention is self-guided and a combination of virtual reality exposure training and text-based modules. Preliminary results indicated a significant reduction in public speaking anxiety symptoms (Kahlon et al., 2019; ClinicalTrials.gov NCT04396392).

The first participant accessed the intervention on 4th of April 2020, as soon as the development of the first two modules were completed. The development of the next six modules were completed at least 5 days before the first participant reached that module (see Table 2 for overview of the modules).

TABLE 2

Table 2. Overview of modules.

Designing and Evaluating the Participant’s Front-End

The development of the intervention was enabled by a digital platform for delivering interventions, previously developed in collaboration with the industry-partner Youwell AS as part of the INTROMAT project. The platform has three front-ends for different means that can be used in an Internet browser: (1) The participant’s front-end for accessing and using the intervention; (2) the therapist’s front-end for monitoring participants’ progress; (3) and an authoring tool for creating and modifying content and customizing the work-flow of the intervention.

The interface design of the participant’s front-end can be customized to any Internet-delivered intervention including idiosyncratic workflows, informed consents and randomization procedures. It is also possible to customize the participant’s front-end for different participant groups in each Internet-delivered intervention. All front-ends of the platform require two-factor authentication. However, it is also possible to make certain modules of the Internet-delivered programs publicly available. This was done to distribute information about the intervention, such as inclusion criteria, information about the research project and eligibility screening for people interested in participating in the trial.

The platform is structured in a hierarchy of programs, modules, pages and tasks. Text, pictures, audio, video, worksheets, and questionnaires can easily be developed and edited by non-technical personnel. The platform can also push automated prompts and reminders to the participant, and it collects and stores health- and interaction-data that may be used for clinical and IT-related research.

Internet-delivered interventions should be fully functional and usable to the general public on all Internet browsers and devices (laptops, tablets, smartphones) (Yogarajah et al., 2020). It should also be accessible to persons with disabilities. To reach this goal, we concurrently performed formal and informal usability evaluations of the participant’s front-end with developing the program content. Evaluation types included: functional testing with predefined tasks (Bruun and Stage, 2012), accessibility evaluation (Mankoff et al., 2005), and ad hoc usability testing (Følstad et al., 2013).

The functional testing was done by pre-defining tasks with descriptions of expected results to be completed within the participant’s front-end of the intervention platform (Bruun and Stage, 2012). Two persons piloted the tests simultaneously and described the results from their tasks in a spreadsheet. In the case of functional failures, these were described and corrected.

Accessibility evaluations were done according to Web Content Accessibility Guidelines (WCAG) 2.0 (Caldwell et al., 2008) and specifications by the World Wide Web Consortium (W3C). The guidelines specify recommendations for making websites accessible for people with disabilities. Web accessibility is protected by law through the EU Web Accessibility Directive and the Norwegian Digitalization Agency. Two persons conducted an accessibility evaluation (Mankoff et al., 2005). A third person then compared the results, compiled a list of accessibility issues and handed these over to the industrial partner who developed and maintained the platform. Due to the time constraints of rapidly answering to the COVID-19 situation, these accessibility tests were done ad hoc. Proper tools for assessing and exploring website accessibility (a screen reader extension and Chrome Developer Tools) were used in a Chrome web browser. Whenever a part of the participant’s front-end did not meet the WCAG 2.0 requirements, which is the level required by the Norwegian Digitalization Agency for public web sites, evaluators made a note detailing the shortcoming. For example, the sequence of elements in questionnaires had to be rearranged to be made accessible for visually impaired users.

Finally, ad hoc usability testing (Følstad et al., 2013) was done while developing the intervention. In this activity, we noted all front-end usability errors and faults that could negatively affect the user experience. These usability tests were organized by email and the chat service Slack.

Based on these evaluations, the interface design of the participant’s front-end was improved by iterative development so that it would be fully responsive to different devices (both stationary and mobile). Existing interface components in the platform were improved for better usability and accessibility (see Figure 1 for an example). New interface components were rapidly sketched and prototyped in the web languages HTML and CSS. Sketches of designs and excerpts of HTML and CSS code were then handed over to the industrial partner who developed and managed the platform, and they in turn implemented the specified designs in the front-end. Following these improvements, the participant’s front-end complied with WCAG 2.0 requirements, and functional testing was completed without errors on all devices.

FIGURE 1

Figure 1. Illustration of the program website.

Recruitment and Procedure

The participants were recruited through social- and traditional media from all over Norway. Interested participants were first directed to an open website with information about the study and an online screening to evaluate eligibility according to the following inclusion criteria: (1) 18 years of age or older, (2) self-reported mild and moderate distress in relation to the COVID-19 pandemic, (3) a score on the Patient Health Questionnaire between 5 and 14 (PHQ-9; Kroenke et al., 2001), and (4) able to read and write in Norwegian. The exclusion criteria were (1) currently receiving psychological or somatic treatment and (2) self-reported severe mental disorder, such as psychosis or substance abuse disorder. Eligible participants were invited to leave their contact information to be contacted by the study team for a phone screening interview. A clinical psychologist or psychiatric nurse conducted the screening interviews for further evaluation and obtained the national identity number of the eligible participants. Eligible participants then were randomized to a treatment group and thereby asked to digitally sign an informed consent before they were given access to the pre-treatment assessment and first treatment module.

Outcome Measures

Primary Clinical Outcome Measure

The Patient Health Questionnaire (PHQ-9; Kroenke et al., 2001) measures depression and consists of nine items rated on a four-point scale scored 0 (“not at all”) to 3 (“nearly every day”). A short version of PHQ-9 with the first two items was used in the weekly measurements. A higher score indicates greater depression severity. Cronbrach’s alpha for PHQ-9 for this sample was 0.58. The low value is partly explained by the fact that PHQ assesses all dimensions of depression, a very heterogeneous disorder.

Secondary Clinical Outcome Measures

Generalized Anxiety Disorder 7 (GAD-7; Spitzer et al., 2006) measures the severity of general anxiety and consists of seven items rated on a four-point scale ranging from 0 (“not at all”) to 3 (“nearly every day”). A short version of GAD-7 with the first two items was used in the weekly measurements. Cronbrach’s alpha for GAD-7 for this sample was 0.80.

The positive and negative affect scale (PANAS; Watson et al., 1988) measures the positive and negative effect experienced and consists of ten adjectives related to emotions rated on a five-point Likert scale ranging from 1 (“very slightly/not at all”) to 5 (“very much”). The total sum score of both positive and negative affect is then calculated. In the current sample, Cronbach’s alpha for the positive affect scale was 0.87 and for the negative affect scale it was 0.74.

Treatment Satisfaction

Each module was evaluated by the participants with the following questions: (1) “To what extent do you find this module useful?” ranging from 1 = not useful at all, 10 = very useful; and (2) “To what degree of certainty would you recommend this module to a friend?” ranging from 1 = not certain at all, 10 = very certain. Participants also provided written feedback at the end of each module on the following questions: (1) “What did you like the most or what did you find as most useful in this module?,” and (2) “What did you miss or what disappointed you in this module?”

In addition, the Client Satisfaction Questionnaire (CSQ-8; Attkisson and Greenfield, 1994), an eight-items self-report questionnaire, was used to assess the overall satisfaction of the service on a scale from 1 = Not satisfied to 4 = Very satisfied.

Statistical Analysis

SPSS Statistics 26 was used to perform the statistical analyses. Intention to treat analyses (ITT) of the treatment outcomes from pre-, post-, and follow-up assessments were performed using linear mixed models including random effects and random intercept. PHQ, GAD, PANAS positive, and PANAS negative served as the dependent variables. Missing data were handled according to Chakraborty and Gu (2009), as they were estimated and not imputed, by using restricted maximum likelihood (RML). This is assumed to affect the power of a clinical trial to a lesser degree compared to other strategies such as Last Observation Carried Forward or imputation for handling missing data (Chakraborty and Gu, 2009). All items in the PHQ-9 (primary measure) and GAD-7 and PANAS were assessed at pre-treatment, post-treatment, and 6 weeks follow-up, leaving three data points for the Mixed Linear Model analysis.

Effect sizes calculated as Cohen’s d were based on both observed and estimated means from pre-treatment to post- and follow-up treatment. Pre_m − Post_m/SD_pre. Standard deviation was calculated by standard error x√n.

As this randomized controlled trial had two relatively similar active arms and no previous relevant data were identified, no power calculator for between groups effects were conducted prior to the study. However, a post hoc power analysis was conducted.

Results

Participants

The recruitment started April 4th, 2020 and ended on December 14th, 2020. In this period a total of 1256 individuals accessed the pre-screening survey (see the Flow Chart in Figure 2), 407 were eligible according to the online screening, 92 provided contact information.

FIGURE 2

Figure 2. Flow Chart.

A total of 82 individuals were eligible to participate in the study and signed informed consent. The sample had a mean age of 40 (20–80, SD = 14.19). A total of 44 participants were randomized to the 3-day group and 38 participants to the 5-day group. Participant characteristics are shown in Table 3.

TABLE 3

Table 3. Participant characteristics (N = 82).

Adherence

A total of 43 out of 82 participants (52.4%) completed the post-measurements, and 42 out of 82 participants (51.2%) completed the 6 weeks follow-up assessment. In terms of module completion, there was no significant difference between the 3-day group (M = 4.91, SD = 3.07) and 5-day group (M = 4.42, SD = 3.02) group; [t(90) = 0.72, p = 0.472]. An overview of completed modules shows that 29 of the participants (66%) in the intensive group completed four modules or more, with 19 participants (43%) completing all eight modules, whereas in the 5-day group, 21 of the participants (55%) completed four modules or more, with 14 participants (37%) completing all eight modules. For the total sample (n = 82), 50 participants (61%) completed four modules or more, and 33 participants (40.2%) completed all modules.

Clinical Outcomes

No significant between-group differences were identified between the 3-day and 5-day group (Table 4).

TABLE 4

Table 4. Module completion among participants.

The following analyses were conducted for both groups combined in order to enhance power.

Overall, the observed means shows small and moderate within effect sizes (Table 5). Estimated means and effect sizes are reported in Table 6. The intention-to-treat analyses were as follows.

TABLE 5

Table 5. Observed Means, Standard Deviations, and Effect sizes for Primary and Secondary Outcome Measures at Pre, Post, and Follow-up.

TABLE 6

Table 6. Estimated means values and effect sizes.

Depressive Symptoms

The linear mixed model showed a significant decrease in PHQ-9 from pre to post (p < 0.05), with an effect size of 0.42 (Tables 6, 7). However, when comparing pre with follow up measurement, the decrease in symptoms was not significant (p = 0.201).

TABLE 7

Table 7. Intention to treat analyses from pre to post, and pre to follow up.

Anxiety Symptoms

The linear mixed model showed a significant decrease in GAD-7 from pre to post (p < 0.001) and from pre to follow up (p < 0.05) with an effect size of 0.67 and 0.33 (Tables 6, 7).

Positive and Negative Affect

The linear mixed model of the PANAS showed a small but significant increase in positive emotions from both pre to post (p < 0.05) and from pre to follow up (p < 0.05), with an effect size of 0.29 and 0.35 (Tables 6, 7), and a significant decrease in negative emotions over time from both pre to post (p < 0.001) and from pre to follow up (p < 0.001), with effect sizes 0.74 and 0.72 (Tables 6, 7).

Treatment Satisfaction

Usefulness of the modules was rated between 6.58 and 7.65 for all participants. When asked if they would recommend the module to a friend, the participants gave a score between 6.74 and 7.85 (Table 8). Participants generally offered positive feedback on the content, and most participants mentioned the practical exercises (e.g., breathing exercises and activity planning) as the most useful elements in the modules. Few participants provided feedback on missing or disappointing elements. On the Client Satisfaction scale, the participants (n = 41) gave the intervention an overall rating of 2.60–3.34 (Table 9).

TABLE 8

Table 8. Satisfaction of each module.

TABLE 9

Table 9. Client Satisfaction Questionnaire (n = 41).

Post hoc Power Analysis

The post hoc power analysis showed that with a probability level of 0.05 with two groups with 44 and 38 participants the study was underpowered to detect significant differences between the two groups.

Discussion

This study reports on the development and the clinical effects of a self-guided Internet-delivered intervention targeting psychological distress during the first months of the COVID-19 pandemic in Norway. In particular, the study assessed whether the intervention was associated with a reduction in depression symptoms, anxiety symptoms, negative emotions, and increased positive emotions. Overall, 82 participants were included in the study and 43 completed the intervention. A relatively large proportion of the sample (36 of 82) had previous experience with psychological treatment. No difference was found between the groups receiving the intervention modules every third or fifth day. When examining the overall clinical effects, the linear mixed model analyses showed significant reductions with moderate effects for depressive symptoms (d = 0.42), anxiety symptoms (d = 0.67), and negative emotions (d = 0.74), and a small significant increase (d = 0.29) was observed in positive emotions.

No between-group differences were identified between those who automatically received a new module every third day and those who automatically received a new module every fifth day. This could indicate that reducing the time spent in treatment does not have a negative effect on treatment outcomes. However, the lack of significant difference could also be a result of the sample being too underpowered to detect such a difference. Hence, more data concerning different time intervals for module delivery in this intervention is needed in order to contribute to knowledge regarding the optimal interval of automated self-guided psychological interventions.

Regarding clinical outcomes for the total sample, the findings showed a moderate decrease in depressive and anxiety symptoms during the intervention. This is a somewhat higher effect compared to previous studies of self-guided Internet-delivered interventions (Karyotaki et al., 2017). The effects were somewhat lower at the 6-week follow-up. The change from post to 6-week follow-up may be related to changes in the direct or indirect consequences of the pandemic. One factor may be uncertainty related to the duration of the pandemic and its consequences (Brooks et al., 2020). Another factor may be the increased attention on the negative economic impact of the COVID-19 pandemic, i.e., people becoming temporarily laid off, companies going bankrupt, or savings running out.

The findings showed a moderate and significant decrease in negative emotions over time, and a small but significant increase for positive emotions. The latter might be due to the challenge in finding activities that can increase positive emotions during quarantine. Activities that involve social interaction and physical contact are important for key positive emotions, and such activities were difficult to engage in during the pandemic (Moroń and Biolik-Moroń, 2021).

The adherence in the current self-guided intervention was relatively high, as 35 participants (38.5%) completed all eight modules. This is considerably higher compared to a study by Lillevoll et al. (2014) who explored the feasibility and effects of a self-guided mental health intervention for Norwegian youths where 8.5% of the eligible participants logged into the intervention and few completed all modules. However, the discrepancy could be a result of different target populations and recruitment procedures as adults participating in the current study themselves made contact for participation, while the youths in the Lillevoll et al. (2014) study were recruited through a survey conducted in Norwegian high schools.

Overall, the participants in the present study reported being generally satisfied with the intervention and all modules were rated as helpful. These findings indicate that the current intervention is perceived as acceptable by the target group and thus have the potential to support other people in their psychological distress during the COVID-19 pandemic and possibly future pandemics.

Regarding the recruitment, there was much interest for the intervention, as 1256 individuals accessed the screening. However, only 407 individuals were found eligible, 92 provided contact information to the study team and 82 were included to receive the intervention. The relatively high number of individuals that were not eligible could indicate that the inclusion criteria were too narrow. This may also have affected the treatment response as a floor-effect should be expected in a sample where people with moderate and high symptom levels were excluded (Bower et al., 2013). Still, including participants with higher levels of depressive symptoms might not be appropriate because of issues such as risk of suicide (Simon et al., 2013). This is in line with experts recommending that those having a risk of suicidality should be excluded from participation in self-guided interventions, or that guided elements should be included to monitor such symptoms (Sander et al., 2020). We suggest that future self-guided psychological interventions aiming to respond quickly to future or ongoing crises, such as COVID-19, should be considered for a wider range of the population, including those with moderate levels of psychological symptoms.

The current study has several limitations. The main limitation was that the study was underpowered for a two-arm randomized controlled trial with two active arms and therefore underpowered to detect a difference between the 3-day and 5-day group. Another limitation is the lack of a waitlist control group. It would have been beneficial to include a waitlist control group to investigate natural recovery in a period where time is important. For example, seasonal changes and the trajectories of the pandemic could affect the results. However, this was not implemented due to ethical considerations at a time with need for, and very limited access to, psychological interventions. Another limitation is the fact that only 82 out of 1256 individuals who accessed the online screening were granted access to the intervention. This disproportionate number of included participants indicates a highly selected sample based on the inclusion/exclusion criteria and the number of participants leaving their contact information. Importantly, this limits the generalizability of these findings to a larger population. Furthermore, Cronbrach’s alpha for PHQ-9 for this sample was 0.58. The low value is partly explained by the fact that PHQ assesses all dimensions of depression, a very heterogeneous disorder.

Another limitation is the high number of eligible individuals that did not leave their contact information in order to be included in the study. This could be a result of them not being motivated to participate or having enough information to commit to participation. This could be addressed by providing more detailed information about the intervention and strengthening the rationale for treatment on the project web-page where participants self-screened. More research is needed to understand the conversion rates of the number of people starting the pre-screening survey versus the number of people completing the survey. Understanding the factors involved in successful onboarding can be of benefit for future digital interventions.

Taken together, the results of our study indicate that the present self-guided intervention for adults experiencing psychological distress during the COVID-19 pandemic may lead to reduction in symptoms of anxiety and depression, as well as negative emotions. The majority of the participants were satisfied with the treatment and the overall adherence was good when compared to other self-guided interventions. The lack of significant differences between the 3-day and 5-day group may also indicate that a shorter treatment duration can still produce similar treatment outcomes. Moving forward, the present non-commercial COVID-19 intervention was made publicly available free of charge from 2021 via the national eHealth platform in Norway (helsenorge.no), where all inhabitants in Norway can access their public eHealth services. In this context, the login still includes two-factor authentication, but without screening and the inhabitants get access to the intervention immediately after registration. The intervention will be continuously adapted to the trajectories of the ongoing COVID-19 pandemic. The results of the present study can be applied in preparence and response for future pandemic crises where the development of digital interventions for handling psychological distress are needed.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by Regional Committees for Medical and Health Research Ethics West. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

TN have lead the work with the development of the intervention, planned the study, and wrote the manuscript. SK have lead the work with the analyses and contributed to the development of the intervention, planned the study, and wrote the manuscript. SM have lead the work related to the person-based approach and have contributed to the development of the intervention, planned the study, and wrote the manuscript. EF have lead the work with the user-testing and contributed to the development of the intervention, and wrote the manuscript. EN have contributed to the development of the intervention, planned the study, and wrote the manuscript. All authors contributed to the article and approved the submitted version.

Funding

The intervention was developed and carried out in a cross-disciplinary and cross-sectorial team within the framework of the research project Introducing Mental Health through Adaptive Technology (INTROMAT, 259293), an ICT Lighthouse research project, funded by the Norwegian Research Council.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We want to thank all who participated in the trial.

References

Andersson, G., and Titov, N. (2014). Advantages and limitations of Internet-based interventions for common mental disorders. World Psychiatry. 13, 4–11. doi: 10.1002/wps.20083

PubMed Abstract | CrossRef Full Text | Google Scholar

Attkisson, C. C., and Greenfield, T. K. (1994). “Client Satisfaction Questionnaire-8 and Service Satisfaction Scale-30,” in The use of psychological testing for treatment planning and outcome assessment, ed. M. E. Maruish (Mahwah: Lawrence Erlbaum Associates, Inc), 402–420.

Google Scholar

Bower, P., Kontopantelis, E., Sutton, A., Kendrick, T., Richards, D. A., Gilbody, S., et al. (2013). Influence of initial severity of depression on effectiveness of low intensity interventions: meta-analysis of individual patient data. BMJ 346:f540. doi: 10.1136/bmj.f540

PubMed Abstract | CrossRef Full Text | Google Scholar

Brooks, S. K., Webster, R. K., Smith, L. E., Woodland, L., Wessely, S., Greenberg, N., et al. (2020). The psychological impact of quarantine and how to reduce it: rapid review of the evidence. Lancet 395:10227. doi: 10.1016/S0140-6736(20)30460-8

CrossRef Full Text | Google Scholar

Bruun, A., and Stage, J. (2012). The effect of task assignments and instruction types on remote asynchronous usability testing. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York City: Association for Computing Machinery, doi: 10.1145/2207676.2208364

CrossRef Full Text | Google Scholar

Caldwell, B., Cooper, M., Reid, L. G., and Vanderheiden, G. (2008). Web content accessibility guide-lines (WCAG) 2.0. Available Online at: https://www.w3.org/TR/2008/REC-WCAG20-20081211/ (accessed June 10, 2020).

Google Scholar

Chakraborty, H., and Gu, H. (2009). A Mixed Model Approach for Intent-to-Treat Analysis in Longitudinal Clinical Trials With Missing Values. Research Triangle Park, NC: RTI Press, doi: 10.3768/rtipress.2009.mr.0009.0903

PubMed Abstract | CrossRef Full Text | Google Scholar

Craske, M. G., Meuret, A. E., Ritz, T., Treanor, M., Dour, H., and Rosenfield, D. (2019). Positive affect treatment for depression and anxiety: A randomized clinical trial for a core feature of anhedonia. J. Consult. Clin. Psychol. 87, 457–471. doi: 10.1037/ccp0000396

PubMed Abstract | CrossRef Full Text | Google Scholar

Craske, M. G., Meuret, A. E., Ritz, T., Treanor, M., and Dour, H. J. (2016). Treatment for anhedonia: a neuroscience driven approach. Depress. Anxiety 33, 927–938. doi: 10.1002/da.22490

PubMed Abstract | CrossRef Full Text | Google Scholar

Cuijpers, P., Van Straten, A., and Warmerdam, L. (2007). Behavioral activation treatments of depression: A meta-analysis. Clin. Psychol. Rev. 27, 318–26. doi: 10.1016/j.cpr.2006.11.001

PubMed Abstract | CrossRef Full Text | Google Scholar

Ebert, D. D., Van Daele, T., Nordgreen, T., Karekla, M., Compare, A., Zarbo, C., et al. (2018). Internet-and mobile-based psychological interventions: applications, efficacy, and potential for improving mental health. Eur. Psychol. 23, 167–187. doi: 10.1027/1016-9040/a000318

CrossRef Full Text | Google Scholar

Ebrahimi, O. V., Hoffart, A., and Johnson, S. U. (2020). Physical distancing and mental health during the COVID-19 pandemic: factors associated with psychological symptoms and adherence to pandemic mitigation strategies. Clin. Psychol. Sci. 9, 489–506. doi: 10.1177/2167702621994545

CrossRef Full Text | Google Scholar

Ehring, T., and Watkins, E. R. (2008). Repetitive negative thinking as a transdiagnostic process. Int. J. Cogn. Ther. 1, 192–205. doi: 10.1521/ijct.2008.1.3.192

CrossRef Full Text | Google Scholar

Fleming, T., Bavin, L., Lucassen, M., Stasiak, K., Hopkins, S., and Merry, S. (2018). Beyond the trial: systematic review of real-world uptake and engagement with digital self-help interventions for depression, low mood or anxiety. J. Med. Internet. Res. 20:e199. doi: 10.2196/jmir.9275

PubMed Abstract | CrossRef Full Text | Google Scholar

Følstad, A., Hornbæk, K., and Ulleberg, P. (2013). Social design feedback: evaluations with users in online ad-hoc groups. Hum. Cent. Comput. Inf. Sci. 3, 18. doi: 10.1186/2192-1962-3-18

CrossRef Full Text | Google Scholar

Hawryluck, L., Gold, W. L., Robinson, S., Pogorski, S., Galea, S., and Styra, R. (2004). SARS control and psychological effects of quarantine, Toronto, Canada. Emerg. Infect. Dis. 10, 1206–1212. doi: 10.3201/eid1007.030703

PubMed Abstract | CrossRef Full Text | Google Scholar

Hinz, A., Klein, A. M., Brähler, E., Glaesmer, H., Luck, T., Riedel-Heller, S. G., et al. (2017). Psychometric evaluation of the Generalized Anxiety Disorder screener GAD-7, based on a large German general population sample. J. Affect. Disord. 210, 338–344. doi: 10.1016/j.jad.2016.12.012

PubMed Abstract | CrossRef Full Text | Google Scholar

Holmes, E. A., O’Connor, R. C., Perry, V. H., Tracey, I., Wessely, S., Arseneault, L., et al. (2020). Multidisciplinary research priorities for the COVID-19 pandemic: a call for action for mental health science. Lancet Psychiatry. 7, 547–560. doi: 10.1016/S2215-0366(20)30168-1

CrossRef Full Text | Google Scholar

Johansson, R., Carlbring, P., Heedman, Å, Paxling, B., and Anderson, G. (2013). Depression, anxiety and their comorbidity in the Swedish general population: point prevalence and the effect on health-related quality of life. Peer J. 1:e98. doi: 10.7717/peerj.98

PubMed Abstract | CrossRef Full Text | Google Scholar

Kahlon, S., Lindner, P., and Nordgreen, T. (2019). Virtual reality exposure therapy for adolescents with fear of public speaking: a non-randomized feasibility and pilot study. Child Adolesc. Psychiatry Mental Health 13:47. doi: 10.1186/s13034-019-0307-y

PubMed Abstract | CrossRef Full Text | Google Scholar

Karekla, M., Kasinopoulos, O., Neto, D. D., Ebert, D. D., Daele, T. M., Nordgreen, T., et al. (2019). Best practices and recommendations for digital interventions to improve engagement and adherence in chronic illness sufferers. Eur. Psychol. 24, 49–67. doi: 10.1027/1016-9040/a000349

CrossRef Full Text | Google Scholar

Karyotaki, E., Ebert, D. D., Donkin, L., Riper, H., Twisk, J., Burger, S., et al. (2018). Do guided internet-based interventions result in clinically relevant changes for patients with depression? An individual participant data meta-analysis. Clin. Psychol. Rev. 63:80. doi: 10.1016/j.cpr.2018.06.007

PubMed Abstract | CrossRef Full Text | Google Scholar

Karyotaki, E., Efthimiou, O., Miguel, C., Bermpohl, F. M. G., Furukawa, T. A., Cuijpers, P., et al. (2021). Internet-based cognitive behavioral therapy for depression: A systematic review and individual patient data network meta-analysis. JAMA Psychiatry. 78, 361–371. doi: 10.1001/jamapsychiatry.2020.4364

PubMed Abstract | CrossRef Full Text | Google Scholar

Karyotaki, E., Riper, H., Twisk, J., Hoogendoorn, A., Kleiboer, A., Mira, A., et al. (2017). Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms: A meta-analysis of individual participant data. JAMA Psychiatry 74, 351–359. doi: 10.1001/jamapsychiatry.2017.0044

PubMed Abstract | CrossRef Full Text | Google Scholar

Kauffman, K., Horvat Davey, C., Dolata, J., Figueroa, M., Gunzler, D., Huml, A., et al. (2020). Changes in self-reported depressive symptoms among adults in the United States from 2005 to 2016. J. Am. Psychiat. Nurses Assoc. 27, 148–155. doi: 10.1177/1078390320906194

PubMed Abstract | CrossRef Full Text | Google Scholar

Kroenke, K., Spitzer, R. L., and Williams, J. B. (2001). The PHQ-9: validity of a brief depression severity measure. J. General Internal Med. 16, 606–613. doi: 10.1046/j.1525-1497.2001.016009606.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Kruse, C., Fohn, J., Wilson, N., Patlan, E. N., Zipp, S., and Mileski, M. (2020). Utilization barriers and medical outcomes commensurate with the use of telehealth among older adults: systematic review. JMIR Med. Inf. 8:e20359. doi: 10.2196/20359

PubMed Abstract | CrossRef Full Text | Google Scholar

Lillevoll, K. R., Vangberg, H. C. B., Griffiths, K. M., Waterloo, K., and Eisemann, M. R. (2014). Uptake and adherence of a self-directed internet-based mental health intervention with tailored e-mail reminders in senior high schools in Norway. BMC Psychiatry. 14:14. doi: 10.1186/1471-244x-14-14

PubMed Abstract | CrossRef Full Text | Google Scholar

Limcaoco, R. S. G., Mateos, E. M., Fernandez, J. M., and Roncero, C. (2020). Anxiety, worry and perceived stress in the world due to the COVID-19 pandemic, March 2020. Preliminary results. MedRxiv. doi: 10.1177/00912174211033710

PubMed Abstract | CrossRef Full Text | Google Scholar

Maideen, S. F. K., Sidik, S. M., Rampal, L., and Mukhtar, F. (2015). Prevalence, associated factors and predictors of anxiety: a community survey in Selangor, Malaysia. BMC Psychiatry 15:262. doi: 10.1186/s12888-015-0648-x

PubMed Abstract | CrossRef Full Text | Google Scholar

Mankoff, J., Fait, H., and Tran, T. (2005). Is your web page accessible? A comparative study of methods for assessing web page accessibility for the blind. In Proceedings of the SIGCHI conference on Human factors in computing systems. Portland: DBLP, doi: 10.1145/1054972.1054979

CrossRef Full Text | Google Scholar

Maske, U. E., Buttery, A. K., Beesdo-Baum, K., Riedel-Heller, S., Hapke, U., and Busch, M. A. (2016). Prevalence and correlates of DSM-IV-TR major depressive disorder, selfreported diagnosed depression and current depressive symptoms among adults in Germany. J. Affect. Disord. 190, 167–177. doi: 10.1016/j.jad.2015.10.006

PubMed Abstract | CrossRef Full Text | Google Scholar

Michl, L. C., McLaughlin, K. A., Shepherd, K., and Nolen-Hoeksema, S. (2013). Rumination as a mechanism linking stressful life events to symptoms of depression and anxiety: Longitudinal evidence in early adolescents and adults. J. Abnor. Psychol. 122, 339–352. doi: 10.1037/a0031994

PubMed Abstract | CrossRef Full Text | Google Scholar

Moroń, M., and Biolik-Moroń, M. (2021). Trait emotional intelligence and emotional experiences during the COVID-19 pandemic outbreak in Poland: A daily diary study. Pers. Individ. Differ. 168:110348. doi: 10.1016/j.paid.2020.110348

PubMed Abstract | CrossRef Full Text | Google Scholar

Myklebost, S. B., Nordgreen, T., and Hammar, Å (2021). An open pilot study of an internet-delivered intervention targeting self-perceived residual cognitive symptoms after major depressive disorder. Appl. Neuropsychol. Adult. 4, 1–10. doi: 10.1080/23279095.2021.1901706

PubMed Abstract | CrossRef Full Text | Google Scholar

Nitschke, J. P., Forbes, P. A., Ali, N., Cutler, J., Apps, M. A., Lockwood, P. L., et al. (2021). Resilience during uncertainty? Greater social connectedness during COVID-19 lockdown is associated with reduced distress and fatigue. Br. J. Health Psychol. 26, 553–569. doi: 10.1111/bjhp.12485

PubMed Abstract | CrossRef Full Text | Google Scholar

Nordby, E. S., Kenter, R. M. F., Lundervold, A. J., and Nordgreen, T. (2021). A self-guided Internet-delivered intervention for adults with ADHD: A feasibility study. Internet. Interven. 25:100416. doi: 10.1016/j.invent.2021.100416

PubMed Abstract | CrossRef Full Text | Google Scholar

Sander, L., Gerhardinger, K., Bailey, E., Robinson, J., Lin, J., Cuijpers, P., et al. (2020). Suicide risk management in research on internet-based interventions for depression: A synthesis of the current state and recommendations for future research. J. Affect. Disord. 263, 676–683. doi: 10.1016/j.jad.2019.11.045

PubMed Abstract | CrossRef Full Text | Google Scholar

Sekse, R. J. T., Nordgreen, T., Flobak, E., Lystrup, M., Braathen, E., and Werner, H. M. J. (2021). Development of a framework and the content for a psychoeducational internet-delivered intervention for women after treatment for gynecological cancer. Nurs. Rep. 11, 640–651. doi: 10.3390/nursrep11030061

CrossRef Full Text | Google Scholar

Simon, G. E., Rutter, C. M., Peterson, D., Oliver, M., Whiteside, U., Operskalski, B., et al. (2013). Does response on the PHQ-9 depression questionnaire predict subsequent suicide attempt or suicide death? Psychiatric Serv. 64:12. doi: 10.1176/appi.ps.201200587

PubMed Abstract | CrossRef Full Text | Google Scholar

Spitzer, R. L., Kroenke, K., Williams, J. B., and Löwe, B. (2006). A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch. Internal Med. 166, 1092–1097. doi: 10.1001/archinte.166.10.1092

PubMed Abstract | CrossRef Full Text | Google Scholar

Terry, P. C., Parsons-Smith, R. L., and Terry, V. R. (2020). Mood responses associated with COVID-19 restrictions. Front. Psychol. 11:3090. doi: 10.3389/fpsyg.2020.589598

PubMed Abstract | CrossRef Full Text | Google Scholar

Titov, N., Blake, D., Nielssen, O., Staples, L., Hadjistavropoulos, H., Nugent, M., et al. (2018). ICBT in routine care: A descriptive analysis of successful clinics in five countries. Internet. Interven. 13, 108–115. doi: 10.1016/j.invent.2018.07.006

PubMed Abstract | CrossRef Full Text | Google Scholar

Topper, M., Emmelkamp, P., Wakins, E., and Ehring, T. (2017). Prevention of anxiety disorders and depression by targeting excessive worry and rumination in adolescents: A randomized controlled trial. Behav. Res. Ther. 90, 123–136. doi: 10.1016/j.brat.2016.12.015

PubMed Abstract | CrossRef Full Text | Google Scholar

Wang, C., Pan, R., Wan, X., Tan, Y., Xu, L., Ho, C. S., et al. (2020). Immediate psychological responses and associated factors during the initial stage of the 2019 coronavirus disease (COVID-19) epidemic among the general population in China. Int. J. Environ. Res. Public Health 17:1729. doi: 10.3390/ijerph17051729

PubMed Abstract | CrossRef Full Text | Google Scholar

Watson, D., Clark, L. A., and Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: the PANAS scales. J. Personal. Soc. Psychol. 54, 1063–1070. doi: 10.1037//0022-3514.54.6.1063

CrossRef Full Text | Google Scholar

Wind, T. R., Rijkeboer, M., Andersson, G., and Riper, H. (2020). The COVID-19 pandemic: The ‘black swan’for mental health care and a turning point for e-health. Internet. Interven. 20:100317. doi: 10.1016/j.invent.2020.100317

PubMed Abstract | CrossRef Full Text | Google Scholar

Yardley, L., Morrison, L., Bradbury, K., and Muller, I. (2015). The person-based approach to intervention development: application to digital health-related behavior change interventions. J. Med. Internet. Res. 17:e30. doi: 10.2196/jmir.4055

PubMed Abstract | CrossRef Full Text | Google Scholar

Yogarajah, A., Kenter, R., Lamo, Y., Kaldo, V., and Nordgreen, T. (2020). Internet-delivered mental health treatment systems in Scandinavia–A usability evaluation. Internet. Interven. 20:100314. doi: 10.1016/j.invent.2020.100314

PubMed Abstract | CrossRef Full Text | Google Scholar