Patterns of engagement in a digital mental health service during COVID-19: a cohort study for children and young people

Introduction: The COVID-19 pandemic increased public use of digital mental health technologies. However, little is known about changes in user engagement with these platforms during the pandemic. This study aims to assess engagement changes with a digital mental healthcare service during COVID-19.

Methods: A cohort study based on routinely collected service usage data from a digital mental health support service in the United Kingdom. Returning users aged 14–25 years that interacted for a maximum of two months were included. The study population was divided into pre-COVID and COVID cohorts. Demographic and usage information between cohorts were compared and usage clusters were identified within each cohort. Differences were tested using Chi-squared, two-sample Kolmogorov–Smirnov tests and logit regressions.

Results: Of the 624,103 users who joined the service between May 1, 2019, and October 1, 2021, 18,889 (32.81%) met the inclusion criteria: 5,048 in the pre-COVID cohort and 13,841 in the COVID cohort. The COVID cohort wrote more journals; maintained the same focus on messaging practitioners, posting discussions, commenting on posts, and having booked chats; and engaged less in writing journals, setting personal goals, posting articles, and having ad-hoc chats. Four usage profiles were identified in both cohorts: one relatively disengaged, one focused on contacting practitioners through chats/messages, and two broadly interested in writing discussions and comments within the digital community. Despite their broad similarities, usage patterns also exhibited differences between cohorts. For example, all four clusters had over 70% of users attempting to have ad-hoc chats with practitioners in the pre-COVID cohort, compared to one out of four clusters in the COVID cohort. Overall, engagement change patterns during the COVID-19 pandemic were not equal across clusters. Sensitivity analysis revealed varying strength of these defined clusters.

Discussion: Our study identified changes in user activity and engagement behavior within a digital mental healthcare service during the COVID-19 pandemic. These findings suggest that usage patterns within digital mental health services may be susceptible to change in response to external events such as a pandemic. Continuous monitoring of engagement patterns is important for informed design and personalized interventions.

Introduction

Research suggests that the COVID-19 pandemic has exacerbated the mental health crisis across many countries, including the United Kingdom (UK) (1). Furthermore, there is a growing body of evidence highlighting effects of the pandemic and consequent lockdowns on children and young people’s (CYP) mental health (2–4). At the same time, contacts and interactions with all types of healthcare services reduced dramatically during the pandemic (5). This was also true for mental health-related contacts, particularly face-to-face contacts, with patients having to access to video and over-the-phone contacts (6, 7). Electronic mental healthcare and telemedicine rapidly became the “new normal” (8). By mid-2020, more than 80% of high-income countries shifted to digital mental health technologies to replace or supplement in-person mental health consultations (1).

The use of digital mental health technologies has provided data for several machine learning studies focusing on patterns of engagement within user-led digital support systems. These studies have illustrated several different ways in which mental health support can be personalized in a digital setting: based on engagement type, frequency of access, session duration, timing, and clinical outcomes (9–11). Segmenting users according to behaviors within digital mental health services can be a first step in personalizing support and improving design effectiveness (12).

However, to the best of our knowledge, there are no studies exploring whether user engagement and behaviors within these platforms are subject to change during major events like the COVID-19 pandemic, in which patterns of engagement can be disrupted and present some challenges to machine learning assumptions or solutions based on this type of information. At the same time, such knowledge could inform the use of digital mental health interventions (an important psychological support component during the COVID-19 pandemic) in future disasters (13) and help to resource and prevent overload or saturation of healthcare provision using data-driven technology and decisions.

We hypothesize that stable engagement types exist within Kooth, which could inform the personalization of services. The COVID-19 pandemic provides a unique opportunity to test this hypothesis, as it was an exceptional situation that could potentially affect user engagement patterns with digital mental health services. This cohort study aims to assess changes in engagement within a digital mental health service in the UK during the COVID-19 pandemic, comparing routinely collected usage data between a pre-COVID and a COVID cohort of users.

Materials and methods

We used data from Kooth Digital Health,¹ the UK’s largest provider to the National Health Service of web-based online mental health support (14). This service provides mental health support and interventions through its pseudonymous platform to CYP aged 11–25 years at no cost to the service users. Users can self-refer and find out about Kooth from school, online promotion, primary and secondary health services, social media or word of mouth.

The service allows CYP to self-direct their experience, interacting with their preferred type of support from a range of service features: personal journals, goal setting, discussion boards, articles, asynchronous therapeutic messaging, and live text-based counseling. Comprehensive safeguarding procedures are adhered to by moderators and practitioners following user interactions with the service. Demographic and usage information is stored across databases that can be linked at an individual level under the legal basis of ‘legitimate interest’ as it informs service improvements (15). In this study, data was used from 1 May 2019 to 31 December 2021.

Study sample

This study relied on data from 1 May 2019 to 31 December 2021. We included users between the ages of 14–25 years, and who had consented to have their non-identifiable demographic and service usage information used for research purposes. Users who were flagged by practitioners as not having Gillick competence (16) were excluded from the analysis.

To ensure sufficient journey and engagement information per service user, only returning users (i.e., with two or more log-ins) were included in the analysis. Users with a journey longer than 56 days were excluded from the dataset to avoid outliers, in that 99.03% of returning users aged 14–25 had a usage period of 56 days or less. To reduce bias from cut-off or cohort-crossing usage periods, users were excluded if their registration was within 56 days of the end date for each cohort dataset.

We divided users into two cohorts: pre-COVID and COVID. The World Health Organization declared COVID-19 a global pandemic on 11 March 2020 (17). Hence, we defined pre-COVID and COVID cohorts as users who signed up from 1 May 2019 to 11 January 2020 (256 days), and from 11 March 2020 to 1 October 2021 (570 days), respectively.

Measures

Demographic variables of interest collected routinely included ethnicity (‘Asian’, ‘Black’, ‘Mixed’, ‘White’ and ‘Other’), gender (‘Female’, ‘Male’ and ‘Non-binary’) and age group (‘14–17’ and ‘18–25’) at the time of registration. We measured interaction with the service through a number of service usage variables: 3 continuous variables (‘usage period’, ‘engagement’ and ‘activeness’) recording the overall level of interaction; and 8 dichotomous variables recording whether users made use of each component of the service (e.g., journals, discussions, ad-hoc chats). Table 1 provides the complete list of variables, including activity type, with details.

TABLE 1

Table 1. Characteristic, usage and experience variables of interest.

Analysis

We performed two-stepped analyses: (1) comparison of pre-COVID and COVID cohorts and (2) identification of usage profiles within pre-COVID and COVID cohorts. All data processing and analyses were done in python v3.9.2 (18). Packages sshtunnel 0.4.0, psycopg2-binary 2.9.5, pymysql 1.0.2, and python-bigquery 3.3.5 (19–22) were used to query data sources and construct the measures. We used scipy 1.9.3 (23) and statsmodels 0.13.2 packages (24) for statistical modeling and tests. The threshold for statistical significance for all value of ps was set at p < 0.05.

The main analyses were preceded by an assessment of the generalizability of our results for the service population, comparing separately for pre-COVID and COVID cohorts the study sample (returning, research consenting Kooth users aged 14–25 with a journey of ≤56 days) with the corresponding wider study population (all returning Kooth users aged 14–25). We used the Mann–Whitney U test (25) to measure differences in signup age as a continuous variable, and the Chi-squared test (26) to measure differences in ethnicity and gender.

In the first step of the analysis, we measured proportion and 95% confidence intervals (CI) estimated by Wilson score with continuity correction (27). We measured variations between the pre-COVID and COVID cohorts using Chi-squared tests (26) for demographic variables; two-sample Kolmogorov–Smirnov tests (28) for continuous usage variables; and logit regressions for dichotomous usage variables (as outcomes) using ‘cohort’ (i.e., pre-COVID or COVID) as the response variable and controlling for demographic and service change covariates.

In the second step, user groups were identified separately for pre-COVID and COVID cohorts through clustering of the usage variables. Prior to clustering, continuous usage variables were transformed to a logarithmic scale to limit the negative impact of large outliers (29). The usage data was then transformed into a binary indicator of whether the user had interacted with each component of the service, to address the sparsity of the data and improve the efficacy of dimensionality reduction (30). We applied Multiple Correspondence Analysis to reduce dimensionality and allow for the use of Euclidean-based clustering (31).

We ran a sensitivity analysis in which we applied KMeans, Birch, DBSCAN, and Gaussian Mixture Models on the resulting dataset to explore differences across clustering algorithms. We computed the Silhouette Coefficient as (b_i – a_i)/max(a_i, b_i), with a_i the mean intra-cluster distance of sample i, b_i the mean nearest-cluster distance of sample i, and N the number of samples. This ranges from −1 (worst) to 1 (best) – values near 0 indicate overlapping clusters (32) (Supplementary Tables S3, S4). The cluster number choice was made by inspecting the silhouette analysis plots for the highest scoring algorithms: Birch and KMeans for 2 to 5 clusters (Supplementary Figures S1–S4). After assessing these plots and clusters, we decided to present the output of Birch in the main text, but we ran the full analysis using KMeans for comparison.

We made a deliberate decision not to include service user demographics in the clustering algorithm to minimize the potential biasing effect of demographic factors on the identification of engagement behavior patterns. By doing so, we ensured that the resulting clusters were based solely on the observed engagement behaviors and not influenced by demographic characteristics. Instead, we were interested in observing whether different engagement profiles naturally had demographic differences.

Clusters were then further explored using Chi-squared tests (26) for demographic variables (not used to compute the clusters), Anderson-Darling tests (33) for k-samples for continuous usage variables; and logit regressions for dichotomous usage variables (as outcomes) with ‘cluster’ as the response and adjusting for covariates age group, ethnicity and gender (an overall value of p for variable ‘cluster’ was calculated through a log-likelihood test). For each cluster, we show proportion, 95% CIs and calculated value of ps per variable.

Results

Study sample

Of the 624,103 individuals who joined Kooth between May 1, 2019, and October 1, 2021, 57,568 (12.82%) were returning users aged 14–25 years. Of these, 18,889 (32.81%) met all the inclusion criteria: 5,048 in the pre-COVID cohort and 13,841 in the COVID cohort. The number of signups per day increased from 19.72 in the pre-COVID cohort to 24.28 in the COVID cohort. The most common place for service users to find out about Kooth is school, which remained consistent across both cohorts but with altered proportions (pre-COVID: 45.38%, COVID: 34.24%). Full details of the cohort selection procedure are in Figure 1.

FIGURE 1

Figure 1. Study cohort flow diagram.

When comparing demographic distributions between the pre-COVID and COVID cohorts and their corresponding population of interest, we found no significant difference in signup age (p = 0.220) and ethnicity (p = 0.999) for the pre-COVID cohort, and a significant difference for the COVID cohort (p < 0.001) with mean signup age of 0.78 years older and a 0.47 percentage points increase in user proportion that did not select an ethnicity option. For both cohorts, we found no significant differences in gender (p > 0.888).

Comparison between pre-COVID and COVID study cohorts

Full results of the comparison between pre-COVID and COVID cohorts are in Table 2.

TABLE 2

Table 2. Variable distributions for pre-COVID and COVID cohorts.

Signup age, and the proportion of users who reported gender as ‘Non-binary’ increased during the pandemic, as did the proportion of users reporting ‘Black’, ‘Mixed’ or ‘Other’ ethnicity against a decrease in those reporting ‘White’. More users had relatively longer usage periods during the pandemic, with similar engagement rates but a more active interaction with the service. The COVID cohort wrote more journals; maintained the same focus on messaging practitioners, posting discussions, commenting on posts, and having booked chats; and engaged less in writing journals, setting personal goals, posting articles, and having ad-hoc chats. A visual representation of similarities and differences in service usage between pre-COVID and COVID periods can be seen in the right-hand side of Supplementary Figures S1–S4.

Identification of usage clusters

During optimization of the clustering algorithms, the silhouette score varied between 0.14 and 0.46 for the pre-COVID cohort and between 0.16 and 0.43 for the COVID cohort across 24 hyperparameter configurations. Full results can be seen in Supplementary Tables S1 and S2 for the pre-COVID and COVID cohorts, respectively. There was no clear best configuration for both pre-COVID and COVID cohorts. Inspecting both the results of the silhouette analysis (Supplementary Figures S1–S4) and the obtained clusters, we decided to continue with the output of 4 usage clusters for each cohort.

Comparison between pre-COVID and COVID usage clusters

Figure 2 and Tables 3–6 show the size, characteristics, and usage profiles for each cluster. We observed some broad similarities between pre-COVID and COVID usage clusters. In both cases, cluster sizes were highly unbalanced, with the largest and smallest clusters containing 53.80 and 3.74% of pre-COVID users, respectively, and 51.29 and 2.15% of COVID users, respectively. The largest cluster in both cohorts was also the one with the shortest enrolment period (1–2 days) and the least engaged in practitioner-based interventions. Similarly, the smallest cluster in both cohorts was also the one with highest proportion of females, older users, and most engaged with practitioner-based interventions. The remaining two clusters where the most interested in community-based interventions.

FIGURE 2

Figure 2. Distribution plots for period, activeness and engagement variables across pre-COVID and COVID Birch clusters.

TABLE 3

Table 3. Control variables across Birch engagement clusters for the pre-COVID cohort.

TABLE 4

Table 4. Usage variables across Birch engagement clusters for the pre-COVID cohort.

TABLE 5

Table 5. Control variables across Birch engagement clusters for the COVID cohort.

TABLE 6

Table 6. Usage variables across Birch engagement clusters for the COVID cohort.

At the same time, there were substantial differences between the pre-COVID and COVID clusters. For example, gender differences were only significant in the COVID cohort. The unengaged cluster (C0) was less active during the pandemic. Over 70% of users in all pre-COVID clusters requested and ad-hoc chat with a practitioner, compared to only one COVID cluster. The two clusters interested in community-based interventions (C1 and C2) showed opposite trends in usage period and engagement. One of these (C2) was also the least engaged on creating articles before COVID, but the second most engaged during COVID. The cluster engaging the most with practitioner-based intervention had the highest proportion of users who selected Asian and Mixed ethnicity pre-COVID, and the lowest during COVID.

Our sensitivity analysis had mixed results (Supplementary Figure S5; Supplementary Tables S3–S5). In the pre-COVID period, the obtained clusters were substantially different to those of Birch, although we still found a cluster of disengaged used (this time even more disengaged). Of the remaining clusters, one was focused on both self-help and community-based interventions, and the other two focused on practitioner-based interventions and had moderate interest in community-based interventions. The size of pre-COVID clusters based on KMeans was also much more valanced, with each accounting for 20–30% of users. Meanwhile, KMeans’ result during COVID was similar to Birch’s, with a disengaged cluster, a cluster focused on practitioner-based interventions, and a cluster focused on community-based interventions. The fourth cluster was also relatively focused on community-based interventions. Cluster sizes were also similar to Birch’s.

Discussion

Key findings

We found changes in the usage of Kooth, a UK mental health digital service, by users aged 14–25 years during the COVID-19 pandemic. While the number of signups per day increased, these users were less engaged with the service, most prominently with less activity within each log-in (albeit usage periods were longer on average) and focusing less on creating articles and discussions and requesting ad-hoc chats with practitioners. This excess of users during the pandemic may be driven by a lack of capacity on traditional mental health services, a desire to ‘protect’ these services, and/or fear of COVID-19 infection in physical settings. We also identified changes in the user experience, with more users being asynchronously contacted and fewer having live chats with practitioners during the pandemic. This is likely the result of service changes implemented to manage the observed increase in the demand, like practitioners actively contacting users.

We conducted cluster analyses individually in each time period (before and during the COVID-19 pandemic) and identified four clusters or usage profiles: one relatively disengaged, one focused on contacting practitioners through chats/messages, and two broadly interested in writing discussions and comments within the digital community. The disengaged profile is likely an extension of our initial observation on the high proportion of users not returning to the system after one visit, as this profile is also the largest of the four (>50% of users), highlighting the importance of this type of interaction and user preference for digital interventions. Users seeking only contact with practitioners returned to the system sporadically. This is a fitting strategy for them, since there are natural idling times between messages and chats. Users more interested on posting articles, discussions and comments seemed to be the most committed overall, with relatively longer usage periods, engagement and activeness metrics. These seemed to be the most valanced users in terms of engagement, showing also high interactions in personal- and practitioner-based interventions. All clusters had over 70% of users requesting ad-hoc chats with practitioners, highlighting the importance of this type of interaction for digital interventions.

Pre-COVID and COVID usage profiles, despite being grossly similar, had some stark differences particularly with the two community-focused clusters. These two clusters exhibited opposite changes on some activity in the platform (e.g., practitioner-based interventions), even swapping their ranking as most/least engaged as a result in some instances. They also swapped the length of usage period and engagement. We originally thought that these differences may have been artificially introduced by moving from three to four clusters, but inspection of the Silhouette plots (Supplementary Figures S1, S2) revealed that this step gave way to the practitioner-focused cluster (i.e., the two community-engaged clusters were already present with three clusters). We also observed differences in demographic variables not used for clustering. These may explain part of the usage profile changes between pre-COVID and COVID within the platform, but the demographic differences pertained to a small proportion of the study sample, and therefore unlikely to explain the full range of such changes. Therefore, significant external events such as pandemics may impact how users interact with digital mental health services and may affect how to effectively identify patterns of engagement to form profiles.

Our sensitivity analysis led to a similar conclusion: that usage profiles are susceptible to significant external events. However, it also showed that the resulting usage profiles are not always strongly defined in our data, and thus the selection of clustering algorithm may have a big impact on the results – this may also be weakness of our data, rather than the methodology itself. As such, the utilization of usage profiles to inform the ongoing design of these services and the recommendations of personalized interventions may not be an optimal strategy – at least not during major events and not without the right data, careful sensitivity analyses and a strong methodology leading to robust outcomes.

Our clustering analysis revealed changes in service usage not readily apparent from the analysis using the full pre-COVID and COVID cohorts. Most prominently, despite community engagement variables decreasing (articles created) or not changing (discussions and comments created) during the COVID-19 pandemic, the influx of users focused on community engagement increased. Therefore, even though community engagement decreased during the COVID-19 pandemic, 2 out of 5 users that registered during this period in fact directed their attention to community-based activity. This effect may have been driven by the lockdown, self-isolation, and social distancing measures in place during the pandemic, and thus reflect users longing for social interaction, especially in young people (34). In general terms, different clusters show different patterns of change during COVID. From a methodological point of view, these results suggest that clustering analysis may be a useful tool in the analysis of service usage and its change over time, as it can provide insight into previously hidden patterns.

Comparison with prior research

Prior work on mental health service usage profiling incorporates time and typically tries to understand where a user is in their lifetime with a service (9–11). However, since the average user’s time with the service studied here is less than two weeks (pre-COVID: 12.35 [12.91], COVID: 12.90 [13.20]), we simplified the analysis by assuming fixed usage profiles throughout the users’ journey.

Prior work on mental health service usage profiling incorporates outcome variables and relies on a single time period for examination (9–11). They typically found between 3 and 5 usage profiles, mostly focused on the level of engagement. Since we did not have access to outcome variables in our analysis, direct comparison with other study results is not possible. However, we found a similar number of usage profiles within each cohort, some overall more engaged than others, but we also found differences in the type of engagement, as discussed above.

Previous mental health studies have shown a widespread deterioration of the population’s mental health during the COVID-19 pandemic (1), but disproportionately so for young adults and minoritized gender and ethnic groups (35, 36). We found corresponding increases in the number of signups to the service (from 19.72 users/day to 24.28 users/day). The proportion of users increased for adults, users who selected ‘Black’, ‘Mixed’ or ‘Other’ ethnicity and users who selected ‘Agender’ or ‘Gender Fluid’ gender, but not in the proportion of females compared to males.

Strengths and limitations

We have assessed changes in the way users interact with a digital mental health service before and after the COVID-19 pandemic started in the UK, using routinely collected usage data from 18,969 users across 30 months, including the first two waves of the pandemic. We explored whether these differences varied across user types, themselves defined using clustering techniques on usage information. To the authors’ knowledge, this is the first work to study how engagement behaviors within a digital mental healthcare service change during a global crisis of this kind.

We approached the use of clustering techniques not as a central part of the research, but as a tool to answer our research question (i.e., whether usage profiles changed during the COVID-19 pandemic). As such, our methodological decisions were not driven by clustering performance, but by domain knowledge (e.g., access routes of users to the different parts of the service, and the way their interactions are recorded) to ensure the relevance of all the included variables. Additionally, we validated and compared the resulting clusters using traditional statistical methods and exploring variable distributions.

There were limitations surrounding the study population, as it included only 32.81% of the total population of users ever using the service (27.93% from the full pre-COVID cohort and 35.04% from the full COVID cohort). Nevertheless, of age, gender and ethnicity, the study sample only differed from the whole population on signup age.

The digital mental health service examined moved through several product and service improvements, potentially influencing usage. This translates to unmeasured impact of changes which prevent us from establishing with complete certainty a relationship between the changes solely attributable to COVID-19 pandemic. Other potential mental health covariates were not available, like socioeconomic status or simultaneous engagement with other services, which has been shown to change during the pandemic (36, 37). The time period of data collection is also not consistent across cohorts, so there is a possibility that changes could be due to seasonality effects or other confounders.

Engagement with digital mental health services may also be subject to variation based on service availability, making it challenging to determine which digital behaviors are genuinely influenced by the service user and not by changes in the platform and resources to provide support. We made efforts to control for changes due to service availability in terms of messages sent to service users by practitioners. However, controlling for this factor becomes exceedingly difficult in an active, naturalistic environment where resource changes can occur at different times and in various regions where the service operates.

Our study was limited to a UK-only service, which restricted our ability to compare engagement data with similar services in other countries. Therefore, the generalizability of our findings is limited to the UK context, and caution should be exercised in extrapolating our results beyond the digital service examined.

Future research

Future research into CYP engagement would benefit from incorporating mental health measures before and after engagements. This would allow us to explore if measure responses predict engagement with digital services when combined with age, ethnicity and gender. There are known barriers in access to mental health services, and therefore understanding engagement patterns with digital mental health services can provide an early look into engagement preferences or barriers. We had data on mental health measures associated with this study, but opportunities for completion of these measures within the system were based on engagement preferences and were therefore biased. Hence, we decided to exclude outcome measures with a view to investigating outcomes in a separate study.

This study focuses only on returning users. For Kooth users aged 14–25, 87.18% of users do not return to the site after initial signup which leaves a large portion of the service user population uninvestigated. This drop-off could be due to implementation barriers such as lack of personalization or human capacity (38), and it is similar to that reported by other digital platforms (39). Future research is needed to understand the difference between returning and non-returning users, and how to maximize the potential of brief engagement vs. more continuous and regular engagement.

Our main finding, that usage profiles are affected by major events, puts into question the stability of usage profiles using clustering methods of data based on engagement. Further analysis over periods without major catastrophic events is required to ascertain whether changes in usage profiles can also occur naturally (i.e., without the influence of major events), but this also highlights the importance of examining and accounting for such events when machine learning algorithms are used for cluster and designing products and services and optimization.

This study focuses on user engagement changes between pre-COVID and COVID cohorts. Future work should investigate the long-term impact of the pandemic on mental health. It would be beneficial to explore whether mental health issues during the pandemic have regressed back to their mean or if they have persisted, as well as the role of community support in mental health during the pandemic. This may involve longitudinal studies to track changes in service engagement and mental health outcomes over time.

Kooth is a standalone digital mental health platform that provides online support for children and young people – it is not specifically designed to be integrated into face-to-face mental health care or other health care systems. While Kooth can be accessed independently by users, it may also be used as part of a blended care approach, where digital interventions are combined with traditional face-to-face services. However, the extent to which Kooth is adaptable to a blended care approach is beyond the scope of this study and warrants further investigation.

Conclusion

The study of the effect of the COVID-19 pandemic on digital mental health services is particularly relevant, as these remained uninterrupted, while face-to-face services paused or changed provision. We explored the user activity and engagement behavior within a digital mental healthcare service and identified changes in these digital profiles during the COVID-19 pandemic. This indicates that usage profiles are not suitable to inform service design or provide personalized interventions yet, as they are susceptible to change due to events like a pandemic. However, usage profiles can provide important insight into the analysis of such changes in digital behavior and can help us better understand digital mental health service user populations and contribute to future disaster management procedures (13).

While digital mental health interventions can be powerful support tools, particularly in periods when traditional face-to-face services lack capacity or space, a better understanding of user engagement with these systems and how it changes over time is needed to fully unlock their potential, alongside other important considerations such as effectiveness, usability, and equity of access.

Data availability statement

The data analyzed in this study is subject to the following licenses/restrictions: Under Kooth’s privacy policy, data can only be shared with trusted partners for research studies dedicated to improving the service. Requests to access these datasets should be directed to AB, abernard@kooth.com.

Ethics statement

The studies involving human participants were reviewed and approved by Swansea University Ethics Committee. Written informed consent from the participants’ legal guardian/next of kin was not required to participate in this study in accordance with the national legislation and the institutional requirements. Study data covered only individuals that provided informed consent for their data to be utilized for research purposes.

Author contributions

AB and MDP-B defined the original concept for the study, conducted the comparison of clusters, and wrote the original draft. AB prepared the data and conducted the cluster analysis. AB, SdOG, and MDP-B interpreted the results. All authors reviewed the original draft and contributed to the final version.

Funding

AJ and MDP-B were funded by UKRI – Medical Research Council through the DATAMIND Hub (MRC reference: MR/W014386/1).

Conflict of interest

AB, SdOG, and LS are employed and receive honorarium by Kooth plc. MDP-B was contracted by Kooth plc as a consultant for this work.

The remaining author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The authors declare that this study received funding from Kooth Digital Health. The funder had the following involvement in the study: study design, data collection and analysis, interpretation of the results, and preparation of the manuscript.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2023.1143272/full#supplementary-material

Abbreviations

CYP, children and young people.

Footnotes

1. ^Kooth.com

References

1. Hewlett, E, Takino, S, Nishina, Y, and Prinz, C. Tackling the mental health impact of the COVID-19 crisis: An integrated, whole-of-society response. Paris: OECD (2021).

Google Scholar

2. Mindel, C, Salhi, L, Oppong, C, and Lockwood, J. Alienated and unsafe: Experiences of the first national UK COVID‐19 lockdown for vulnerable young people (aged 11–24 years) as revealed in Web‐based therapeutic sessions with mental health professionals. Couns Psychother Res. (2022) 14:708–24.

Google Scholar

3. Elliot Major, L, Eyles, A, and Machin, S. Generation COVID: Emerging work and education inequalities. Centre Econ Perform. (2020) 14.

Google Scholar

4. Young Minds. Coronavirus: impact on young people with mental health needs. Young Minds. (2021). Available at: www.youngminds.org.uk

Google Scholar

5. Moynihan, R, Sanders, S, Michaleff, ZA, Scott, AM, Clark, J, To EJ, et al. Impact of COVID-19 pandemic on utilisation of healthcare services: a systematic review. BMJ Open. (2021) 11:e045343. doi: 10.1136/BMJOPEN-2020-045343

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Armitage, R. Antidepressants, primary care, and adult mental health services in England during COVID-19. Lancet Psychiatry. (2021) 8:e3. doi: 10.1016/S2215-0366(20)30530-7

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Bakolis, I, Stewart, R, Baldwin, D, Beenstock, J, Bibby, P, Broadbent, M, et al. Changes in daily mental health service use and mortality at the commencement and lifting of COVID-19 ‘lockdown’ policy in 10 UK sites: a regression discontinuity in time design. BMJ Open. (2021). 11:e049721. doi: 10.1136/BMJOPEN-2021-049721

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Martinez-Martin, N, Dasgupta, I, Carter, A, Chandler, JA, Kellmeyer, P, Kreitmair, K, et al. Ethics of Digital mental Health during COVID-19: crisis and opportunities. JMIR Ment Health. (2020) 7:e23776. doi: 10.2196/23776

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Chien, I, Enrique, A, Palacios, J, Regan, T, Keegan, D, Carter, D, et al. A machine learning approach to understanding patterns of engagement with internet-delivered mental Health interventions. JAMA Netw Open. (2020) 3:e2010791. doi: 10.1001/JAMANETWORKOPEN.2020.10791

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Matthews, P, Topham, P, and Caleb-Solly, P. Interaction and engagement with an anxiety management app: analysis using large-scale behavioral data. JMIR Mental Health. (2018) 5:9235. doi: 10.2196/MENTAL.9235

CrossRef Full Text | Google Scholar

11. Sanatkar, S, Baldwin, PA, Huckvale, K, Clarke, J, Christensen, H, Harvey, S, et al. Using cluster analysis to explore engagement and e-attainment as emergent behavior in electronic mental Health. J Med Internet Res. (2019) 21:e14728. doi: 10.2196/14728

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Fleming, T, Merry, S, Stasiak, K, Hopkins, S, Patolo, T, Ruru, S, et al. The importance of user segmentation for designing digital therapy for adolescent mental Health: findings from scoping processes. JMIR Ment Health. (2019) 6:e12656. doi: 10.2196/12656

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Sheek-Hussein, M, Abu-Zidan, FM, and Stip, E. Disaster management of the psychological impact of the COVID-19 pandemic. Int J Emerg Med. (2021). doi: 10.1186/S12245-021-00342-Z/TABLES/2

CrossRef Full Text | Google Scholar

14. North Central London Clinical Comissioning Group. (2021). Kooth: the UK’s leading mental health & wellbeing platform for children and young people. North Central London GP Website Available at: https://gps.northcentrallondonccg.nhs.uk/education/video/kooth-the-uks-leading-mental-health-wellbeing-platform-for-children-and-young-people-islington (Accessed August, 2022)

Google Scholar

15. Kooth Digital Health. (2022). Privacy policy – Kooth. Kooth.Com. Available at: https://www.kooth.com/privacy (Accessed August, 2022)

Google Scholar

16. Griffith, R. What is Gillick competence? Hum Vaccin Immunother. (2016) 12:244–7. doi: 10.1080/21645515.2015.1091548

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Ghebreyesus, T. (2020). WHO director-General’s opening remarks at the media briefing on COVID-19 – 11 March 2020. World Health Organisation. Available at: https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---11-march-2020

Google Scholar

18. Python. (2021). Python release Python 3.9.2. Python.Org. Available at: https://www.python.org/downloads/release/python-392/ (Accessed April, 2022)

Google Scholar

19. White, P. (2021). Pahaz/sshtunnel: SSH tunnels to remote server. Github.Com. Available at: https://github.com/pahaz/sshtunnel/ (Accessed April, 2022)

Google Scholar

20. Varrazzo, D. Psycopg/psycopg2: PostgreSQL database adapter for the Python programming language. Github.Com (2021). Available at: https://github.com/psycopg/psycopg2 (Accessed April, 2022)

Google Scholar

21. Naoki, I. (2016). PyMySQL/PyMySQL: pure Python MySQL client. Github.Com. Available at: https://github.com/PyMySQL/PyMySQL (Accessed April, 2022)

Google Scholar

22. Google. Releases…googleapis/python-bigquery. Github.Com (2021). Available at: https://github.com/googleapis/python-bigquery/releases (Accessed April 2022)

Google Scholar

23. Virtanen, P, Gommers, R, Oliphant, TE, Haberland, M, Reddy, T, Cournapeau, D, et al. Scipy/scipy: SciPy library main repository. Nature Methods. (2020) 17:261–72. doi: 10.1038/s41592-019-0686-2

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Seabold, S, and Perktold, J. Statsmodels/statsmodels. Github.Com (2010). Available at: https://github.com/statsmodels/statsmodels (Accessed April 2022)

Google Scholar

25. Mann, HB, and Whitney, DR. On a test of whether one of two random variables is stochastically larger than the other. Ann Math Stat. (1947) 18:50–60. doi: 10.1214/AOMS/1177730491

CrossRef Full Text | Google Scholar

26. Karl Pearson, FRS. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Biometrika. (2009) 50:157–75. doi: 10.1080/14786440009463897

CrossRef Full Text | Google Scholar

27. Brown, LD, Cai, TT, and DasGupta, A. Interval estimation for a binomial proportion. Stat Sci. (2001) 16:101–33. doi: 10.1214/SS/1009213286

CrossRef Full Text | Google Scholar

28. Hodges, JL. The significance probability of the mirnovv two-sample test. Ark Mat. 3:469–86. doi: 10.1007/BF02589501

CrossRef Full Text | Google Scholar

29. Fan, C, Chen, M, Wang, J, and Huang, B. A Review on Data Preprocessing Techniques Toward Efficient and Reliable Knowledge Discovery From Building Operational Data. Front. Energy Res. (2021) 9:652801. doi: 10.3389/fenrg.2021.652801

CrossRef Full Text | Google Scholar

30. Jollife, IT, and Cadima, J. Principal component analysis: a review and recent developments. Philos Trans R Soc A Math Phys Eng Sci. (2016) 374:20150202. doi: 10.1098/rsta.2015.0202

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Greenacre, M, and Blasius, J. Multiple correspondence analysis and related methods. Boca Raton, FL: CRC Press (2006).

Google Scholar

32. Rousseeuw, PJ. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J Comput Appl Math. (1987) 20:53–65. doi: 10.1016/0377-0427(87)90125-7

CrossRef Full Text | Google Scholar

33. Stephens, MA. EDF statistics for goodness of fit and some comparisons. J Am Stat Assoc. (1974) 69:730–7. doi: 10.1080/01621459.1974.10480196

CrossRef Full Text | Google Scholar

34. Sundqvist, A, and Hemberg, J. Adolescents’ and young adults’ experiences of loneliness and their thoughts about its alleviation. Int J Adolesc Youth. (2021) 26:238–55. doi: 10.1080/02673843.2021.1908903

CrossRef Full Text | Google Scholar

35. O'Connor, RC, Wetherall, K, Cleare, S, McClelland, H, Melson, AJ, Niedzwiedz, CL, et al. Mental health and well-being during the COVID-19 pandemic: longitudinal analyses of adults in the UK COVID-19 Mental Health & Wellbeing study. Br J Psychiatry. (2021) 218:326–33. doi: 10.1192/BJP.2020.212

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Bécares, L, and Kneale, D. Inequalities in mental health, self-rated health, and social support among sexual minority young adults during the COVID-19 pandemic: analyses from the UK millennium cohort study. Soc Psychiatry Psychiatr Epidemiol. (2022) 57:1979–86. doi: 10.1007/S00127-022-02291-1

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Carr, MJ, Steeg, S, Webb, RT, Kapur, N, Chew-Graham, CA, Abel, KM, et al. Effects of the COVID-19 pandemic on primary care-recorded mental illness and self-harm episodes in the UK: a population-based cohort study. Lancet Public Health. (2021) 6:e124–35. doi: 10.1016/S2468-2667(20)30288-7

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Balcombe, L, and De Leo, D. Evaluation of the use of Digital mental Health platforms and interventions: scoping review. Int J Environ Res Public Health. (2023) 20:362. doi: 10.3390/ijerph20010362

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Grist, R, Porter, J, and Stallard, P. Mental health mobile apps for preadolescents and adolescents: a systematic review. J Med Internet Res. (2017) 19:e176. doi: 10.2196/jmir.7332

PubMed Abstract | CrossRef Full Text | Google Scholar