Patterns of engagement in digital mental health intervention for LGBTQ+ youth: a latent profile analysis

Engagement is a key metric that researchers can use to assess whether participants received the intended dose of a digital health intervention. However, the prevailing approach has predominantly focused on individual paradata metrics, resulting in a fragmented understanding of overall engagement. To address this limitation, our study utilizes person-centered approaches that allow for the simultaneous capture of multiple engagement metrics within imi–a web application specifically designed to support the mental health of lesbian, gay, bisexual, transgender, queer, and other sexual and gender minority youth (LGBTQ+ youth). This person-centered approach enabled us to explore the association between engagement patterns and stress appraisal outcomes within the imi intervention arm. Utilizing latent profile analysis, we classified users into two engagement forms: overall engagement (total number of sessions, pages visited, and external links clicked and their cumulative time spent using imi) and content engagement (number of pages viewed across imi's four core guides: gender, stress, queerness, and stigma). We identified two profiles for each form: a “high engagement” profile and an “average engagement” profile, with the majority of participants assigned to the “average engagement” profile. Our analyses revealed a significant association between overall engagement profiles and stress appraisals, with the “high engagement” profile demonstrating higher challenge appraisals and marginal improvements in threat appraisals compared to the “average engagement” profile. However, no such associations were observed for content engagement profiles and stress appraisal outcomes. The two person-centered approaches used were consistent with prior results utilizing a variable-centered approach, indicating a stronger intervention effect among individuals who exhibit higher engagement in digital health interventions. Although both methods yielded comparable findings, the person-centered approach mitigates concerns related to multi-collinearity and adds additional nuance and context to the study of digital engagement.

1. Introduction

Digital health interventions (DHIs) have become increasingly popular in recent years as a means of promoting healthy behaviors and reducing healthcare costs through their affordability and accessibility. Particularly for younger generations, DHIs have proven to be an excellent resource for improving health outcomes (1). DHIs also can be tailored to meet the particular needs of individuals, making them a promising tool for enhancing health outcomes (2, 3). Furthermore, DHIs have proven to be highly effective, especially within sexual and gender minority (SGM) communities, as DHIs serve as a valuable tool for reaching SGM youth (4–6). Despite their promising outcomes, the true effectiveness of DHIs is a matter of concern. A review of 24 unique DHIs seeking to address mental, physical, or sexual health–related concerns in lesbian, gay, bisexual, transgender, queer, and other sexual and gender minority (LGBTQ+) young people found the methodologies to be lacking in rigor and the quality of evidence to be poor (7). Researchers have noted that the variable efficacy across trials may be dependent on participant engagement with the interventions (8), as opposed to their reach or acceptability.

Engagement can be understood as a multi-faceted concept encompassing three fundamental components—behavior, cognition, and affect—that are commonly shared across various domains (9, 10). Considering the wide variability in engagement with previous DHIs (11), analyzing engagement to assess whether participants received the intended dose of a DHI becomes a crucial effectiveness and implementation outcome. Previous studies that have examined the relationship between engagement and outcomes suggest that engagement positively predicts improved health outcomes (11). A recent meta-analysis of DHIs focused on mental health found greater engagement to be positively associated with therapeutic gains (12). Similarly, greater engagement has been associated with changes in other health domains (e.g., HIV prevention and care). For example, in an online intervention to reduce sexual risk behaviors among young Black men who have sex with men, researchers found a stronger intervention effect among those who used the intervention for more than 60 min over the 3-month intervention period (8).

The behavioral aspect of engagement, often referred to as “usage,” is the most frequently employed dimension. Paradata (i.e., automatically generated process data that captures participants' actions within an application or website) allows for nuanced and interrelated explorations regarding how users' behavioral engagement within a DHI are associated with an intervention's outcomes (13–16). Using paradata, Hightow-Weidman and Bauermeister characterized engagement metrics into four domains: amount, duration, frequency, and depth. This characterization offers a foundation for standardizing the use of paradata to examine behavioral engagement in DHIs (11). To date, however, there has been a tendency to explore individual paradata metrics (i.e., amount, duration, and depth of use) as independent predictors of outcomes in DHI analyses (11). For example, Choi et al. explored the correlation of engagement within an online HIV prevention intervention for single young men who have sex with men using two individual paradata metrics; the number of log-ins and the number of sessions viewed (17). Also, Bonett utilized two individual paradata metrics, the time spent on the intervention components and interactions with features, to examine whether paradata metrics differed by participants' characteristics in a brief online intervention to reduce barriers to healthcare for young men who have sex with men (18). However, these traditional variable-centered approaches, which focus on the predictive power of individual paradata metrics, have limitations in explaining the complex nature of user engagement within DHIs (4, 11) and often result in multiple testing issues, including inflated Type I error rates (19, 20). Therefore, there is a need to examine the utility of alternative methodologies like person-centered approaches (i.e., mixture cluster analysis, latent profile analysis, or latent class analysis) to recognize the multidimensional nature of engagement (21) and correct for these statistical challenges.

Person-centered approaches are extensively utilized in human behavior research as they enable researchers to gain a comprehensive understanding of complex behaviors by considering multiple factors simultaneously (22, 23). Moreover, the person-centered approach can help address multi-collinearity issues that may arise from the variable-centered approach. In the context of DHI engagement, there are various metrics (i.e., amount, depth, and frequency) that contribute to the quality of engagement beyond just duration (e.g., time spent), which is commonly used. Through the adoption of a person-centered approach, researchers can identify unmeasured patterns based on the distribution of paradata metrics concurrently. While a few studies have employed traditional variable-centered approaches to explore individual paradata metrics in DHIs (8, 13, 17, 24), there have been no previous studies that have utilized person-centered approaches to comprehensively understand engagement patterns in DHIs with multiple factors simultaneously. Therefore, it is crucial to investigate the utility of employing a person-centered approach in examining DHI engagement, as this may offer new insights and facilitate a more accurate understanding of engagement.

We utilized imi, a web application (web-app) designed to facilitate LGBTQ+ identity affirmation for teens and young adults (ages 13–19), as a case study (25). The imi application was created with the aim of fostering affirmation of SGM identities, fostering a sense of belonging within the LGBTQ+ community, and motivating the practice of cognitive and behavioral coping skills. Consistent with the Minority Stress Model (26), imi recognizes that addressing the well-being of LGBTQ+ youth must acknowledge that negative health outcomes often arise from a hostile culture towards sexual and gender minorities, which creates stressors unique to minority identity, including harassment, victimization, internalized homophobia, and expectations of rejection. To address these challenges, researchers (27–30) have argued that LGBTQ+ interventions should focus on providing resources for coping with minority stress, including strengthening SGM youth's ability to cope, teaching additional cognitive and behavioral coping skills, affirming SGM identities, and strengthening supportive social connections. Therefore, the imi web-application offered content, interactive activities, stories, images and videos from LGBTQ+ youth across four content areas: (1) exploring gender identity (gender), (2) exploring sexual orientation and broader LGBTQ+ identities (queerness), (3) managing externalized SGM stressors (e.g., discrimination) and learning coping skills (stress), and (4) dealing with internalized homophobia and transphobia (stigma). Results from the imi trial (25) supported the value of this approach, with participants assigned to the intervention arm reporting improved challenge appraisals (i.e., belief in one's coping abilities) than participants in the attention-control arm. Moreover, main outcome results found that specific individual paradata metrics (≥5 sessions, >10 min, or >10 pages) were associated with greater benefits among imi intervention arm participants (25).

In this analysis, we used a person-centered approach to characterize the paradata collected in the imi intervention and examine whether different typologies of engagement are associated with participants' stress appraisals. We proposed two person-centered approaches: an overall engagement approach and a content engagement approach. The overall engagement approach explores how users interact with a DHI by employing four behavioral engagement indicators: (1) users' cumulative time spent on imi, (2) their total number of sessions, (3) pages visited, and (4) external links clicked. The content engagement approach, on the other hand, examines participants' number of page views across the four topic-specific guides within imi: (1) gender, (2) stress, (3) queerness, and (4) stigma. The content engagement approach allowed for an assessment of whether individual differences in the specific nature of the content consumed were predictive of the benefits participants derived from the intervention. Using these two distinct forms of engagement (overall engagement and content engagement), we examined the relationship between engagement profiles and changes in stress appraisals, which is the primary outcome of the imi study, across the course of the 4-week intervention (25). We also compare our person-centered approach results with those previously reported using the traditional variable-centered approach.

2. Materials and methods

Data for this analysis came from a pilot randomized controlled trial (RCT) of imi, a web application designed to improve mental health by supporting lesbian, gay, bisexual, transgender, queer, and other LGBTQ+ youth identity affirmation, coping self-efficacy, and coping skills. A detailed procedure for the imi study has been previously outlined elsewhere (25).

2.1. Study procedures

This is a 4-week pilot RCT to evaluate the efficacy of the imi web-application. Participants were randomized to either the intervention arm (called “imi”), which has full access to the application or the control arm which had access to curated resource pages which were mirrored on imi (see Figure 1; currently accessible at https://imi.guide/). We collected survey data via web-based self-completed Qualtrics surveys administered at baseline and a 4-week follow-up. Participants' actions in the application were collected as paradata. Participants were recruited across the United States through advertisements on Instagram. Participants had to (1) be 13 to 19 years of age (inclusive); (2) self-identify as LGBTQ+; (3) reside within the United States; (4) have English literacy; (5) have access to a device with internet access, a web browser, and SMS text messaging capabilities; and (6) be willing to participate in study activities for 4 weeks. To ensure that our results generalize to racial/ethnic minority LGBTQ+ youth, who often face multiple forms of intersecting minority stress, we over-recruited racial/ethnic minority participants such that they comprise approximately 75% of our sample.

FIGURE 1

Figure 1. Screenshots from the imi (treatment) web applications [from Bauermeister et al. (25)].

Interested participants completed an online screener between October and November 2021. If eligible, they were shared an email link to the consent form and baseline survey. Participants had two weeks to complete the survey. 162 of the 488 screeners who were invited to consent and complete the baseline survey did not participate during the allowable two-week period. In addition to established best practices for participant verification (31–33), we compared participants’ screener data to their baseline responses across various questions asked on both assessments. If inconsistencies were identified, individuals were emailed and asked to respond via email or phone to resolve the issue. 56 participants did not pass the participant verification procedure (i.e., there were discrepancies between information entered on screener and baseline surveys). In total, a sample of 270 LGBTQ+ youth (mean age 16.49; SD 1.49) was enrolled and randomized into the intervention arm and control arm in a 1:1 fashion.

2.2. Intervention description

The imi web-application is a tool that provides curated content, interactive activities, videos, stories, and imagery from LGBTQ+ youth across guides covering four content areas. Content was tailored based on youth feedback and contributions (i.e., interviews, focus group, and co-design sessions). These areas are: (1) exploring gender identity (gender), (2) exploring sexual orientation and broader LGBTQ+ identities (queerness), (3) managing externalized SGM stressors (e.g., discrimination) and learning coping skills (stress), and (4) dealing with internalized homophobia and transphobia (stigma). Participants in the intervention group also had access to resource webpages that linked to the additional crisis and non-crisis resources, which were also available to the control group. Participants were asked to use imi at least once per week.

2.3. Ethics approval

All study procedures were approved by the Institutional Review Board at the University of Pennsylvania (protocol 849509), and the study was registered on ClinicalTrials.gov (NCT05061966). We received a waiver of parental consent for youth who may not have disclosed their gender identity or sexuality to their parents or who may have limited parental support to participate in the study.

2.4. Measures

2.4.1. Engagement profiles

This study analyzed the intervention arm (n = 135) paradata over the 4 week intervention period and the main outcome at the end of 4 weeks. Over the 4 week intervention period, participants' actions in imi were collected as paradata. Paradata can be transformed to characterize the amount, frequency, duration, and depth of engagement with a web-based intervention. In this study, we present two distinct forms of engagement, (1) overall engagement and (2) content engagement to characterize their potential to inform the evaluation of DHIs in the future.

2.4.1.1. Overall engagement

We derived four paradata metrics to define overall engagement: (1) counts of user sessions, (2) time spent in the intervention, (3) the number of pages visited, and (4) external links clicked.

2.4.1.2. Content engagement

We also derived the number of pages visited across theoretically-anchored guides: (1) gender identity exploration (gender), (2) sexual orientation and broader LGBTQ+ identity exploration (queerness), (3) stress and coping (stress), and (4) internalized homophobia and transphobia (stigma).

2.4.2. Primary outcomes

The primary outcome in the imi RCT was stress appraisals. The Stress Appraisal Measure for Adolescents (34) captures stress appraisals across 3 dimensions (challenge, threat, and resources). The 3-item Challenge subscale assesses perceptions of stress as a surmountable challenge (Cronbach α = .67). The 7-item Threat subscale measures perceptions of stress as having lasting, negative repercussions (α = .83). The 3-item Resources subscale assesses the belief that one has the necessary internal and external resources to cope with stress (α = .81). Responses to all items are recorded on a 5-point scale (1 = strongly disagree to 5 = strongly agree). A mean score was computed for each subscale, with higher values indicating greater endorsement of each respective stress appraisal.

2.4.3. Demographic characteristics

We asked the participants to report characteristics regarding their age, race, ethnicity, highest level of education, subjective socio-economic status, gender identity, and sexual orientation. For gender identity and sexual orientation, respondents were given the option to select multiple identities. Subsequently, we categorized gender identity as either “mutually exclusive cisgender” or “other.” Similarly, we dichotomized sexual orientation into “single identity” or “multiple identities.” Further details about non-mutually exclusive patient characteristics can be found elsewhere (25).

2.5. Statistical analysis

Descriptive statistics were used to summarize the study participants' demographic characteristics and engagement characteristics, including total session completed, total time spent, time of engagement, engagement devices (phone or computer), unique page viewed, and the number of links clicked.

We utilized latent profile analysis to determine the number of engagement profiles (35). For overall engagement, we used the z-scores for four sub-constructs: (1) counts of user sessions, (2) time spent on each intervention, (3) the number of pages visited, and (4) external links clicked. For content engagement profiles, we used the mean score for four sub-constructs: (1) the number of pages visited in the gender guide, (2) the number of pages visited in the queerness guide, (3) the number of pages visited in the stress guide, and (4) the number of pages visited in the stigma guide. The most appropriate model was chosen using several criteria, including the Akaike Information Criteria (AIC), Bayesian Information Criteria (BIC), Vuong-Lo-Mendell-Rubin Likelihood Ratio Test (LRT), and entropy (36). Lower AIC and BIC scores indicate a better model fit, and a higher entropy score indicates better class separation. A significant LRT between the k-1 profile model and the k profile model indicates that the model fit improved from the former to the latter. Once the best-fitting model was selected, each profile was defined based on the mean scores of the four sub-constructs. Latent profile analysis was conducted using Mplus version 8 (37). After finalizing the best-fitting model, Cohen's kappa coefficient was used to evaluate the degree of agreement between the two engagement profiles (38).

Finally, we used linear regression to test the main effect of engagement (latent engagement profiles) on primary outcomes at week 4, adjusting for the baseline value of each respective outcome as a covariate. We also tested for changes over time within each engagement profile. We used SAS 9.4 to examine the associations between engagement profiles and primary outcomes (Cary, NC: SAS Institute Inc.).

3. Results

3.1. Sample characteristics

Participants (n = 135) had a mean age of 16.6 (SD 1.46) years. Most participants lived in a metropolitan area (91.1%) and identified as racial/ethnic minorities (76.3%). Over a third of participants identified with multiple genders (42.2%) and sexual identities (39.3%).

3.2. Engagement

Users completed a median of four sessions, viewed 25 unique pages (the maximum number of unique pages for imi was 73), and spent 11.6 min in imi. More than half of the participants accessed imi between 9am and 6pm. Participants had the option to access the web-app using either mobile phones or computers, and it was observed that a higher percentage of participants (81.1%) chose to use mobile phones, while a smaller percentage (38.6%) used computers (not mutually exclusive; some participants had access to both). Given external links were the sole component of the control arm whereas they were embedded within a lot of other content within the intervention arm, the median number of links clicked was 0 (range 0–6).

3.3. Latent profile analysis

3.3.1. Overall engagement

We iteratively compared models with increasing numbers of profile solutions using AIC and BIC. AIC and BIC get smaller as the number of profiles increases. Entropy was similar across the different solutions. The difference in LRT between a 2-class solution and a 3-class solution was not statistically significant (two-times-the-log likelihood difference = 133.48, df = 5, p = 0.166) (see Table 1). Therefore, a 2-class latent profile solution was selected from the empirical and theoretical perspective as the optimal model; the results of the 2-profile solution are shown in Figure 2. The “average engagement” profile represents the degree to which individuals reported mean engagement with imi. This profile accounted for the majority of the sample (n = 104; 77.0%) and was characterized by having comparable scores across the four derived paradata metrics (sessions, time, pages, and external links). The “high engagement” profile (n = 31; 22% of the sample) was characterized by being at least one standard deviation (range 1.06–1.97) above the mean for all metrics. Pages had the greatest difference (SD 1.91) between profiles, followed by external links (SD 1.81), minutes (SD 1.79), and sessions (SD 1.38).

TABLE 1

Table 1. Fit indices for overall engagement profile analysis.

FIGURE 2

Figure 2. Mean Z-scores for the four engagement metrics by overall engagement profiles. Session: total number of sessions; Time: cumulative time spent; page: total number of pages visited; links: total number of external links clicked.

3.3.2. Content engagement

Similar to the overall engagement results, analysis of content engagement showed that the values of AIC and BIC decreased with an increase in the number of profiles, indicating better model fit. Furthermore, the measure of model fit known as Entropy, remained consistent across the different solutions. The difference in LRT between a 2-class solution and a 3-class solution was not statistically significant (two-times-the-log likelihood difference = 199.1, p = 0.222; see Table 2), therefore, a 2-class latent profile solution was optimal for the content engagement (see Figure 3). The “compliant engagement” profile accounted for the majority of the sample (n = 105; 77.8%) and was characterized by comparable engagement across the four derived paradata metrics (gender, stress, queerness and stigma pages). Among four different content pages, participants read stress-related pages the least. The “high engagement” profile (n = 30; 21.2% of the sample) was characterized by around one standard deviation (range 0.83–1.59) above the mean for all metrics. The number of stress pages had the biggest difference (SD 2.05) between the two profiles.

TABLE 2

Table 2. Fit indices for the content engagement profile analysis.

FIGURE 3

Figure 3. Mean Z-scores for the number of pages viewed in four guides by content engagement profiles. Gender: the number of pages visited in the gender guide; stress: the number of pages visited in the stress guide; queer: the number of pages visited in the queerness guide; stigma: the number of pages visited in the stigma guide.

3.3.3. Agreement between overall engagement profile and content profile

A Kappa agreement was employed to assess the level of agreement between the overall engagement profile and the content engagement profile. Classification of users between the overall engagement profile and the content engagement profile was high (Kappa = 0.81).

3.4. Engagement profiles, demographic characteristics, time of engagement, and modes of delivery

We examined the association between demographic characteristics and engagement profiles and none of the demographic characteristics differed by engagement profiles (see Table 3). We also analyzed time of engagement (i.e., the time of day in which participants' used imi) across the different engagement profiles (see Table 4). Participants in the “high engagement” profile, both for overall engagement and content profiles, exhibited greater engagement with the intervention during the time frames of 6am–9am and 6pm–12am local time, compared to the “compliant engagement” profile (p < 0.001). However, mode of delivery (phone vs. computer) was not associated with engagement profiles.

TABLE 3

Table 3. Demographic characteristics by engagement profile (N = 135).

TABLE 4

Table 4. Engagement metrics by engagement profile (N = 135).

3.5. Engagement profiles on primary outcomes

The observed associations between three stress appraisals and each engagement profile varied. For overall engagement, we identified a significant association with two stress appraisal outcomes: challenge appraisals and threat appraisals (see Table 5). The “high engagement” profile reported significantly higher challenge appraisals (Cohen d = 0.25; p = 0.034) compared to the “average engagement” profile. However, we did not find significant associations between overall engagement profiles and threat (Cohen d = −0.31; p = 0.057) and resource appraisals (Cohen d = 0.41; p = 0.086). None of the three stress appraisal outcomes were associated with the content engagement profiles (see Table 6).

TABLE 5

Table 5. Changes in primary and secondary outcomes from baseline to the 4-week follow-up between overall engagement profiles.

TABLE 6

Table 6. Changes in primary and secondary outcomes from baseline to the 4-week follow-up between content engagement profiles.

4. Discussion

We employed a person-centered approach to analyze engagement patterns within a randomized control trial of imi, a web-application for LGBTQ+. Based on overall and content engagement with the app, we identified two engagement profiles: “average engagement” and “high engagement.” For overall engagement, most participants fell into the “average engagement” profile, while only 33% were classified as “high engagement.” We also examined the distribution of four metrics (total number of sessions, pages visited, external links clicked, and time spent) within each profile and identified correlations among them, indicating similar distributions between the metrics. For content engagement, we also identified two profiles (“high engagement” and “average engagement”), with the number of stress-related pages viewed most clearly differentiating the two profiles. Considering that engagement with DHIs positively predicts health outcomes, further research is necessary to explore the factors that drive participants to engage more with certain content compared to others. Understanding these drivers will enable researchers to implement content in real time that allows customization for participants' needs, thereby enhancing their engagement.

We observed a strong agreement between overall engagement and content engagement. This suggests that participants who engaged with the intervention content in a general sense exhibited similar patterns of engagement with specific content pages. However, it remains significant to examine both types of engagement profiles. Our exploration of content-specific engagement revealed that participants in the “high engagement” profile in content engagement were motivated to read the stress-related page more frequently than other pages. This suggests that overall engagement in terms of duration, frequency, and quantity may be influenced by pages addressing minority stress. Since our study was cross-sectional, we were unable to determine whether participants experiencing higher levels of stress were more engaged with imi or if the stress section was simply more attractive to engage with compared to other pages. Obtaining this information will enable researchers to develop more tailored interventions that meet the specific needs of individuals and promote effective engagement strategies (4).

The findings of this study underscore the significance of engagement in achieving positive outcomes in DHIs. Our analysis revealed a noteworthy association between overall engagement patterns and stress appraisals. Moreover, both the “high engagement” and “average engagement” profiles exhibited improvements in stress appraisals, with the “high engagement” profile showing greater enhancements compared to the “average engagement” profile. These results align with previous studies (8, 25), highlighting the potential of increased engagement with DHIs as an effective strategy for achieving improved intervention outcomes. This emphasizes the importance of promoting greater utilization of DHI tools to maximize their potential in supporting health and well-being. Moreover, efforts to determine the optimal dosage that facilitates desired outcomes without surpassing a threshold that may contribute to participant burden or compulsive behaviors.

Furthermore, our analysis revealed that participants in the “high engagement” profile tended to use imi more frequently between 6am–9am and 6pm–12am compared to those in the “average engagement” profile. This observation is closely tied to non-schooling time, indicating that engagement patterns are heavily influenced by LGBTQ+ youth who can comfortably use imi during periods outside of school hours. This difference in the time of engagement between profiles highlights the need to adopt precision health approaches to DHIs to promote health based on a better understanding of individual circumstances. For instance, DHIs could allow users to customize their preferred usage times based on when they feel most comfortable using them, and receive reminder SMS to encourage engagement during those times. Furthermore, future qualitative research could delve into understanding the underlying factors that contribute to the higher usage of certain hours by “high engagement” users compared to “average engagement” users.

Our approach to conceptualizing engagement using alternative statistical methodologies has improved our understanding of engagement by recognizing that four domains in paradata metrics contribute to the level of engagement simultaneously. Traditionally, researchers have employed single paradata metrics independently, assuming that each metric independently influences the level of engagement, even when these metrics are highly correlated (8, 17). In the original analysis of the imi RCT data, individual paradata metrics such as session count (≥5 sessions), duration (>10 min), or page views (>10 pages) were associated with improved outcomes among participants in the imi intervention group (25). These findings align with our study results, which indicate that participants in the “high engagement” profile were more likely to demonstrate greater improvements in stress appraisals compared to the “average engagement” profile. While our person-centered approach adds extra nuance and context, it is worth noting that the results observed in our study were not significantly different from those in the original imi paper. This comparison underscores that, despite their greater comprehensiveness, person-centered approaches can still yield similar conclusions to the methodologies that examine measurements of individual metrics, thus demonstrating the robustness of the findings.

Furthermore, our study contributes to researchers' understanding of different levels of engagement thresholds. Determining the minimum engagement threshold required to induce behavioral changes has been challenging for researchers (17). However, this study defined levels of engagement using latent profile analysis, identifying a distinct “high engagement” profile characterized by participants exhibiting a specific range of paradata metrics within the use of imi. This nuanced approach to understanding engagement can provide valuable insights for designing interventions and tailoring strategies to meet engagement thresholds.

Additionally, our study highlighted two distinct conceptual paradata categories: overall engagement and content engagement. It is noteworthy that although there was a high level of agreement among users in terms of these two forms of engagement, they exhibited different associations with our primary outcome. The reasons behind stress appraisals being more strongly associated with overall engagement, as opposed to content engagement, could be influenced by the limited sample size. Although there was substantial agreement between overall engagement and content engagement, we need to highlight that the baseline stress appraisal scores for the high engagement profile in content engagement were higher than those for overall engagement. The relatively small sample size of only 30 participants in the high engagement profile could impact the results. Therefore, this discrepancy in baseline scores with small sample size led to non-significant results for content engagement. Further research with a larger sample size and incorporating additional longitudinal analyses would be beneficial for delving deeper into any discrepancies in the results. It is also possible that the total amount of the intervention consumed was a better predictor of stress appraisals than the specific type of content consumed because all sections of the guide focused on scaffolding identity affirmation and coping with minority stress, and therefore, the degree of engagement with the different sections may not have been differentially effective.

This study has several limitations. First, we were unable to examine real-time changes in participants' behaviors based on their engagement. Although paradata was collected continuously during the study, the measurement of behavior and outcomes was collected at only two time points, limiting our understanding of real-time behavior changes in response to engagement. Second, while we conceptually framed the paradata metrics based on existing literature, there is a possibility that the metrics researchers consider most important for engagement may not align with what participants themselves perceive as most significant. For example, the way users navigate modules or components within DHIs (i.e., mouse movements or engagement sequence), though not measured in the current study, may yield additional insights and could be included as additional paradata metrics. Further research is needed to compare qualitative and quantitative analyses of engagement in DHI in order to determine the most impactful metrics for measuring engagement. Third, the study sample size was limited, which restricted our ability to explore more detailed engagement patterns. Future studies with larger sample sizes would be valuable in helping researchers comprehend the various nuanced engagement patterns, ultimately aiding in the promotion of effective engagement strategies. Finally, our study obtained informed consent from participants for the collection of engagement data, thereby minimizing ethical concerns related to paradata usage. However, it is crucial to acknowledge that the utilization of paradata in different contexts may entail additional ethical considerations that need to be addressed (39).

Given the influence engagement has on DHI outcomes, it is crucial to explore ways of gaining a comprehensive understanding of user engagement. This requires efforts to operationalize paradata both conceptually and statistically, as different approaches can lead to diverse outcomes and conclusions. In our study, we employed latent profile analysis as a statistical method to classify engagement, which proves promise in addressing concerns related to multi-collinearity of paradata in DHI research. Furthermore, by pooling interrelated paradata and examining engagement across multiple domains, we were able to identify two distinct types of users based on their engagement patterns. This holistic categorization of users can be valuable for monitoring users' compliance with recommended DHI doses or creating targeted engagement campaigns aimed at transitioning users from one category to another (e.g., shifting from average engagement to high engagement). In future DHIs, it is advisable to explore user engagement using methods such as latent profile analysis that allow for the consideration of interrelated paradata across multiple domains. This approach, combined with different aspects of operationalization, will enable researchers to gain a deeper understanding of the complexities associated with engagement.

Data availability statement

The data analyzed in this study is subject to the following licenses/restrictions: The data gathered during this study is not publicly available and cannot be shared due to confidentiality. Further inquiries can be directed to the corresponding authors. Requests to access these datasets should be directed to SC, skchoi@nursing.upenn.edu.

Ethics statement

The studies involving humans were approved by The University of Pennsylvania Institutional Review Board approved all study procedures (protocol 849509). The studies were conducted in accordance with the local legislation and institutional requirements. The ethics committee/institutional review board waived the requirement of written informed consent from legal guardians/next of kin for underage participants. This waiver of parental consent was granted to ensure that youth who might not yet be out to their parents or have less parental support, and thus could benefit from an identity-affirming tool, could participate in the study.

Author contributions

SC: Data curation, Formal Analysis, Methodology, Visualization, Writing – original draft. EB: Conceptualization, Investigation, Project administration, Resources, Writing – review & editing. AG: Conceptualization, Writing – review & editing. JL: Data curation, Project administration, Resources, Writing – review & editing. JB: Conceptualization, Project administration, Supervision, Writing – review & editing.

Acknowledgments

The authors are grateful to all their partners who participated in the co-design of the imi application and the study reported herein. In particular, the authors would like to thank Fred Dillon, Lionel Ramazzini, Ariana Taboada, Jesse Golinkoff, Jana Haritatos, Chris McCarthy, Denise Ho, Danielle Ramo, Janise Chan, Grace Greene, Cassandra Marzke, Primo Lagaso Goldberg, Griffin Somaratne, Starr Jiang, Rayan Madjidi, Adriana Allman, Sam Dinga, Will Coleman, Jett Allen, Amanda Keller, Lauren Jacobs, Katie Suppes, Meryl Franklin, Sam Aguilar, Kevin McCloskey, Forrest Marie Linsell, David Biele, Tess Allen, Audrey Ford, Brooks Banker, Deborah Levine, and Justin Tindall, who were instrumental in the design, development, and testing of the imi application. The authors would also like to thank their scientific advisors Annesa Flentje, Shelley Craig, Edward Alessi, Ashley Austin, Alison Cerezo, and Amaranta Ramirez for their thoughtful input and feedback.

Conflict of interest

imi is a progressive web application co-developed by Hopelab, CenterLink, and It Gets Better Project. Although all 3 organizations were involved in the development of the product, CenterLink is responsible for the operation and distribution of imi. As a free digital tool created and distributed by non-profit organizations, none of the organizations involved stand to profit financially from the product. The research reported here, as well as the development of imi, were supported by the non-profit foundation, Hopelab. Hopelab develops behavioral interventions to improve the health and well-being of young people. The design, conduct, analysis, and reporting of this study represent a scientific collaboration between Hopelab and the Eidos LGBTQ+ Health Initiative at the University of Pennsylvania. EB, AG, and JL are employed by Hopelab. The study sponsor was involved in the study design; collection, analysis, and interpretation of data; writing of the article; and decision to submit it for publication.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Hollis C, Falconer CJ, Martin JL, Whittington C, Stockton S, Glazebrook C, et al. Annual research review: digital health interventions for children and young people with mental health problems—a systematic and meta-review. J Child Psychol Psychiatry. (2017) 58(4):474–503. doi: 10.1111/jcpp.12663

CrossRef Full Text | Google Scholar

2. O’Cathain A, Croot L, Duncan E, Rousseau N, Sworn K, Turner KM, et al. Guidance on how to develop complex interventions to improve health and healthcare. BMJ Open. (2019) 9(8):e029954. doi: 10.1136/bmjopen-2019-029954

CrossRef Full Text | Google Scholar

3. Yardley L, Morrison L, Bradbury K, Muller I. The person-based approach to intervention development: application to digital health-related behavior change interventions. J Med Internet Res. (2015) 17(1):e4055. doi: 10.2196/jmir.4055

CrossRef Full Text | Google Scholar

4. Hightow-Weidman LB, Horvath KJ, Scott H, Hill-Rorie J, Bauermeister JA. Engaging youth in mHealth: what works and how can we be sure? mHealth. (2021) 7:23. doi: 10.21037/mhealth-20-48

CrossRef Full Text | Google Scholar

5. Muessig KE, Nekkanti M, Bauermeister J, Bull S, Hightow-Weidman LB. A systematic review of recent smartphone, Internet And Web 2.0 interventions to address the HIV continuum of care. Curr HIV/AIDS Rep. (2015) 12(1):173–90. doi: 10.1007/s11904-014-0239-3

CrossRef Full Text | Google Scholar

6. Hightow-Weidman LB, Muessig KE, Bauermeister J, Zhang C, LeGrand S. Youth, technology, and HIV: recent advances and future directions. Curr HIV/AIDS Rep. (2015) 12(4):500–15. doi: 10.1007/s11904-015-0280-x

CrossRef Full Text | Google Scholar

7. Gilbey D, Morgan H, Lin A, Perry Y. Effectiveness, acceptability, and feasibility of digital health interventions for LGBTIQ+ young people: systematic review. J Med Internet Res. (2020) 22(12):e20158. doi: 10.2196/20158

CrossRef Full Text | Google Scholar

8. Hightow-Weidman LB, LeGrand S, Muessig KE, Simmons RA, Soni K, Choi SK, et al. A randomized trial of an online risk reduction intervention for young black MSM. AIDS Behav. (2019) 23:1166–77. doi: 10.1007/s10461-018-2289-9

CrossRef Full Text | Google Scholar

9. Kelders SM, van Zyl LE, Ludden GDS. The concept and components of engagement in different domains applied to eHealth: a systematic scoping review. Front Psychol. (2020) 11:926. doi: 10.3389/fpsyg.2020.00926

CrossRef Full Text | Google Scholar

10. Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med. (2017) 7(2):254–67. doi: 10.1007/s13142-016-0453-1

CrossRef Full Text | Google Scholar

11. Hightow-Weidman L, Bauermeister JA. Engagement in mHealth behavioral interventions for HIV prevention and care: making sense of the metrics. mHealth. (2020) 6:7. doi: 10.21037/mhealth.2019.10.01

CrossRef Full Text | Google Scholar

12. Gan DZQ, McGillivray L, Han J, Christensen H, Torok M. Effect of engagement with digital interventions on mental health outcomes: a systematic review and meta-analysis. Front Digit Health. (2021) 3:764079. doi: 10.3389/fdgth.2021.764079

CrossRef Full Text | Google Scholar

13. Bauermeister JA, Golinkoff JM, Muessig KE, Horvath KJ, Hightow-Weidman LB. Addressing engagement in technology-based behavioural HIV interventions through paradata metrics. Curr Opin HIV AIDS. (2017) 12(5):442. doi: 10.1097/COH.0000000000000396

CrossRef Full Text | Google Scholar

14. Couper M. Measuring survey quality in a CASIC environment. Proc Surv Res Methods Sect ASA JSM. (1998):41–9.

Google Scholar

15. Couper MP, Kreuter F. Using paradata to explore item level response times in surveys. J R Stat Soc Ser A Stat Soc. (2013) 176(1):271–86. doi: 10.1111/j.1467-985X.2012.01041.x

CrossRef Full Text | Google Scholar

16. Nicolaas G. Survey Paradata: a review. (2011). ESRC National Centre for Research Methods.

17. Choi SK, Golinkoff J, Michna M, Connochie D, Bauermeister J. Correlates of engagement within an online HIV prevention intervention for single young men who have sex with men: randomized controlled trial. JMIR Public Health Surveill. (2022) 8(6):e33867. doi: 10.2196/33867

CrossRef Full Text | Google Scholar

18. Bonett S, Connochie D, Golinkoff JM, Horvath KJ, Bauermeister JA. Paradata analysis of an eHealth HIV testing intervention for young men who have sex with men. AIDS Educ Prev Off Publ Int Soc AIDS Educ. (2018) 30(5):434–47. doi: 10.1521/aeap.2018.30.5.434

CrossRef Full Text | Google Scholar

19. Bender R, Lange S. Adjusting for multiple testing—when and how? J Clin Epidemiol. (2001) 54(4):343–9. doi: 10.1016/S0895-4356(00)00314-0

CrossRef Full Text | Google Scholar

20. Hightow-Weidman L, Muessig KE, Egger JR, Vecchio A, Platt A. Epic allies: a gamified Mobile app to improve engagement in HIV care and antiretroviral adherence among young men who have sex with men. AIDS Behav. (2021) 25(8):2599–617. doi: 10.1007/s10461-021-03222-y

CrossRef Full Text | Google Scholar

21. Scotto Rosato N, Baer JC. Latent class analysis: a method for capturing heterogeneity. Soc Work Res. (2012) 36(1):61–9. doi: 10.1093/swr/svs006

CrossRef Full Text | Google Scholar

22. Christens BD, Collura JJ, Tahir F. Critical hopefulness: a person-centered analysis of the intersection of cognitive and emotional empowerment. Am J Community Psychol. (2013) 52(1–2):170–84. doi: 10.1007/s10464-013-9586-2

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Christens BD, Peterson NA, Reid RJ, Garcia-Reid P. Adolescents’ perceived control in the sociopolitical domain: a latent class analysis. Youth Soc. (2015) 47(4):443–61. doi: 10.1177/0044118X12467656

CrossRef Full Text | Google Scholar

24. Baltierra NB, Muessig KE, Pike EC, LeGrand S, Bull SS, Hightow-Weidman LB. More than just tracking time: complex measures of user engagement with an internet-based health promotion intervention. J Biomed Inform. (2016) 59:299–307. doi: 10.1016/j.jbi.2015.12.015

CrossRef Full Text | Google Scholar

25. Bauermeister J, Choi SK, Bruehlman-Senecal E, Golinkoff J, Taboada A, Lavra J, et al. An identity-affirming web application to help sexual and gender minority youth Cope with minority stress: pilot randomized controlled trial. J Med Internet Res. (2022) 24(8):e39094. doi: 10.2196/39094

CrossRef Full Text | Google Scholar

26. Meyer IH. Prejudice, social stress, and mental health in lesbian, gay, and bisexual populations: conceptual issues and research evidence. Psychol Bull. (2003) 129(5):674. doi: 10.1037/0033-2909.129.5.674

CrossRef Full Text | Google Scholar

27. Craig SL, Austin A, McInroy LB. School-based groups to support multiethnic sexual minority youth resiliency: preliminary effectiveness. Child Adolesc Soc Work J. (2014) 31(1):87–106. doi: 10.1007/s10560-013-0311-7

CrossRef Full Text | Google Scholar

28. Craig SL, Leung VWY, Pascoe R, Pang N, Iacono G, Austin A, et al. AFFIRM online: utilising an affirmative cognitive–behavioural digital intervention to improve mental health, access, and engagement among LGBTQA+ youth and young adults. Int J Environ Res Public Health. (2021) 18(4):1541. doi: 10.3390/ijerph18041541

CrossRef Full Text | Google Scholar

29. Pachankis JE, Hatzenbuehler ML, Rendina HJ, Safren SA, Parsons JT. LGB-affirmative cognitive-behavioral therapy for young adult gay and bisexual men: a randomized controlled trial of a transdiagnostic minority stress approach. J Consult Clin Psychol. (2015) 83(5):875–89. doi: 10.1037/ccp0000037

CrossRef Full Text | Google Scholar

30. Craig SL, Eaton AD, Leung VWY, Iacono G, Pang N, Dillon F, et al. Efficacy of affirmative cognitive behavioural group therapy for sexual and gender minority adolescents and young adults in community settings in Ontario, Canada. BMC Psychol. (2021) 9(1):94. doi: 10.1186/s40359-021-00595-6

CrossRef Full Text | Google Scholar

31. Teitcher JEF, Bockting WO, Bauermeister JA, Hoefer CJ, Miner MH, Klitzman RL. Detecting, preventing, and responding to “fraudsters” in internet research: ethics and tradeoffs. J Law Med Ethics. (2015) 43(1):116–33. doi: 10.1111/jlme.12200

CrossRef Full Text | Google Scholar

32. Bauermeister JA, Pingel E, Zimmerman M, Couper M, Carballo-Diéguez A, Strecher VJ. Data quality in HIV/AIDS web-based surveys: handling invalid and suspicious data. Field Methods. (2012) 24(3):272–91. doi: 10.1177/1525822X12443097

CrossRef Full Text | Google Scholar

33. Guest JL, Adam E, Lucas IL, Chandler CJ, Filipowicz R, Luisi N, et al. Methods for authenticating participants in fully web-based Mobile app trials from the iReach project: cross-sectional study. JMIR MHealth UHealth. (2021) 9(8):e28232. doi: 10.2196/28232

CrossRef Full Text | Google Scholar

34. Rowley AA, Roesch SC, Jurica BJ, Vaughn AA. Developing and validating a stress appraisal measure for minority adolescents. J Adolesc. (2005) 28(4):547–57. doi: 10.1016/j.adolescence.2004.10.010

CrossRef Full Text | Google Scholar

35. Berlin KS, Williams NA, Parra GR. An introduction to latent variable mixture modeling (part 1): overview and cross-sectional latent class and latent profile analyses. J Pediatr Psychol. (2014) 39(2):174–87. doi: 10.1093/jpepsy/jst084

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Nylund KL, Asparouhov T, Muthén BO. Deciding on the number of classes in latent class analysis and growth mixture modeling: a Monte Carlo simulation study. Struct Equ Model. (2007) 14(4):535–69. doi: 10.1080/10705510701575396

CrossRef Full Text | Google Scholar

37. Muthén LK, Muthén BO. Mplus. User’s guide. Compr Model Program Appl Res. (2015) 5.

Google Scholar

38. McHugh ML. Interrater reliability: the kappa statistic. Biochem Medica. (2012) 22(3):276–82. doi: 10.11613/BM.2012.031

CrossRef Full Text | Google Scholar

39. Struminskaya B, Sakshaug JW. Ethical considerations for augmenting surveys with auxiliary data sources. Public Opin Q. (2023) 87(S1):619–33. doi: 10.1093/poq/nfad030

CrossRef Full Text | Google Scholar