Ishaq Malik1
Arnaldo Perez2
Elaine Toombs1Fred Schmidt1,3Janine V. Olthuis4Jaidyn Charlton1Elizabeth Grassia1Crystal Squier5Kristine Stasiuk5Tina Bobinski5
Aislin R. Mushquash1*
- 1Department of Psychology, Lakehead University, Thunder Bay, ON, Canada
- 2Faculty of Medicine and Dentistry, University of Alberta, Edmonton, AB, Canada
- 3Children’s Centre Thunder Bay, Thunder Bay, ON, Canada
- 4University of New Brunswick, Fredericton, NB, Canada
- 5Dilico Anishinabek Family Care, Fort William First Nation, ON, Canada
Introduction: Mobile health (mHealth) apps are a promising adjunct to traditional mental health services, especially in underserviced areas. Developed to foster resilience in youth, the JoyPop™ app has a growing evidence base showing improvement in emotion regulation and mental health symptoms among youth. However, whether this novel technology will be accepted among those using or providing mental health services remains unknown. This study aimed to evaluate the JoyPop™ app's acceptance among (a) a clinical sample of youth and (b) mental health service providers.
Method: A qualitative descriptive approach involving one-on-one semi-structured interviews was conducted. Interviews were guided by the Technology Acceptance Model and were analyzed using a deductive-inductive content analysis approach.
Results: All youth (n = 6 females; Mage = 14.60, range 12–17) found the app easy to learn and use and expressed positive feelings towards using the app. Youth found the app useful because it facilitated accessibility to helpful coping skills (e.g., journaling to express their emotions; breathing exercises to increase calmness) and positive mental health outcomes (e.g., increased relaxation and reduced stress). All service providers (n = 7 females; Mage = 43.75, range 32–60) perceived the app to be useful and easy to use by youth within their services and expressed positive feelings about integrating the app into usual care. Service providers also highlighted various organizational factors affecting the app's acceptance. Youth and service providers raised some concerns about apps in general and provided recommendations to improve the JoyPop™ app.
Discussion: Results support youth and service providers' acceptance of the JoyPop™ app and lend support for it as an adjunctive resource to traditional mental health services for youth with emotion regulation difficulties.
Introduction
Mental health difficulties are increasing among youth in Canada, and barriers to receiving services (e.g., limited community-based programs, long wait times) are prominent (1). These access barriers are particularly salient in rural and remote areas like Northwestern Ontario, Canada, compared to more populous urban areas (2, 3). Limited access to mental health services and long wait times can profoundly impact youth by increasing social dysfunction, decreasing treatment motivation, prolonging physical and emotional distress, and exacerbating mental health conditions (3, 4). Novel interventions are required to address these challenges.
The Changing Directions, Changing Lives, Mental Health Strategy for Canada, developed by the Mental Health Commission of Canada, has recommended increasing e-mental health (eHealth) interventions to help meet the increasing demand for mental health supports (2). Notably, the Mental Health Commission of Canada reports that youth in Canada are open to using technology for mental health care and that eHealth interventions may benefit Indigenous people and youth in rural and remote areas (5). A recent systematic review of the acceptability and feasibility of eHealth interventions tailored toward the mental health of Indigenous youth found that service users (ages 12–25) and providers (e.g., Elders, frontline workers) report positive attitudes towards eHealth interventions (6).
Mobile health (mHealth) apps (i.e., medical and public health practices delivered via mobile device applications) are one type of eHealth intervention discussed as a possible solution in addressing unmet needs for youth mental health support in Canada (3, 5). App-based solutions have some advantages over traditional services as they provide increased availability and access to support and reduced geographical barriers (7–9). mHealth apps can also be effective as adjuncts to regular services for improving self-management of mental health related difficulties among youth, adolescents, and adults (9–14). However, there are significant gaps between the large number of mHealth apps available and data supporting claims made about their effectiveness (12, 15). Further, the evidence on the effectiveness of mHealth apps designed for youth (12–25) is limited and mixed (11, 14). This was highlighted by a recent systematic review of 11 RCTs (attention controls or wait-listed peers) examining the effectiveness of mHealth apps tailored toward youth [1,706 adolescents and young adults; (11)]. The results showed that four of eight studies on apps targeting depression showed moderate to large improvements, four studies examining overall mental health reported significant small to medium positive effects, three of seven studies found significant and positive effects in reducing anxiety, four studies targeting distress only showed small and non-significant effects, and two studies examining stress showed significant positive effects (11). The authors concluded that more long-term evaluations and studies are needed within diverse clinical, gender, and ethnic youth populations before mental health services can adopt mHealth apps into care (11).
Another consideration related to mHealth app integration is the issue of app engagement. Showing app effectiveness via change in quantitative outcome measures does not guarantee that an app will be used or accepted, and the evidence-based content of apps is often unrelated to their popularity (16). Research has shown that approximately 70% of patients who used or were invited to use mHealth technologies stopped prematurely or declined, with lack of engagement cited as a major contributor (17). Furthermore, the unique needs of service providers (e.g., managers, clinicians) and organizations, which are often highly regulated and have budgetary concerns, can facilitate or prevent app integration into healthcare services (18, 19). Consistent with mHealth app assessment frameworks (20) and Canadian recommendations, future research on developing and evaluating mHealth apps requires a multimethod approach (5, 20). While establishing effectiveness of an app for improving target outcomes is important via quantitative methods, qualitative research is needed to examine user perspectives, identify facilitators and barriers of use, and to reveal specific features, designs, and functions that help with engagement and long-term uptake (21, 22). Examining user perspectives is essential as perceived challenges can slow the uptake of mHealth into services (17, 23).
The Technology Acceptance Model (TAM) is one of the most used frameworks in qualitative studies examining factors influencing users' and service providers’ acceptance of mHealth apps (18, 24). This framework posits that perceived usefulness and perceived ease of use are strong determinants of attitudes, which in turn influence the intention to use technology (24, 25). Extensions are often added to the TAM framework when evaluating app acceptance in complex healthcare settings as these extensions better capture important organizational factors (e.g., available resources and time, consistency with values and needs) (25, 26). For example, the facilitating conditions construct from the Unified Theory of Acceptance and Use of Technology [UTAUT; (26)] can account for organizational factors affecting service providers' acceptance of new technology (18, 26, 27). The present study sought to evaluate the acceptance of an mHealth app (JoyPop™) among a clinical sample of youth and service providers in Northwestern Ontario using the TAM framework.
The JoyPop™ app is a resilience-building mHealth app designed for youth (12–25) (28–30) that was conceptualized and developed using resilience theory. Resilience theory is a strength-based approach that focuses on increasing promotive factors rather than reducing youth deficits or risk exposure (31, 32). Promotive factors are social, individual, and environmental factors that can reduce the likelihood that an individual exposed to stress or adversity progresses to pathology (31, 32). Promotive factors are further understood as assets, which occur within an individual (e.g., emotion regulation) and resources, which occur outside an individual [e.g., social connectedness; (32)]. Emotion regulation and social connectedness are recognized as some of the most important facets of resilience (33, 34).
The JoyPop™ app (see Figure 1) focuses primarily on improving promotive assets by helping individuals learn, practice, and implement evidence-based emotion regulation skills that increase self-monitoring and self-awareness (28, 29). More specifically, the Rate My Mood feature helps youth identify, differentiate, and manage emotions (35). The Journal feature provides an opportunity to express thoughts and emotions in words or emojis, and the Calendar provided an opportunity to reflect on past Journal entries (36, 37). Breathing Exercises support relaxation and refocusing (38). A Tetris-like game called SquareMoves provides a helpful distraction (39). The Art feature allows users to express their creativity freely (40). The SleepEase feature provides users with best-practice sleep tips and soothing water sounds to aid sleep onset (41). The app also aims to increase promotive resources by facilitating opportunities for individuals to connect with social and mental health supports. The Circle of Trust feature facilitates quick and easy access to personal social support networks. The Call for Help feature provides instant access to 24-h helplines. When used over time, the improved emotion regulation skills can become habitual responses to stress and improve overall resilience (33, 34, 42). Strength-based mHealth apps, like the JoyPop™ app, can provide youth with 24-h access to evidence-based skills and resources to foster their inherent resilience by up-regulating the positive (e.g., social support), down-regulating the negative (e.g., emotion labelling), and transcending the self [e.g., mindful breathing; (34)]. Moreover, the JoyPop™ app's focus on emotion regulation aligns with recommendations that mHealth apps emphasize the transdiagnostic treatment of mental health conditions (43) and target overall well-being and coping instead of specific mental health disorders (44, 45).
Figure 1. The JoyPop app.
The framework for developing the JoyPop™ app involved a cumulative research and parallel consultation approach. Essential to app development was research showing that improved self-reflection and self-regulation can increase resilience and reduce the association between adversity and adverse mental health outcomes (46, 47). App features were developed based on these findings and consequent consultations by team members, youth, service providers, and clinician-scientists (28). After an initial version of the app was created, revisions of features and designs were assessed and implemented using input from youth involved in child welfare and victim services and providers working with those youth (28).
The JoyPop™ app is particularly relevant in Northwestern Ontario where access to mental health services are especially limited (2, 3) and local data showing that many youth seeking services present with emotion regulation difficulties (48). Moreover, Indigenous youth in this region also show significant challenges related to past trauma and adverse childhood experiences [ACEs; (49)]. This is consistent with evidence from a recent systematic review showing that Indigenous people experience more ACEs (mean score of 2.5–3.05) than non-Indigenous people [mean of 1.36; (50)]. Research has shown that experiencing adversity during childhood is associated with the development of autonomic nervous system alterations that contribute to heightened sensitivity to environmental demands, increased biological and emotional reactivity in response to stress, and less capability for adaptive self-regulation (51, 52). The JoyPop™ app's strength-based approach in fostering resilience and improving emotion regulation skills may help to mitigate the impact of this past trauma and adversity (53, 54).
The JoyPop™ app has a growing multi-method evidence base supporting its use with youth and young adults in Northwestern Ontario. For example, first-year undergraduates at a university in Northwestern Ontario [N = 156, 78.8% female, Mage = 19.02 (SD = 2.90), range 16–38, 87.8% were 19 years or younger] used the app for four weeks and showed significant improvements in depressive symptoms and emotion regulation (29). Notably, greater improvements in emotion regulation were evident for those with early adversity and trauma (e.g., abuse, neglect) (29). A follow-up qualitative descriptive study was conducted with 30 undergraduates [80% female, Mage = 18.77 (SD = 2.30), range 16–29, 93% being 19 years or younger] who had used the app over four weeks (30). Participants highlighted important facilitators of use (e.g., increased self-monitoring and expressive opportunities), barriers to use (e.g., lack of variety and editing), positive outcomes of use (e.g., improved self-awareness and emotion regulation), and recommendations for improvement (e.g., adding and enhancing features). This study provided key insight into features of the app that were useful and engaging to users, along with recommendations to continue improving user experience while promoting long-term engagement (30). While promising, initial evaluations of the JoyPop™ app were limited to older youth within a university setting (29), and qualitative feedback was gathered without a guiding theoretical framework (30). Moreover, it is unknown how service providers felt about the app and its use or relevance for the clients presenting for mental health services.
To better understand the relevance and utility of the JoyPop™ app for youth seeking mental health services in Northwestern Ontario, this study evaluated the acceptance of the JoyPop™ app among a clinical sample of youth and service providers at the two largest mental health agencies in Northwestern Ontario. Specifically, we evaluated (1) youth acceptance of the JoyPop™ app using the TAM (i.e., perceived usefulness, perceived ease of use, and attitudes towards use); and (2) service providers' acceptance of the JoyPop™ app in a mental health setting using the TAM plus the facilitating conditions construct from the UTAUT (26). In line with the TAM, we defined acceptance as youth and service providers' behavioural intention to use the app based on perceived usefulness, perceived ease of use, and overall attitudes towards use. With the lack of a consensus and clear definitions of acceptability, acceptance, and adoption in the mHealth literature (55), this study used recommendations suggested by the Technology Acceptance Lifecycle guidelines to improve the quality of communication in technology use research. These guidelines recommend using the term acceptance when a technology is in the initial use phase and when perceptions are being assessed after using a technology (55).
Materials and methods
Design
The present study was part of a larger pilot study of the JoyPop™ app. For the pilot study, youth 12–18 years old who were seeking or receiving mental health services from [blinded for peer review] were provided with the app and asked to use it across four weeks. Youth completed outcome assessments at baseline, after two weeks, and after four weeks. The pilot study included 41 youth (63.4% female; 70.4% Indigenous, 24.4% White) with a mean age of 15.0 (SD = 1.41). Twenty participants who completed all outcome assessments during the pilot study were invited to participate in the present qualitative study. Service providers affiliated with youth services from pilot study sites were invited to participate as well. This qualitative study was guided by qualitative description, which is best suited when a straightforward description of a phenomena is desired (56). Conceptually, the TAM and UTAUT informed the study design, especially data collection and analysis. The present study was reviewed and approved by [blinded University's] Research Ethics Board and the ethics committee/advisory boards of [blinded for peer review; partner mental health agencies].
Participant sampling and recruitment
Youth
All youth were recruited using purposeful sampling, which identifies and selects individuals with rich knowledge or experience related to the primary purpose of a research study (57). Consequently, we sought feedback and invited youth with significant experience using the JoyPop™ app to obtain comprehensive feedback on its acceptance. Youth were contacted via email, asking if they were interested in participating in the qualitative study following their completion of the pilot study.
Service providers
We used purposeful sampling to identify service providers who were informed about the JoyPop™ app and interested in participating. Service providers at both pilot study sites were initially informed about the JoyPop™ app during the launch and promotion of the pilot study (e.g., through presentations and weekly emails from the research team). Some service providers at the sites had referred youth to the pilot study whereas others had less direct involvement with the pilot study. We also used snowball sampling by asking participating service providers if they knew other service providers that may be interested in the study (57).
Data collection
Youth interested in participating were provided general information about the study and contacted to schedule a time and location to complete the interview. Youth provided informed consent prior to participating. Service providers interested in participating communicated with the research team to arrange a date for the interview and were sent an additional information email, which included a consent form, PowerPoint document and video describing the rationale and function of each app feature, a link to the JoyPop™ website and eBook, and multiple videos describing the app. Before each interview began, service providers had the option of viewing a 4-min JoyPop™ feature review video if they wanted a reminder about the design and features (58).
The first, second, and last author created the youth interview guide in consultation with partner mental health agencies and based on the TAM. The guide included questions such as “How did using the app affect your mental health, if at all?” and “What would make the app more useful to you?” The first, second, and last author developed the service provider interview guide in consultation with partner mental health agencies and based on the TAM and the facilitating conditions construct of the UTAUT. The service provider interview included questions such as “What is your impression of the usefulness of the app as an adjunct to usual services for youth in and/or seeking mental health care?” and “What factors in your organization would make integrating this app into usual care easier or more difficult?” The first author pilot tested the interview guides (see Supplementary Material S1) with two youth and two service providers. The first author reviewed the pilot interview audio recordings and transcripts with the last author and discussed participant experiences during these interviews with the second and last author. Interview guides were deemed adequate upon review and discussion because they were understood clearly, facilitated participant comfort to provide feedback in a nonbiased manner, and elicited appropriate information related to the TAM constructs of interest and study purpose. The first author conducted all interviews between December 2021 to March 2022. Interviews were audio recorded, anonymized, and transcribed verbatim. Interviews were conducted virtually (i.e., via Zoom), by phone, or in person based on participant preference. All participants received $10 CAD upon completing the interview.
Data analysis
The first, second, and last authors were the only research team members involved in data analysis. The first author checked transcripts for accuracy before data analysis, imported them into NVivo Software 12, and coded all the data. We used deductive-inductive content analysis to analyze youth and service provider data to provide a condensed and thorough description of app acceptance in the form of a conceptual map (59). This approach consists of three main phases: preparation, organizing, and reporting (59). We used constructs from the TAM and UTAUT to provide the organizing categories during data analysis.
In the preparation phase, we selected individual interviews as the unit of analysis. The first author also immersed himself in the data by transcribing each interview and reading through each transcript several times (59, 60). In the organizing phase, the first author coded the relevant data line-by-line into organizing categories (constructs) of the chosen frameworks. The first author inductively coded the data and grouped it into generic categories and sub-categories within the organizing categories. Additional organizing categories were inductively developed to create generic categories, and sub-categories to account for the data that did not fit into the constructs suggested by the TAM and UTAUT. Participants' terms were used to name the developed categories and sub-categories. Finally, in the reporting phase, two conceptual maps were developed to summarize the analysis of the youth and service provider data. The first author performed all the steps of the analysis, including coding, categorization, and mapping, and discussed all steps with the second and last authors who have extensive experience in qualitative research. We resolved discrepancies throughout the analysis process by consensus.
Data saturation was based on category saturation and the richness (quality) and thickness (quantity) of the data (61). Interview guides were structured to elicit information relevant to app acceptance and pre-determined theoretical categories to help achieve data saturation (61, 62), and as a result of the small sample size of the population (i.e., youth with significant experience using the app, service providers with sufficient knowledge of the app) and ethical concern of reducing participant burden, data saturation was assessed after four interviews from each group. The first author then discussed the commonalities, quality, and quantity of data emerging in each generic and sub-category nested within relevant pre-determined main categories with the second author. Because of the common data from youth and service providers, it was determined by consensus that two more interviews would be conducted to assess if saturation had been reached. After two more interviews, it was determined that category saturation was reached (i.e., interviews and analytic procedures provided no new material for analysis).1
Reflexivity and trustworthiness
We completed and used the Consolidated Criteria for Reporting Qualitative Research (COREQ; see Supplementary Table S2) as a guide to create a detailed and comprehensive report of the research study (63). To ensure trustworthiness throughout the data collection and analysis processes, we applied various strategies (i.e., credibility, dependability, confirmability, transferability) outlined by Elo (64) and Lincoln (65). We have highlighted specific strategies and methods used to establish trustworthiness throughout data collection and analysis in Table 1.
Table 1. Trustworthiness strategies used throughout data collection and analysis.
Results
Youth
Participant demographics
Seven youth, out of the 20 invited, agreed to participate in the study. One youth chose to stop participating after the first two interview questions due to discomfort engaging in a one-on-one interview. This resulted in a final sample of 6 female youth with a mean age of 14.60 (SD = 1.63; range 12–17 years). Three participants self-identified as White, and three self-identified as Indigenous. Five participants were in high school, and one was in elementary school. The 6 youth that participated were similar in age, gender, and ethnicity to those that did not complete an interview or choose to participate [N = 14, Mage = 15.0 (SD = 1.84), range 12–18, 85.7% female, 64.3% Indigenous, 35.7% White]. Interviews averaged 26 min in length (SD = 9.31; range 10–42 min) and occurred virtually over Zoom (n = 4), by phone (n = 1), and in-person (n = 1).
Youth acceptance of the JoyPop™ app
The conceptual map (with several generic categories and sub-categories under each organizing category) drawn from the deductive-inductive analysis is presented in Figure 2. We describe each organizing category and its associated generic categories and sub-categories below. Quote numbers (Q#) have been provided within the text to guide readers to specific quotes in Table 2 that illustrate participant experiences and the sub-categories.
Figure 2. Conceptual map summarizing youth acceptance of the JoyPop app using the technology acceptance model. The technology acceptance model includes the constructs of perceived usefulness, perceived ease of use, and attitudes towards use. Organizing categories (left), generic categories (middle), and sub-categories (right).
Table 2. Categories and illustrative quotes summarizing youth acceptance of the JoyPop™ app.
Perceived usefulness
Overall, all participants described helpful facilitators that promoted use of the JoyPop™ app and beneficial mental health outcomes following use, along with barriers preventing the perceived usefulness of the app.
Facilitators of use
All participants described how the accessibility of the app was essential in facilitating their continued use. Participants primarily described how the app was always available to provide quick, efficient, and timely support when needed, such as when they experienced mental health difficulties, had no one to talk to, or were bored (Q1). Some participants specifically discussed how the app could provide needed support between counselling sessions (Q2). Others talked about how the app simplified access to relevant crisis services via the Call for Help feature (Q3). Half the participants described how the app helped them develop a useful routine. For example, some participants began using coping skills regularly when experiencing stress or difficult moods; others talked about the app facilitating the development of a structured routine in their daily lives (Q4 + 5). Four participants discussed how the app was helpful because it provided a safe space to work on their mental health. The safe space was discussed in two ways: to have their own space and time for themselves or to work on their mental health privately without judgment (Q6 + 7).
Outcomes of use
Four participants discussed how using the app and specific features, such as the Journal, Rate My Mood, and Calendar features, helped them increase their self-awareness by improving their abilities to identify, understand, and reflect on their emotions and thoughts. For example, participants discussed how they would reflect on past journal entries using the Calendar feature to understand the antecedents of their feelings or how they would use the Rate My Mood feature to help them understand their emotions and implement coping skills to manage them (Q8 + 9). All participants described how they used the app to foster self-expression of their emotions and thoughts by using either the Art feature to visually express themselves or the Journal feature to vent and expressively write about their days (Q10 + 11). All participants discussed how the features of the app (particularly the Journal and Breathing) helped them increase relaxation and reduce anxiety and stress (Q12 + 13). Three participants talked about how the SleepEase feature helped improve their overall sleep. Sleep quality and onset were improved by integrating the sleep tips and using the sleep sounds (Q14 + 15). Three participants mentioned how the Art, Journaling, and SquareMoves features provided a helpful distraction in stressful situations or how they could foresee the features being used as a helpful distraction in situations (e.g., interpersonal, educational) that might result in stress and anxiety (Q16 + 17).
Barriers
All youth participants discussed additions and enhancements to improve the app's usefulness. Suggested additions included having more features (e.g., educational games and coping skills), adding helpful evidence-based daily tips in the form of notifications (e.g., reminders to exercise), and adding text options for crisis lines (Q18). Participants also recommended enhancements to the app that might make it more useful, such as adding more pencil sizes to the Art feature, increasing the number of emotions in the Rate My Mood feature, and expanding the accessibility of the app across mobile platforms (Q19).
Each participant described personal and contextual factors that reduced the utility of specific features because of their relevance to their current needs. These perceptions were not critiquing the usefulness of the features to others who may benefit but were descriptions of why a specific feature was not helpful for their specific mental health needs (Q20) or that they already had other tools that helped fulfill a feature's function (Q21).
Perceived ease of use
Youth participants described aspects of the app that either facilitated the ease of learning and/or using the app or barriers that made the app more difficult to learn and/or use.
Facilitators
Five participants commented on the app's overall design, layout and simplicity, and how it facilitated their adjustment to using the JoyPop™ app. Participants specifically discussed the ease of navigating the app and locating various features. They highlighted how the app's aesthetics (e.g., colour scheme), layout and simplicity made it convenient to use, that everything was organized in one area, and that nothing in the layout was ambiguous. Some participants also found the app easier to use than previous mHealth apps they had tried (Q22 + 23). Five participants mentioned that it was easy to learn skills and features within the app and implement them to their benefit, such as using and combining the Journal and Calendar features to reflect on and gain insight into past experiences (Q24). Others discussed how it was easy to use breathing exercises to manage experiences of stress and panic because the feature provided instructions and guidance (Q25).
Barriers
Three participants noted difficulties with functionality due to technical issues within features that made them challenging to use. These issues were related to an error when saving moods while using the Rate My Mood feature and design complications in the SquareMoves and Art features (e.g., no undo button; Q26 + 27). All participants discussed personal challenges associated with their mental health, psychomotor difficulties, and personal situations (e.g., time constraints) that made the app difficult to use (Q28 + 29).
Attitudes towards use
Positive feelings
Four participants expressed positive feelings about the design and ease of learning and using the JoyPop™ app. Many participants found the colours and design of the app aesthetically appealing (Q30 + 31). Some participants expressed positive feelings towards the ease of using specific features because of their built-in supports (e.g., journal prompts) and the simplicity of using them during times of stress (Q32). All participants communicated positive feelings toward the usefulness of features in improving their mental health. For example, some participants found the Circle of Trust feature helpful because it facilitated quick access to social support and/or was a helpful reminder of their social support network (Q33). Participants also discussed how much they enjoyed the SleepEase feature because it helped them resolve their sleep issues, the Journal feature because it helped them reflect on positive experiences and mitigate negative feelings, and the Rate My Mood feature because it increased their insight into their emotions (Q34). All participants expressed positive feelings about continued and future use of the app. Many talked about how they felt good about using the app during the study and would continue using it because of the mental health benefits they acquired. Some participants also discussed how they would recommend the app to friends and family members with mental health difficulties (Q35 + 36). Each participant expressed positive feelings about their overall experience with the JoyPop™ app (Q37). One participant compared their overall experience to previous efforts to improve their mental health (Q38).
Negative feelings
Although no subcategories emerged under this generic category and most feelings expressed towards the JoyPop™ app were positive, four participants expressed negative feelings primarily directed to the app's lack of variety and the SquareMoves feature because the design made it challenging to guide blocks down (Q39 + 40). Two participants expressed disappointment that the app was only available on iOS devices, preventing them from using it on their personal phones throughout the pilot study (Q41).
Supplementary analysis
To compare youth that participated and completed an interview in this qualitative study (N = 6) to invited youth that did not complete the interview (N = 14), we examined scores on the satisfaction and acceptability construct of the User Experience Questionnaire for Internet-based Interventions [UEQII; (71)], where higher scores indicate greater satisfaction and acceptability (maximum score = 32) that was included for all participants in the pilot study. An unequal independent samples t-test showed that scores for the six participants who completed the qualitative study (M = 25.5; SD = 2.83) did not significantly differ from the 14 that did not complete an interview (M = 25.0; SD = 0.66).
Service providers
Participant demographics
Seven female service providers (Mage = 43.75, SD = 8.93, range 32–60) participated in the study. All participants self-reported their ethnicity as White. Positions held included managers, counsellors, and program directors. Most participants served clients between the ages of 6–18, with four primarily providing services to Indigenous clients. Participants had an average of 16.25 years of experience working in mental health services (SD = 8.23). Three service providers were providing counselling for clients who participated in the pilot study, and one was providing counselling for a client who completed an interview. Interviews averaged 32 min in length (SD = 3.08, range 27–48 min), and occurred virtually over Zoom (n = 6) and by phone (n = 1).
Service provider acceptance of the JoyPop™ app
The conceptual map (with several generic categories and sub-categories under each organizing category) drawn from the deductive-inductive analysis is presented in Figure 3. We describe each organizing category and its associated generic categories and sub-categories below. See quote numbers within the text which specify associated quotes in Table 3.
Figure 3. Conceptual map summarizing service providers’ acceptance of the JoyPop app using an extended model of the technology acceptance model. The extended model of the technology acceptance model includes constructs of perceived usefulness, perceived ease of use, attitudes towards use, and facilitating conditions. Organizing categories (left), generic categories (middle), and sub-categories (right).
Table 3. Categories and illustrative quotes summarizing service providers acceptance of the JoyPop™ app.
Perceived usefulness
Perceptions of the app's usefulness in improving their clients' mental health and/or enhancing current services were predominantly related to benefits for clients, benefits to services, and barriers preventing the app's usefulness for clients/or services.
Benefits to clients
All participants described the general benefits the JoyPop™ app could provide to clients. For example, service providers discussed how the app could benefit youth and underserved populations by increasing access to mental health supports, coping skills, and social support (e.g., Circle of Trust feature; Q1). Three participants additionally discussed how the app could help clients develop healthy habits and autonomy outside of services (Q2 + 3). All participants described specific benefits that the app could provide clients, such as improved coping skills and self-regulation, increased relaxation, improved self-awareness and self-monitoring, and being a helpful distraction when experiencing stress (Q4). Consequently, due to high rates of trauma-related symptoms among the clients they serve, many saw the app as being a beneficial adjunct to treatment for clients to cope with trauma-related difficulties (e.g., avoidance, difficulties with sleep and concentration, negative affect, hyperarousal) (Q5 + 6).
Benefits to services
All participants described how the app could benefit pre-treatment services, such as supporting clients on the waitlist and decreasing waitlists by preventing a client's need for intensive services or supporting low-risk youth who may not need intensive services (Q7). Many participants also described how the app could help clients build comfort before seeking more intensive services, which was considered necessary because of the high rates of trauma among their clients and the potential dangers of delaying treatments (Q8). All participants described how the app could facilitate and improve outcomes for clients and clinicians during active treatment. Many participants discussed how the app could help clients between sessions by facilitating homework and skill reinforcement (Q9). Participants further described how the app could bolster in-person sessions by providing clinicians with more tools to meet clients' diverse needs and by improving clients' ability to track their moods and experiences, which could provide the clinician with more information to help support them (Q10 + 11). Lastly, participants discussed how the app might reduce time spent in counselling and the need for acute care services (e.g., hospitalizations). Five participants discussed how the JoyPop™ app could benefit post-treatment services as a discharge recommendation to ease a client's transition out of services and support youth with evidence-based coping skills after counselling (Q12 + 13).
Barriers
Every participant discussed useful additions that would improve the app for youth and their services, such as increasing the variety of coping skills and activities, adding notifications and rewards, and integrating different language options (Q14 + 15). Three participants talked about difficulties relating to youth engagement based on their experiences trying to engage youth to try the app throughout the pilot study (Q16). Others discussed the general problems in having youth clients follow treatment recommendations and how this would likely generalize towards the JoyPop™ app if it became a treatment option (Q17). Although these participants highlighted the app's benefits, engaging youth to try the app was mentioned as a significant barrier. Almost all participants commented on the need to ensure the appropriate use of the app by clients and service providers to supplement usual services to minimize risk and harm. These participants were cautious about the potential risks of using the app to replace treatment or with clients relying on an app and forgoing other interventions (Q18).
Perceived ease of use
Participants highlighted aspects of the app's design, functions, and features that could either improve or create barriers to the ease of using the app for clients and services.
Facilitators
All participants described the ease for clients in their services to learn and use the app based on their initial perceptions of its simple design and layout and because of clients' familiarity and comfort with using technology (Q19 + 20). Six participants discussed the ease for service providers to use the app and integrate it into regular services. Overall, participants did not foresee any serious resistance toward integrating the app. Reasons were associated with the ease with which they integrated mHealth apps in the past (Q21). Others referred to the simplicity of setting up clients with the app for the pilot study (Q22).
Barriers
Five participants commented on barriers for clients that would make learning and using the app more challenging. Because of significant socioeconomic barriers, especially for Indigenous clients, most participants talked about how the cost and lack of appropriate devices could make accessing the app challenging for some clients (Q23). Six participants described barriers to integration that may reduce acceptance of the app into usual services. Perceptions centred around workload concerns and service provider feelings (e.g., distrust of apps in general) and characteristics (e.g., personality) towards integrating a mHealth app into services (Q24 + 25).
Attitudes towards use
Positive feelings
Consistent with their perceptions of the JoyPop™ app's usefulness, each service provider expressed positive feelings toward various client benefits, including improved accessibility, confidential support, and enhanced coping skills in various life domains (Q26 + 27). Two participants highlighted the positive feelings that their client, who participated in the pilot study, had while using the app during active treatment (Q28). All participants expressed positive feelings towards the JoyPop™ app's integration into usual services, often because of its usefulness to improve their services at different stages of treatment and their overall impressions of the app as a tool to help clients (Q29 + 30).
Concerns
Six participants highlighted concerns for youth and clients using a mHealth app. Participants were concerned that the JoyPop™ app might confuse clients who do not speak the same language or have lower literacy levels (Q31). Others had concerns over the adverse effects (e.g., avoiding more practical skills, isolation) of youth spending more time on their phones (Q32). Five participants had concerns with integration of the app into usual services. Participants expressed concerns about potential difficulties engaging clients in long-term app use to harness its effectiveness (Q33). Some participants also had concerns about protecting client privacy and the ownership of the information gathered from the app (Q34).
Facilitating conditions
Service providers identified various organizational and technological factors that would influence the overall support and integration of the JoyPop™ app into services. Factors included the organizations' accountability in providing quality, ethical, and equitable services, whether the app is consistent with their organization's current values and needs, and their responsibilities in engaging service providers to support the continued and appropriate use of the app.
Accountability
All participants described the importance and difficulties for the organization in ensuring clients were provided equitable access to the JoyPop™ app via devices and funding, especially for clients that may face increased inequities and lack the money or devices to use the app (Q35 + 36). However, participants were optimistic about adapting and finding ways that the organization could support clients in getting access to devices and the app. For example, two participants discussed working with schools and developers (e.g., creating a joint purchase plan) to increase client access; some believed their organization could cover client costs (Q37). Four participants highlighted the importance of establishing evidence and safety behind the app for their organizations to ensure clients were provided with a low-risk and effective intervention. A clear picture of the safety and evidence associated with the JoyPop™ app was essential for the organization to support its integration into usual services (Q38 + 39).
Compatibility
Participants described how the app aligned with the values and needs of their organization because of the clear connection between their organization's continuous quality improvement, strength-based, and client-centred approaches and the JoyPop™ app (Q40 + 41). Four participants (working with primarily Indigenous clients) described barriers to compatibility. They suggested changes to the app that could increase its compatibility with the values and needs of their Indigenous-health-specific organization. The main barriers to compatibility were the lack of cultural adaptations within the app, which did not align with the organization's fundamental values (Q42).
Engaging service providers
Four participants described the importance of ensuring reminders were provided to service providers about utilizing the JoyPop™ app. Reminders were discussed in the form of team meetings, sharing opportunities within organizational departments, and encouragement to staff from managers (Q43). Two participants talked about email reminders they received from their organization about offering clients the opportunity to participate in the pilot study, which increased their engagement with the JoyPop™ app (Q44). Six participants described the importance of having training and technical support with the JoyPop™ app. The training included educating and ensuring new staff members are competent in using the JoyPop™ app with clients (Q45 + 46). Participants also discussed having one member in their organization with experience and training that could provide technical support for others if needed (Q47). Four participants described the significance of having the organization explain the value and rationale of integrating the JoyPop™ app into services, such as a member within the organization (e.g., manager) describing the benefits of the JoyPop™ app in order to engage service providers (Q48 + 49).
Discussion
National Canadian eHealth guidelines recommend a multi-method approach to develop and evaluate mHealth interventions, including gleaning the perspectives of service providers and users to assess acceptance (5, 20). These perspectives are critical in tailoring apps to the local needs of those engaging in or working within mental health systems and supporting long-term acceptance and uptake of mHealth apps into models of care (20, 24). Therefore, this study aimed to evaluate the acceptance of the JoyPop™ app among a clinical sample of youth and service providers using the TAM, at the two largest mental health agencies in Northwestern Ontario.
Acceptance of the JoyPop™ app
Overall, the results support the acceptance of the JoyPop™ app among a clinical sample of youth and mental health service providers. Because important feedback/recommendations were identified using theoretical frameworks of technology acceptance, results will help developers ensure that future changes to the app are consistent with local needs and established constructs proven to increase acceptance and continued use (17, 19). Recommendations can also be leveraged to increase the overall utility of the app in future studies with youth and service providers.
TAM constructs
The perceived usefulness of the JoyPop™ app to improve mental health and well-being was highlighted by all youth and service providers. Most participants described how the app was useful because it facilitates accessibility to coping skills and mental health support for individuals who face barriers (e.g., geographical distance, long wait times) when accessing services. Each group of participants also reported that the app improved, or could improve, mental health via improved sleep hygiene, self-awareness, self-expression, and emotion regulation. The usefulness of the JoyPop™ app as a novel tool for mental health services in Northwestern Ontario was also evident. All service providers perceived the JoyPop™ app as a valuable addition for their diverse clientele and services because of its potential utility at various stages of treatment (i.e., pre/active/post-treatment). These results are promising because one of the primary purposes of the app is to provide a transdiagnostic, easily accessible intervention for youth and mental health agencies that targets general well-being and coping via improved emotion regulation. Importantly, these results are consistent with research suggesting that the usefulness and benefit (e.g., added value) of using mHealth technology are among the most important factors influencing youth and service providers' acceptance of mHealth apps (17, 19, 22). Although not all youth used every feature available in the JoyPop™ app, each youth identified specific features relevant to their needs. These results support the usefulness of the JoyPop™ app because it provides a range of features that can help meet the various needs of youth and is consistent with research highlighting the importance variety plays in the long-term use and acceptance of mHealth apps (21, 22).
Both youth and service providers perceived the JoyPop™ app as easy to use and/or integrate because of its simple design, aesthetic appearance, and practical functionality. These results are consistent with the broader literature demonstrating increased acceptance and long-term uptake of mHealth apps with simple designs, straightforward functions, and a pleasant appearance (17, 19, 22). Attitudes expressed by youth and service providers also support the acceptance of the JoyPop™ app. Both youth and service providers converged in their positive feelings and attitudes about the usefulness and ease of using the JoyPop™ app. These findings are promising because expressed positive attitudes toward mHealth apps are among the strongest predictors of technology acceptance and long-term engagement (24, 26). These results also align with research suggesting that individuals are more likely to accept and express positive feelings toward efficient and helpful apps with an intuitive, straightforward design (19, 22). One interpretation of these findings is that the positive attitudes expressed by youth and service providers highlight fundamental aspects, designs, and functions within the JoyPop™ app that are essential to its overall acceptance.
Youth and service providers converged on their recommendations to improve the JoyPop™ app. Both discussed the need for specific additions and enhancements of current features that could increase the usefulness and ease of the app for usual services and youth mental health. Additions included ensuring cross-platform compatibility to increase accessibility for clients who do not have iOS devices, adding a built-in notification system, a tracking system (e.g., tracking mood over time), and a larger variety of games to increase usage and engagement. Further, adding a more diverse array of coping skills (e.g., mindfulness meditations) was recommended to support youth's diverse needs better. Enhancements to features were suggested, such as adding more pencil sizes to the Art feature and grid lines to the SquareMoves feature, along with increasing the number of moods in the Rate My Mood feature to increase feature utility (e.g., increased mood monitoring), ease of use, and long-term engagement. Incorporating user feedback is needed because the design, variety, and functionality play a critical role in the long-term acceptance and continued use of mHealth apps (21, 22). Research demonstrates that app users are unlikely to continue using mHealth apps that do not engage them immediately (72). Failing to incorporate app feedback suggested and validated by primary users in the development and evaluation of mHealth apps can also result in developers missing vital technical improvements and set back evaluative research and the overall clinical effectiveness of an app (17, 72). Developers must consider, implement, and evaluate these recommendations to increase acceptance among diverse samples of youth and service providers while providing a more user-friendly and user-centred mHealth app tailored to local needs (5, 20, 72). Developers of the JoyPop™ app have and will continue to consider the perspectives and recommendations from users at each stage of its development and implementation. This process will ensure the app is user-focused, engaging, and effective over long periods.
Facilitating conditions
Essential to the comprehensive evaluation in the present study was the addition of the facilitating conditions construct from the UTAUT to capture important organizational factors influencing mHealth acceptance among service providers. All participants found the app compatible with their organization's current values and needs because of its strength-based and client-centred approach. These findings support the acceptance of the JoyPop™ app in broader healthcare contexts because the compatibility of an app with an organization's values and needs is often associated with increased acceptance among service providers (18, 19).
Relevant needs were identified within each organization that could increase or decrease future acceptance of the JoyPop™ app into regular services. For example, because some clients may be unable to afford appropriate devices to use the app, ensuring that the organization could provide clients equal access to the app was a key factor influencing acceptance. These findings are consistent with research showing that lack of proper funding, costs, and reimbursement issues can hinder service providers' acceptance of mHealth apps (19, 73). It will be vital for developers and researchers of the JoyPop™ app to continue working with service providers and their accompanying organizations to find new ways to improve equitable access. For instance, studies have shown that payment models and integrating mHealth tools into healthcare plans can increase acceptance among service providers (5, 19, 20).
The need for a clinical evidence base and adequate training and education to engage service providers were also identified as important organizational needs influencing the acceptance of the JoyPop™ app. These needs are consistent with research highlighting the important influence clinical effectiveness and appropriate training and education have on service providers' acceptance of mHealth apps (18, 19). The results suggest that continuing to evaluate the JoyPop™ app's clinical effectiveness is important to establish its evidence base and promote acceptance, trust, and quality interventions among service providers (18, 19). Also, continuous support from developers in terms of guidance and the provision of education to service providers is an important first step to maximize acceptance and continued use if the app is integrated into usual care (18, 19). Implementation science will be critical in achieving optimal use, acceptance and long-term adoption of the app among mental health organizations. For example, identifying “champions” (dedicated individuals supportive of the app) within organizations may bolster the promotion, proper use, and eventual acceptance of the JoyPop™ app in complex healthcare settings (74).
The JoyPop™ app and coping with adversity and trauma
All youth, including the three Indigenous youth, expressed how the app was useful in improving mental health and emotion regulation. One participant specifically noted that using the app helped address emotion regulation difficulties associated with their past trauma (e.g., they used Rate My Mood to document heightened emotional arousal). These findings are important given that many youth presenting for mental health services in Northwestern Ontario do so with difficulties related to emotion regulation. This is also relevant as Indigenous youth within this region experience particular difficulties with adjusting to past trauma (49). It is well established that neurocognitive and neurobiological changes occur throughout development in response to childhood trauma and adversity (51, 52). These changes influence the functioning of neurobiological systems associated with emotional arousal and emotion regulation and leave children with response propensities that affect how they respond to stress (75, 76). Developing certain response styles (e.g., hypervigilance, avoidance) related to early trauma and adversity may provide some protective benefits within the original context (e.g., recognizing and attempting to avoid contact with the threat), however, they may be a liability in other context in which they are not required (e.g., at school) and influence future resilience (75, 77).
In addition, many service providers in the present study were willing and eager to integrate the JoyPop™ app into usual care because of its potential utility in improving trauma-related symptoms at various stages of treatment. These perspectives align with research demonstrating the utility of mHealth apps in treating trauma, especially when used as an adjunct treatment (10, 14). The results of the present study (e.g., self-reported improved emotion regulation) and evidence that the JoyPop™ app confers the greatest benefits on emotion regulation for those who have experienced more childhood adversity (29), suggest that the app may be able to bolster an individual's capacity to manage the impact of trauma by facilitating resilience through adaptive coping. The results are consistent with research showing that mHealth apps can be a helpful standalone intervention for young people (11) and that resilience can be harnessed and increased through interventions that target emotion regulation (78), training programmes [e.g., mixed interventions, mindfulness; (79)], and strength-based primary prevention efforts (80). Future research specifically testing these speculations is required.
Limitations and future directions
There are some significant limitations concerning the results of this study. First is that all participants were female. Research suggests there are sex and gender differences in the acceptance of new technology for youth and service providers (81, 82); therefore, future research with other gender identities (including boys and men) is needed. A second limitation is that out of 20 youth who were invited to participate in the qualitative study, only six completed an interview and 1 youth (representing 14% of the initial sample) left the study because of discomfort. It is also likely that many service providers were aware of their opportunity to participate in the study but chose not to. This may result in a non-response sample bias, influencing the results and reducing the representativeness of the sample and generalizability of the results (83). For example, participants in the pilot study who felt more positively about the app and/or interacted with it more often may have been more likely to participate in this qualitative study. Further, important perspectives of youth who may be less comfortable with discussing their experiences during an interview may have been missed. However, upon comparison, youth who completed the qualitative study and youth that were invited but did not complete it were similar in age, sex, and ethnicity. Youth ratings of satisfaction and acceptance were also similar. These results provide some reassurance that the sample of youth who participated in the qualitative study were similar to those that participated in the pilot study but chose not to participate in the qualitative study.
Another limitation is that only those that completed the pilot study were eligible to participate in this qualitative study. Also, no demographic data were available for service providers who did not participate in the study. Assessing acceptance among youth who did not complete the pilot study and service providers who did not participate may have resulted in more negative feedback. Negative feedback is essential for informing continued adaptation and tailoring of the app to meet the diverse needs of youth and healthcare organizations (17, 19). Research has shown that there are certain factors (e.g., socioeconomic status, age, mental health diagnoses, personality traits, culture) that influence youth and service providers' perspectives on mHealth app acceptance either negatively or positively (17, 19, 84). Future research should gather more information about these factors to better understand youth and service providers' acceptance of the JoyPop™ app. Usage of the app can also influence acceptance of mHealth apps and unfortunately we were unable to obtain and make use of objective usage data from the app (i.e., details about how often and how long each feature was used). Gathering and examining data on overall app usage including patterns and duration of feature usage would provide important insight into the characteristics of youth who use each feature and acceptance of each feature to better serve the diverse needs of youth. Future research should also consider strategies to explicitly seek out more critical feedback (e.g., contact youth who dropped out of pilot study early, as they may be more likely to hold more negative views). It may also be beneficial to seek out a diverse group of participants to watch a video or test out the app for a shorter period of time (e.g., a few days) and then ask them about their experiences.
A third limitation is related to the broad age range of the youth sample (12–17 years). Adolescents is marked by significant changes in biological (e.g., puberty), psychological (e.g., emotion regulation), behavioural (e.g., risk behaviours) and social development [e.g., changes in school environment; (85)] and age has been shown to moderate app acceptance (17, 84). Examining larger samples of youth from early adolescents to later adolescence may reveal specific positive and negative feedback on the acceptance of the JoyPop™ app. For example, youth in the upper ranges (17–18) may be more likely to express themselves verbally (e.g., using the Journal feature) vs. youth in the lower ranges (12–13), who may be more visual (e.g., using the Art feature). However, it is important to note that the JoyPop™ app purposefully has various features to capture these potential differences and meet the needs of all ages of youth. However, future research would still benefit from understanding app acceptance among older and young youth.
A final limitation is that we only had one coder which could impact the interpretations derived from the data (86). Though having more than one coder to examine the inter-coder reliability can help validate the data analysis, more important is having multiple researchers evaluate the content of codes to highlight disagreements and provide insight to refine the coding process (86, 87). In addition, having detailed notes of all decisions made throughout the coding process and the organization of the data into categories is essential to facilitate the consistency and trustworthiness of coding procedures if only one coder is possible (87). We reduced this limitation by having the first, second, and last authors meet throughout each step of data analysis to highlight disagreements and reach a consensus. The first author also kept detailed memos and notes of all their decisions in the form of an audit trail.
Conclusion
This qualitative study guided by the TAM provides insight into the factors that influence acceptance of the JoyPop™ app among youth and service providers in Northwestern Ontario. All youth and service providers found the app useful and easy to use and expressed positive attitudes toward using the app, along with recommendations to improve acceptance in future iterations. Service providers also acknowledged and highlighted important organizational factors that may influence acceptance of the JoyPop™ app into usual care. The present research contributes to a growing body of evidence supporting the JoyPop™ app as an accessible, helpful, and timely intervention to reduce barriers to mental health services and support diverse youth in Canada, especially in underserved areas like Northwestern Ontario. The recommendations and suggestions derived from the results are essential in ensuring that future iterations of the JoyPop™ app meet the needs of youth and service providers.
Data availability statement
The original contributions presented in the study are included in the article/Supplementary Materials, further inquiries can be directed to the corresponding author.
Ethics statement
The studies involving humans were approved by Lakehead University Research Ethics Board. The studies were conducted in accordance with the local legislation and institutional requirements. Written informed consent for participation was not required from the participants or the participants’ legal guardians/next of kin because consistent with the protocol reviewed and approved through our institution and community partner research ethics boards, youth participants were deemed capable of providing consent independently. For youth under 16, the research coordinator encouraged youth of the desirability of informing their parent(s)/caregiver(s) about their involvement in the research, however parental/caregiver involvement/consent was not required. This approach is consistent with the The Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS2), with the practices in place at each partner agency, and with youth 12 and up being able to consent to counselling services independently via the Child, Youth and Family Services Act (2017; https://www.ontario.ca/Laws/statute/17c14).
Author contributions
IM led the data collection, analyses, and writing. AP and AM provided support and supervision throughout the process. AM conceptualized the initial study in consultation with all co-authors. AM, ET, FS, JO, CS, KS, and TB collaborated in developing funding applications to support the project. e.g., and JC supported data collection. All authors contributed to the article and approved the submitted version.
Acknowledgments
Funding to support this study was provided through the Thunder Bay Community Foundation—General Fund (principal investigator: AM) and the Canadian Institutes of Health Research (subgrant agreement between Lakehead University and McMaster University; TE3-138302). The research team is grateful for the collaboration with community partners Children's Centre Thunder Bay and Dilico Anishinabek Family Care.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fdgth.2023.1197362/full#supplementary-material
Footnote
1A supplementary analysis was conducted to examine app acceptance on a quantitative measure used in the pilot study in an effort to compare youth that completed an interview in the qualitative study vs. those that chose not to participate or did not complete the interview in the qualitative study. This comparison is mentioned at the end of the results section.
References
1. MHASEF Research Team. The mental health of children and youth in Ontario: a baseline scorecard. Toronto, ON: Institute for Clinical Evaluative Sciences (2015).
2. Mental Health Commission of Canada. Changing directions, changing lives: the mental health strategy for Canada. Calgary, AB: MHCC (2012).
3. Children’s Mental Health Ontario. Kids can’t wait: 2020 report on wait lists and wait times for child and youth mental health care in Ontario (2020). Available at: https://cmho.org/wp-content/uploads/CMHO-Report-WaitTimes-2020.pdf (Accessed March 15, 2022).
4. Brown SA, Parker JD, Godding PR. Administrative, clinical, and ethical issues surrounding the use of waiting lists in the delivery of mental health services. J Behav Health Serv Res. (2002) 29(2):217–28. doi: 10.1007/BF02287708
5. Mental Health Commission of Canada. E-Mental Health in Canada: Transforming the Mental Health System Using Technology - a briefing document. Ottawa, ON: MHCC (2014).
6. Toombs E, Kowatch KR, Dalicandro L, McConkey S, Hopkins C, Mushquash CJ. A systematic review of electronic mental health interventions for indigenous youth: results and recommendations. J Telemed Telecare. (2021) 27(9):539–52. doi: 10.1177/1357633X19899231
7. Fletcher R, Kay-Lambkin F, May C, Oldmeadow C, Attia J, Leigh L. Supporting men through their transition to fatherhood with messages delivered to their smartphones: a feasibility study of SMS4dads. BMC Public Health. (2017) 17(1):953. doi: 10.1186/s12889-017-4978-0
8. Olff M. Mobile mental health: a challenging research agenda. EJPT. (2015) 6(1):27882. doi: 10.3402/EJPT.V6.27882
9. Dingwall KM, Puszka S, Sweet M, Nagel T. “Like drawing into sand”: acceptability, feasibility, and appropriateness of a new e-mental health resource for service providers working with aboriginal and Torres Strait Islander people. Aust Psychol. (2015) 50(1):60–9. doi: 10.1111/ap.12100
10. Hallenbeck HW, Jaworski BK, Wielgosz J, Kuhn E, Ramsey KM, Taylor K, et al. PTSD Coach version 3.1: a closer look at the teach, use, and potential impact of this updated mobile health app in the general public. JMIR Ment Health. (2022) 9(3):e34744. doi: 10.2196/34744
11. Leech T, Dorstyn D, Taylor A, Li W. Mental health apps for adolescents and young adults: a systematic review of randomised controlled trials. Child Youth Serv Rev. (2021) 127:106073. doi: 10.1016/j.childyouth.2021.106073
12. Donker T, Petrie K, Proudfoot J, Clarke J, Birch MR, Christensen H. Smartphones for smarter delivery of mental health programs: a systematic review. JMIR. (2013) 15(11):e247. doi: 10.2196/jmir.2791
13. Marshall JM, Dunstan DA, Bartik W. The digital psychiatrist: in search of evidence-based apps for anxiety and depression. Front Psychiatry. (2019) 10:831. doi: 10.3389/fpsyt.2019.00831
14. Lecomte T, Potvin S, Corbière M, Guay S, Samson C, Cloutier B, et al. Mobile apps for mental health issues: meta-review of meta-analyses. JMU. (2020) 8(5):e17458. doi: 10.2196/17458
15. Lui JH, Marcus DK, Barry CT. Evidence-based apps? A review of mental health mobile applications in a psychotherapy context. Prof Psychol Res Pr. (2017) 48(3):199. doi: 10.1037/pro0000122
16. Peng W, Kanthawala S, Yuan S, Hussain SA. A qualitative study of user perceptions of mobile health apps. BMC Public Health. (2016) 16(1):1. doi: 10.1186/s12889-016-3808-0
17. Jacob C, Sezgin E, Sanchez-Vazquez A, Ivory C. Sociotechnical factors affecting patients’ adoption of mobile health tools: systematic literature review and narrative synthesis. JMU. (2022) 10(5):e36284. doi: 10.2196/36284
18. Jacob C, Sanchez-Vazquez A, Ivory C. Understanding clinicians’ adoption of mobile health tools: a qualitative review of the most used frameworks. JMU. (2020) 8(7):e18072. doi: 10.2196/18072
19. Jacob C, Sanchez-Vazquez A, Ivory C. Social, organizational, and technological factors impacting clinicians’ adoption of mobile health tools: systematic literature review. JMU. (2020) 8(2):e15935. doi: 10.2196/15935
20. Zelmer J, van Hoof K, Notarianni M, van Mierlo T, Schellenberg M, Tannenbaum C. An assessment framework for e-mental health apps in Canada: results of a modified Delphi process. JMU. (2018) 6(7):e10016. doi: 10.2196/10016
21. Wei Y, Zheng P, Deng H, Wang X, Li X, Fu H. Design features for improving mobile health intervention user engagement: systematic review and thematic analysis. JMIR. (2020) 22(12):e21687. doi: 10.2196/21687
22. Widnall E, Grant CE, Wang T, Cross L, Velupillai S, Roberts A, et al. User perspectives of mood-monitoring apps available to young people: qualitative content analysis. JMU. (2020) 8(10):e18140. doi: 10.2196/18140
23. Kissi J, Dai B, Dogbe CS, Banahene J, Ernest O. Predictive factors of physicians’ satisfaction with telemedicine services acceptance. J. Health Inform. (2020) 26(3):1866–80. doi: 10.1177/1460458219892162
24. Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. (1989) 1:319–40. doi: 10.2307/249008
25. Davis FD. User acceptance of information technology: system characteristics, user perceptions and behavioral impacts. Int J Man-Mach Stud. (1993) 38(3):475–87. doi: 10.1006/imms.1993.1022
26. Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: Toward a unified view. MIS Q. (2003) 27(3):425–78. doi: 10.2307/30036540
27. Holtz BE, McCarroll AM, Mitchell KM. Perceptions and attitudes toward a mobile phone app for mental health for college students: qualitative focus group study. JMIR Form Re. Res. (2020) 4(8):e18347. doi: 10.2196/18347
28. Wekerle C. JoyPop™: A resilience intervention for youth mental health (2021). https://youthresilience.net/joypop-app (Accessed April 22, 2023).
29. MacIsaac A, Mushquash AR, Mohammed S, Grassia E, Smith S, Wekerle C. Adverse childhood experiences and building resilience with the JoyPop app: evaluation study. JMU. (2021) 9(1):e25087. doi: 10.2196/25087
30. Mushquash AR, Pearson ES, Waddington K, MacIsaac A, Mohammed S, Grassia E, et al. User perspectives on a resilience-building app (JoyPop): qualitative study. JMU. (2021) 9(7):e28677. doi: 10.2196/28677
31. Zimmerman MA, Brenner AB. Resilience in adolescence: overcoming neighborhood disadvantage. In: Reich J, Zautra AJ, Hall JS, editors. Handbook of adult resilience. New York: Guilford Press (2010). p. 283–308.
32. Zimmerman MA, Stoddard SA, Eisman AB, Caldwell CH, Aiyer SM, Miller A. Adolescent resilience: promotive factors that inform prevention. Child Dev Perspect. (2013) 7(4):215–20. doi: 10.1111/cdep.12042
33. Fritz J, De Graaff AM, Caisley H, Van Harmelen AL, Wilkinson PO. A systematic review of amenable resilience factors that moderate and/or mediate the relationship between childhood adversity and mental health in young people. Front Psychiatry. (2018) 9:230. doi: 10.3389/fpsyt.2018.00230
34. Tabibnia G. An affective neuroscience model of boosting resilience in adults. Neurosci Biobehav Rev. (2020) 115:321–50. doi: 10.1016/j.neubiorev.2020.05.005
35. Hilt LM, Hanson JL, Pollak SD. Emotion dysregulation. In: Brown BB, Prinstein MJ, editors. Encyclopedia of adolescence 3. San Diego: Academic Press (2011). p. 160–9
36. Pascual-Leone A, Yeryomenko N, Morrison O, Arnold R, Kramer U. Does feeling bad, lead to feeling good? Arousal patterns during expressive writing. Rev Gen Psychol. (2016) 20(3):336–47. doi: 10.1037/gpr0000083
37. King L. The health benefits of writing about life goals. Pers Soc Psychol Bull. (2001) 27(7):798–807. doi: 10.1177/0146167201277
38. Arch JJ, Craske MG. Mechanisms of mindfulness: emotion regulation following a focused breathing induction. Behav Res Ther. (2006) 12:1849–58. doi: 10.1016/j.brat.2005.12.007
39. Rankin K, Walsh LC, Sweeny K. A better distraction: exploring the benefits of flow during uncertain waiting periods. Emotion. (2019) 19(5):818–28. doi: 10.1037/emo0000479
40. Andrade J. What does doodling do? Appl Cognit Psychol. (2010) 24(1):100–6. doi: 10.1002/acp.1561
41. Schutte-Rodin S, Broch L, Buysse D, Dorsey C, Sateia M. Clinical guideline for the evaluation and management of chronic insomnia in adults. J Clin Sleep Med. (2008) 4(5):487–504. doi: 10.5664/jcsm.27286
42. Ungar M. Resilience after maltreatment: the importance of social services as facilitators of positive adaptation. Child Abuse Negl. (2013) 37(2–3):110–5. doi: 10.1016/j.chiabu.2012.08.004
43. Barlow DH, Ellard KK, Fairholme CP. Unified protocol for transdiagnostic treatment of emotional disorders: workbook. New York: Oxford University Press (2011).
44. Jadhakhan F, Blake H, Hett D, Marwaha S. Efficacy of digital technologies aimed at enhancing emotion regulation skills: literature review. Front Psychiatry. (2022) 13:809332. doi: 10.3389/fpsyt.2022.809332
45. Bakker D, Kazantzis N, Rickwood D, Rickard N. Mental health smartphone apps: review and evidence-based recommendations for future developments. JMIR Ment Health. (2016) 3(1):e7. doi: 10.2196/mental.4984
46. Park T, Thompson K, Wekerle C, Al-Hamdani M, Smith S, Hudson A, et al. Posttraumatic stress symptoms and coping motives mediate the association between childhood maltreatment and alcohol problems. JTS. (2019) 32(6):918–26. doi: 10.1002/jts.22467
47. Sanders J, Munford R, Thimasarn-Anwar T, Liebenberg L, Ungar M. The role of positive youth development practices in building resilience and enhancing wellbeing for at-risk youth. Child Abuse Negl. (2015) 42:40–53. doi: 10.1016/j.chiabu.2015.02.006
48. Kowatch KR. Analysis of the child and adolescent needs and strengths assessment in a first nation population [Dissertation]. Thunder Bay (ON): Lakehead University.
50. Radford A, Toombs E, Zugic K, Boles K, Lund J, Mushquash CJ. Examining adverse childhood experiences (ACEs) within indigenous populations: a systematic review. J. Child Adolesc Trauma. (2022) 15(2):401–21. doi: 10.1007/s40653-021-00393-7
51. Loman MM, Gunnar MR. Early experience and the development of stress reactivity and regulation in children. Neurosci Biobehav Rev. (2010) 34(6):867–76. doi: 10.1016/j.neubiorev.2009.05.007
52. Lupien SJ, McEwen BS, Gunnar MR, Heim C. Effects of stress throughout the lifespan on the brain, behaviour and cognition. Nat Rev Neurosci. (2009) 10(6):434–45. doi: 10.1038/nrn2639
53. McLaughlin KA, Lambert HK. Child trauma exposure and psychopathology: mechanisms of risk and resilience. Curr Opin Psychol. (2017) 14:29–34. doi: 10.1016/j.copsyc.2016.10.004
54. Thompson NJ, Fiorillo D, Rothbaum BO, Ressler KJ, Michopoulos V. Coping strategies as mediators in relation to resilience and posttraumatic stress disorder. JAD. (2018) 225:153–9. doi: 10.1016/j.jad.2017.08.049
55. Nadal C, Sas C, Doherty G. Technology acceptance in mobile health: scoping review of definitions, models, and measurement. JMIR. (2020) 22(7):e17256. doi: 10.2196/17256
56. Sandelowski M. Whatever happened to qualitative description? Res Nurs Health. (2000) 23(4):334–40. doi: 10.1002/1098-240X(200008)23:4-334::AID-NUR9%3.0.CO;2-G
57. Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm Policy Ment Health. (2015) 42(5):533–44. doi: 10.1007/s10488-013-0528-y
58. JoyPop App. JoyPop app scroll through video (2021). Available at: https://www.youtube.com/watch?v=CnDt-_4Mr2k&ab_channel=JoyPopApp (Accessed April 1, 2023).
59. Elo S, Kyngäs H. The qualitative content analysis process. J Adv Nurs. (2008) 62(1):107–15. doi: 10.1111/j.1365-2648.2007.04569.x
60. Burnard P. A method of analysing interview transcripts in qualitative research. Nurse Educ Today. (1991) 11(6):461–6. doi: 10.1016/0260-6917(91)90009-Y
61. Fusch P, Ness L. Are we there yet? Data saturation in qualitative research. Qual. Rep. (2015) 20(9):1408–16. doi: 10.46743/2160-3715/2015.2281
62. Francis JJ, Johnston M, Robertson C, Glidewell L, Entwistle V, Eccles MP, et al. What is an adequate sample size? Operationalising data saturation for theory-based interview studies. Psychol Health. (2010) 25(10):1229–45. doi: 10.1080/08870440903194015
63. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. (2007) 19(6):349–57. doi: 10.1093/intqhc/mzm042
64. Elo S, Kääriäinen M, Kanste O, Pölkki T, Utriainen K, Kyngäs H. Qualitative content analysis: a focus on trustworthiness. SAGE Open. (2014) 4(1):2158244014522633. doi: 10.1177/2158244014522633
66. Koch T. Establishing rigour in qualitative research: the decision trail. J Adv Nurs. (1994) 19(5):976–86. doi: 10.1111/j.1365-2648.1994.tb01177.x
67. Kornbluh M. Combatting challenges to establishing trustworthiness in qualitative research. Qual Res Psychol. (2015) 12(4):397–414. doi: 10.1080/14780887.2015.1021941
68. Thomas E, Magilvy JK. Qualitative rigor or research validity in qualitative research. J Spec Pediatr Nurs. (2011) 16(2):151–5. doi: 10.1111/j.1744-6155.2011.00283.x
69. Schreier M. Qualitative content analysis in practice. London, UK: Sage Publications Ltd. (2012).
70. Anfara VA, Mertz NT. Setting the stage. In: Anfara VA, Mertz NT, editors. Theoretical frameworks in qualitative research. Thousand Oaks, CA: Sage (2015). p. 1–20.
71. Radomski A, Bagnell A, Curtis S, Hartling L, Newton A. Examining the usage, user experience, and perceived impact of an internet-based cognitive behavioral therapy program for adolescents with anxiety: randomized controlled trial. JMIR Ment Health. (2020) 7(2):e15795. doi: 10.2196/15795
72. McCurdie T, Taneva S, Casselman M, Yeung M, McDaniel C, Ho W, et al. Mhealth consumer apps: the case for user-centered design. Biomed Instrum Technol. (2012) 46(s2):49–56. doi: 10.2345/0899-8205-46.s2.49
73. Gagnon MP, Ngangue P, Payne-Gagnon J. Desmartis M. m-health adoption by healthcare professionals: a systematic review. JAMIA. (2016) 23(1):212–20. doi: 10.1093/jamia/ocv052
74. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. (2009) 4(1):1–5. doi: 10.1186/1748-5908-4-50
75. Thompson RA. Socialization of emotion and emotion regulation in the family. In: Gross JJ, editor. Handbook of emotion regulation. 2nd ed. New York, NY: Guildford Press (2014). p. 173–86.
76. Blair C. Stress and the development of self-regulation in context. Child Dev Perspect. (2010) 4(3):181–8. doi: 10.1111/j.1750-8606.2010.00145.x
77. Morgan CA, Chang YH, Choy O, Tsai MC, Hsieh S. Adverse childhood experiences are associated with reduced psychological resilience in youth: a systematic review and meta-analysis. Children. (2021) 9(1):27. doi: 10.3390/children9010027
78. Polizzi CP, Lynn SJ. Regulating emotionality to manage adversity: a systematic review of the relation between emotion regulation and psychological resilience. Cogn Ther Res. (2021) 45:577–97. doi: 10.1007/s10608-020-10186-1
79. Joyce S, Shand F, Tighe J, Laurent SJ, Bryant RA, Harvey SB. Road to resilience: a systematic review and meta-analysis of resilience training programmes and interventions. BMJ Open. (2018) 8(6):e017858. doi: 10.1136/bmjopen-2017-017858
80. Oral R, Ramirez M, Coohey C, Nakada S, Walz A, Kuntz A, et al. Adverse childhood experiences and trauma informed care: the future of health care. Pediatr Res. (2016) 79(1):227–33. doi: 10.1038/pr.2015.197
81. Goswami A, Dutta S. Gender differences in technology usage—a literature review. Open J Bus Manag. (2015) 4(1):51–9. doi: 10.4236/ojbm.2016.41006
82. Venkatesh V, Morris MG. Why don’t men ever stop to ask for directions? Gender, social influence, and their role in technology acceptance and usage Behavior. MIS Q. (2000) 24(1):115–39. doi: 10.2307/3250981
83. Groves RM, Peytcheva E. The impact of nonresponse rates on nonresponse bias: a meta-analysis. Public Opin Q. (2008) 72(2):167–89. doi: 10.1093/poq/nfn011
84. Zhao Y, i Q N, Zhou R. What factors influence the mobile health service adoption? A meta-analysis and the moderating role of age. IJIM. (2018) 43:342–50. doi: 10.1016/j.ijinfomgt.2017.08.006
85. Jaworska N, MacQueen G. Adolescence as a unique developmental period. JPN. (2015) 40(5):291. doi: 10.1503/jpn.150268
86. Castleberry A, Nolen A. Thematic analysis of qualitative research data: is it as easy as it sounds? Curr Pharm Teach Lear. (2018) 10(6):807–15. doi: 10.1016/j.cptl.2018.03.019
Keywords: mobile health (mHealth) app, the JoyPop app, qualitative, Technology Acceptance Model (TAM), youth, service providers, mental health, resilience
Citation: Malik I, Perez A, Toombs E, Schmidt F, Olthuis JV, Charlton J, Grassia E, Squier C, Stasiuk K, Bobinski T and Mushquash AR (2023) Female youth and mental health service providers' perspectives on the JoyPop™ app: a qualitative study. Front. Digit. Health 5:1197362. doi: 10.3389/fdgth.2023.1197362
Received: 30 March 2023; Accepted: 12 September 2023;
Published: 27 September 2023.
Edited by:
Amanda Gilmore, Georgia State University, United StatesReviewed by:
Claire Spears, Georgia State University, United StatesHollie Granato, The University of Texas at Austin, United States
© 2023 Malik, Perez, Toombs, Schmidt, Olthuis, Charlton, Grassia, Squier, Stasiuk, Bobinski and Mushquash. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Aislin R. Mushquash aislin.mushquash@lakeheadu.ca