Acceptability and Usability of Mobile Apps for Smoking Cessation Among Young Adults With Psychotic Disorders and Other Serious Mental Illness

Background: Young adults with serious mental illness (SMI) are over twice as likely to smoke cigarettes than those in the general population, but little research has evaluated the efficacy of interventions for this group. While smartphone apps are a promising tool to address this need, their usability should be evaluated among young adults with psychotic disorders, whose symptoms and cognitive impairments may be a barrier to app use.

Methods: We compared usability and acceptability of National Cancer Institute apps (QuitGuide and quitSTART) between young adult smokers with SMI psychotic disorders and other SMI diagnoses. We evaluated objective app usability at the initial study visit and following 2 weeks of independent use via a video-recorded task-completion protocol. Perceptions of usability and acceptability were assessed with semi-structured interviews. Engagement was assessed with backend app use data.

Results: Participants had a mean age of 29 years old (SD = 4). Of the participants without psychotic disorders (n = 10), all were diagnosed with SMI post-traumatic stress disorder (SMI-PTSD). QuitGuide objective task completion rates were high and similar between diagnosis groups, whereas quitSTART task completion was initially lower among users with psychotic disorder compared to users with SMI-PTSD at Visit 1, and improved by Visit 2. Mean app interactions, mean days of use, and median completed notifications were dramatically higher among quitSTART users compared to QuitGuide users. Compared to quitSTART users with SMI-PTSD, quitSTART users with psychotic disorders had similar daily app interactions over the first week of use (mean 3.8 ± 2.4 interactions), and numerically lower mean daily app interactions during the second week (1.9 ± 1.5 vs. 3.4 ± 2.5), whereas completed notifications remained stable among quitSTART users in both diagnosis groups over time. Qualitative comments indicated general acceptability of both apps among both diagnosis groups.

Conclusions: Both QuitGuide and quitSTART were usable and appealing among young adult smokers with psychotic disorders and SMI-PTSD, although quitSTART engendered a dramatically greater level of engagement compared to QuitGuide. Initial coaching to support initiation and app notifications to promote prolonged engagement may be important for young adult smokers with psychotic disorders. Replication and efficacy testing for quitSTART is warranted.

Introduction

While the overall rate of smoking in the United States continues to decline, smoking rates remain higher and cessation rates remain lower for people with serious mental illness (SMI; such as schizophrenia and severe mood or anxiety disorders) compared to the general population (1–3). Quitting at a younger age can provide greater health benefits than quitting later (4), making young adulthood an important target for cessation interventions. Significant work has been done on smoking cessation interventions for young adults in the general population (5, 6), and for middle-aged adults with SMI [e.g., (3, 7–9)], however more work is needed to address smoking in young people with SMI. Smoking cessation apps are a promising tool based on testing among general population adults [e.g., (10, 11)], but no research has evaluated their impact in young adults with SMI.

We previously conducted focus groups among young adults with SMI that identified an interest in using smoking cessation apps during a quit attempt as well as an interest in a variety of app features related to smoking and quitting (12). These young adults did not specify an interest in addressing their mental illness within a smoking cessation app, suggesting that apps with evidence-based smoking cessation content designed for the general public may be appealing to this group. However, usability has been identified as a possible problem for people with SMI, particularly those with psychotic disorders such as schizophrenia (13, 14), and therefore warrants additional consideration.

People with psychotic disorders experience a greater degree of cognitive impairments and amotivation than people with other types of SMI (15–18). Such cognitive impairments and amotivation may influence how they interact with digital tools (13, 14). We and others have identified design features that may overcome these barriers (9, 19–21), but it is unknown to what extent general smoking cessation tool designs are problematic for young people with psychotic disorders. Smartphone technology continues to gain popularity across demographic groups (22), and recent data demonstrate that nearly 80% of young adults with SMI own smartphones and most are interested in using their smartphones for health interventions (23). Given that young adults have grown up in an era of smartphone technology and report greater comfort with technology than older age groups (24), it is possible that their familiarity with digital technology may mitigate the effect of cognitive impairments associated with psychotic disorders.

We therefore conducted a study with the primary goal of assessing the usability and acceptability of two smoking cessation smartphone apps designed for general population adults (QuitGuide) and teens (quitSTART) among young adults with SMI (25). The purpose of this secondary analysis was to examine the apps' usability and acceptability among young adult smokers with psychotic disorders compared to smokers with other serious mental illness diagnoses in order to determine whether usability and acceptability differed between these two groups.

Methods

Participants and Recruitment

Study methods have been previously reported in detail (25) and will be described briefly here. We recruited young adult smokers who were in treatment for SMI (assessed by the mental health clinic as a serious mental illness of any diagnosis causing substantial disability). Clinicians referred potentially eligible subjects and some self-referred from waiting room flyers. Inclusion criteria included: age 18-35 years old, English-speaking, stable in outpatient mental health treatment for SMI (i.e., stable in treatment with no hospitalization in past 30 days per chart review), self-reported regular tobacco smokers (daily and non-daily) confirmed by breath carbon monoxide (CO) > 7 parts per million (ppm) (26), and smartphone users (either Apple or Android). We selected the relatively broad young adult age range of 18-35 in order to assess the apps in the largest user group that maximizes the potential for the long-term health benefits of smoking cessation, since quitting by age 35 has been associated with life expectancy similar to that of a never smoker (4, 27). Exclusion criteria were current pregnancy or current, unstable substance use disorder per chart review or per the participant's mental health center clinician. Interest in quitting or reducing their smoking was not required to join this study. Prior usability research has demonstrated that over 80% of usability issues can be identified after the first five participants (28), therefore, we aimed to recruit at least five participants with psychotic disorders and five with other SMI diagnoses to use each app (20 total).

Interventions

QuitGuide and quitSTART are smoking cessation apps available without cost at Smokefree.gov via the Apple Store or Google Play. While QuitGuide was designed for adults and quitSTART was designed for teens, the apps share a number of commonalities, including encouraging the user to set a quit date within 14 days, providing informational and motivational content related to quitting smoking, providing users the opportunity to set notifications based on time or location, allowing users to track information such as “slips” (cigarettes smoked) and cravings, and providing feedback to the user based on information they had previously entered into the app.

The two apps differ in several aspects of their content and design (see Table 1 and Figure 1). This includes their layout (linear vs. complex), color palette (dark vs. bright), balance of text and symbols (text-heavy vs. symbol-based), and delivery of information (black and white text document vs. colorful swipeable “cards”). Additionally, quitSTART also automatically sends check-in notifications that ask users how many cigarettes they have smoked since the last check-in, whereas QuitGuide does not have an analogous check-in feature.

TABLE 1

Table 1. Comparison of QuitGuide and quitSTART characteristics.

FIGURE 1

Figure 1. Selected Screen Shots from QuitGuide (A–C) and quitSTART (D–F). Represented here are the home screens (A,D), progress screens (B,E), and information on how to quit smoking (C,F).

Procedures

Participation in the study lasted 2 weeks. At the first study visit (Visit 1), participants were randomly assigned to one of the two apps. They completed questionnaires regarding demographics, tobacco use, technology use, and app feature preferences. They then completed the usability protocol described below, followed by a semi-structured interview and usability and acceptability questionnaire items. At the end of Visit 1, participants were instructed to use the app independently for 2 weeks before returning for Visit 2 and repeating the usability protocol, questionnaires, and semi-structured interview.

Measures

Demographics, Technology Use, Diagnosis

Researchers obtained participants' demographics at Visit 1, and history of technology use (e.g., frequency of internet use and app use) at Visits 1 and 2. Researchers reviewed medical record at Visit 1 to obtain Psychiatric diagnosis, mental health stability, and insurance status.

Tobacco History and Smoking

With a structured interview at Visit 1, researchers obtained participants' tobacco history (e.g., duration and frequency of smoking, product use, prior quit attempts). At Visits 1 and 2, researchers used the Fagerström test for nicotine dependence (29), a six-item scale shown to be reliable and valid among smokers with SMI (30), to obtain a rating of level of dependence. Smoking status was confirmed with exhaled breath CO > 7 ppm (measured with a Covita Smokerlyzer^®, Santa Barbara, CA) at both visits (26).

Observed App Usability

We developed and administered a usability protocol with basic principles of usability testing (31, 32). Using a “Think Aloud” procedure (33), participants were given up to 5 min to freely explore the app, and then were asked to complete nine specified tasks within the app (Appendix 1). Tasks were chosen based on U.S. Clinical Practice Guidelines (34), as well as prior studies that evaluated frequently used features, desired app features, and features that have been correlated with point prevalence abstinence (12, 35, 36). The participants' phone screens and hand motions were video-recorded as they completed the tasks.

Video recordings were scored as “completed” if the participant was able to reach the requested end point, or “not completed” if the participant requested to skip the task or did not reach the requested end point. “Usability challenges” were defined either as actions performed in the app that could not be used to reach the requested endpoint, or difficulty reaching the requested endpoint identified either by researcher review or by participant verbalization during the task.

System Usability Scale

This validated questionnaire (37) is commonly used to assess the usability of many types of technologies, including apps among people with SMI (8, 38, 39). Scores range from 0 to 100, with values of 68-70 representing average usability (40, 41).

Perceived Ease of Use and Acceptability Questionnaire

Using questions from the Post-Study System Usability Questionnaire (PSSUQ) (42) and the Usefulness, Satisfaction, and Ease of Use (USE) Questionnaire (43), we administered a 14 item questionnaire, using a five-point Likert-type scale, to assess ease of use and acceptability. We chose a subset of questions from these scales that have been previously used in people with SMI (44). The complete list of questions is presented in Appendix 2.

App Perceptions Semi-Structured Interview

In addition to structured questionnaires, we conducted semi-structured qualitative interviews at each visit. We posed open-ended questions to elicit participant feedback on their assigned app's features, content, and ease of use. While we recognized mental illness could affect participants' perceptions of the apps, we elected to use general open-ended questions about the apps rather than specific questions about mental illness or other topics. This allowed participants to discuss what they felt was most important to their experience with their assigned app.

App Utilization

We obtained backend app usage data from NCI (45), including date and time of app use, buttons clicked in the app, and response to notifications.

Participant Flow

Ninety-eight potentially eligible individuals with any SMI diagnosis were screened. Thirty-four were ineligible based on pre-screening criteria, seven refused due to work or child care responsibilities, one due to an upcoming move. Ten were not eligible because they did not have working smartphones, and 27 declined to participate. Researchers obtained informed consent from 19 participants. Of these, two were ultimately deemed ineligible due to low breath CO, resulting in 17 study participants. All 17 participants completed Visits 1 and 2 (100% retention); 15 provided backend app use data (home app use data was not available for two quitSTART participants due to phone problems, and these participants were excluded from the app utilization analyses).

Data Analysis

Quantitative Analyses

All quantitative analyses were completed using descriptive statistics. One usability task was missing at Visit 2 for one participant, thus these data were omitted from the analysis.

For the 15 participants with backend app use data available, we analyzed home app use on days 2-14, thus excluding days with study visits. Complete data was available for all 15 participants on all days except day 14, since one participant was assessed a day early.

Qualitative Analyses

Transcriptions of the audio-recorded semi-structured interview responses were compared to the original audio files to ensure accuracy, and then analyzed using thematic analytic techniques (46). Themes were readily apparent due to the descriptive nature of the data, and researchers reached consensus regarding these themes after a single discussion without discrepancies. Negative case analysis was used to ensure the entire data set was represented in the emerging themes.

We reviewed video recordings from the usability task completion protocol to assess participants' ability to reach each task's end point, and to identify usability challenges. Participants' comments during the session were also used to identify usability challenges. During the initial coding of the videos, definitions regarding user challenges were refined until a final set of definitions was reached. Final coding of the video recordings was performed using this set of definitions.

Results

Participant Characteristics

Participant characteristics are shown in Table 2. All were daily smokers, 7 (41%) were female, and 16 (94%) were White. Just under half (n = 7, 41%) were diagnosed with psychotic disorders (schizophrenia spectrum disorders; hereafter used interchangeably with “psychosis” for readability), and the remainder (n = 10, 59%) were diagnosed with SMI post-traumatic stress disorder (SMI-PTSD). Overall, participants smoked an average of 15 cigarettes per day (SD = 7). Most participants (n = 15, 88%) had previously engaged in a quit attempt, and all but one (n = 16, 94%) used smartphone apps on a daily basis. The psychosis and SMI-PTSD groups were similar, though there was a lower proportion of female participants in the psychosis group (n = 1, 14% vs. n = 6, 60%).

TABLE 2

Table 2. Participant Characteristics.

Usability

QuitGuide task completion rates, a key marker of usability, were similar between diagnosis groups, and were high at both at both visits (Figure 2). quitSTART task completion rates were lower at Visit 1 as compared to QuitGuide, particularly for users with psychosis. However, by Visit 2, completion improved for quitSTART users, and rates were similar between diagnosis groups and compared to QuitGuide users.

FIGURE 2

Figure 2. Task completion rates for each app for participants with psychosis and SMI-PTSD.

Self-reported usability (mean SUS scores; not shown) for QuitGuide at both Visit 1 and Visit 2 were higher for participants with psychotic disorders (76 ± 1.4 and 74 ± 7.6, respectively) compared to participants with SMI-PTSD (59 ± 19 and 63 ± 21, respectively). Mean SUS scores for quitSTART at Visit 1 and Visit 2 were similar between groups, and increased after 2 weeks among participants with psychotic disorders (Psychosis: 53 ± 29 and 68 ± 22, respectively; SMI-PTSD: 58 ± 9.6 and 61 ± 6.9, respectively). Responses to other ease of use questions followed this same pattern.

Acceptability

Favorable responses to the structured interview acceptability questions were higher for participants with psychotic disorders than for those with SMI-PTSD for both apps (Figure 3). While responses among QuitGuide users were generally stable from Visit 1 to Visit 2, quitSTART users' responses were low and generally improved at the second visit for both diagnosis categories.

FIGURE 3

Figure 3. Percent of participants assigned to QuitGuide and quitSTART who agree or strongly agree with the corresponding acceptability statements.

Responses to open-ended questions regarding the apps indicated a higher level of acceptability. A detailed description of participants' semi-structured interview responses is presented elsewhere (25). In summary, participants assigned to both apps expressed positive attitudes toward the apps, though reactions to QuitGuide were generally subdued, while reactions to quitSTART tended to be more exuberant (both positive and negative). Participants perceived that both apps were positive and supportive, and appreciated the motivational content. They noted these aspects of the apps were very important to their ongoing use of the apps. However, both QuitGuide and quitSTART users wished the apps had the ability to track the number of cigarettes smoked per day and felt that the feedback the apps provided to them was inaccurate. While some users did not recall receiving any notifications, and some preferred not to receive notifications, most users assigned to both apps were either satisfied with the number of notifications they received or wished they had received more. The themes of responses to semi-structured interview questions were similar among participants in both diagnosis groups.

App Utilization

App utilization is a key indicator of both usability and acceptability. Participants interacted with quitSTART dramatically more that QuitGuide over the 13-day study period (Table 3). QuitGuide users in both diagnosis groups opened the app on a similar number of days out of the 2-week period. quitSTART users in both diagnosis groups opened their app on a higher number of days than QuitGuide users, with the greatest days of use among those with SMI-PTSD. Total app interactions followed a similar pattern, with greater interactions among quitSTART users than QuitGuide users, and greater interactions among the SMI-PTSD group.

TABLE 3

Table 3. Summary of home app use measures over the two-week trial period.

Mean daily app interactions are shown in Figure 4. QuitGuide users interacted with their app dramatically less than quitSTART users [less than once vs. 3.6 (SD = 2.4) times per day on average during the first week of the trial]. quitSTART app interactions declined among participants with psychosis during the second week compared to participants with SMI-PTSD (1.9 ± 1.5 vs. 3.9 ± 2.4).

FIGURE 4

Figure 4. Mean daily app interactions for participants with psychosis and SMI-PTSD.

Responses to notifications are shown in Figure 5. QuitGuide users in both groups completed very few notifications over the trial period. In contrast, quitSTART users steadily completed more notifications than QuitGuide users overall, with slightly higher completion rates for participants with SMI-PTSD (psychosis 1.2 ± 1.2 vs. SMI-PTSD 1.9 ± 0.9).

FIGURE 5

Figure 5. Mean daily completed notifications for participants with psychosis and SMI-PTSD.

Tobacco Use at Follow-Up

While participants were not required to quit or reduce their smoking for this study, over three-quarters of participants (13/17, 77%) reported attempting to do so during the 2-week trial period. This included 6/7 (86%) of participants with psychosis and 7/10 (70%) of participants with SMI-PTSD. The proportion of participants who reported attempting to quit or reduce their smoking was similar for each app: 7/9 (78%) QuitGuide users, and 6/8 (75%) quitSTART users. Notably, two quitSTART participants reported that they were no longer smoking at Visit 2 (confirmed with breath CO < 7 ppm) after switching completely to e-cigarettes (one with psychosis and one with SMI-PTSD). Both stated that they believed the app had helped them quit smoking.

Discussion

Main Findings

In this secondary analysis of usability and acceptability of two smoking cessation apps designed for the general population, young adults with and without psychotic disorders had high objective task completion rates for both apps. Participants with psychotic disorders rated both apps more favorably than those with SMI-PTSD, yet back end app use data demonstrated low engagement with QuitGuide and high engagement with quitSTART among both groups. Participants with psychosis demonstrated declining interactions with quitSTART during their second week of use, while responses to notifications were stable during this time, suggesting that the decline in app interactions was due to a decline in spontaneous app use. Most participants reported attempting to cut down or quit, and two quitSTART participants, one in each diagnosis group, demonstrated biologically verified abstinence from smoking after 2 weeks of app use. Notably, while questionnaire assessment of acceptability suggested low acceptability of the apps, interview responses, user engagement with the apps, quit attempts, and abstinence demonstrated favorable acceptability, especially for quitSTART, among people with psychotic disorders and with SMI-PTSD.

These results suggest several important points regarding app use for smoking cessation among young adults with psychotic disorders, whose mental-illness related cognitive impairments can reduce functional capacity beyond that seen in other SMI diagnoses. First, this form of intervention is very appealing for both young adults with psychotic disorders and young adults with SMI-PTSD. The high level of engagement aligns with our previous survey in which 75% of young adults with SMI were interested in using the devices for health interventions (23). Second, young adults with psychotic disorders were able to easily learn how to use these apps despite several design problems we have previously described (25), indicating that high quality “off the shelf” apps may have utility in this population. Third, young adults with psychotic disorders were activated by the content of the quitSTART app in a manner similar to smokers with SMI-PTSD, suggesting that tailoring content specific for psychotic disorders in this age group may not be required. Finally, the pattern of quitSTART use in those with psychotic disorders (declining spontaneous use while responses to notifications persisted) suggests that using push notifications may be a key strategy to maintain engagement among users with psychotic disorders that warrants testing in future research.

Comparison With Prior Research

One other small study has assessed usability of QuitGuide, in which QuitGuide was compared to an Acceptance and Commitment Therapy (ACT)-based app developed for people with SMI (Learn To Quit) among middle-aged adults (39). This study included four participants with psychotic disorders who used QuitGuide during the trial, with an average SUS score for QuitGuide of 61.9 (range 35-82.5). While not reported by the presence or absence of psychotic disorder diagnosis, the authors found that in general, participants used QuitGuide self-tracking features more frequently than they responded to Learn to Quit's ecological momentary assessment (EMA) features, leading the authors to hypothesize that notifications may not improve user engagement. These results contrast with our findings among young adults with SMI, in which QuitGuide engagement remained very low over time, and quitSTART users engaged extensively and steadily with notifications. Our findings would suggest that notifications can effectively engage users, particularly among people with psychotic disorders.

Previous evaluations of the usability and acceptability of apps in middle-aged adults with SMI (20, 47) demonstrated that participants found “text-heavy” apps (such as QuitGuide) to be unappealing and difficult to understand, aligning with our finding of low spontaneous use of QuitGuide over the study period. Our participants' report of the importance of a positive, motivational tone also overlaps with findings among middle-aged adults with SMI (9). However, our young adult participants did not mention the need for an app specifically tailored to address mental illness. Whether, digital interventions for smoking among people with SMI require specific mental health content is a topic needing further study.

A secondary analysis of app engagement among middle-aged adults in the general population found that a number of participant characteristics predicted lower app engagement, including lower education level (high school or less), heavier smoking (>10 cigarettes per day), and greater depressive symptoms (48). While our study did not directly assess the impact of these factors on engagement, people with psychotic disorders tend to have lower educational attainment and higher levels of nicotine dependence than those without psychotic disorders, and may be at risk for lower levels of engagement with cessation apps. Thus, our finding that young adults with psychotic disorders exhibited strong engagement with quitSTART further supports quitSTART's potential role in this population.

Limitations

This usability study was small by design, thus the finding of differences between participants with and without psychotic disorder diagnoses require confirmation in larger studies. Additionally, the 2-week study period was relatively short. Given that smoking and app use patterns change over time, future studies should include longer follow-up. We also did not ask participants about how their mental illness affected their perceptions and use of the apps, nor were we able to assess cognitive abilities and symptom severity in order to examine the relationship between these symptoms and usability. Nevertheless, these novel findings generally align with previous work, which supports their validity.

Conclusions

Both QuitGuide and quitSTART were usable and acceptable among young adult smokers with psychotic disorders and SMI-PTSD. quitSTART engendered a greater level of engagement compared to QuitGuide, although spontaneous engagement declined while notification engagement persisted among smokers with psychotic disorders. Initial coaching to support initiation and app notifications to promote ongoing engagement may be advantageous for use of quitSTART among young adult smokers with psychotic disorders. Replication in larger study samples and testing for efficacy of quitSTART are warranted.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by New Hampshire Department of Health and Human Services Institutional Review Board. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

MG and MB designed the study and prepared the original draft of the manuscript. MG collected the data and performed quantitative and qualitative data analysis with oversight of these activities provided by MB. KA provided expert consultation for data interpretation and reviewed and provided comments on the manuscript before submission. All authors contributed to the article and approved the submitted version.

Funding

MG's work was partially supported by National Institute on Drug Abuse (NIDA) (Grant Number: P30DA029926) via a pilot award through the Center for Technology and Behavioral Health at Dartmouth, and by a Health Resources Services Administration (HRSA) postdoctoral fellowship grant (T32HP32520). MB's work was partially supported by Substance Abuse and Mental Health Services Administration (SAMHSA) (Grant Number: 1H79SM080245-01). KA's work was partially supported by National Institute of Mental Health (NIMH) (Grant Numbers: NCI P50CA244433 and NIMH R01MH110965). These funding organizations had no involvement in the study design; the collection, analysis, or interpretation of the data; the preparation of the report; or the decision to submit the article for publication.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors would like to thank Susan Guarino, Jason Welsh, and Michael Borsari for their assistance with recruiting participants.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2021.656538/full#supplementary-material

References

1. Smith PH, Mazure CM, McKee SA. Smoking and mental illness in the U.S. population. Tob Control. (2014) 23:e147–53. doi: 10.1136/tobaccocontrol-2013-051466

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Wang TW, Asman K, Gentzke AS, Cullen KA, Holder-Hayes E, Reyes-Guzman C, et al. Tobacco product use among adults - United States, 2017. MMWR Morb Mortal Wkly Rep. (2018) 67:1225–32. doi: 10.15585/mmwr.mm6744a2

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Anthenelli RM, Benowitz NL, West R, St Aubin L, McRae T, Lawrence D, et al. Neuropsychiatric safety and efficacy of varenicline, bupropion, and nicotine patch in smokers with and without psychiatric disorders (EAGLES): a double-blind, randomised, placebo-controlled clinical trial. Lancet. (2016) 387:2507–20. doi: 10.1016/S0140-6736(16)30272-0

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Doll R, Peto R, Boreham J, Sutherland I. Mortality in relation to smoking: 50 years' observations on male British doctors. BMJ. (2004) 328:1519. doi: 10.1136/bmj.38142.554479.AE

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Villanti AC, McKay HS, Abrams DB, Holtgrave DR, Bowie JV. Smoking-cessation interventions for U.S. young adults: a systematic review. Am J Prev Med. (2010) 39:564–74. doi: 10.1016/j.amepre.2010.08.009

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Suls JM, Luger TM, Curry SJ, Mermelstein RJ, Sporer AK, An LC. Efficacy of smoking-cessation interventions for young adults: a meta-analysis. Am J Prev Med. (2012) 42:655–62. doi: 10.1016/j.amepre.2012.02.013

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Evins AE, Cather C, Pratt SA, Pachas GN, Hoeppner SS, Goff DC, et al. Maintenance treatment with varenicline for smoking cessation in patients with schizophrenia and bipolar disorder: a randomized clinical trial. JAMA. (2014) 311:145–54. doi: 10.1001/jama.2013.285113

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Vilardaga R, Rizo J, Palenski P, Mannelli P, Oliver JA, McClernon FJ. Pilot randomized controlled trial of a novel smoking cessation app designed for individuals with co-occurring tobacco dependence and serious mental illness. Nicotine Tob Res. (2019) 22:1533–42. doi: 10.1093/ntr/ntz202

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Klein P, Lawn S, Tsourtos G, van Agteren J. Tailoring of a smartphone smoking cessation app (kick.it) for serious mental illness populations: qualitative study. JMIR Hum Factors. (2019) 6:e14023. doi: 10.2196/14023

PubMed Abstract | CrossRef Full Text

10. Bricker JB, Watson NL, Mull KE, Sullivan BM, Heffner JL. Efficacy of smartphone applications for smoking cessation: a randomized clinical trial. JAMA Intern Med. (2020) 180:1–9. doi: 10.1001/jamainternmed.2020.4055

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Iacoviello BM, Steinerman JR, Klein DB, Silver TL, Berger AG, Luo SX, et al. Clickotine, a personalized smartphone app for smoking cessation: initial evaluation. JMIR Mhealth Uhealth. (2017) 5:e56. doi: 10.2196/mhealth.7226

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Gowarty MA, Kung NJ, Maher AE, Longacre MR, Brunette MF. Perceptions of mobile apps for smoking cessation among young people in community mental health care: qualitative study. JMIR Form Res. (2020) 4:e19860. doi: 10.2196/19860

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Brunette MF, Ferron JC, Devitt T, Geiger P, Martin WM, Pratt S, et al. Do smoking cessation websites meet the needs of smokers with severe mental illnesses? Health Educ Res. (2012) 27:183–90. doi: 10.1093/her/cyr092

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Rotondi AJ, Sinkule J, Haas GL, Spring MB, Litschge CM, Newhill CE, et al. Designing websites for persons with cognitive deficits: design and usability of a psychoeducational intervention for persons with severe mental illness. Psychol Serv. (2007) 4:202–24. doi: 10.1037/1541-1559.4.3.202

PubMed Abstract | CrossRef Full Text | Google Scholar

15. David AS, Zammit S, Lewis G, Dalman C, Allebeck P. Impairments in cognition across the spectrum of psychiatric disorders: evidence from a Swedish conscript cohort. Schizophr Bull. (2008) 34:1035–41. doi: 10.1093/schbul/sbn028

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Zammit S, Allebeck P, David AS, Dalman C, Hemmingsson T, Lundberg I, et al. A longitudinal study of premorbid IQ Score and risk of developing schizophrenia, bipolar disorder, severe depression, and other nonaffective psychoses. Arch Gen Psychiatry. (2004) 61:354–60. doi: 10.1001/archpsyc.61.4.354

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Reichenberg A, Weiser M, Rabinowitz J, Caspi A, Schmeidler J, Mark M, et al. A population-based cohort study of premorbid intellectual, language, and behavioral functioning in patients with schizophrenia, schizoaffective disorder, and nonpsychotic bipolar disorder. Am J Psychiatry. (2002) 159:2027–35. doi: 10.1176/appi.ajp.159.12.2027

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Sharma T, Antonova L. Cognitive function in schizophrenia: deficits, functional consequences, and future treatment. Psychiatr Clin North Am. (2003) 26:25–40. doi: 10.1016/S0193-953X(02)00084-9

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Ferron JC, Brunette MF, McHugo GJ, Devitt TS, Martin WM, Drake RE. Developing a quit smoking website that is usable by people with severe mental illnesses. Psychiatr Rehabil J. (2011) 35:111–6. doi: 10.2975/35.2.2011.111.116

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Ben-Zeev D, Kaiser SM, Brenner CJ, Begale M, Duffecy J, Mohr DC. Development and usability testing of FOCUS: a smartphone system for self-management of schizophrenia. Psychiatr Rehabil J. (2013) 36:289–96. doi: 10.1037/prj0000019

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Rotondi AJ, Spring MR, Hanusa BH, Eack SM, Haas GL. Designing eHealth applications to reduce cognitive effort for persons with severe mental illness: page complexity, navigation simplicity, and comprehensibility. JMIR Hum Factors. (2017) 4:e1. doi: 10.2196/humanfactors.6221

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Mobile Fact Sheet: Pew Research Center. Available online at: https://www.pewresearch.org/internet/fact-sheet/mobile/ (accessed June 12, 2019).

23. Brunette MF, Achtyes E, Pratt S, Stilwell K, Opperman M, Guarino S, et al. Use of smartphones, computers and social media among people with SMI: opportunity for intervention. Community Ment Health J. (2019) 55:973–8. doi: 10.1007/s10597-019-00431-7

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Andersen M, Perrin A. Tech Adoption Climbs Among Older Adults: Pew Research Center. Available online at: https://www.pewinternet.org/2017/05/17/barriers-to-adoption-and-attitudes-towards-technology/ (accessed May 17, 2017).

25. Gowarty MA, Longacre MR, Vilardaga R, Kung NJ, Maher AE, Brunette MF. Usability and acceptability of two smartphone apps for smoking cessation among young adults with serious mental illness: mixed methods study. JMIR Ment Health. (in press). doi: 10.2196/26873

CrossRef Full Text | Google Scholar

26. Benowitz NL, Ahijevych K, Jarvis MJ, Hall S, LeHouezec J, Lichtenstein E, et al. Biochemical verification of tobacco use and cessation. Nicotine Tob Res. (2002) 4:149–59. doi: 10.1080/14622200210123581

CrossRef Full Text | Google Scholar

27. Jha P, Ramasundarahettige C, Landsman V, Rostron B, Thun M, Anderson RN, et al. 21st-century hazards of smoking and benefits of cessation in the United States. N Engl J Med. (2013) 368:341–50. doi: 10.1056/NEJMsa1211128

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Neilsen J, Landeur TK editors. A mathematical model of the finding of usability problems. In: CHI '93 Conference on Human Factors in Computing Systems. Amsterdam: Association for Computing Machinery (1993).

Google Scholar

29. Heatherton TF, Kozlowski LT, Frecker RC, Fagerström KO. The Fagerström Test for Nicotine Dependence: a revision of the Fagerström Tolerance Questionnaire. Br J Addict. (1991) 86:1119–27. doi: 10.1111/j.1360-0443.1991.tb01879.x

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Weinberger AH, Reutenauer EL, Allen TM, Termine A, Vessicchio JC, Sacco KA, et al. Reliability of the Fagerstrom Test for Nicotine Dependence, Minnesota Nicotine Withdrawal Scale, and Tiffany Questionnaire for Smoking Urges in smokers with and without schizophrenia. Drug Alcohol Depend. (2007) 86:278–82. doi: 10.1016/j.drugalcdep.2006.06.005

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Rubin J, Chisnell D. Handbook of Usability Testing, Second Edition: How to Plan, Design, and Conduct Effective Tests. Indianapolis, IN: Wiley Publishing, Inc. (2008).

32. Usability.gov. Improving the User Experience: U.S. General Services Administration Technology Transformation Service. Available online at: https://www.usability.gov/how-to-and-tools/methods/index.html (accessed June 6, 2019).

33. van Someren MW, Barnard YF, Sandberg JAC. The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes. London: Academic Press (1994).

Google Scholar

34. Fiore MC, Jaén CR, Baker TB, Bailey WC, Benowitz NL, Curry SJ, et al. Treating Tobacco Use and Dependence: 2008 Update. Clinical Practice Guideline. Rockville, MD: U.S. Department of Health and Human Services. Public Health Service (2008).

PubMed Abstract | Google Scholar

35. Heffner JL, Vilardaga R, Mercer LD, Kientz JA, Bricker JB. Feature-level analysis of a novel smartphone application for smoking cessation. Am J Drug Alcohol Abuse. (2015) 41:68–73. doi: 10.3109/00952990.2014.977486

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Oliver JA, Hallyburton MB, Pacek LR, Mitchell JT, Vilardaga R, Fuemmeler BF, et al. What do smokers want in a smartphone-based cessation application? Nicotine Tob Res. (2018) 20:1507–14. doi: 10.1093/ntr/ntx171

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Brooke J. SUS - a quick and dirty usability scale. In: Jordan PW, Thomas B, McClelland IL, Weerdmeester B, editors. Usability Evaluation in Industry. Londong: CRC Press (1996). p. 189–94.

Google Scholar

38. Vilardaga R, Rizo J, Kientz JA, McDonell MG, Ries RK, Sobel K. user experience evaluation of a smoking cessation app in people with serious mental illness. Nicotine Tob Res. (2016) 18:1032–8. doi: 10.1093/ntr/ntv256

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Vilardaga R, Rizo J, Ries RK, Kientz JA, Ziedonis DM, Hernandez K, et al. Formative, multimethod case studies of learn to quit, an acceptance and commitment therapy smoking cessation app designed for people with serious mental illness. Transl Behav Med. (2019) 9:1076–86. doi: 10.1093/tbm/iby097

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Bangor A, Kortum P, Miller J. Determining what individual sus scores mean: adding an adjective rating scale. J Usability Stud. (2009) 4:114–23.

Google Scholar

41. Sauro J. A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices. Denver, CO: CreateSpace Independent Publishing Platform (2011).

42. Lewis JR. Psychometric evaluation of the post-study system usability questionnaire: The PSSUQ. In: Proceedings of the Human Factors Society Annual Meeting. vol. 36. Washington, DC (1992). p. 1259–60.

Google Scholar

43. Gao M, Kortum P, Oswald F. Psychometric evaluation of the use (usefulness, satisfaction, and ease of use) questionnaire for reliability and validity. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting. (2018) 62:1414–8. doi: 10.1177/1541931218621322

CrossRef Full Text | Google Scholar

44. Whiteman KL, Lohman MC, Gill LE, Bruce ML, Bartels SJ. Adapting a psychosocial intervention for smartphone delivery to middle-aged and older adults with serious mental illness. Am J Geriatr Psychiatry. (2017) 25:819–28. doi: 10.1016/j.jagp.2016.12.007

PubMed Abstract | CrossRef Full Text | Google Scholar

45. National Cancer Institute. DCCPS, Behavioral Research Program. Bethesda, MD: National Cancer Institute (2020).

46. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology. (2006) 3:77–101. doi: 10.1191/1478088706qp063oa

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Ferron JC, Brunette MF, Geiger P, Marsch LA, Adachi-Mejia AM, Bartels SJ. Mobile phone apps for smoking cessation: quality and usability among smokers with psychosis. JMIR Hum Factors. (2017) 4:e7. doi: 10.2196/humanfactors.5933

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Zeng EY, Vilardaga R, Heffner JL, Mull KE, Bricker JB. Predictors of utilization of a novel smoking cessation smartphone app. Telemed J E Health. (2015) 21:998–1004. doi: 10.1089/tmj.2014.0232

PubMed Abstract | CrossRef Full Text | Google Scholar