Ayumi Takano1*
Toshihiko Matsumoto2- 1Department of Mental Health and Psychiatric Nursing, Tokyo Medical and Dental University, Tokyo, Japan
- 2Department of Drug Dependence Research, National Center of Neurology and Psychiatry, Tokyo, Japan
This secondary analysis of a pilot randomized controlled trial assessed the usability and acceptability of a web-based relapse prevention and self-monitoring program, which was newly developed for Japanese drug users. We analyzed quantitative and qualitative data using a mixed-method approach among 43 study participants. We assessed the usability of the programs using the Web Usability Scale (WUS) and satisfaction and acceptance using the Client Satisfaction Questionnaire 8-item version (CSQ-8). We also collected qualitative feedback using semi-structured open-ended questions. The results of WUS indicated that the usability of the programs was sufficient. According to the CSQ-8 results, the self-monitoring program had less satisfaction, while the web-based relapse prevention program met satisfaction. Some possible improvements for a future version of the web-based programs were suggested based on participants' feedback.
Introduction
Despite evidence supporting psychosocial treatment for drug abuse and dependence, there is still a treatment gap between potential treatment needs and actually provided treatment (1). To deal with challenges in treatment implementation, e-health interventions, which are therapeutic interventions using information and communication technologies (ICT), have been adapted to drug use problems (2). Various types of e-health interventions, such as computer-assisted, web-, or mobile-based programs for drug users, have been developed based on psychosocial approaches as well as for use in face-to-face treatment. These e-health programs have demonstrated benefit for abstinence, treatment retention, and cost-effectiveness with small to moderate effect sizes (2–4). However, the majority of these programs targeted opioid, cocaine, or cannabis users in western countries (5–11). There have been few e-health programs in Asian countries, where the national drug policy is likely to be very strict, and there are obstacles to seeking treatment because of a strong stigma.
The authors developed a web-based treatment program named “e-learning Serigaya Methamphetamine Relapse Prevention Program (e-SMARPP)” for Japanese drug users (12). The content of e-SMARPP was based on an existing face-to-face relapse prevention program, the Serigaya Methamphetamine Relapse Prevention Program (SMARPP), which is in widespread use in Japan. The web-based program was developed to be user friendly, includes a user guide, support information, FAQ, and contact form. The website was designed to support all devices (PC, mobile phone, tablet). We tested the effectiveness of e-SMARPP with a pilot randomized controlled trial (RCT) (13, 14). We found a medium effect on abstinence, although the statistically-significant effect was not confirmed because of the small sample size (14). In the RCT, the intervention group was asked to complete a web-based 6-session relapse prevention program with 8-week self-monitoring. The control group was only asked to complete a web-based 8-week self-monitoring program. The relapse prevention program involved interactive therapist support. The completion rates of the intervention were high; relapse prevention program: 73.9%, self-monitoring: 82.6% in the intervention group, 100.0% in the control group (14). However, the usability and acceptability of e-SMARPP have not been evaluated in detail.
The aim of this secondary analysis of the pilot RCT was to assess the usability and acceptability of the web-based relapse prevention and self-monitoring program, which was newly developed for Japanese drug users. We analyzed quantitative and qualitative data using a mixed-method design to obtain information to improve future implementation.
Methods
Trial Design
This study analyzed the usability and satisfaction of a previous RCT, which evaluated the effectiveness of a web-based treatment program (14). The design was a two-arm (one-to-one allocation ratio), parallel-group, non-blinded, and multicenter randomized controlled trial. Further methodology and results detailing the efficacy and completion rate of the original RCT can be found in our published articles (13, 14). The Ethics Committee of the Faculty of Medicine and Graduate School of Medicine of the University of Tokyo and the Institutional Review Boards of each recruiting hospital and clinic approved this study. The study protocol was registered with the University Hospital Medical Information Network clinical trial registry (UMIN000016075).
Participants
Participants were recruited from six psychiatric hospitals located in large cities of Japan, with all hospitals providing specialized treatment for those with substance use disorders. Outpatients were required to be diagnosed with substance use disorder by DSM-IV/DSM-5 in addition to having used any type of illicit drug in the past year to be eligible for this study. Participants also had to be able to use the internet in order to access the e-SMARPP website and communicate via email. Exclusion criteria were patients with severe physical diseases, high suicide risk, severe substance-induced psychotic disorder symptoms, or impaired cognitive function. Eligible participants were then randomly allocated to either the e-SMARPP intervention group or the control group utilizing the permuted block method. The researchers and staff working at the psychiatric hospitals were blinded to the results of the randomization.
Intervention
The main intervention modules of e-SMARPP were the cognitive behavioral relapse prevention sessions, and web-based self-monitoring practice. The relapse prevention sessions included six sessions, and each had three videos and homework assignments. Participants were required to submit two homework assignments and a diary after each assignment via the e-SMARPP website. Participants would then receive tailored feedback comments from healthcare professionals trained to support patients with substance use disorder to enhance user motivation and provide individual support. Automated notification emails were also sent to the participant when submitting homework or receiving feedback. The web-based self-monitoring was a self-guided module in which the participants recorded their daily drug use on a calendar format for 8 weeks.
Participants assigned to the intervention group received the complete content of the e-SMARPP intervention. Although participants could progress at their own pace, all sessions were expected to be completed within an 8-week intervention period. A daily record of drug use was also kept. An email reminder was sent to the participant by the web-therapist if they missed a weekly deadline. Those in the control group only participated in the self-monitoring module.
Measures and Statistical Analysis
Demographic characteristics included age, sex, education level, employment status, and internet use (frequency and device). Clinical characteristics included primary abused drug, total abstinent days from all substances in the past 28 days, and drug dependence severity assessed using Drug Abuse Screening Test-20 (15, 16).
After the intervention, the participants were asked to complete a feedback survey. Participants evaluated the usability and satisfaction of the program using the following measurements. The e-SMARPP website was evaluated using the Web Usability Scale (WUS) (17). WUS is comprised of 21 items measured on a 5-point scale (1: disagree to 5: agree) and seven subscales: ease of use, website structure, visual, response speed, favorability, helpfulness, and credibility. Subscale average scores were also calculated, with higher scores indicating higher website usability. The Client Satisfaction Questionnaire 8-item version (CSQ-8) was used to assess perceived program satisfaction and acceptance (18). CSQ-8 consists of 8 items measured on a 4-point scale. A higher score indicates higher satisfaction with service use. Demographic and clinical characteristics, WUS, and CSQ-8 were compared by the intervention condition using a t-test (the relapse prevention program vs. self-monitoring).
For an in-depth evaluation of usability and acceptability, qualitative feedback was also collected using the following semi-structured open-ended questions: general impressions of the program, the settings in which they had used the program, technical difficulties, program features they felt were the most and least useful, and suggestions for future versions of the program. We summarized the positive feedback comments by program content or function and categorized suggestions for improvements using a content analysis approach. The initial codes were extracted from text data of feedback comments. Related codes were merged together into sub-categories and main categories according to similarities. When analyzing the feedback comments, we combined the comments from both groups because this facilitated the collection of major points about how to revise the program, rather than determining the differences in responses between groups.
Results
Participant Demographics
The participants' mean age was 38.3 (SD: 7.4), and 68.5% were male. Approximately 54% were unemployed. Among the participants, 83.3% used the internet every day, and 73.2% accessed the internet using their smartphone. About half of the participants mainly used methamphetamine. There was no significant difference in the characteristics between the two groups. The participants' demographic characteristics by group are provided in the Supplementary Table 1.
Usability and Satisfaction
In total, 43 participants completed the feedback survey. Among the participants, 83.7% (n = 36) accessed e-SMARPP from their home, and 9.3% (n = 4) used it when away from home. To access e-SMARPP, 48.8% (n = 21) used their smartphone and 44.2% (n = 19) used a PC.
Table 1 shows the results of the usability and satisfaction scores of e-SMARPP by the intervention condition. All scores of the WUS subscales were over three points for program usability, except for the favorability subscale in the self-monitoring group. In the relapse prevention group, the subscales of ease of use, visual, and credibility were over four points. In the self-monitoring group, only the response speed sub-scale was over four points. The subscales of favorability and credibility of the relapse prevention group were significantly higher than those of the self-monitoring group. As for program satisfaction, the total score of CSQ-8 in the relapse prevention group was significantly higher than the self-monitoring group. Figure 1 shows detailed responses for each item in CSQ-8. All participants assigned to the relapse prevention program felt the program was “mostly satisfied” or “very satisfied.” They said they would come back to the program if they seek treatment again. Moreover, they said they would recommend it to their friends who need similar help. On the other hand, about half of the participants assigned to the self-monitoring program felt the program did not meet their needs and were less satisfied.
Table 1. Program usability and satisfaction by intervention condition.
Figure 1. Detailed program satisfaction assessed using CSQ-8.
Qualitative Feedback
The results of the qualitative feedback are shown in Tables 2, 3. Overall, the participants provided positive feedback comments, especially for therapist involvement and relapse prevention program content. Meanwhile, potential challenges and areas for improvement were revealed. Some participants stated that a smartphone App would be more useful and interesting. Non-active drug users stated the self-monitoring program was not helpful because they had already quit drugs for several months and did not feel drug cravings. We received some suggestions for a future version of e-SMARPP. As for self-monitoring, some participants suggested that structured self-monitoring for triggers would be of great help. Additional content and functions were suggested, such as relaxation exercises, information for LGBTQ, or peers' narrative stories. As for interaction, some participants proposed online sessions with therapists or peers. A portion of participants felt the 2-month program duration was short, and that a longer duration (e.g., 6 months) would be better.
Table 2. Positive feedback comments for the web-based program.
Table 3. Suggestions for a future version.
Discussion
This study evaluated the usability and acceptability of a web-based relapse prevention and self-monitoring program among 43 Japanese drug users who participated in a pilot RCT. The usability of both programs was good. The majority of the participants felt these programs were easy to use, helpful, and credible. The participants who used the relapse prevention program were satisfied with the content. However, participants who only used the self-monitoring program were likely to be unsatisfied. The participants provided helpful suggestions for a future version of e-SMARPP.
Usability
WUS results indicated that both web-based relapse prevention and self-monitoring programs were user-friendly. The interface of e-SMARPP was simple and it was easy to find desired content. In the self-monitoring calendar, the daily record was displayed in three colors (red: primary drug use, yellow: other drugs/alcohol use, blue: no drug/alcohol use). Through this display, participants could see at a glance their drug use patterns. Moreover, the participants quickly became familiar with actions on the website because automated notification emails were sent when submitting homework or receiving feedback. These functions might help to increase participant use of the programs. However, some participants felt it would be easier to use the program through a smartphone app, as apps are now increasingly common. In Japan, over 95% of people aged 20–49 use the internet, and about 80% use their smartphone when accessing the internet (19). Globally, smartphone-based applications have been developed as educational tools, self-monitoring for substance use and craving, screening and brief intervention, harm reduction strategy for people with at-risk for addictive behavior, support for recovery from addiction issues, or linkage to peers in self-help group (20–22). A smartphone app might be better if we develop an internet-based intervention with frequent use or a long intervention duration, such as daily self-monitoring. Most existing smartphone-based apps for addiction problems are for alcohol problems and few are for drug problems, and the efficacy of apps for drug problems is unclear (21, 23). Attractive features might be needed because mobile-based interventions are used less and less as time passes unless regular contact and prompts are provided (23).
Satisfaction and Acceptability
The results of the CSQ-8 and qualitative feedback suggested that participants were satisfied with the web-based relapse prevention program, and especially interactions with the therapist and program content. Previous studies revealed that therapist involvement in computer-delivered intervention improved treatment outcomes and retention (24–27). The interaction with the therapist might enhance the participants' motivation to continue treatment and to improve satisfaction. It might be a good idea to include additional interactive functions such as an online peer meeting or counseling according to the intervention purpose. There might still be room for improvement in the web-based self-monitoring program. The participants might have at times felt that the program was less helpful and insufficient to motivate behavioral change because the web-based self-monitoring program was too simple and did not include interaction with the therapist. Additional useful content (e.g., self-monitoring of drug craving and triggers, text boxes for free comments) or regular automated prompts might be needed if the web-based self-monitoring program does not include interactions with others, especially for drug users who have maintained abstinence for several months.
Limitations
There are some possible limitations to this study. First, the sample size in this study was small, and the participants were recruited in limited areas of Japan. These limitations could lead to uncertainty in the representativeness and the generalization of the results. Second, satisfaction with the web-based self-monitoring program might be affected by the primary abused drug because the participants used different drugs with different frequencies. Further study with a large sample is needed to assess the usability and acceptability of the web-based program.
Data Availability Statement
The datasets presented in this article are not readily available because we cannot share the data unless the ethics committee approves data sharing. Requests to access the datasets should be directed to Ayumi Takano, YXl1bWktdGFrYW5vJiN4MDAwNDA7dW1pbi5hYy5qcA==.
Ethics Statement
The studies involving human participants were reviewed and approved by The Ethics Committee of the Faculty of Medicine and Graduate School of Medicine of the University of Tokyo. The patients/participants provided their written informed consent to participate in this study.
Author Contributions
AT contributed to data collection and analysis and drafted the manuscript. TM approved the use of the existing face-to-face program and advised on development of the web-based program. AT and TM contributed to interpretation and critically reviewed the manuscript. All authors approved the final version of the manuscript.
Funding
This study was supported by the Japan Society for the Promotion of Science KAKENHI Grant No. JP16K20813.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
We would like to acknowledge Dr. Norito Kawakami, Dr. Yuki Miyamoto, and Dr. Tomohiro Shinozaki for the design and conducting the pilot RCT. We would also like to acknowledge Dr. Nobuya Naruse, Dr. Ohoji Kobayashi, Dr. Nozomu Hashimoto, Dr. Takashi Sunami, Dr. Satoshi Sakakibara, Dr. Arisa Kadowaki, Dr. Sachiko Yamada, and Dr. Asuka Hida for helping us to recruit patients for this study.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2021.625480/full#supplementary-material
References
1. United Nations Office on Drugs and Crime. World Drug Report 2018. United Nations Office on Drugs and Crime (2018). Available online at: https://www.unodc.org/wdr2018/prelaunch/WDR18_Booklet_2_GLOBAL.pdf (accessed January 28, 2021).
2. Boumparis N, Karyotaki E, Schaub MP, Cuijpers P, Riper H. Internet interventions for adult illicit substance users: a meta-analysis. Addiction. (2017) 112:1521–32. doi: 10.1111/add.13819
3. Moore B, Fazzino T, Garnet B, Cutter CJ, Barry DT. Computer-based interventions for drug use disorders: a systematic review. J. Subst. Abuse Treat. (2011) 40:215–23. doi: 10.1016/j.jsat.2010.11.002
4. Portnoy DB, Scott-Sheldon LJ, Johnson BT, Carey MP. Computer-delivered interventions for health promotion and behavioral risk reduction: a meta-analysis of 75 randomized controlled trials, 1988-2007. Prev. Med. (2008) 47:3–16. doi: 10.1016/j.ypmed.2008.02.014
5. Carroll KM, Ball SA, Martino S, Nich C, Babuscio A, Nuro KF, et al. Computer-assisted delivery of cognitive-behavioral therapy for addiction: a randomized trial of CBT4CBT. Am. J. Psychiatry. (2008) 165:881–8. doi: 10.1176/appi.ajp.2008.07111835
6. Carroll KM, Ball SA, Martino S, Nich C, Babuscio TA, Rounsaville BJ. Enduring effects of a computer-assisted training program for cognitive behavioral therapy: a 6-month follow-up of CBT4CBT. Drug Alcohol Depend. (2009) 100:178–81. doi: 10.1016/j.drugalcdep.2008.09.015
7. Kay-Lambkin FJ, Baker AL, Lewin TJ, Carr VJ. Computer-based psychological treatment for comorbid depression and problematic alcohol and/or cannabis use: a randomized controlled trial of clinical efficacy. Addiction. (2009) 104:378–88. doi: 10.1111/j.1360-0443.2008.02444.x
8. Kay-lambkin FJ, Baker AL, Kelly B, Lewin TJ. Clinician-assisted computerised versus therapist-delivered treatment for depressive and addictive disorders: a randomised controlled trial. Med. J. Aust. (2011) 195:S44–50. doi: 10.5694/j.1326-5377.2011.tb03265.x
9. White A, Kavanagh D, Stallman H, Klein B, Kay-Lambkin F, Proudfoot J, et al. Online alcohol interventions: a systematic review. J. Med. Internet Res. 19:e62. doi: 10.2196/jmir.1479
10. Ondersma SJ, Svikis DS, Schuster CR. Computer-based brief intervention a randomized trial with postpartum women. Am. J. Prev. Med. (2007) 32:231–8. doi: 10.1016/j.amepre.2006.11.003
11. Rooke S, Copeland J, Norberg M, Hine D, McCambridge J. Effectiveness of a self-guided web-based cannabis treatment program: randomized controlled trial. J. Med. Internet Res. (2013) 15:e26. doi: 10.2196/jmir.2256
12. Takano A, Miyamoto Y, Kawakami N, Matsumoto T. Web-based cognitive behavioral relapse prevention program with tailored feedback for people with methamphetamine and other drug use problems: development and usability study. JMIR Ment. Heal. (2016)3:e1. doi: 10.2196/mental.4875
13. Takano A, Miyamoto Y, Kawakami N, Matsumoto T, Shinozaki T, Sugimoto T. Web-based cognitive behavioral relapse prevention program with tailored feedback for people with methamphetamine and other drug use problems: protocol for a multicenter randomized controlled trial in Japan. BMC Psychiatry. (2016) 16:87. doi: 10.1186/s12888-016-0793-x
14. Takano A, Miyamoto Y, Shinozaki T, Matsumoto T, Kawakami N. Effect of a web-based relapse prevention program on abstinence among Japanese drug users: a pilot randomized controlled trial. J. Subst. Abuse Treat. (2020) 111:37–46. doi: 10.1016/j.jsat.2019.12.001
15. Shimane T, Imamura A, Ikeda K, Yamamoto M, Tsuji M, Nagayo Y, et al. Reliability and validity of the Japanese version of the DAST-20. Nihon Arukoru Yakubutsu Igakkai Zasshi. (2015) 50:310–24.
16. Skinner HA. The drug abuse screening test. Addict. Behav. (1982) 7:363–71. doi: 10.1016/0306-4603(82)90005-3
17. Nakagawa K, Suda T, Zempo H, Matsumoto K. The Development of Questionnaire for Evaluating Web Usability. Available online at: http://u-site.jp/wp-content/uploads/his_10th_paper.pdf (accessed November 1, 2020).
18. Tachimori H, Ito H. Reliabilty and validity of the Japanese version of the client satisfaction questionnaire-8. Seishin Igaku. (1999) 41:711–7.
19. Ministry of Internal Affairs and Communications (2018). Available online at: https://www.soumu.go.jp/johotsusintokei/whitepaper/ja/h30/html/nd142110.html (accessed November 1, 2020).
20. Zhang MWB, Ho RCM. Tapping onto the potential of smartphone applications for psycho-education and early intervention in addictions. Front. Psychiatry. (2016) 17:40. doi: 10.3389/fpsyt.2016.00040
21. Tofighi B, Chemi C, Ruiz-Valcarcel J, Hein P, Hu L. Smartphone apps targeting alcohol and illicit substance use: systematic search in in commercial app stores and critical content analysis. JMIR mHealth uHealth. (2019) 7:e11831. doi: 10.2196/11831
22. Colbert S, Thornton L, Richmond R. Smartphone apps for managing alcohol consumption: a literature review. Addict. Sci. Clin. Pract. (2020) 15:17. doi: 10.1186/s13722-020-00190-x
23. Kazemi DM, Borsari B, Levine MJ, Li S, Lamberson KA, Matta LA. A systematic review of the mHealth interventions to prevent alcohol and substance abuse. J. Health Commun. (2017) 22:413–32. doi: 10.1080/10810730.2017.1303556
24. Spek V, Cuijpers P, Nyklícek I, Riper H, Keyzer J, Pop V. Internet-based cognitive behaviour therapy for symptoms of depression and anxiety: a meta-analysis. Psychol. Med. (2007) 37:319–28. doi: 10.1017/S0033291706008944
25. Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn. Behav. Ther. (2009) 38:196–205. doi: 10.1080/16506070903318960
26. Rooke S, Thorsteinsson E, Karpin A, Copeland J, Allsop D. Computer-delivered interventions for alcohol and tobacco use: a meta-analysis. Addiction. (2010) 105:1381–90. doi: 10.1111/j.1360-0443.2010.02975.x
Keywords: web-based intervention, drug dependence, relapse prevention, self-monitoring, usability, acceptability
Citation: Takano A and Matsumoto T (2021) A Mixed-Methods Evaluation of Usability and Acceptability of Web-Based Relapse Prevention and Self-Monitoring Program: Secondary Analysis of a Pilot Randomized Controlled Trial. Front. Psychiatry 12:625480. doi: 10.3389/fpsyt.2021.625480
Received: 03 November 2020; Accepted: 19 January 2021;
Published: 11 February 2021.
Edited by:
Grant Christie, The University of Auckland, New ZealandReviewed by:
Mei Yang, Shenzhen Mental Health Centre, ChinaEmilien Jeannot, Centre Hospitalier Universitaire Vaudois (CHUV), Switzerland
Copyright © 2021 Takano and Matsumoto. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Ayumi Takano, YXl1bWktdGFrYW5vJiN4MDAwNDA7dW1pbi5hYy5qcA==