Evaluating the Implementation of a Mental Health App for Overseas Filipino Workers in Macao China: A Mixed-Methods Study of Stakeholders’ Perspectives

Background: Overseas Filipino workers (OFWs) is one of the largest communities of international migrant workers. They face systemic barriers to fulfilling their health needs. The COVID-19 pandemic worsened this condition and provided a context to evaluate the utility of a digital mental health intervention delivered within a stepped-care model to address OFW mental health. Using an implementation science framework, this study aimed to evaluate stakeholders’ perspectives on the implementation of Kumusta Kabayan, a mobile phone-based mental health app, for OFWs in Macao.

Methods: A mixed-methods convergent design was used by conducting two parallel steps, including quantitative and qualitative measures. The quantitative data was collected from Filipino team members and local non-governmental organization (NGO) staff members (N = 12). The qualitative data were gathered from interviews with OFWs in Macao who used the app (N = 25; 80% females, 76% domestic workers).

Results: From the online survey, the staff members of the local partner NGO and the Filipino team members strongly perceived that their organization could adapt Kumusta Kabayan to their program and generally evaluated that Kumusta Kabayan achieved its goal and was received well by OFWs. In the interviews, the OFW app users shared their experiences in using Kumusta Kabayan, which was thematically organized into six aspects of the participants’ experience: (1) promotional channel and expectation; (2) when to use the app and in what language; (3) lessons learnt; (4) memorable aspects; (5) key facilitators and barriers; and (6) suggestions.

Conclusion: Kumusta Kabayan was well accepted and shows potential to be integrated into the existing support services for OFWs in Macao. This app has the promise of being scaled-up for OFWs in other countries by collaborating with local and overseas stakeholders. Lessons learnt from this evaluation could also be implemented in wider digital mental health services in different settings.

Introduction

Overseas Filipino workers (OFWs) is one of the largest international migrant worker communities, with an estimated number of over 10 million worldwide (1). They work in both high-skill and low-skill sectors across countries to improve their lives and the livelihoods of their family members who remain in the Philippines (2). However, OFWs and other migrant workers struggle to fulfill their health needs because of systemic barriers they experience including language difficulties, adverse working conditions, lack of available healthcare providers, and inadequate social protection for migrant workers in the host countries (3, 4). Consequently, migrant workers, including OFWs, experience higher risks for developing mental disorders like anxiety and depression than local people in the host countries (5–7).

The COVID-19 pandemic worsened this condition as migrant workers were forced to take temporary unpaid leaves from employment or suddenly lost their jobs and social protections (8, 9). Additionally, OFWs and other migrants who worked as domestic workers encountered other challenges like excessive work load from their employers who began to work from home and some were not permitted to take their normal day-off due to employer’s fears of COVID-19 contamination and lack of trust in the domestic workers’ hygiene (10, 11). Due to cultural and language barriers, migrant workers were also at risk of having inadequate knowledge and awareness of the latest COVID-19 situation and policies in their host countries (12). Consequently, OFWs and other migrant workers’ mental health deteriorated and they have to survive with the limited mental health support in the host countries (13).

The inequity in healthcare access during the pandemic may be reduced by implementing digital mental health interventions within a stepped-care approach, which includes providing the least intensive intervention based on clients’ needs and monitoring them regularly (14). Digital mental health interventions, including via the internet and mobile phones, were found to be consistently effective in reducing mental disorders like depression and anxiety (15, 16). Furthermore, mobile phone-based health apps can also accurately screen mood disorder symptoms (17). Furthermore, a previous study on OFWs’ attitude toward digital health interventions found that the community showed their acceptance and willingness in accessing mobile phone-based mental health interventions (18). However, mobile phone-based mental health applications for improving mental health of OFWs and other migrant workers were still limited as previous interventions were mostly developed for inpatient adults (15).

With the limited internet and mobile phone-based mental health interventions for non-clinical groups, the World Health Organization (WHO) developed Step-by-Step, an online guided self-help intervention for people with depressive symptoms (19). Step-by-Step is an online version of the WHO’s offline transdiagnostic psychological intervention for common mental health problems named Problem Management Plus (PM+) (19, 20). The Step-by-Step website version has been piloted among Lebanese, Palestinian, and Syrian adults in Lebanon (21) and the mobile application version was tested among adult Syrian refugees residing in Germany, Sweden and Egypt (22). These studies showed that Step-by-Step was a promising stepped-care digital mental health intervention for OFWs. The Step-by-Step program was also culturally adapted for use among Filipino overseas workers.

The current study evaluated stakeholders’ perspectives on the implementation of Step-by-Step for OFWs in Macao using a mixed-methods design. The study utilized an implementation science approach, which emphasized the practical aspects of how to scale-up an evidence-based intervention (24). The current study presents the results of the implementation evaluation of the program utilizing questionnaire data collected from our local non-governmental organization (NGO) partner and e-helpers (digital lay health workers) in the Philippines as well as interviews with OFWs in Macao as end users of the Filipino version of Step-by-Step named Kumusta Kabayan, which means “Hello, fellow Filipino.” The roles of each stakeholder member are presented in Figure 1 below. Using an implementation science framework, this study was designed to accelerate the translation of evidence-based intervention into real-world system of care and address inequities in healthcare delivery for migrant workers (25, 26).

FIGURE 1

Figure 1. Stakeholder members’ roles.

Materials and Methods

This implementation science evaluation study used a mixed-methods parallel convergent design (27). The evaluation was conducted in two parallel steps, using quantitative and qualitative measures, which complemented each other and provided more comprehensive evaluation results (28). Details of the study is described in sub-sections below. Ethical approval for the study was granted from the Research Ethics Committee at the University of Macau (SSHRE19-APP074-FSS).

Sampling and Procedure

For the quantitative measure, total population sampling (29) was used where the lead author sent message invitations to the anonymous online evaluation survey to all e-helpers, clinical supervisors, staff and volunteers of the local NGO partner. The online survey for staff and volunteers of the local NGO partner were provided in English and Chinese to increase the participants’ understanding. Before completing the survey, electronic informed consent forms were provided to the participants.

For the qualitative measure, participants were recruited with a convenience census-approach sampling. All participants who participated in the Kumusta Kabayan (N = 206) and were screened with the Patient Health Questionnaire (PHQ)-9 for depressive symptoms (6) were considered as eligible respondents and were asked if they were willing to participate in the interview. Participants who either completed or did not complete the program were purposively invited to represent diverse experiences. Not all potential interviewees joined the interviews as illustrated in Figure 2.

FIGURE 2

Figure 2. Participants recruitment for interviews from both completer and non-completer groups.

Before the interviews were audio-recorded, participants were informed about the aim of interview and verbally asked for their consent. One-to-one audio interviews from October 2020 to February 2021 were conducted by e-helpers using voice over IP services. Interviews lasted an average of 25 min (SD = 8) and were conducted in English or Tagalog. Participants were compensated with a food voucher worth MOP 100 (≈USD $12) for their time. In order to improve participants’ openness, they were assigned to a different e-helper for the interview if they were previously assisted by an e-helper during the intervention period. Moreover, e-helpers were also trained for doing the interviews using a role play method prior to the data collection.

Participants

The demographic information (e.g., age and sex) of participants in the survey (N = 12) was not asked to protect their anonymity, given the small sample sizes of e-helpers and NGO partner members. Participants in this quantitative measure consisted of a team in the Philippines (n = 8), including two clinical psychologists, two Master of Psychology students, and four Bachelor of Psychology students; and a team in Macao (n = 4), including three social workers and one logistical staff.

The interviewed OFW participants’ demographic information (N = 25) was retrieved from their biodata in the app. The interviewed participants were mostly female (n = 20), with an average age of 39 years old (SD = 8), and dominated by a domestic work job (n = 19). On average, they have been working as OFWs for 59 months (SD = 78), having 66 working hours per week (SD = 16), and with a monthly salary of MOP 4,664 (SD = 1,182) (≈USD $580, exchange rate in July 2021). The interviewed participants consisted of both completers and non-completers. The demographic information of interviewees’ and non-interviewees were comparable (Supplementary Material).

Instruments

The online survey was adapted from previous studies on evaluation of innovative health programs promotion and implementation (30–33). The measures included organizational climate scale, awareness and concern scale, perceived advantage scale, perceived complexity scale, and level of success scale. Also, one open-ended item to ask for general feedback and suggestion was located at the end of the online survey (see Supplementary Material).

The 11-item organizational climate scale consisted of three parts: mission, teamwork, and infrastructure support. An item from this measure is “Our team is willing to take a chance on a good idea” that was measured with Likert’s scale from 1 (strongly disagree) to 5 (strongly agree), where higher summed scores represent more positive organizational climate in adapting a new program. This scale was given only to the team in Macao due to the nature of the items. The internal reliability (Cronbach’s alpha) of this measure was 0.94. The 9-item awareness and concern scale consisted of three parts: awareness, concern, and interest. An item example of this measure is “I am aware that Kumusta Kabayan addresses depressive symptoms.” Item responses were on a Likert-type scale from 1 (not at all true) to 4 (very true), where higher summed scores represent higher awareness and concern of participants to the program. The Cronbach’s alpha of this measure was 0.77.

The perceived advantage scale was measured with a 3-item Likert scale from 1 (strongly disagree) to 5 (strongly agree), where higher summed scores represent greater perceived level of program’s advantage. An item from this measure is “The Kumusta Kabayan program can improve the mental health of Filipino workers in Macao.” The Cronbach’s alpha of this measure was 0.71. The perceived complexity scale was measured with a single item of organizational level (Part A, “The Kumusta Kabayan would be difficult for our organization/institution to explain to potential Filipino migrant worker users”) and 2-item Likert scale (Part B) from 1 (strongly disagree) to 5 (strongly agree), where higher total score represents higher perceived complexity of the program. A sample item from Part B is “The Kumusta Kabayan program would be hard for Filipino migrant workers to understand.” The Cronbach’s alpha of this measure (Part B) was 0.74. These reliability coefficients around 0.70 are considered moderately reliable (34). The last question, level of success, asked the participants to choose one of eight continuum options based on their experience and observation. It started from Option 1 (The Kumusta Kabayan program not only failed to meet its goals, but it caused a loss of resources and create other problems) to complete successful perception of Option 8 (The Kumusta Kabayan program fulfilled all of its goals and also provided other benefits for Filipino migrant workers and/or for other stakeholders).

For the qualitative measure, all e-helpers followed the interview guide (Supplementary Material) that was developed for this study. The guideline was pilot tested among e-helpers and with three participants. During the interviews, e-helpers introduced themselves and asked for participants’ verbal consent. Consented participants were asked about the following themes: referral channel and initial thoughts about the program, period of use, lessons learnt, key facilitators and barriers, and improvement suggestions for the program.

Data Analysis

Results from the online survey were descriptively analyzed. The inferential analysis was not conducted due to the limited number of responses and demographic variables of online survey participants were not collected to protect participants’ anonymity. Data from the research team in Macao and the Philippines were visually compared to show any contrast or similarity between the stakeholders in evaluating the program.

Interview audio-recordings were transcribed by the e-helper who did the interview within 2 weeks from the interview then sent to the clinical supervisor and research coordinators. Following a Rapid Evaluation and Assessment Method (35) that was used in a similar evaluation study (36), the lead author with a doctoral degree in psychology and mixed-method research experience read each transcript and summarized the information into a table for each participant using a deductive content analysis approach (i.e., promotional channel and initial expectation). This summary table was sent to other research team members and clinical supervisors for iterative analysis and discussion using an inductive content analysis method, in which the interpretation focused on the explicit data given by the participants (37, 38). Steps in this method included open coding, creating categories, and abstraction for each aspect. Data saturation was reached in the interview with the 14th participant where the responses shared similarity with previous responses. The Consolidated Criteria for Reporting Qualitative Studies (COREQ) (39) was used as a guideline for the qualitative portion of the study.

Reflexivity

This study was part of an ongoing implementation study that involved research team members from multiple sectors and countries that might have different perspectives and cultural backgrounds. Therefore, at the beginning of the study, regular intensive meetings were conducted to build mutual understanding and improve transparency, including the challenges that might be faced and alternative solutions. Along the way, regular meetings were also conducted among e-helpers, clinical supervisors, and research coordinators to discuss any experiences and reflective thoughts during the interviews. The research coordinators emphasized that all opinions shared within the meeting were valid and important to encourage members to share both positive and negative experiences. The e-helpers also had their regular meetings with their clinical supervisor, in which the information in the minutes could be anonymized when necessary so they could openly share their thoughts without worry of being judged. The same step was also conducted when collecting quantitative feedback by using an anonymous online survey.

Results

Findings were presented in the two sub-sections below. The quantitative results from local NGO staff members and research team members in the Philippines (N = 12) cover organizational climate, awareness and concern, the app’s advantage and complexity, and level of success. Findings from the interviews with OFWs in Macao (N = 25) cover six aspects from promotional channel and expectation to suggestions.

Online Evaluation of Local and Overseas Partner Member Perspectives

Table 1 displays the quantitative findings from the local NGO partner and overseas research team members. The median score of organizational climates from the teams in Macao and the Philippines are 4 and 5, respectively. The local partner NGO staff members and overseas members strongly perceived that their organization could positively incorporate Kumusta Kabayan into its program. However, staff members in Macao indicated their worry on infrastructure support, with a relatively lower score on this sub-scale than on other sub-scales and when compared with e-helpers in the Philippines. On the awareness and concern, both groups were highly aware of the app’s purpose, with score median of 4 in Macao and the Philippines. Both groups agreed that Kumusta Kabayan would be helpful for the OFWs. However, their perception toward the complexities of the app was ambiguous. For level of success, the teams in Macao and the Philippines generally evaluated that Kumusta Kabayan had achieved its goal.

TABLE 1

Table 1. Characteristics of participants in online survey.

Interviews With Overseas Filipino Workers in Macao

Findings from the interviews with the completers and non-completers are organized into six aspects of the participants’ experience, including (1) promotional channel and expectation; (2) when to use and in what language; (3) lessons learnt; (4) memorable aspects; (5) key facilitators and barriers; and (6) suggestions. Relatable quote(s), regardless of participants’ completion status, were provided for each aspect with information of participant’s number at the end of the quote.

Promotional Channel and Expectation

Participants learned about the program from several promotional channels including social media like Facebook and flyers that were distributed in churches and restaurants where OFWs gather on weekends (Supplementary Material). Some of them read the news about about Kumusta Kabayan’s launching and talked about it with friends. Participants were also informed about the program from The Philippines Consulate in Macau and the local NGO partner, Caritas Macau. Participants commonly expected that the program could assist them in managing emotions, sharing experiences, or connecting with other OFWs, particularly because they recognized the name of the program in Tagalog. However, some participants who learned about the program from the Consulate thought that Kumusta Kabayan was part of the employment and visa process or it would provide monetary aid.

“I just knew about this application from my friends here in Macao so I installed it like them. My first impression about the Kumusta Kabayan is that it would be helpful for us, our fellow Filipinos to check up on each other, to check up on how they are doing. At first, I really do not know. But, while reading the questionnaires, it is really related to the lives of the OFW and also to our personal life, so in a way it touches us.” (Participant #94).

When to Use and in Which Language

Participants who completed the program typically used the app in the morning before work, sometimes in the afternoon, or at night before sleeping, depending on their free time. They used the mobile app almost every day to check for new content or use the mood tracker. Participants who partially completed or discontinued the program used the app only on their day-off on weekends. However, during weekends, they still had difficulties finding time to use the app as they were also busy with extra works or were too tired. Both English and Filipino language versions were used by participants. Some words related to psychological questions were reported hard to understand, especially in English.

“Sometimes I was able to open it but I did not continue answering… Recently I wasn’t able to use it because I got busy with my work.… When I answer it, when I want to finish the questionnaire, there are times that I wasn’t able to finish when my boss calls me, and also when their child [who I take care of] needs me. I answer it during my rest hour. When the child is sleeping, that’s when I open it.” (Participant #28).

“English, for me it’s easy. But, I can’t speak for other. Ay Ineng [young girl], I don’t want to sound overconfident. But, I somehow understand the questions since I’m also a college graduate. That means I understand it. But, for those that didn’t aim a higher degree, they may not be able to answer it well. I’m not generalizing. But, in my opinion, the question and the English are okay. It’s easier for me to understand if it’s in English.” (Participant #40).

Lessons Learnt

Participants explained some lessons that they learnt from the program. The basic lesson is doing reflective thinking or self-checking and managing their emotions and feelings. Participants learnt how emotions and feelings are connected to their physical state so they also understand why doing exercise is important. When feeling stressed, participants learnt from the app to divide their focus and to do self-care by doing activities they enjoy like cooking or applying breathing exercise. Moreover, they learnt the importance of making, opening themselves to others, and finding support from the local society.

“Before when I hadn’t installed the Kumusta Kabayan, I was always stressed. I don’t go out of the boarding house. I just go from work to home all the time. But after opening Kumusta Kabayan, it was a huge help for me especially the things I’m reading from the app, including the questions from Kumusta Kabayan. There were times, when I was reading the questions or stories, then suddenly I started crying. I said to myself “This Kumusta Kabayan is such a big help for me because before I’ve forgotten to interact with other people. But this time, I’m doing well. Now I’m able to socialize with other people all because of Kumusta Kabayan.” (Participant #207).

Memorable Aspects

Participants felt grateful for the program that they never knew existed before joining the study. The name of the program, Kumusta Kabayan, also made it easy for participants to understand the program’s aim. They praised the story and the characters that highly resembled participants’ lives as OFWs. Having someone like them overcome life’s challenges as an OFW made them hopeful and boosted their confidence. The psychological questions encouraged participants to reflect and disclose their feelings to their friends, including things they never told others before. From the content, participants remember breathing exercises that they could apply in their daily lives, especially when they get stressed.

“What I like about the app are the stories that are really, sometimes it gives me a boost of self-confidence most especially that we live alone and slowly losing hope. But after reading the stories there is still hope. Something like that, we just have to keep fighting. You can do it. It’s just a problem and it will eventually end. Just like the saying that goes in every problem there is a solution.” (Participant #50).

“Contents related to my experiences, experiences of an OFW… about missing your family, and also stress.… and there were times when you encountered a misunderstanding with your employer. Something like that, those kinds of situations. I liked the story, I feel excited also. The character [Ate Sonia] is very inspiring too. The activity is very relaxing, especially the exercising part, Inhale-Exhale [breathing exercise]. It helped me relax my mind.” (Participant #194).

Key Facilitators and Barriers

The main facilitators for successful implementation of Kumusta Kabayan were the accessibility of this app on Play Store and App Store, and the app’s small size. On another side, four key barriers for participants using the app were recognized. First is the time barrier that the majority of participants were migrant domestic workers with only one day-off. Some of them had extra work to do during their day-off and some used it for resting, and they had minimal time to prioritize the app. Second are technological barriers including lack of phone memory to accommodate the app, incompatible operating system, poor signal reception, limited or no mobile data, limited or unstable Wi-Fi, and technical glitch where the app always return to the start and did not record their responses to questionnaires. Third is the lack of the digital literacy where participants changed their phone or formatted it so they had to re-install the app. However, they could not re-install by themselves and needed assistance by research assistants. Fourth are human factors including forgetting their username or password that was case sensitive, changing phone numbers so they could not receive access code when they forgot username or password, and psychological barriers because the questions were triggering of homesickness, as explained by Participant 94 below:

“… while answering the questionnaire, we are also in another country, it feels difficult, you really take in the emotions of each question that you will be answering. It is like you are going back to the time before you leave the Philippines, you are thinking about the past and how can you be better compared before.” (Participants #94).

Suggestions

Participants suggested several points for the improvement of Kumusta Kabayan, namely (1) connectivity, (2) connectedness, (3) content, (4) feedback or brief explanation on participants’ responses to the questionnaires, (5) add-ons, and (6) promotional methods. First was about connectivity, which they recommended to connect the app with their social media account so they do not need to remember another username and password. Participants also shared their expectation for Kumusta Kabayan that does not require an internet connection because they did not always have mobile data or Wi-Fi. However, on another side, some participants disclosed that their mobile phone memory was full so they had to use the website version. Second was about connectedness, where participants would like to see a video call feature with the e-helper and group chat feature with other participants. This suggestion was motivated by participants’ need to be connected and to share stories with other OFWs and people in their home country, which would ease their homesickness. Third was about the content, which participants suggested to also integrate their positive and uplifting and positive experiences of being OFWs besides the struggles to balance the negative feelings as illustrated below. In addition, participants who came from various occupational backgrounds recommended making several storylines and characters that users could choose base on their type of job. Fourth was about feedback or brief explanation on their responses to the questionnaires so they could understand their condition. Fifth was about add-ons, which participants suggested to add entertainment features and news on the app so they still could use the app while waiting for the next session to be opened. Sixth was about the promotional methods, which participants suggested to involve the OFWs who are also YouTubers or TikTokers to promote Kumusta Kabayan because many OFWs watch these social media celebrities.

“… maybe give many examples of the OFWs’ lives, not only the sad stories.… they should put stories about the bright side of being an OFW so that people will be encouraged to use the application.… because sometimes when the story is sad, the delivery of it is also negative, it should be something nice too.” (Participant #28).

Discussion

This study evaluated the stakeholders’ perspectives on the Kumusta Kabayan implementation for OFWs in Macao using a mixed-methods design. Three stakeholders were involved in this study: staff members of local NGO partner, research team members in the Philippines, and OFWs in Macao as the app users. The quantitative measure among NGO staff members and Filipino team members found that Kumusta Kabayan was perceived fit to be integrated into their organizations. In particular, the NGO staff members strongly agreed that Kumusta Kabayan was in line with the NGO’s aim of assisting migrant workers.

On the other hand, infrastructure support was a concern to both NGO staff members and Filipino members. Infrastructure support covered the available organization’s equipment and technology, human resources, and financial situation. Their concerns might be associated with the sustainability of the program as staff numbers in the NGO were limited and Filipino e-helpers were volunteer students. Moreover, the app maintenance fee was relatively high (≈USD $2,367/year) for the NGO.

Generally, the NGO staff members and overseas members perceived that Kumusta Kabayan achieved its goals. From the survey, some Filipino members chose an evaluation option that despite this success, the app caused other problems. This finding might be associated with the difficulty experienced by users when they forgot their passwords. For safety reasons, a one-time-password (OTP) would be sent to user’s phone number registered in the system if user forgot their password. However, some app users changed mobile phone numbers during the program so they could not retrieve the OTP and could not access their account privacy settings. This unfortunately led to users being unable to use the app in some instances. Future iterations of the application should provide easier and more stable access to the application.

Moreover, the OFWs also used a variety of smartphone brands and not all of the operating systems were compatible with the app. Users with inadequate digital literacy could not follow the instructions to check their smartphone configuration nor to download and install the app directly which frustrated them. This finding reflected similar results in Step-by-Step testing among refugees in European countries where technical literacy was the main barrier for them for accessing the app (22). Therefore, OFWs in the interviews suggested an easier log-in method such as linking it with their social media accounts. However, this recommendation should be thought through carefully so it will not compromise user’s data privacy. An alternative is posting FAQs, including the steps in accessing the app, in the Kumusta Kabayan program’s social media account.

Users also suggested that video calls and group chat features should be incorporated into the app. This suggestion might be driven by social isolation and loneliness among people from communal cultures (40). Moreover, the mobility restriction during the pandemic amplified the social isolation because migrant workers, particularly the migrant domestic workers, were not allowed by their employers to take the day-off or asked to stay at home during their day-off (9). This also explains why users were very touched and appreciative of the storyline because it was relatable and educational. This finding highlighted the importance of cultural adaptation as the storyline was developed carefully by involving the OFWs themselves in the adaptation process (23). However, from the interviews it was found that some users would like to see more characters from different types of jobs because of different challenges faced by these OFWs.

In the interviews we also explored the main barriers for OFWs when using the app, including from the users who discontinued using it. The primary reason was they did not have time to use the app because they were busy with their jobs, which some also did extra work during their day-off. The demographic information showed that our participants could work up to 14 h per day on average. OFWs who worked as domestic workers had difficulty opening the app because they were also taking care of the children or older adults in the family. This finding supported previous study on migrant workers’ health that migrant domestic workers were at high risk for injuries, mental problems, and low quality of life because of long working hours, low wages, and inadequate health insurance (41). Furthermore, the working condition of OFWs shared by the users also explained their suggestion about entertainment add-ons so they could relax when using the app. Moreover, many users opened the app at night after long working hours and could not concentrate on reading on their screen. Therefore, a format of video or animation could be an alternative in delivering the content for OFWs.

Lastly, the interviews enlightened us about the promotional channels and methods. It was clear that using social media as an online method and face-to-face promotion at church and restaurants worked well in recruiting potential users. Moreover, collaboration with the local NGO that was well-known for its programs in assisting migrant workers enhanced the trust of OFWs for the app. However, when the app was launched and covered by the mass media, the local people showed divided attitude toward this program as some viewers commented that the citizens should be taken care more than the foreigners [i.e., (42)]. This resistance and sentiment toward migrant workers confirmed a previous study on fear and disgust toward migrant workers, particularly domestic workers, as their professions were perceived as low-end 3D jobs (dangerous, difficult, and dirty) (43). The insight emerged from the interviews was to collaborate with OFWs who were popular YouTubers and TikTokers to promote the app on their channels. Also, the interviews revealed that collaborations with the Government’s representatives either from the host country or sender country, was like a double-edged sword because it could reach many OFWs but not all of them trust these entities and were thinking that the app was a surveillance program.

Implications for Practice and Future Studies

The findings from this study could be extended to the improvement of mobile mental health apps implementation, particularly for migrant workers. Moreover, lessons learnt from this evaluation could also be useful to be implemented in wider digital mental health services for other group of users. For example, the very appreciated storyline and characters were designed from a cultural adaptation process, which also being a critical step for in designing digital health intervention for Chinese young adults (44). The local partner and Filipino e-helpers contributed substantially to the successful implementation of Kumusta Kabayan. Therefore, it is also important to involve the stakeholders at the earliest possible time during the planning phase, to minimize the potential barriers in implementing digital mental health. Also, it is crucial to assess the digital literacy level of potential users when designing the app as having a smartphone does not guarantee that the owner could use it easily.

Another practical implication, the users’ recommendations of adding additional features including feedback on questionnaires, video chat, group chat, entertainment, and news could be integrated with gamification principles to improve user engagement and increase retention (45). For example, an app user might creatively design an avatar for their profile picture that would increase the connectedness to the app. Leaderboard and progress bar (46) could also be introduced to encourage users to complete the challenges during the sessions such as planning and executing the social activities as a part of behavioral activation intervention. Users might collect virtual coins from completing activities or just from opening the app that these coins can be used to access the suggested add-ons. Future studies could investigate the effectivity of these gamification principles in mobile phone app for migrant workers. Additionally, a future study could explore the virtual connections among app users within the app online community using a digital ethnography method (47).

Limitations

This study has two limitations. First, despite its anonymity, evaluation from the local and overseas partners relied on self-report questionnaires that might lead to social desirability. Second, despite data saturation and engagement with discontinued users, evaluation from the OFWs might be affected by self-selection bias as disappointed users might ignore the interview invitation so their perspective was less represented. Regardless of these limitations, the quantitative and qualitative findings in our study were complimentary of one another and provided key directions to improve subsequent implementation of Kumusta Kabayan among OFWs in Macao and elsewhere.

Conclusion

This study reported the implementation evaluation of Kumusta Kabayan, a mobile mental health app for OFWs, from the perspective of stakeholders including staff members of local NGO as a collaborator, team members in the Philippines, and OFWs in Macao as the app users. The Kumusta Kabayan program was accepted by the users because the character’s story highly resonated with their lives as OFWs and the app was relatively easy to use. The Kumusta Kabayan has the potential of being integrated into the existing support services for migrant workers in Macao by collaborating with the local NGO stakeholder. This app could be scaled-up for OFWs in other countries by paying attention to the key facilitators, barriers, and the suggestions shared by the users that reported in this study.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by the ethical approval for the study was granted from the Research Ethics Committee at the University of Macau (SSHRE19-APP074-FSS). The patients/participants provided their written informed consent to participate in this study.

Author Contributions

AL, HS, and BH: study conceptualization and design. AIFL and BH: secured funding. AL, AIFL, BH, HS, KP, MG, and SB: data collection and final draft writing, review, and editing. AL, KP, and MG: data analysis. AL: draft manuscript writing. All authors contributed to the article and approved the submitted version.

Funding

BH and AIFL were supported by the Research Services and Knowledge Transfer Office (RSKTO), University of Macau and the Macau Foundation. Funding for this work was provided by the Li Ka Shing Foundation.

Conflict of Interest

BH is co-editing the Frontiers special topic “Global mental health among marginalized communities in pandemic emergencies” but was not directly involved in the review of this article.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We thank Caritas Macau, the Macau Foundation, Centre for Macau Studies – University of Macau, and the Philippine Consulate General in Macau (SAR) for their support and partnership in this research; also Hybrid Heroes, the application developer team for the technical assistance and research assistants at the Global and Community Mental Health Research Group for their support in the research and team members from De La Salle University, particularly Professor Allan B. Bernardo, Alma Grace De Vera, Andrae Joseph Yap, Angelica Julienne Tan, Genevive Koa Chua, Jamilla Marie B. Luces, Jesus Antonio D. Marcelo, and Michaella Lobaton. We also thank the overseas Filipino migrant workers in Macao for their time, effort, and guidance in the adaptation of Step-by-Step.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2022.836156/full#supplementary-material

References

1. Cleofas JV, Eusebio MCSC, Pacudan EJP. Anxious, apart, and attentive: a qualitative case study of overseas Filipino workers’ families in the time of COVID-19. Fam J. (2021). doi: 10.1177/10664807211006339 [Epub ahead of print].

CrossRef Full Text | Google Scholar

2. Garabiles MR, Ofreneo MAP, Hall BJ. Towards a model of resilience for transnational families of Filipina domestic workers. PLoS One. (2017) 12:e0183703. doi: 10.1371/journal.pone.0183703

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Hall BJ, Garabiles MR, Latkin CA. Work life, relationship, and policy determinants of health and well-being among Filipino domestic workers in China: a qualitative study. BMC Public Health. (2019) 19:229. doi: 10.1186/s12889-019-6552-4

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Hall BJ, Yang X, Huang L, Yi G, Chan EW, Tucker JD, et al. Barriers and facilitators of rapid HIV and syphilis testing uptake among Filipino transnational migrants in China. AIDS Behav. (2019) 24:418–27. doi: 10.1007/s10461-019-02449-0

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Garabiles MR, Lao CK, Yip PSY, Chan EWW, Mordeno IG, Hall BJ. Psychometric validation of PHQ–9 and GAD–7 in Filipino migrant domestic workers in Macao (SAR), China. J Pers Assess. (2019) 102:833–44. doi: 10.1080/00223891.2019.1644343

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Hall BJ, Patel AR, Lao L, Liem A, Mayawati EH, Tjipto S. Structural validation of the patient health questionnaire-9 (PHQ-9) among Filipina and Indonesian female migrant domestic workers in Macao. Psychiatry Res. (2021) 295:1–7. doi: 10.1016/j.psychres.2020.113575

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Liem A, Renzaho AMN, Hannam K, Lam AIF, Hall BJ. Acculturative stress and coping among migrant workers: a global mixed-methods systematic review. Appl Psychol Health Well Being. (2021) 13:491–517. doi: 10.1111/aphw.12271

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Carvalho R. Stressed, Isolated: Migrants Face Increased Mental Health Risks Amid Coronavirus Crisis. South China Morning Post. (2020). Available online at: https://www.scmp.com/week-asia/people/article/3052929/stressed-isolated-migrants-face-increased-mental-health-risks-amid (accessed February 20, 2020).

Google Scholar

9. Liem A, Wang C, Wariyanti Y, Latkin CA, Hall BJ. The neglected health of international migrant workers in the COVID-19 epidemic. Lancet Psychiatry. (2020) 7:e20. doi: 10.1016/S2215-0366(20)30076-6

CrossRef Full Text | Google Scholar

10. Carvalho R. ‘It’s Chaos’: Hong Kong’s Domestic Workers Call for Help Amid Coronavirus Outbreak. South China Morning Post. (2020). Available online at: https://www.scmp.com/print/week-asia/health-environment/article/3048893/its-chaos-hong-kongs-domestic-workers-call-help-amid (accessed February 8, 2020).

Google Scholar

11. Kaur-Gill S, Qin-Liang Y, Hassan S. Negotiating mental health during the COVID-19 pandemic: performing migrant domestic work in contentious conditions. Am Behav Sci. (2021) 65:1406–25. doi: 10.1177/00027642211000394

CrossRef Full Text | Google Scholar

12. Liem A, Wang C, Dong C, Lam AIF, Latkin CA, Hall BJ. Knowledge and awareness of COVID-19 among Indonesian migrant workers in the greater China region. Public Health. (2021) 197:28–35. doi: 10.1016/j.puhe.2021.05.029

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Fan BE. Migrant workers with COVID-19: recognizing the crucial role non-governmental organizations perform. Lancet Reg Health West Pac. (2021) 9:100145. doi: 10.1016/j.lanwpc.2021.100145

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Proudfoot J, Gale N, O’Moore K, Faraj M, Gieng C, Anderson J. Implementation of an augmented stepped mental health care service in Australian primary care: a mixed method study. Adv Ment Health. (2021) 19:75–93. doi: 10.1080/18387357.2019.1662318

CrossRef Full Text | Google Scholar

15. Hennemann S, Farnsteiner S, Sander L. Internet- and mobile-based aftercare and relapse prevention in mental disorders: a systematic review and recommendations for future research. Internet Interv. (2018) 14:1–17. doi: 10.1016/j.invent.2018.09.001

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Lecomte T, Potvin S, Corbière M, Guay S, Samson C, Cloutier B, et al. Mobile apps for mental health issues: meta-review of meta-analyses. JMIR Mhealth Uhealth. (2020) 8:1–14. doi: 10.2196/17458

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Burchert S, Kerber A, Zimmermann J, Knaevelsrud C. Screening accuracy of a 14-day smartphone ambulatory assessment of depression symptoms and mood dynamics in a general population sample: comparison with the PHQ-9 depression screening. PLoS One. (2021) 16:e0244955. doi: 10.1371/journal.pone.0244955

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Hall BJ, Shi W, Garabiles MR, Chan EWW. Correlates of expected eMental health uptake among Filipino domestic workers in China. Glob Ment Health. (2018) 5:e33–46. doi: 10.1017/gmh.2018.25

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Carswell K, Harper Shehadeh M, Watts S, van’t Hof E, Abi Ramia J, Heim E, et al. Step-by-step: a new WHO digital mental health intervention for depression. Mhealth. (2018) 4:1–7. doi: 10.21037/mhealth.2018.08.01

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Sijbrandij M, Acarturk C, Bird M, Bryant RA, Burchert S, Carswell K, et al. Strengthening mental health care systems for Syrian refugees in Europe and the Middle East: integrating scalable psychological interventions in eight countries. Eur J Psychotraumat. (2017) 8(Suppl. 2):1–11. doi: 10.1080/20008198.2017.1388102

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Harper Shehadeh MJ, Abi Ramia J, Cuijpers P, El Chammay R, Heim E, Kheir W, et al. Step-by-step, an e-mental health intervention for depression: a mixed methods pilot study from Lebanon. Front Psychiatry. (2020) 10:986. doi: 10.3389/fpsyt.2019.00986

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Burchert S, Alkneme MS, Bird M, Carswell K, Cuijpers P, Hansen P, et al. User-centered app adaptation of a low-intensity e-mental health intervention for Syrian refugees. Front Psychiatry. (2019) 9:663. doi: 10.3389/fpsyt.2018.00663

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Garabiles MR, Harper Shehadeh M, Hall BJ. Cultural adaptation of a scalable World Health Organization e-mental health program for overseas Filipino workers. JMIR Form Res. (2019) 3:e11600–12. doi: 10.2196/11600

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Theobald S, Brandes N, Gyapong M, El-Saharty S, Proctor E, Diaz T, et al. Implementation research: new imperatives and opportunities in global health. Lancet. (2018) 392:2214–28. doi: 10.1016/S0140-6736(18)32205-0

CrossRef Full Text | Google Scholar

25. Baumann AA, Cabassa LJ. Reframing implementation science to address inequities in healthcare delivery. BMC Health Serv Res. (2020) 20:190. doi: 10.1186/s12913-020-4975-3

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Fogarty International Center. Toolkit Part 1: Implementation Science Methodologies and Frameworks. (2016). Available online at: https://www.fic.nih.gov/About/center-global-health-studies/neuroscience-implementation-toolkit/Pages/methodologies-frameworks.aspx (accessed March 25, 2021).

Google Scholar

27. Liem A, Garabiles MR, Pakingan KA, Chen W, Lam AIF, Burchert S, et al. A digital mental health intervention to reduce depressive symptoms among overseas Filipino workers: protocol for a pilot hybrid type 1 effectiveness-implementation randomized controlled trial. Implement Sci Commun. (2020) 1:1–16. doi: 10.1186/s43058-020-00072-y

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Creswell JW, Creswell JD. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Singapore: SAGE (2018).

Google Scholar

29. Etikan I, Musa SA, Alkassim RS. Comparison of convenience sampling and purposive sampling. Am J Theor Appl Stat. (2016) 5:1–4. doi: 10.11648/j.ajtas.20160501.11

CrossRef Full Text | Google Scholar

30. Aarons GA, Cafri G, Lugo L, Sawitzky A. Expanding the domains of attitudes towards evidence-based practice: the evidence based practice attitude scale-50. Adm Policy Ment Health. (2012) 39:331–40. doi: 10.1007/s10488-010-0302-3

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Chinman M, Young AS, Rowe M, Forquer S, Knight E, Miller A. An instrument to assess competencies of providers treating severe mental illness. Ment Health Serv Res. (2003) 5:97–108. doi: 10.1023/a:1023281527952

CrossRef Full Text | Google Scholar

32. Schoenwald SK, Chapman JE, Kelleher K, Hoagwood KE, Landsverk J, Stevens J, et al. A survey of the infrastructure for children’s mental health services: implications for the implementation of empirically supported treatments (ESTs). Adm Policy Ment Health. (2008) 35:84–97. doi: 10.1007/s10488-007-0147-6

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Steckler A, Goodman RM, McLeroy KR, Davis S, Koch G. Measuring the diffusion of innovative health promotion programs. Am J Health Promot. (1992) 6:214–24. doi: 10.4278/0890-1171-6.3.214

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Tavakol M, Dennick R. Making sense of Cronbach’s alpha. Int J Med Educ. (2011) 2:53–5. doi: 10.5116/ijme.4dfb.8dfd

PubMed Abstract | CrossRef Full Text | Google Scholar

35. McNall M, Foster-Fishman PG. Methods of rapid evaluation, assessment, and appraisal. Am J Eval. (2007) 28:151–68. doi: 10.1177/1098214007300895

CrossRef Full Text | Google Scholar

36. Dopp AR, Mapes AR, Wolkowicz NR, McCord CE, Feldner MT. Incorporating telehealth into health service psychology training: a mixed-method study of student perspectives. Dig Health. (2021) 7:1–15. doi: 10.1177/2055207620980222

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Elo S, Kyngäs H. The qualitative content analysis process. J Adv Nurs. (2008) 62:107–15. doi: 10.1111/j.1365-2648.2007.04569.x

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Watfern C, Heck C, Rule C, Baldwin P, Boydell KM. Feasibility and acceptability of a mental health website for adults with an intellectual disability: qualitative evaluation. JMIR Ment Health. (2019) 6:e12958. doi: 10.2196/12958

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. (2007) 19:349–57. doi: 10.1093/intqhc/mzm042

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Hudiyana J, Lincoln TM, Hartanto S, Shadiqi MA, Milla MN, Muluk H, et al. How universal is a construct of loneliness? Measurement invariance of the UCLA loneliness scale in Indonesia, Germany, and the United States. Assessment. (2021). doi: 10.1177/10731911211034564 [Epub ahead of print].

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Hargreaves S, Rustage K, Nellums LB, McAlpine A, Pocock N, Devakumar D, et al. Occupational health outcomes among international migrant workers: a systematic review and meta-analysis. Lancet Glob Health. (2019) 7:e872–82. doi: 10.1016/S2214-109X(19)30204-9

CrossRef Full Text | Google Scholar

42. TDM. Non-Local Employees Seeking Help From Financial Difficulties Increase. (2020). Available online at: https://www.facebook.com/tdmchinesenews/videos/%E8%BF%8E%E9%9B%81%E8%BB%92%E5%A4%96%E5%9C%B0%E5%83%B1%E5%93%A1%E7%B6%93%E6%BF%9F%E5%9B%B0%E9%9B%A3%E6%B1%82%E5%8A%A9%E5%A2%9E/632647380691255/ (accessed August 8, 2020).

Google Scholar

43. Shi W. Cultural politics of emotions in households: migrant domestic workers in Macau. Inter Asia Cult Stud. (2017) 18:464–81. doi: 10.1080/14649373.2017.1387090

CrossRef Full Text | Google Scholar

44. Sit HF, Ling R, Lam AIF, Chen W, Latkin CA, Hall BJ. The cultural adaptation of step-by-step: an intervention to address depression among Chinese young adults. Front Psychiatry. (2020) 11:650. doi: 10.3389/fpsyt.2020.00650

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Litvin S, Saunders R, Maier MA, Lüttke S. Gamification as an approach to improve resilience and reduce attrition in mobile mental health interventions: a randomized controlled trial. PLoS One. (2020) 15:e0237220. doi: 10.1371/journal.pone.0237220

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Cechetti NP, Bellei EA, Biduski D, Rodriguez JPM, Roman MK, De Marchi ACB. Developing and implementing a gamification method to improve user engagement: a case study with an m-health application for hypertension monitoring. Telemat Inf. (2019) 41:126–38. doi: 10.1016/j.tele.2019.04.007

CrossRef Full Text | Google Scholar

47. Oreg A, Babis D. Digital ethnography in third sector research. Voluntas. (2021). doi: 10.1007/s11266-021-00397-9 [Epub ahead of print].

CrossRef Full Text | Google Scholar