Healthcare professionals' perception and satisfaction with mental health tele-medicine during the COVID-19 outbreak: A real-world experience in telepsychiatry

Background: The use of telemedicine is increasingly being implemented, showing numerous benefits over other methods. A good example of this is the use of telemedicine following the breakdown caused by the COVID-19 pandemic. Previous experiences with telemedicine (TM) have not been significantly explored in relation to the professionals' own perspectives.

Objective: Identify and explore the perceptions and interests of mental health professionals who have performed TM during the period of pandemia.

Methods: A questionnaire on mental health professionals' perceptions of and satisfaction of TM, the Font Roja Work Satisfaction Questionnaire, was adapted and used. Data collected included 112 Psychiatric Service professionals who conducted TM in March 2020, after the country had been under lockdown for 10 weeks. Over 12.000 medical consultations were carried out by the phone, showing an overwhelming response to this method.

Results: High levels of satisfaction were recorded amongst professionals. TM would function as a complement to the traditional system of face-to-face visits (n-112, f-109, 96.5%). Only 9.7% (f-11) believed that digital or virtual interventions would completely replace face-to-face visits. 60.8% did not consider this monotonous work. The older the health workers were, the more satisfied they felt during their follow-up telephone consultation. The greater the previous experience, the more satisfaction was shown. There were gender differences: female mental health workers reported a greater level of comfort.

Conclusion: TM can be implemented with less effort, but it requires time, methods, and resources to be managed. Satisfaction among professionals is high, especially among those with more clinical experience. Patient satisfaction must be contrasted against this.

Introduction

The COVID-19 pandemic is the most critical public health issue this society has faced in the last 50 years. By October 27, 2021, more than 244 million people have been infected worldwide (5006675 in Spain), with 4968445 deaths (1, 2). This eventful issue forced mental health services to quickly modify their usual procedures to face the new situation (3, 4). The National Health System in Spain is made up of a set of health services dependent on public authorities. It is free and universal: consultations, access to emergencies and medicines are free. For this reason, the majority of Spaniards turn to the public health system on a regular basis. The method usually used is face-to-face attendance. Telemedicine (TM) has become an essential tool to guarantee mental health in this new situation (4, 5). However, it must be carefully analyzed to be considered effective in the future because as experiences in this field are remain limited (6, 7), even though they have increased in recent years. Little is known about the subjective perception, satisfaction, and interest in TM of mental health professionals who have conducted it both in a the pre-pandemic context (8) and during the pandemic outbreak (4, 5). Knowing professionals' perspectives is valuable since they diagnose and treat diseases based on the resources, they have access to (9–11). Based on these facts, TM's use should be planned and implemented for future scenarios or clinical development for the post-COVID-19 stage.

There are increasingly more studies based on the assistance provided during the pandemic COVID-19 (12–14) however, this study highlights the point of view of the professionals who have performed these services. From this perspective, gender differences among mental health professionals still remain under-researched and under-reported, despite the fact that most of the task force in mental health are women (2).

This study describes how mental health professionals of the Psychiatry Department at the University of Salamanca Health Care Complex (USHCC) adapted and perceived TM after switching from the traditional face-to-face model to the new widely-available TM system on March 16 2020, after the lockdown that lasted 10 weeks (15). We also focus on socio-demographic and gender differences, factors that must be analized (16). This point of view is interesting because it helps to analyze the perceived self-efficacy of the workers, and the difficulties they encounter in their work, and contributes to improving the quality of the services provided.

Based on previous information, the main hypotheses are:

- Telemedicine acceptance and satisfaction will increase among professionals exposed to this modality (way of working).

- Professionals carried out tele-matic consultations (telemedicine) will be more willing to use it as part of their future practice.

- Professionals' perception of patients' satisfaction will be more significant in patients attended by more satisfied and experienced professionals.

- Age or gender differences exist in professional satisfaction when applying telemedicine.

Materials and methods

Data were obtained from a sample of mental health professionals of the Psychiatry Service (PS) of the USHCC, including 45 different medical services and more than 900 beds. The PS includes most Salamanca Mental Health Network resources, including part of the drug dependence care network, community outpatient services, inpatient units and community flats. It also includes psychiatric emergency services and on-call doctors at the University Hospital (UH), plus a transversal program for suicide prevention and attention for refractory mental disorders.

One hundred and twelve participants participated in the study, 75 of them provided TM services while confined. During the first 10 weeks of the COVID-19 outbreak, over 12,000 medical consultations were conducted by phone. A summary of the data collected can be found in Figure 1.

FIGURE 1

Figure 1. A summary of the data collection.

Each professional was able to participate in the study; however, those professionals who did not apply TM only answered 18 questions from the questionnaire.

An observational descriptive, cross-sectional study was carried out to understand TM's expectations and satisfaction by PS members during the COVID-19 pandemic confinement. The analysis was performed in a single phase using a questionnaire developed and carried out by the authors.

A questionnaire on mental health professionals' perceptions and satisfaction of TM, the Font Roja Work Satisfaction Questionnaire, was adapted and used. The authors designed an adapted questionnaire based on the literature's references, including 49 closed-ended questions. This questionnaire includes some questions, grouped into nine factors, which exploring different areas involved in job satisfaction: (1) work monotony; (2) job satisfaction; (3) stress/stress from work; (4) competition; (5) labor pressure; (6) promotional capacity; (7) relationship with superiors; (8) relationship with colleagues; (9) extrinsic characteristics of work (17). Four additional items were included: 2 to assess the work environment (18) and 2 to measure the professional's perception of the patient's satisfaction. The user satisfaction was measured on a scale of 1 (not at all satisfied) to 10 (maximum satisfaction). An additional open question was added at the end of the questionnaire to include qualitative information and personal feedback.

The online questionnaire was sent by email to all the staff members on May 15, 2020, and was available online until May 25, 2020. There were no exclusion criteria; every member from the PS received the e-mail. The procedures followed ethical standards and the Organic Law 3/2018, of the 5th of December, on the Personal Data Protection guaranteeing digital rights. The questionnaire was confidential, and subjects accepted by informed consent with the possibility to withdraw voluntarily at any time.

For quantitative data, normality tests were performed through either the Kolmogorov- Smirnov or Shapiro-Wilk Test. The described data was measured with the dispersion measures of mean and standard deviation for normal samples and the Median and Interquartile Range (IQR) for models that did not comply with the normality criteria.

Qualitative variables were explained with tables of good percentage or frequencies, which corresponded to be more appropriate. For parametric variables, an analysis was performed through either the Student's T-test for independent sample differences or the ANOVA test, when more than two groups existed. Mann-Whitney's U-test was used for non-parametric variables for two independent groups or Kruskal-Wallis when there were more than two separate groups. Qualitative variables were correlated by Fisher or Chi-Square tests, depending on which was appropriate in each case. The results were considered statistically significant for all analyses when the p-value was < 0.05 (significance level α = 0.05). SPSS software, v.26.0, was used for Mac.

Results

Sample description

The sample involved 112 participants, representing 70% of the PS total members (n = 160). 82.1% of the sample was working during COVID-19, and 66.4% carried out TM. In this sub-sample, as data to mention, 13 subjects (11.6%) lived with people who had COVID-19 while working remotely. Furthermore, 84.69% of all those who carried out TM, had never done TM or did without previous training.

On another front, concerning other professionals' characteristics, it should be said that most of the subjects who performed telemedicine were Doctors (32.1%) and 20.5% of the sample had management responsibility. The professionals were homogeneously distributed by different age groups and years of experience; however, gender differences were not found in the distribution of the sample (Table 1).

TABLE 1

Table 1. Sociodemographic and professional profile among respondents.

Regarding the TM method used, there were no prior resources or data protection systems to facilitate videoconferencing be realized, so the telephone was the most popular. Medical consultation by phone is considered as an immediate and direct method of telematic assistance, which allows the user to contact the health professional from anywhere and in any circumstance. Most subjects tracked digital follow-up used this method, followed by chat/email (40%) and video calling (4/75, 9.3%). Most of health professionals who used these methods had direct access to the hospital health platforms, and could carry out the health service with computer support to consult and update the medical history of the patients.

It could be said the amount of TM accomplished was significant as 32 % of the sample made more than 40 connections per week.

Professional's subjective perceptions about telemedicine and the future

This new TM service was considered an innovative practice for most participants. Most full sample participants (85%) pointed out that TM would increase its use in the coming years and found it very useful. Meanwhile, 60% of the sample thought that TM was not yet well-defined and 4.4% of the sample felt that this option would not be implemented and did not consider it worthwhile. Despite the fact that only eight subjects (7.1%) stated that TM would not cause any side effects, most professionals thought TM would complement the traditional face-to-face visits model (97.3%). Furthermore, some gender differences were found (Table 2).

TABLE 2

Table 2. Professional's perceptions about TeleMedicine.

Job satisfaction perceived during COVID-19

More than half of the respondents (60.8%) did not consider this experience as monotonous work, and this result was significantly related to having prior knowledge of TM (P = 0.02).

Concerning professionals' feelings about this practice, in our study, a majority of participants (71.6%) perceived some responsibility when making decisions and 14.9% of the subjects felt very tired at the end of the day, while 50% reported the opposite feeling. Furthermore, 16.3% of the sample had mood disturbance, in comparison to half of the respondents that did not experience any mood alteration (50%) related to the change from face-to-face to TM. Secondly, while a small number of participants (6.8%) indicated an over-strain, 35.5% of participants indicated they could disconnect at the end of the day. In our study, the failure to disconnect at the end of the day has been related to the confinement situation (living alone vs. merging social and family life 90.4%) and a significant relationship between the two variables was found (p = 0.041).

Almost all subjects (83.2%) considered themselves independent to do their work, 75.6% were satisfied with their work, 50% felt their work was well-recognized and 68.9% perceived that they had had the opportunity to learn new aspects. The Work Satisfaction Questionnaire category “I have gained recognition for my work” also showed a statistically significant correlation depending on whether the subjects reconciled work and family life or lived alone (P = 0.02). Among all the participants, only one perceived that thet could not do their job (1.4%), while 85.1% felt qualified.

Under other matters, 47.3% considered having a salary suitable for the work done and 67.5% an acceptable position. Competitiveness did not appear to be a problem in these circumstances, as only three subjects reported stress or tension because of that fact. Some professionals (39.2%) perceived that they received enough time to organize their work, while 23% lacked time. More than half of the staff (62.1%) knew what was expected of them at work, while the 16.2% had some doubts about their task.

There is a relationship between the category: “how problems of other colleagues can affect you” and the situation of coexistence during the confinement (living alone vs. merging social and family life) P = 0.008. It should be noted that 30 subjects of the total sample (n = 160) were infected by COVID, although sick leaves did not appear to correlate with any other items (categories).

Relating the extrinsic characteristics of the work, the same number of subjects (36.5%) seemed to have the right resources to perform their work which is statistically related to a better adaptation of the work (P < 0.001), better working environment (P = 0.002) and the physical structure of work performed at home (P = 0.02). Most respondents (67.6%) did not consider that the work's physical structure at home interfered with their ability to carry it out.

Some gender differences were detected in satisfaction showing that females were more prone to TM and more satisfied and adapted (Table 3).

TABLE 3

Table 3. The satisfaction of the professionals performing telemedicine and differences according to gender.

Telemedicine general satisfaction

Respondents scored a high perceived satisfaction with their work done with TM (Me = 8; IQR = 8 (n = 73) being the highest score 9/10 (very satisfied) (Figure 2).

FIGURE 2

Figure 2. Level of general satisfaction on telemedicine (range 0–10).

Previous experience with TM seems to be related to more satisfaction in its performance (P = 0.029). There were no statistically significant differences between the different professional categories concerning perceived satisfaction when conducting TM. According to our results, satisfaction with telemedicine was also not related to professional responsibility. However, the group with more than 30 years of experience showed the highest satisfaction rates. Linear regression was generated to predict participants' satisfaction with their therapy performed according to their age. The older the health workers were, the more satisfied they felt with their telephone follow-up model. This model would have great goodness of fit index (R-squared-0.94), so it would be appropriate to predict one variable over the other. Moreover, TM's positive opinions for the future were significantly related to the respondents' general satisfaction (P = 0.002) and with more social connections, professionals had more satisfaction with the activity performed (P = 0.007).

Professional's perception of patient's satisfaction

The mental health care providers also positively assessed the degree of satisfaction and comfort perceived from the patients receiving TM (Me = 8, IQR = 7 (n = 74), indicating that patients were “quite satisfied” with the treatment received. It is remarkable that five participants point out the risk of dehumanization in open questions and that human contact is needed for a good therapeutic relationship. On the other hand, the perceived professional satisfaction of the patients' approval was related to age (P = 0.01), according to the previous linear regression model (Me = 8; IQR = 7; average score = 7.76 (DT = 1,451; n = 74). This subjective perception of the patient's comfort was also statistically related to the gender variable (P = 0.009).

Discussion

The main hypotheses of this study were confirmed: Telemedicine acceptance and satisfaction increased among professionals exposed to this modality (who participated in it's use); Most of the Professionals who carried out telemedicine stated that they had expectations about the increased use of telemedicine as part of their future practice; Professionals' perception of patients' satisfaction was related to experienced professionals. However, it was not more significant in patients attended by more satisfied professionals. There were age and gender differences in professional satisfaction when applying telemedicine.

The mental health professionals showed, in general, an excellent clinical experience with TM, even though they had little any previous experience with it. This new model was performed through multiple calls and was carried out unexpectedly with little resources or legal regulation and no adaptation or prior training. Professionals mostly managed to reconcile family and work-life despite the pandemic circumstances, and in general, they were quite satisfied with the TM they had performed. The work was done autonomously, adapting personal preferences and peculiarities to the new task and employment situation. Older age had a positive relationship over the satisfaction with the job done, and simultaneously, professionals perceived the approval and satisfaction from most of their patients. Therefore, the higher the clinical experience, the greater the satisfaction. This could be related to more increased safety and confidence in their prolonged professional activity.

Matching previous studies, TM's satisfaction and efficiency in professionals, including psychiatrists, is high (8). Some gender differences were detected, showing that females are more likely to carry out TM (19) and are more satisfied and adapted when conducting this modality. This fact has not been described previously, but it is a well-known that females are more likely to engage in different e-health behaviors than males (20).

Professionals felt that patients were satisfied, which overlaps previous studies addressing patients' perception of the use of TM (21). There is evidence of high patients satisfaction when receiving TM in other sanitary fields (22, 23).

In general, patients and providers/professionals are satisfied with telepsychiatry services, even though providers/professionals tend to express more concerns about the telepsychiatry's potentially adverse effects. The therapeutic relationship with TM is comparable to face-to-face services in terms of reliability of clinical evaluations and treatment outcomes (24), especially in the post-COVID era when the psychopathological presentation will be different (25).

This sample identifies TM as a new and widely generalized tool that will continue to grow in the future after adopting the methods, time, resources, professional stress, and some essential features like family reconciliation. This aspect became important when considering the high prevalence of anxiety and depression due to the COVID-19 pandemic (22).

There is some data on clinical interventions in severe mental disorders and TM, such as care of patients with schizophrenia (24–26) and on the benefits and costs when applying internet cognitive behavioral therapy (I-CBT) (27). Neither of our results is comparable to previous research describing phone calls for the training of functional skills (28); the use of mobile devices to decrease hospitalizations (29) to promote treatment adherence in outpatients in remission (25). These experiences can respond to link scientific evidence and clinical reality (28). Other pathologies have received care attention/treatment through TM, including patients with obsessive-compulsive disorder (29), mental health care for patients with intercultural difficulties (30), early detection and diagnosis in patients with Alzheimer's disease or mild cognitive impairment (31), and crisis interventions (32).

The use of telepsychiatry in the COVID-19 pandemic has been scarcely discussed (3, 5, 33, 34) with some experiences among psychologist, psychotherapists and clinicians. Professionals tend to prefer face-to-face care to telematics, although their experiences with telematic care during the COVID-19 pandemic were better than expected (35, 36).

This research helps the populations living far from the mental health clinics. We can agree with some of TM's advantages, such as better quality and access to care and related cost decreased. Moreover, according to the pharmaceutical companies' plans, an increase in TM's annual growth rate (between 13 and 27%, with valuation growing over 20 billion US dollars in the next several years) is expected (3). In this sense, combining TM with psychoeducation shows greater efficacy than the regular visits in the overall functioning of patients with bipolar disorder (37).

Nevertheless, our results emphasized that not all features are positive. TM shows some inconveniences and difficulties in the real world (38). Some participants (6.8%) perceived this new model as tiring at the end of the day. Most of the sample believed telemedicine could have side effects and suggested dehumanization risks in the assistance. Chakrabarti et al. (39) referred to perceived difficulties in different areas, such as communication or building a good relationship, and uncertainty over legal, regulatory, and ethical issues could influence somewhat negative attitudes regarding this model among physicians. A possible disadvantage could be that there were many failed phone calls (around 25%); additionally, some patients were either at work or accompanied by relatives when receiving the phone call, so their availability and intimacy could not be guaranteed.

Limitations

The use of the telephone restricts the observation of the patient's non-verbal and qualitative behavior. The gold standard of e-health services should be video-conference, but in the context of the pandemic, its use was marginal due to lack of resources. The use of the telephone was considered a previous step to digital tools despite of its limitations.

The stressful state may have influenced the professional opinions of the sample during the pandemic.

In this naturalistic study, the collected data's standard practice features would not differentiate the first consultations' weight vs. follow-ups. On the other hand, the results have been obtained after a short time of performing TM (2 months), and their evolution over time remains unknown.

The previous data acquired is not underpinned by clinical trials or pilot studies. Overall, previous works support the non-inferiority of remote psychiatric care in the evaluation and treatment compared to face-to-face care (40), not regarding the age, kind of population, or mental disorder. However, specific people (children and adolescents) do need some adjustments (41). However, we describe the difficulties of applying telemedicine in the real world.

The pandemic brought about a radical change in the way of treating patients. The priority after this has been to give normality and continuity to patients who were generally willing to return to face-to-face consultations, so no more data on satisfaction with telemedicine has been collected since that date. In the future, it would be interesting to assess whether patients and professionals would be interested in giving continuity to telematic services in a context of regular assistance. As well, the mental health of the professionals should be taken into account in order to evaluate the adaptation to the new technologies use (42, 43).

Conclusion

Telemedicine is a tool that has been identified as a complement in clinical practice and a world-class resource in pandemic situations such as COVID-19's. It can be deployed without difficulties, although these equate to procedures, time, methods, and resources. Satisfaction between professionals was high, especially those with more clinical experience and those with previous telemedicine practice. Gender and professional differences should be carefully studied. However, some disadvantages should be considered, and perhaps therapeutic relationships could become impaired. Patient satisfaction should be contrasted, and the planning and decision-making of when to use face-to-face vs. online techniques should be widely considered.

The feasibility of performing telemedicine consultations has been generalized with all types of patients, even with a shallow level of training and preparation in high-pressure situations (lockdown). This condition has been representative of many psychiatry services around the world. Therefore, training and access to resources that allow fair use of telemedicine should be promoted; thus, the barriers in patient care would be reduced (44, 45). On the other hand, professionals' perceptions must be taken into account because they are the ones who prescribe the treatments (9–11). Additionally, it should be noted that patients' and mental health providers' satisfaction is crucial for telemedicine success (46). Therefore, a comprehensive and long term evaluation for measuring professionals' satisfaction in delivering telemedicine is vital for the future (47).

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author contributions

CR, BB-V, and NC-E conceptualized, designed, and supervised the study. DR-G, BB, and LA collected the data, performed data processing, and statistical analysis. BB, DR-G, and SG-L wrote the manuscript. CR and BB-V supervised the project. AM, MG, and LA critically revised and edited the manuscript. All authors have read and agreed to the published version of the manuscript.

Funding

This research project was supported by Castile and León's (Spain) Regional Management of Health (GRS 2075/A/2019) Scholarship for the project Clinical characterization of psychotic symptoms and their relationship with addiction severity among patients consulting for alcohol and cocaine intake and alcohol intake. This 12-month scholarship was awarded to CR (Main Researcher) and his research team. This research project was also supported by Castile and Leon's (Spain) Regional Management of Health (GRS COVID 59/A/20) Scholarship for the project Impact and approach on the mental health of patients affected by COVID, their families, and the health professionals who care for them. This scholarship was awarded to CR (principal researcher) and his research team.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. University JH. COVID-19 Dashboard. (2020). Available online at: https://coronavirus.jhu.edu/map.html (accessed October 27, 2020).

Google Scholar

2. Roncero C, Garcia-Ullán L., de la Iglesia-Larrad J, Javier I, Martín C, Andrés P, et al. The response of the mental health network of the Salamanca area to the COVID-19 pandemic: the role of the telemedicine. Psychiatry Res. (2020) 291:113–252. doi: 10.1016/j.psychres.2020.113252

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Corruble E. A viewpoint from Paris on the COVID-19 pandemic: a necessary turn to telepsychiatry. J Clin Psychiatry. (2020) 81. doi: 10.4088/JCP.20com13361

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Mukherjee A, Bandopadhyay G, Chatterjee SS. COVID-19 pandemic: mental health and beyond - the Indian perspective. Ir J Psychol Med. (2020) 38:140–4. doi: 10.1017/ipm.2020.63

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Olwill C, Mc Nally D, Douglas L. Psychiatrist experience of remote consultations by telephone in an outpatient psychiatric department during the COVID - 19 pandemic. Ir J Psychol Med. (2020) 38:132–9. doi: 10.1017/ipm.2020.51

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Góngora AS, Fumero GV, Morón LN, Sainz BA. de la Torre ID, Franco M. Usability analysis of a system for cognitive rehabilitation, “gradior”, in a spanish region. Telemed J E Health. (2020) 26:671–82. doi: 10.1089/tmj.2019.0084

PubMed Abstract | CrossRef Full Text | Google Scholar

7. De Las Cuevas C, Arredondo MT, Cabrera MF, Sulzenbacher H, Meise U. Randomized clinical trial of telepsychiatry through videoconference versus face-to-face conventional psychiatric treatment. Telemed J E Health. (2006) 12:341–50. doi: 10.1089/tmj.2006.12.341

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Kachnic J, Wojtuszek M, Wutke J, Krysta K, Krystanek M. Telemedicine – how does it work in practice? European Psychiatry. (2017) 41:S147–8. doi: 10.1016/j.eurpsy.2017.01.1995

CrossRef Full Text | Google Scholar

9. Roncero C, Szerman N, Terán A, Pino C, Vázquez JM, Velasco E, et al. Professionals' perception on the management of patients with dual disorders. Patient Prefer Adherence. (2016) 10:1855–68. doi: 10.2147/PPA.S108678

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Roncero C, Vega P, Grau-López L, Martínez-Raga J, Barral C, Basurte-Villamor I, et al. Relevant differences in perception and knowledge of professionals in different spanish autonomous communities regarding availability of resources for patients with dual disorders. Actas Esp Psiquiatry. (2016) 44:1–12.

PubMed Abstract | Google Scholar

11. Roncero C, Vega P, Martínez-Raga J, Barral C, Basurte-Villamor I, Rodríguez-Cintas L, et al. Professionals' perceptions about healthcare resources for co-occuring disorders in Spain. Int J Ment Health Syst. (2014) 8:35. doi: 10.1186/1752-4458-8-35

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Elhadi M, Elhadi A, Bouhuwaish A, Alshiteewi FB, Elmabrouk A, Alsuyihili A, et al. Telemedicine awareness, knowledge, attitude, and skills of health care workers in a low-resource country during the COVID-19 pandemic: cross-sectional study. J Med Internet Res. (2021) 23:e20812. doi: 10.2196/20812

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Martínez Pérez DC, Agudelo García ÁM, Andrade Carrillo R, García Cano JF, Porras Cataño SM. Telepsiquiatría: una experiencia exitosa en Antioquia, Colombia. Revista Colombiana de Psiquiatría. (2020) 49:239–45. doi: 10.1016/j.rcp.2019.06.005

PubMed Abstract | CrossRef Full Text | Google Scholar

14. O'Brien M, McNicholas F. The use of telepsychiatry during COVID-19 and beyond. Ir J Psychol Med. (2020) 37:250–5. doi: 10.1017/ipm.2020.54

PubMed Abstract | CrossRef Full Text | Google Scholar

15. European Union Promoting Gender Equality in Research Innovation. (2020). Available online at: https://ec.europa.eu/programmes/horizon2020/en/h2020-section/promoting-gender-equality-research-and-innovation (accessed March 31, 2021).

Google Scholar

16. Ministerio de Sanidad, C,.y. BS. Situación COVID-19 en España. (2020). Available online at https://cnecovid.isciii.es/covid19/ (accessed June 15, 2020).

Google Scholar

17. Aranaz J. Cuestionario Font Roja. Un instrumento de medida de la satisfacción en el medio hospitalario. Todo hospital. (1988) 52:63–8.

Google Scholar

18. Xiong J, Lipsitz O, Nasri F, Lui LM, Gill H, Phan L, et al. Impact of COVID-19 pandemic on mental health in the general population: A systematic review [published online ahead of print, 2020 August 8]. J Affect Disord. (2020) 277:55–64. doi: 10.1016/j.jad.2020.08.001

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Luo J, Tong L, Crotty BH, Somai M, Taylor B, Osinski K, et al. Telemedicine Adoption during the COVID-19 pandemic: gaps and Inequalities. Appl Clin Inform. (2021) 12:836–44. doi: 10.1055/s-0041-1733848

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Escoffery C. Gender Similarities and differences for e-health behaviors among U:S adults. Telemed J E Health. (2018) 24:335–43. doi: 10.1089/tmj.2017.0136

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Buwalda V. EPA-1426 – Implementation of new systems as part of the clinical care processes: Knowledge of operations management and patients' attitude can make the difference. European Psychiatry. (2014) 29:1. doi: 10.1016/S0924-9338(14)78627-4

CrossRef Full Text | Google Scholar

22. Lin CH, Tseng WP, Wu JL, Tay J, Cheng MT, Ong HN, Chen SY. A double triage and telemedicine protocol to optimize infection control in an emergency department in Taiwan during the COVID-19 pandemic: retrospective feasibility study. J Med Int Res. (2020) 22. doi: 10.2796/20586

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Ohannessian R, Duong TA, Odone A. Global telemedicine implementation and integration within health systems to fight the COVID-19 pandemic: a call to action. JMIR Public Health Surveill. (2020) 6:e18810. doi: 10.2196/18810

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Castro A, Larrain A, Fritsch E, Rojas G. Telepsiquiatría: una revisión sistemática cualitativa. Rev Méd Chile. (2012) 140:789–96. doi: 10.4067/S0034-98872012000600015

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Depp CA, Mausbach B, Granholm E, Cardenas V, Ben-Zeev D, Patterson TL, et al. Mobile interventions for severe mental illness: design and preliminary data from three approaches. J Nerv Ment Dis. (2010) 198:715–21. doi: 10.1097/nmd.0b013e3181f49ea3

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Firth J, Torous J. Smartphone apps for schizophrenia: a systematic review. JMIR Mhealth Uhealth. (2015) 3:e102. doi: 10.2196/mhealth.4930

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Zhang MW, Ho RC. Moodle: the cost effective solution for internet cognitive behavioral therapy (I-CBT) interventions. Technol Health Care. (2017) 25:163–5. doi: 10.3233/THC-161261

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Krzystanek M, Krzeszowski D, Jagoda K, Radzio B, Krysta K. Treatment of schizophrenia using tablet and smartphone based applications (Polish Study). Eur Psychiatry. (2016) 33:S27. doi: 10.1016/j.eurpsy.2016.01.845

CrossRef Full Text | Google Scholar

29. Spaniel F, Vohlídka P, Hrdlicka J, KoŽený J, Novák T, Motlová L, et al. Itareps: information technology aided relapse prevention programme in schizophrenia. Schizophr Res. (2008) 98:312–7. doi: 10.1016/j.schres.2007.09.005

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Mucic D. The use of a telemedicine model and its logistics to reach as many European refugees as possible. Eur Psychiatry. (2017) 41:S20. doi: 10.1016/j.eurpsy.2017.01.116

CrossRef Full Text | Google Scholar

31. Costanzo M, Signorelli M, Aguglia E. EPA-0715 – Telemedicine and alzheimer: a systematic review. Eur Psychiatry. (2014) 29:1. doi: 10.1016/S0924-9338(14)78073-3

CrossRef Full Text | Google Scholar

32. Swanson CL, Trestman RL. Rural assertive community treatment and telepsychiatry. J Psychiatr Pract. (2018) 24:269–73.

PubMed Abstract | Google Scholar

33. Rabinowitz T, Murphy KM, Amour J, Ricci MA, Caputo MP, Newhouse P. Benefits of a telepsychiatry consultation service for rural nursing home residents. Telemed J E Health. (2010) 16:34–40. doi: 10.1089/tmj.2009.0088

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Tonn P, Reuter S, Kuchler I, Reinke B, Hinkelmann L, Stoeckigt S, et al. Development of a questionnaire to measure the attitudes of laypeople, physicians, and psychotherapists toward telemedicine in mental health. JMIR Ment Health. (2017) 4:e39. doi: 10.2196/mental.6802

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Humer E, Stippl P, Pieh C, Pryss R, Probst T. Experiences of psychotherapists with remote psychotherapy during the COVID-19 pandemic: cross-sectional web-based survey study. J Med Internet Res. (2020) 22:e20246. doi: 10.2196/20246

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Zentner K, Gaine G, Ethridge P, Surood S. Abba-Aji A. Clinicians' attitudes toward telepsychology in addiction and mental health services, and prediction of postpandemic telepsychology uptake: cross-sectional study. JMIR Form Res. (2022) 6:e35535. doi: 10.2196/35535

PubMed Abstract | CrossRef Full Text | Google Scholar

37. García-Alocén A, Bermudez-Ampudia C, Martínez-Cengotitabengoa M, et al. Telemedicine and bipolar disorder. Eur Psychiatry. (2016) 33:S121. doi: 10.1016/j.eurpsy.2016.01.148

CrossRef Full Text | Google Scholar

38. Roncero C, Vicente-Hernández B, Casado-Espada NM, Aguilar L, Gamonal-Limcaoco S, Garzón MA, et al. The impact of COVID-19 pandemic on the Castile and Leon Addiction Treatment Network: a real-word experience. Front Psychiatry. (2020) 11:57555. doi: 10.3389/fpsyt.2020.575755

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Chakrabarti S. Usefulness of telepsychiatry: a critical evaluation of videoconferencingbased approaches. World J Psychiatry. (2015) 5:286–304. doi: 10.5498/wjp.v5.i3.286

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Drago A, Winding TN, Antypa N. Videoconferencing in psychiatry, a meta-analysis of assessment and treatment. Eur Psychiatry. (2016) 36:29–37. doi: 10.1016/j.eurpsy.2016.03.007

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Llorca-Bofí V, Buil-Reiné E, Adrados-Pérez M, Del Agua E, Palacios-Garrán R. Psychotic depression in healthcare workers during the COVID-19 healthcare crisis: a case series. Actas Esp Psiquiatr. (2022) 50:163–6.

PubMed Abstract | Google Scholar

42. Andrés-Olivera P, García-Aparicio J, Lozano López MT, Benito Sánchez JA, Martín C, Maciá-Casas A, et al. Impact on sleep quality, mood, anxiety, and personal satisfaction of doctors assigned to COVID-19 units. Int J Environ Res Public Health. (2022) 19:2712. doi: 10.3390/ijerph19052712

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Hilty DM, Shoemaker EZ, Myers K, Snowdy CE, Yellowlees PM, Yager J. Need for and steps toward a clinical guideline for the telemental healthcare of children and adolescents. J Child Adolesc Psychopharmacol. (2016) 26:283–9. doi: 10.1089/cap.2015.0129

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Torous J, Jän KM, Rauseo-Ricupero N, Firth J. Digital mental health and COVID-19: using technology today to accelerate the curve on access and quality tomorrow. JMIR Ment Health. (2020) 7:e18848. doi: 10.2196/18848

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Rodriguez-Villa E, Naslund J, Keshavan M, Patel V, Torous J. Making mental health more accessible in light of COVID-19: scalable digital health with digital navigators in low and middle-income countries. Asian J Psychiatr. (2020) 54:102433. doi: 10.1016/j.ajp.2020.102433

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Kissi J, Dai B, Dogbe CSK, Banahene J, Ernest O. Predictive factors of physicians' satisfaction with telemedicine services acceptance. Health Informatics J. (2019) 26:1866–80. doi: 10.1177/1460458219892162

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Law T, Cronin C, Schuller K, Jing X, Bolon DP. Conceptual framework to evaluate health care professionals' satisfaction in utilizing telemedicine. J Am Osteopath Assoc. (2019) 119:435–45. doi: 10.7556/jaoa.2019.080

PubMed Abstract | CrossRef Full Text | Google Scholar