Mental Health Needs Assessment During the COVID-19 Pandemic: Consensus Based on Delphi Study

The COVID-19 pandemic has revealed significant gaps in mental health in terms of unrecognized and unmet needs. The goal was to accurately assess the needs and identify gaps in this area during the epidemiological crisis. A Delphi study to identify the needs was conducted with a group of decision-makers, experts, and users of mental health services. A starting point of the Delphi study was prepared in two working groups, based on recognizable international recommendations and experiences of the practitioners from the field situation. This initial set of emergency measures was supplemented through the first Delphi round, and consensus about the importance was reached in the second round. A total of 41 activities were derived, the vast majority of which were rated with a score of 4 or more. Mental health activities, which should be addressed in terms of needs, can be divided into systemic measures and service measures. This study recognizes a need to reorganize services in the direction of improving local accessibility and strengthening the network of services for immediate responses to the psychological, health, and social needs of individuals, including those arising from crisis situations, such as COVID-19 pandemic. The results of this study are in line with the international recommendations and also influenced the formulation of the Action Plan of the National Mental Health Program, while some of the measures were already implemented during the publication of the research results.

Introduction

The COVID-19 pandemic has affected the mental health of populations around the world. Evidence from the outbreaks of SARS, Ebola, Influenza A H1N1, and MERS has shown their impact on psychological distress (1–5), which was also recognized from the beginning of the COVID-19 pandemic (6–12). Feelings of bereavement, isolation, and fear during pandemics, as well as loss of income, increased the risk of developing insomnia, anxiety, and depressive symptoms. People at the risk of mental disorders may experience a relapse with regard to a range of mental and substance-use disorders (6, 10, 13). Pre-existing mental disorders increase the risk of becoming severely physically ill or even dying, as well as the risk of having long-term complications due to COVID-19 (14, 15). Widespread psychological distress was reported in many populations during the reported period, and a long-term increase in the number and severity of mental health problems is possible (16). The effects of the COVID-19 pandemic on mental health are as important as understanding its clinical features, transmission patterns, and the overall management of the outbreak (17).

While the COVID-19 pandemic increased the demand for mental health services, the WHO nevertheless reported that mental health services have been disrupted or halted in 93% of countries worldwide, including Slovenia. The WHO identified mental health as an integral component of COVID-19 responses, and recommended allocating more resources in order to maintain or increase mental health services during the pandemic (13). Moreover, based on the growing trend of mental health problems in many European countries, it is necessary to scale-up mental health services (18), so they can better match the actual mental health needs.

In 2018, the National Mental Health Program (NMHP) 2018–2028 was adopted in Slovenia (19) to reform the mental health provisions with the aim of achieving comprehensive, community-oriented, and needs-led services according to the international recommendations. During 2020, the National Institute of Public Health in Slovenia started to coordinate the second Action Plan to implement the NMPH from 2021 to 2023 including a crisis plan regarding the COVID-19 pandemic. Psychiatric services are currently organized in outpatient clinics, psychiatric hospitals, and in the newly developed community-based mental health centers at the primary health care level. During the crisis, improved accessibility was recognized as an essential factor in reducing the mental health burden, and a general recommendation was made to organize services in the community.

Given the increased needs caused by the pandemic, different problems with regard to accessing care can be addressed through the organization of efficient, affordable, and human rights protective community-based services as a part of any national COVID-19 recovery plan (16). Obstacles to accessing mental health services should be removed. The stigma and discrimination (20) that seemed to increase during the times of crisis should also be reduced (21). An assessment process with all stakeholders, i.e., service users and providers from different professional groups and purchasers (primarily politicians), is needed to provide evidence-based solutions for an effective consensus-based planning (22). User involvement is essential both in decision-making as well as within the implementation (23), through the involvement of peer support (24) and with addressing the needs of different vulnerable groups. Mental health services should also be scaled up because the COVID-19 pandemic produced greater social isolation which increases the risk of domestic violence and poor mental health (25). Educational programs and humanitarian assistance supporting vulnerable groups are needed. Organized support should also be accessible for overburdened frontline workers (26). The majority of the actions mentioned above are also suggested in the United Nations guidelines (16), which define a set of actions to urgently minimize the mental health consequences of the pandemic. Specifically, these call for a whole-society approach to promote, protect, and care for mental health, ensure the widespread availability of emergency mental health and psychosocial support, and support recovery from COVID-19 by building mental health services for the future.

This study aimed to identify and design a set of emergency mental health activities needed in the Slovenian mental health care system through a consensus-based process. The present study serves as an evidence-based approach to plan further actions in this context. The results are also a part of a formative evaluation process for further steps in the mental health field. Some Slovenian research also examines the health field using a Delphi study (27), as do some international studies (28–34), along with addressing the mental health of an individual (35), but none has carried out a mental health needs assessment during the pandemic. Research about mental health services in this context is also an opportunity (36) for service organization improvements and to address mental health needs with the engagement of professional organizations, a political review of mental health plans and policies, and consultation with service users (37).

Materials and Methods

To gain the highest level of consensus among professionals working in the field of mental health, the Delphi method for gathering data from respondents focusing on their domain of expertise (38) and achieving consensus on measures was used.

Two interdisciplinary working groups, formed within the implementation of the Resolution on the Mental Health Program 2018–2028, were asked to suggest potential participants to take part in the Delphi study. The participants, invited from all key stakeholder groups, were chosen due to their reputation, skills, and role in the field of mental health. The selection of the participants was additionally supplemented by the snowball technique to prepare the final list. Therefore, we invited 128 participants from whom the majority (53.9%) were service providers, and the rest were decision-makers (25.8%) and service users (20.3%). None of the invited participants were excluded, only those who did not want to participate and did not sign the agreement were not included in the first round of the Delphi study.

The first set of priority activities during the COVID-19 pandemic was listed by two interdisciplinary working groups (Figure 1) gathered together to prepare starting points for mental health needs assessment. The first group addressed the network of services and the second addressed the mental health education. Together they had a total number of 30 members, including decision-makers, service providers, and service users equally represented. Those groups prepared a list of proposed activities, representing different measures in the field of systemic regulation and management of mental health. This initial list was sent to 129 participants who were invited to participate in the Delphi study, to make a review, and add additional comments. The purpose of this first round of the Delphi study was to obtain comments on the proposed activities and gather proposals for additional activities considered necessary in the system in the long run, and emergency ones that need to be implemented as soon as possible. After the first round, all comments and suggestions were reviewed and taken into consideration when designing the final set of activities, including proposals for additional measures. All commented and proposed activities were combined in terms of content and meaningfully formed into the final set of activities that was sent to the second and last rounds of the Delphi study to all participants who participated in the first round. In the second round, participants rated the importance of the listed activities on a 5-point Likert scale from 1 (not important at all) to 5 (very important). The aim of the last round of the Delphi study was to reach a consensus (at least 70% agreement) on the importance (score 4 or 5) of each activity. For the purpose of the last round, the statistical analysis was used. We calculated arithmetical means and modes for all activities that were assessed by the participants in the second round. Consensus was based on the percentage of those who assessed the activity with the grade of 4 or higher.

FIGURE 1

Figure 1. Workflow of the Delphi study.

Results

With regard to the first Delphi round, from all the invited participants, more women (40) than men (9) responded to the invitation and participated in the study, and the overall response rate of the first round was 37.9%. There were 11 (22.4%) representatives of policy- or decision-makers, 28 (57.2%) representatives of professionals in the field of mental health (all professions), and 10 (20.4%) representatives of service users or relatives. All those who participated in the first round were asked to assess the importance of each activity. We received 38 responses in the second round, representing a high (77.6%) response rate. There were 7 (18.4%) representatives of decision-makers or politicians, 23 (60.5%) representatives of professionals, and 8 (21.1%) representatives of service users or relatives, with a total of 33 women and 5 men participating in the second round.

The initial list contained 53 activities, representing 12 measures covering different fields of service organization supplementing the network of services and education. The list was sent to the first Delphi round. The final set was composed of 41 activities within 16 different measures. The final set was included in the last Delphi round to be assessed with regard to the level of importance.

The study highlighted the most important activities according to the recognized needs in the field of mental health during and after the COVID-19 pandemic. These are as follows: (A) Local action groups should be developed or scaled-up to achieve the delivery of food, medicines, and protective equipment to the homes of the people, as well as companionship for those who are lonely, isolated, and excluded because of poverty. The core actions are to be taken through NGO networks and social-care centers. Protocols for connected and continuous outreach are needed, especially for vulnerable groups, such as children, the elderly, and people with preexisting mental disorders. Online and telephone access should be provided to ensure connections among service providers and users. (B) Regular and mandatory consultations should be established among health, social, and educational services, among all sectors with the inclusion of humanitarian and civil society organizations to assess the needs for mental health interventions in the regions, and to enable these to be delivered locally and in good time. (C) A single entry point is necessary to access the psychological support interventions with an accessible telephone number and 24-h information, support, and crisis counseling. (D) Accessible and highly professional psychological assistance for traumatized and overstressed individuals working in the frontlines of treatment and care is needed, as well as specialized care for those infected with COVID-19. The solution chosen as the most effective was to establish educational programs and support for different intensity levels of care (from self-help and mutual support to highly qualified psychotherapy). In addition to lay training about the management of problems related to insecurity, fear, and stress, regular supervision of professionals is necessary, especially during a crisis. More specifically, continuous training and supervision are needed for professionals in health, social, and educational services, especially for those working with vulnerable groups (the lonely, elderly, children and adolescents, victims of violence, people with long-term mental health problems, and other vulnerable groups). Peer support has proven to be successful in Slovenia and is also listed among the priority actions. (E) Destigmatization campaigns need to be continuously implemented to improve help-seeking behaviors. (F) Services addressing domestic violence need to be strengthened. Safe points for reporting such violence need to be established and their visibility must be improved. (G) In times of crisis, the risk of human rights violations is high, so there is an important call to strengthen the advocacy and representation services for people with mental health problems and to monitor the protection of their rights, including institutional and hospital settings. (H) To expand the network of accommodation facilities with different levels of support for people in need, to increase the number of (smaller) group homes or housing units for people with mental disorders (especially in areas where these facilities do not exist), and to strengthen the network of day-care centers. (I) To strengthen the network of vocational rehabilitation and supported employment facilities as well as the promotion of social entrepreneurship, because of the deep economic crisis. (J) The need to provide accessible transport for a group of people with mental health problems was also listed among the priorities. (K) Protective equipment must be provided for the related field work (for health workers, social workers, including those in accommodation programs, and for relatives or carers of people with mental health problems), as well as equipment for remote access.

All the given activities were rated on average with a score of 4.3 on the 1-to-5 scale of importance, but most often with a score of 5 (mode of all overall scores). On average, the percentage of agreement or consensus level was 84.3% (from 71.1 to 97.4%). In the vast majority of cases, all participants evaluated all the statements, and only one person did not give a rating for only three statements. On average, all but one of the activities were rated 4 or higher. Only one activity, addressing the work of mental health councils, had an average score of 3.9. Participants, on average, expressed the opinion that all activities are important or very important. Given the high level of consensus reached, the third round was not necessary.

Discussion

The article describes the Delphi study conducted to identify the needs and gaps in mental health service delivery during the period of the COVID-19 pandemic in Slovenia. The study identified various systemic and service measures that need to be addressed and/or strengthened. Reports of mental health interventions during the COVID-19 pandemic are multiplying rapidly. Several studies (39–43) have so far suggested different activities, measures, and recommendations to address mental health challenges during the pandemic, but to the best of our knowledge, none has used a scientifically supported method, such as the Delphi technique. The results of our Delphi study are in line with the international guidelines (44) using the existing knowledge about the problem from reports and scientific databases (45), whereas in our study, the Delphi-based consensus was used, i.e., an open and thorough discussion with stakeholders to highlight the essential activities according to the recognized needs in mental health during and after the COVID-19 pandemic, and to identify various systemic and service measures that need to be addressed. Based on the results, the recognized needs in mental health during and after the COVID-19 pandemic are as follows (Figure 2): (1) Professional level: better management (coordinated work of mental health councils and local health action groups for accessible, needs-based support) and quality (addressing highly professional support through adequate educational programs and supervisions). (2) System and service level: upgrade of the services network (mental health centers organized locally, accommodation facilities, vocational rehabilitation, and day centers), improved access to mental health services (coordinated protocols for different stakeholders, transportation, and protective equipment within the COVID-19 pandemic and a single entry point for people with mental health problems), user involvement, and service extension (lay assistance and promoting peer support advocacy as well as addressing the risk of human rights violations bearing in mind all the vulnerable groups, improvement of services working against domestic violence and other forms of abuse). (3) Reducing discrimination and promoting destigmatization.

FIGURE 2

Figure 2. Essential activities during and after the COVID-19 pandemic.

The COVID-19 pandemic revealed many gaps in service provision and put a new emphasis on the community mental health care model (46). A locally accessible service network was seen as one of the highest priorities in this context, together with local and central government and the coordination of services, including protocols for communication among different services. The need to address the needs of the people at their homes was also an important priority because of the obviously inadequate protection of people with mental health disorders in institutions (47). The consensus reached by the Delphi study is in line with this recognition, and is as such, another important call for mental health reform in Slovenia. The need for social protection of service users is included in the Delphi results, including the terms of accommodation, provision of life necessities, and vocational rehabilitation, transport, and employment, as stressed in the last WHO recommendations about community care (47). Human rights protection with adequate direct involvement of service users in the planning and delivery of services was once again stressed, but with little reflection in the subsequent political decisions (48). Moreover, before the Delphi study in Slovenia, mental health experts from different backgrounds along with volunteers provided a single entry point which was established as a national helpline (49), which is supported and complemented by the local mental health psychological support offered within health promotion centers (50). Furthermore, information and support materials, including guidelines, were developed for professionals and the general public on a large-scale basis, addressing different aspects of mental health promotion and prevention during the pandemic, some also in the languages of the various national communities in Slovenia (51).

The major strength of this study is the inclusion of policy-makers, service users, and mental health professionals. The Delphi results call for reforming mental health services in line with international recommendations (52, 53) and measures abroad, and we find our results comparable to those reported in other countries, as well as applicable to our context. The Delphi consultation achieved the participation of service users, professionals, and providers in the codesign of emergency measures. Particular attention should be focused on additional services, especially for vulnerable groups. The findings are important to address the knowledge gap in the field of mental health needs during pandemics. The results of the current research call for good collaboration among the existing services and sectors, as well as a shift toward a community-based approach. However, despite the rigorous methodology that was used, the present research has some limitations, as follows. Namely, the Delphi study did not include health- and social-care professionals working in the frontlines with COVID-19 patients at the beginning of the pandemic. It is possible that we addressed more individuals who are already engaged in this field but do not necessarily know all the disadvantages of the current system that should be addressed. We also do not know if we included a large group of people with mental health needs to address their different views on the focal problems in a sufficiently comprehensive manner. Nevertheless, we repeatedly searched for representatives of all stakeholder groups and tried to obtain a more balanced group of participants. Another limitation is that the primary-care teams working during the pandemic were not specifically addressed, with regard to the training and guidelines in the mental health field. With the increase in telemedicine services across the globe, there is also a need to address the problems that can arise with these. In the discussion part of this study, the need for better online and telephone access to improve the connections between services and users were mentioned, but the limitations and challenges that can occur with this were not addressed. There is likely to be a need for new guidelines that can help manage this new context for the provision of services.

Conclusions

Our main finding is that a consensus about emergency mental health activities during a pandemic or other natural disaster is needed, and this can be constructed with the Delphi method. The actions already launched to meet the mental health needs of the population in Slovenia during the COVID-19 pandemic correspond well with the results of this study. Nevertheless, continuous consultation among experts, politicians, and service users is needed to make the right decisions at the right time, according to our experience. If the Delphi recommendations were strictly followed in Slovenia, and followed in a timely manner, we could overcome many challenges.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

IM lead methodology and carried out field work. IM and VŠ recoded answers and make analysis. ŠS wrote introduction. IM wrote methodology section. VŠ wrote conclusion. All authors contributed to the article and approved the submitted version.

Funding

This work was supported by the Slovenian Research Agency, Project No.: Z3-2652.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Maunder RG, Peladeau N, Leszcz M, Romano D, Savage D, Rose M, et al. Applying the lessons of SARS to pandemic influenza an evidence-based approach to mitiga-ting the stress experienced by healthcare workers. Can J Public Health. (2008) 99:486–8. doi: 10.1007/BF03403782

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Goodwin R, Haque S, Neto F, Myers LB. Initial psychological responses to Influenza A, H1N1 (“Swine flu”). BMC Infect Dis. (2009) 9:166. doi: 10.1186/1471-2334-9-166

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Khalid I, Khalid TJ, Qabajah MR, Barnard AG, Qushmaq IA. Healthcare workers emotions, perceived stressors and coping strategies during a MERS-CoV outbreak. Clin Med Res. (2016) 14:7–14. doi: 10.3121/cmr.2016.1303

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Mohammed A, Sheikh TL, Gidado S, Poggensee G, Nguku P, Olayinka A, et al. An evaluation of psychological distress and social support of survivors and contacts of Ebola virus disease infection and their relatives in Lagos, Nigeria: a cross sectional study. BMC Public Health. (2015) 15:824. doi: 10.1186/s12889-015-2167-6

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Brand J, Mckay D, Wheaton MG, Abramowitz JS. The relationship between obsessive compulsive beliefs and symptoms, anxiety and disgust sensitivity, and Swine Flu fears. J Obsessive Compuls Relat Disord. (2013) 2:200–6. doi: 10.1016/j.jocrd.2013.01.007

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Pierce M, Hope H, Ford T, Hatch S, Hotopf M, John A, et al. Mental health before and during the COVID-19 pandemic: a longitudinal probability sample survey of the UK population. Lancet Psychiat. (2020) 7:883–92. doi: 10.1016/S2215-0366(20)30308-4

PubMed Abstract | CrossRef Full Text | Google Scholar

7. World Health Organization. COVID-19 Disrupting Mental Health Services in Most Countries, WHO Survey. (2020). Available online at: https://www.who.int/news/item/05-10-2020-covid-19-disrupting-mental-health-services-in-most-countries-who-survey (accessed February 10, 2021).

Google Scholar

8. Kumar S. Caring for people who care: supporting health workers during the COVID 19 pandemic. EClinicalMedicine. (2020) 28:100667. doi: 10.1016/j.eclinm.2020.100667

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Pan KY, Kok AAL, Eikelenboom M, Horsfall M, Jörg F, Luteijn RA, et al. The mental health impact of the COVID-19 pandemic on people with and without depressive, anxiety, or obsessive-compulsive disorders: a longitudinal study of three Dutch case-control cohorts. Lancet Psychiat. (2021) 8:121–9. doi: 10.1016/S2215-0366(20)30491-0

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Liang L, Ren H, Cao R, Hu Y, Qin Z, Li C, Mei S. The effect of COVID-19 on Youth Mental Health. Psychiatr Q. (2020) 21:1–12. doi: 10.1007/s11126-020-09744-3

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Huang Y, Zhao N. Generalized anxiety disorder, depressive symptoms and sleep quality during COVID-19 outbreak in China: a web-based cross-sectional survey. Psychiatry Res. (2020) 288:112954. doi: 10.1016/j.psychres.2020.112954

PubMed Abstract | CrossRef Full Text | Google Scholar

12. McGinty EE, Presskreischer R, Han H, Barry CL. Psychological distress and loneliness reported by US adults in 2018 and April 2020. JAMA. (2020) 324:94–4. doi: 10.1001/jama.2020.9740

PubMed Abstract | CrossRef Full Text | Google Scholar

13. World Health Organization. The Impact of COVID-19 on Mental, Neurological and Substance Use Services. (2020). Available online at: https://www.who.int/publications/i/item/978924012455 (accessed February 10, 2021).

Google Scholar

14. Li L, Li F, Fortunati F, Krystal JH. Association of a prior psychiatric diagnosis with mortality among hospitalized patients with coronavirus disease 2019 (COVID-19). JAMA Netw Open. (2020) 3:e2023282. doi: 10.1001/jamanetworkopen.2020.23282

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Volkow ND. Collision of the COVID-19 and addiction epidemics. Ann Intern Med. (2020) 2:M20–1212. doi: 10.7326/M20-1212

PubMed Abstract | CrossRef Full Text | Google Scholar

16. United Nations. Policy Brief: COVID-19 and the Need for Action on Mental Health. (2020). Available online at: https://unsdg.un.org/resources/policy-brief-covid-19-and-need-action-mental-health (accessed February 10, 2021).

17. Javed B, Sarwer A, Soto EB, Mashwani ZUR. Impact of SARS-CoV-2 (Coronavirus) Pandemic on Public Mental Health. Front Public Heal. (2020) 8:292. doi: 10.3389/fpubh.2020.00292

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Organisation for Economic Co-operation and Development. Tackling the Mental Health Impact of the COVID-19 Crisis: An Integrated, Whole-of-Society Response. (2021). Available online at: https://www.oecd.org/coronavirus/policy-responses/tackling-the-mental-health-impact-of-the-covid-19-crisis-an-integrated-whole-of-society-response-0ccafa0b/ (accessed February 10, 2021).

19. Resolucija o nacionalnem programu duševnega zdravja 2018–2028. Uradni list RS, št. 24/18; (2018).

20. Mayoclinic. Mental health: Overcoming the Stigma of Mental Illness. (2017). Available online at: https://www.mayoclinic.org/diseases-conditions/mental-illness/in-depth/mental-health/art-20046477 (accessed February 10, 2021).

21. Tehrani H. Mental health stigma related to novel coronavirus disease (COVID-19) in older adults. Geriatr Gerontol Int. (2020) 20:796–7. doi: 10.1111/ggi.13985

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Niederberger M, Spranger J. Delphi technique in health sciences: a map. Front Public Heal. (2020) 8:457. doi: 10.3389/fpubh.2020.00457

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Tait L, Lester H. Encouraging user involvement in mental health services. Adv Psychiatr Treat. (2005) 11:168–75. doi: 10.1192/apt.11.3.168

CrossRef Full Text | Google Scholar

24. Mental Health Foundation. Peer support. Available online at: https://www.mentalhealth.org.uk/a-to-z/p/peer-support (accessed February 10, 2021).

Google Scholar

25. Usher K, Bhullar N, Durkin J, Gyamfi N, Jackson D. Family violence and COVID-19: Increased vulnerability and reduced options for support. Int J Ment Health Nurs. (2020). 2020, 1–11. doi: 10.1111/inm.12735

PubMed Abstract | CrossRef Full Text | Google Scholar

26. El-Hage W, Hingray C, Lemogne C, Yrondi A, Brunault P, Bienvenu T, et al. Health professionals facing the coronavirus disease 2019 (COVID-19) pandemic: what are the mental health risks? Encephale. (2020) 46:73–80. doi: 10.1016/j.encep.2020.04.008

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Susič TP, Švab I, Kolšek M. Community actions against alcohol drinking in Slovenia—a Delphi study. Drug Alcohol Depend. (2006) 83:255–61. doi: 10.1016/j.drugalcdep.2005.11.021

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Howarth E, Vainre M, Humphrey A, Lombardo C, Hanafiah AN, Anderson JK, et al. Delphi study to identify key features of community-based child and adolescent mental health services in the East of England. BMJ Open. (2019) 9:e022936. doi: 10.1136/bmjopen-2018-022936

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Bovopoulos N, Jorm AF, Bond KS, LaMontagne AD, Reavley NJ, Kelly CM, et al. Providing mental health first aid in the workplace: A Delphi consensus study. BMC Psychol. (2016) 4:41. doi: 10.1186/s40359-016-0148-x

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Lauriks S, de Wit MAS, Buster MCA, Arah OA, Klazinga NS. Composing a core set of performance indicators for public mental health care: a modified delphi procedure. Adm Policy Ment Heal Ment Heal Serv Res. (2014) 41:625–35. doi: 10.1007/s10488-013-0506-4

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Hemmings CP, Underwood LA, Bouras N. Services in the community for adults with psychosis and intellectual disabilities: a Delphi consultation of professionals' views. J Intellect Disabil Res. (2009) 53:677–84. doi: 10.1111/j.1365-2788.2008.01146.x

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Langlands RL, Jorm AF, Kelly CM, Kitchener BA. First aid for depression: a Delphi consensus study with consumers, carers and clinicians. J Affect Disord. (2008) 105:157–65. doi: 10.1016/j.jad.2007.05.004

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Kirkham EJ, Crompton CJ, Iveson MH, Beange I, McIntosh AM, Fletcher-Watson S. Co-development of a best practice checklist for mental health data science: a Delphi study. Front Psychiatry. (2021) 12:643914. doi: 10.3389/fpsyt.2021.643914

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Jorm AF. Using the Delphi expert consensus method in mental health research. Aust N Z J Psychiatry. (2015) 49:887–97. doi: 10.1177/0004867415600891

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Cleverley K, Bartha C, Strudwick G, Chakraborty R, Srivastava R. The development of a client care needs assessment tool for mental health and addictions settings using a modified delphi approach. Nurs Leadersh. (2018) 31:52–65. doi: 10.12927/cjnl.2018.25603

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Moreno C, Wykes T, Galderisi S, Nordentoft M, Crossley N, Jones N, et al. How mental health care should change as a consequence of the COVID-19 pandemic. Lancet Psychiat. (2020) 7:813–24. doi: 10.1016/S2215-0366(20)30307-2

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Ghebreyesus TA. Addressing mental health needs: an integral part of COVID-19 response. World Psychiatry. (2020) 19:129–30. doi: 10.1002/wps.20768

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Hsu C-C. The Delphi technique: making sense of consensus. Pract Assess Res Eval. (2007) 12:10. doi: 10.7275/pdz9-th90

CrossRef Full Text | Google Scholar

39. Talevi D, Socci V, Carai M, Carnaghi G, Faleri S, Trebbi E, et al. Mental health outcomes of the covid-19 pandemic. Riv Psichiatr. (2020) 55:137–44. doi: 10.1708/3382.33569

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Antoine B, Nicolas D, Laurent D. Reshaping community mental health services during the COVID-19 epidemic—report from the 59G21 Service in Lille, France. Health Serv Insights. (2020) 13:1178632920954876. doi: 10.1177/1178632920954876

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Pelizza L, Pupo S. The COVID-19 pandemic and Italian Public mental health services: experience and future directions. J Patient Exp. (2020) 7:642–4. doi: 10.1177/2374373520956857

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Stašević-Karličić I, Dordević V, Stašević M, Subotić T, Filipović Z, Ignjatović-Ristić D, et al. Perspectives on mental health services during the COVID-19 epidemic in serbia. Srp Arh Celok Lek. (2020) 148:28–28. doi: 10.2298/SARH200504028S

CrossRef Full Text | Google Scholar

43. Kuzman MR, Curkovic M, Wasserman D. Principles of mental health care during the COVID-19 pandemic. Eur Psychiatry. (2020) 63:e45. doi: 10.1192/j.eurpsy.2020.54

PubMed Abstract | CrossRef Full Text | Google Scholar

44. World Health Organization. Mental Health and COVID-19. (2020). Available online at: https://www.who.int/teams/mental-health-and-substance-use/covid-19 (accessed February 10, 2021).

Google Scholar

45. Almeda N, Garcia-Alonso C, Salvador-Carulla L. Mental health planning at a very early stage of the COVID-19 crisis: a systematic review of online international strategies and recommendations. BMC Psychiatry. (2021) 21:43. doi: 10.1186/s12888-020-03015-y

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Thornicroft G, Tansella M. The balanced care model: the case for both hospital- and community-based mental healthcare. Br J Psychiatry. (2013) 202:246–8. doi: 10.1192/bjp.bp.112.111377

PubMed Abstract | CrossRef Full Text | Google Scholar

47. World Health Organization. Long-Stay Mental Health Care Institutions and the COVID-19 Crisis: Identifying and Addressing the Challenges for Better Response and Preparedness. (2020). Available online at: https://www.euro.who.int/en/health-topics/noncommunicable-diseases/mental-health/publications/2020/long-stay-mental-health-care-institutions-and-the-covid-19-crisis-identifying-and-addressing-the-challenges-for-better-response-and-preparedness-2020 (accessed August 30, 2021).

Google Scholar

48. Ažman R, Švab V. Mental Health in the Community: the Case of Slovenia. Ljubljana: Šent, Slovensko združenje za duševno zdravje. (2015).

49. Nacionalni inštitut za javno zdravje. Telefon za Psihološko Podporo ob Epidemiji COVID-19. (2021). Available online at: https://www.nijz.si/sl/zazivel-nov-telefon-za-psiholosko-podporo-ob-epidemiji-covid-19-080-51-00 (accessed August 10, 2021).

50. Vinko M, Roškar S, Šubara TN, Grum AT. Local Mental Health Support in Slovenia During COVID-19: Setting Up Primary Health Care Helplines. EuroHealthNet magazine (2020). Available online at: https://eurohealthnet-magazine.eu/local-mental-health-support-in-slovenia-during-covid-19-setting-up-primary-health-care-helplines/ (accessed February 10, 2021).

51. Nacionalni inštitut za javno zdravje. Skrb za duševno zdravje v času širjenja novega koronavirusa SARS-CoV-2. (2020). Available online at: https://www.nijz.si/sl/koronavirus-dusevno-zdravje (accessed February 10, 2021).

52. World Health Organization. Substantial Investment Needed to Avert Mental Health Crisis. (2020). Available online at: https://www.who.int/news/item/14-05-2020-substantial-investment-needed-to-avert-mental-health-crisis (accessed February 10, 2021).

53. World Health Organization. Looking Back, Looking Forward: Rapid Assessment of the Mental Health System in Slovenia. Report of a virtual mission by the WHO Regional Office for Europe. Geneva: WHO (2020),1–22.