Translation, Validity, and Reliability of Mental Health Literacy and Help-Seeking Behavior Questionnaires in Indonesia

Background and Aim: Mental health is an integral part of adolescent wellbeing. However, only few adolescents understand the importance of mental health and are aware of the right time to seek help. Lack of knowledge and stigma may impede help-seeking behavior. To assess these aspects, three questionnaires have been developed in the English language. This study aims to assess the validity and reliability of an Indonesian version of the Mental Health Literacy and Help-Seeking Behavior set of questionnaires among adolescents in Indonesia.

Methods: This is a cross-sectional study that used The Mental Health Literacy and Help-Seeking Behavior set of questionnaires developed by Kutcher and Wei. The set consists of three questionnaires: the Mental Health Knowledge, Attitude Toward Mental Health, and Help-Seeking Behavior questionnaire. The study was conducted between October 2020 and January 2021 with 68 first-year medical students at the University of Indonesia, who represented adolescents in a transitional phase. The questionnaires were translated into the Indonesian language by a bilingual psychiatrist and reviewed by 10 expert psychiatrists to determine content validity [Item-Level Content Validity Index (I-CVI) and Scale-Level Content Validity Index (S-CVI)]. Cronbach's alpha values were used to assess internal consistency (reliability).

Results: The content validity test produced positive results with an I-CVI scores of 0.7–1.0 and S-CVI scores of 0.87, 0.90, and 0.99 for the knowledge, attitude, and help-seeking behavior questionnaires, respectively. For the reliability test, Cronbach's alpha values were 0.780 for the attitude questionnaire and 0.852 for the help-seeking behavior questionnaire, while the value for the knowledge questionnaire was 0.521.

Conclusion: The ability to properly measure mental health through the availability of accessible, valid, and understandable tools plays an important role in addressing mental health issues among adolescents. In the current study, the Indonesian translations of all three questionnaires examining knowledge, attitude, and help-seeking behavior were considered to be valid and reliable.

Introduction

Adolescents suffer from mental health problems just like everyone else. Approximately 18% of the Indonesian population suffers from several types of mental disorders (1). Wiguna et al. (2) demonstrated that 33.5% of adolescents who sought help exhibited mood disorders. Self-harm is also prevalent among adolescents, reaching its peak around puberty. Nevertheless, adolescents do not always seek help. One study indicated that only one in four adolescents with depression or one in three adolescents with a mental illness sought help, whereby those who did otherwise may be at risk of drug abuse, risky behavior, decreased quality of life, and lowered life expectancy (3–6). Adolescents may develop mental health problems when facing the experience of transitioning to young adulthood. Their mental health may be affected not only by their biological or executive function development but also by their social environments, for example, family, peers, and schools (7). Adolescents transitioning to adulthood may also need to adapt to new situations and challenges as they pursue higher education. Entering college may promote a sense of accomplishment. However, there may also be new stressors that could trigger mental distress (8).

The recent World Mental Health Surveys conducted by the World Health Organization revealed that 20.3% of college students had 12-month mental health disorders assessed with Diagnostic and Statistical Manual of Mental Disorders, fourth edition (DSM-IV)/Composite International Diagnostic Interview. Of that group, 83.1% of the students experienced mental health disorders before enrolling in college. Those who experienced the onsets of mental health disorders before entering college had a higher chance of dropping out. In addition, the study revealed that only 16.4% of the students with 12-month disorders had received mental healthcare treatment (9). Among college students, medical students have been reported to have higher rates of perceived stress and emotional distress (10). Atkinson (11) found that one-third of medical students showed stress and depressive symptoms, while half of the students were anxious upon entering their first year. Adolescents entering medical schools while transitioning to adulthood may need to be assessed for their mental health knowledge, attitudes to mental health, and their help-seeking behavior to develop specific measures of early intervention to address their psychological distress and promote help-seeking behavior.

Being able to recognize specific mental illnesses, knowing how to obtain information about treatment and adequate professional help, and performing help-seeking behavior are crucial. One study revealed that among its 2,000 subjects, only 39% were able to identify symptoms of depression, and only 27% were able to identify symptoms of schizophrenia (12, 13). Another study involving adolescents found that only 16.4% of respondents could be classified as adequately informed to identify and intend to seek help for certain issues (14). In 2007, the World Health Organization (WHO) defined health literacy as the cognitive and social skills that determine the motivation and ability of individuals to gain access to, understand, and use information in ways that promote and maintain good health. Mental health literacy was defined as the understanding of how to obtain and maintain good mental health, understanding mental disorders and their treatments, developing capacities to decrease stigma, and developing capacities to enhance help-seeking efficacy (15).

The most powerful determinant of actual help-seeking behavior is help-seeking intentions, with studies demonstrating that help-seeking attitudes could predict help-seeking intentions. Sub-factors of help-seeking attitudes are recognition of the need for help, stigma tolerance, interpersonal openness, and therapist confidence (3). The attitude shown by adolescents toward mental health is important, as overall negative attitudes toward mental health may impede help-seeking behavior. Penn et al. (16) demonstrated that adolescents and young adults had greater mental health knowledge, yet they believed those with mental illnesses to be different to those who are “normal.”

There are several factors that facilitate or impede help-seeking behavior: a preference to look to informal sources, e.g., friends and family; a fear of others reacting to their story; and a lack of knowledge of where to seek help (17). Another study found impeding factors related to help-seeking behavior to include a lack of information; biased recognition of severity, cost, shame, time, and distance; and distrust in healthcare professionals (18). According to another study, factors determining help-seeking behavior in adolescents included perceived level of benefit, general health motivation, extraversion, social support, and perceived barriers (cost) (19). Mental health knowledge and attitudes and help-seeking behavior must be assessed to understand the capacity of individuals to seek help, including its accessibility.

Therefore, a comprehensive understanding of mental health literacy is essential for more effective and targeted interventions or programs to address the rise in mental health issues among adolescents (20, 21). Subsequently, utilizing proper and relevant tools would be beneficial in obtaining correct measurements that portray the latest information regarding mental health literacy in adolescence (20). Nevertheless, recent studies on mental health literacy have used limited measures focusing on specific subpopulations. Sorensen et al. (22) mentioned a widely used instrument called the Health Literacy Survey European Questionnaire 47 (HLS-EU-QS47), which assesses a broad spectrum of mental health literacy topics. In Indonesia, there was a self-reporting questionnaire called Willingness to Seek Professional Counseling Outside the university (WSPCO), which was used as a tool to obtain data regarding contributing factors impeding help-seeking behavior to reach out for psychological help and facilities (18). However, both questionnaires feature more general items and do not portray the division of mental health literacy into three aspects: knowledge, attitude, and help-seeking behavior.

Stan Kutcher and Yifeng Wei developed a questionnaire to measure mental health knowledge and attitudes toward mental health among adolescents, which was further mentioned as a mental health literacy questionnaire by Carr, Wei, Kutcher, and Heffernan (15). Milin et al. (23) then modified and validated a questionnaire first used by the Youth Opinion Survey on attitudes toward mental health in adolescents. Of all items contained in the questionnaire, this mental health literacy questionnaire could help to address the issue of transitional age adolescents in terms of stress and psychological distress (11). However, the questionnaire has not yet been used widely across Indonesia as part of child and adolescent psychiatric services, mainly due to the absence of an Indonesian language version. This study aimed to translate and assess the validity and reliability of the mental health literacy questionnaires designed to assess mental health knowledge, attitudes, and help-seeking behavior among adolescents in Indonesia.

Materials and Methods

Design

This was a cross-sectional study, as part of a project to develop a mental health module for university students involving first-year medical students, and was conducted between October 2020 and January 2021 at the University of Indonesia campuses in Depok and Jakarta, Indonesia. The study was approved by the Cipto Mangunkusumo National Hospital, Jakarta, Indonesia, and ethically approved by the Health Research Ethics Committee, Faculty of Medicine, University of Indonesia, numbered KET-527/UN2.F1/ETIK/PPM.00.02/2020 under protocol number 20-05-0538.

Participants

Ten experts in child and adolescent psychiatry were included in the study to determine the internal validity of the questionnaires. There is no consensus on how sample size should be measured in psychometric validation studies. Various sources state that it should be 2–20 times the number of items in the questionnaire (24). In this study, to assess construct validity and reliability, sample size was determined by the proportion difference equation n = (Zα² × p × q)/d², used primarily for cross-sectional studies (25). In October 2020, all three questionnaires were administered to 68 first-year medical students who were selected randomly from 170 students to meet the minimum sample number required for the study based on the calculation formula. The proportion (p) was assumed to be 20% taken from a previous study on transitional youth mental health problems (26). Meanwhile, q is 1 – p, d is the limit from error, and α is the confidence degree. Therefore, n = (1.96² × 0.2 × 0.8)/0.1² = 61.4. We added 10% and rounded the number up to 68 to ensure that the required sample size was met. We conducted the test on first-year medical students, as they are representative of the group of adolescents in the transition to adulthood who require assessment in terms of their mental health literacy and help-seeking behavior.

Measures

The study used The Mental Health Literacy and Help-Seeking Behavior questionnaires developed by Stan Kutcher, Yifeng Wei, et al., consisting of Mental Health Knowledge, Attitude Toward Mental Health, and Help-Seeking Behavior questionnaires (15). All respondents provided their informed consent to complete the questionnaires and several demographic questions. The Mental Health Knowledge questionnaire consisted of 13 items, in which participants were scored one point for correct answers and zero points for wrong or unsure answers. The maximum and minimum total scores were 13 and 0, respectively. The Attitude Toward Mental Health questionnaire consisted of 12 items, with a 5-point Likert Scale ranging from strongly agree to strongly disagree. The maximum and minimum scores were 60 and 12, respectively. The Help-Seeking Behavior questionnaire consisted of 24 items. Results were deemed satisfactory if the participant scored above average, as determined by the pilot mean/median score.

Translation and Pilot Study

The Mental Health Literacy and Help-Seeking Behavior questionnaires were translated into the Indonesian language by a bilingual psychiatrist. A few adjustments were made in accordance with the input provided by 10 first-year medical student respondents during the trial test of the questionnaires. An additional example was provided for item 7 of the questionnaire, which stated that depression can be treated effectively using alternative treatment, in order to clarify that alternative treatment is not medical treatment, as the questionnaire may be used for students from a non-medical background. The questionnaires were then back translated into English and communicated back to the cross-cultural transition module team, which included the authors of the questionnaires, and then set into the final questionnaire.

Validity

Content Validity

To perform the content validity test, each item of the questionnaire was assessed by 10 experts on child and adolescents psychiatry. The scoring was conducted quantitatively, whereby experts determined its relevance using the following scale: (1) not relevant, (2) quite relevant, (3) relevant, and (4) highly relevant. The scoring sheet was transferred into MS Excel. A score of (1) and (2) were deemed irrelevant and given a value of 0, while (3) and (4) were deemed relevant and given a value of 1. Furthermore, the calculation was based on the Item-Level Content Validity Index (I-CVI) and Scale-Level Content Validity Index (S-CVI). The I-CVI and S-CVI were used to demonstrate validity. S-CVI/Ave, the mean of I-CVI value for each item of the questionnaire, was also used. Polit et al. (27) suggested that with 10 or more experts, there is little need to compute values for k^* and that any I-CVI > 0.78 would be considered excellent. Meanwhile, Torkian et al. (28) mention in their study that Content Validity Indexes were considered to be acceptable when I-CVI and S-CVI were at least 0.78 and 0.90, respectively.

Construct Validity

The aim of construct validation is to measure whether the instrument is able to capture what it intends to measure (24). This could be by measuring the relation between each item and the total scores of each category to establish its relation to other variables with which it should, theoretically, be associated positively, negatively, or practically not at all. To quantify construct validity, correlation coefficients and confirmatory factor analysis or structural equation modeling, or other statistical evaluations, can be used (29). Construct validity of The Indonesian Mental Health Literacy and Help-Seeking Behavior Questionnaires was measured using the Pearson correlation coefficient values between each item and the total scores of each category and furthermore between each item (30, 31). The Pearson coefficients are classified as medium (0.30–0.49) and large (>0.50) (30). Similar translation adaptation validity studies have used this method (32).

Reliability

The reliability was determined using the Cronbach's alpha measurement (33). As the questionnaire is a self-report, the inter-rater reliability was deemed unnecessary.

Results

Descriptive Findings

The study conducted in October 2020 involved 68 adolescents who were studying as first-year medical students at the University of Indonesia. The mean age was 18.08 for male and 17.81 for female respondents. All respondents were able to understand English and Bahasa Indonesia. Demographic information of the respondents can be found on Table 1.

TABLE 1

Table 1. Demographic information.

Content Validity

The study tested content validity by having 10 experts on child and adolescent psychiatry assess each item of the questionnaire and calculate the I-CVI and S-CVI. The complete I-CVI and S-CVI data are presented in Table 2.

TABLE 2

Table 2. Validity test of the Indonesian mental health literacy and help-seeking behavior set of questionnaires.

Based on the validity test, the study produced I-CVI scores ranging between 0.7 and 1.0, with S-CVI scores of 0.87, 0.90, and 0.99 on the Mental Health Knowledge, Attitude Toward Mental Health, and Help-Seeking Behavior questionnaires, respectively. The results show that the questionnaires were deemed relevant in measuring the knowledge, attitude, and help-seeking behavior of an individual regarding mental health.

Construct Validity

For the knowledge questionnaire, overall, each item had a medium correlation (over 0.3) with the exception of item 4. The attitude questionnaire had a medium to large correlation ranging from 0.41 to 0.76, except for item 10. Finally, for the help-seeking behavior questionnaire, most items had a medium to large correlation ranging from 0.30 to 0.69, as seen in Table 3.

TABLE 3

Table 3. Item-total correlation coefficients for the Indonesian mental health literacy and help-seeking behavior set of questionnaires.

Reliability

A reliability test was performed to identify correlative reliability between items and internal consistency. The study revealed that the Cronbach's alpha values for the reliability test were 0.521, 0.780, and 0.852 for the Mental Health Knowledge, Attitude Toward Mental Health, and Help-Seeking Behavior questionnaires, respectively. The results of the reliability test on each questionnaire are provided in Table 4. The inter-item correlations for each item are presented in Tables 5–7. The mean score for the Mental Health Knowledge questionnaire was 6.97 (SD 2), 47.26 (SD 5.397) for the Attitude Toward Mental Health questionnaire, and 46.66 (SD 6.825) for the Help-Seeking Behavior questionnaire.

TABLE 4

Table 4. Internal consistency of the Indonesian mental health literacy and help-seeking behavior set of questionnaires.

TABLE 5

Table 5. Inter-item correlation coefficients for Indonesian adolescents mental health knowledge questionnaire.

TABLE 6

Table 6. Inter-item correlation coefficients for Indonesian adolescents' attitude toward mental health questionnaire.

TABLE 7

Table 7. Inter-item correlation coefficients for mental health help-seeking behavior of Indonesian adolescents questionnaire.

Discussion

The current study aimed to provide evidence of the validity and reliability of tools for assessing mental health among Indonesian adolescents in three categories: mental health knowledge, attitude toward mental health, and help-seeking behavior. An Indonesian version of the Mental Health Literacy and Help-Seeking Behavior questionnaires, which could be used to assess the mental health literacy and help-seeking behavior of Indonesian adolescents, is yet to exist. Youth well-being is an important component attributing to improved mental health services across the nation. Therefore, by having access to valid and reliable questionnaires, health professionals could better understand the mental health situation among adolescents within the community before conducting further studies or interventions.

Content Validity

Content validity was assessed based on relevancy, clarity, and ambiguity. For Mental Health Knowledge, the internal validity test showed an I-CVI ranging between 0.7 and 1.0, with a S-CVI of 0.87, meaning that 87% of the items were deemed relevant, clear, and unambiguous by the panel of experts. Similarly, the internal validity test on the questionnaire measuring Attitude Toward Mental Health had an I-CVI of 0.8–1.0, with an S-CVI of 0.90, showing that 90% of the items were also deemed understandable. As for Help-Seeking Behavior questionnaire, each item showed an I-CVI of either 0.9 or 1.0, with an S-CVI of 0.99, meaning that 99% of the items were deemed relevant and clear.

Seven of the 10 experts deemed all items to be appropriate. As the study could be used as the pilot study for further research with similar purposes, there was no precise cutoff point to set the baseline for the questionnaire being considered valid.

Construct Validity

A construct validity test was conducted to assess whether the questionnaires measured the intended constructs, which were knowledge, attitudes, and help-seeking behavior. To measure whether each questionnaire is homogeneous, we conducted Pearson r item-total correlation analysis. The Pearson correlation is used to assess the linear association between variables. Although most items revealed a moderate to high correlation, a few of the items had a low correlation and were statistically insignificant. It has been noted that it would be helpful if further evidence of construct validity could be obtained, for example, convergent evidence and discriminant evidence, which could be retrieved from factor analysis (32, 34).

Reliability

In order to evaluate the internal consistency of the research tool, the Cronbach's alpha coefficient was used. This method has been the most widely employed method in validation studies (35). An alpha value >0.70 is considered good, whereby when reliability increases, the fraction of a test score attributable to error will decrease. Out of the questionnaires, the Attitude Toward Mental Health questionnaire and Help-Seeking Behavior questionnaire had an alpha value of above 0.70, demonstrating their effectiveness and acceptability.

The set of questionnaires was disseminated to 68 respondents, surpassing the minimum sample number of 62. Wei et al. (36) also suggested in their findings that, in order to examine the internal consistency and the dimensionality of the tool, a sample size of at least 30 individuals may be required. The Cronbach's alpha value for the Mental Health Knowledge questionnaire obtained in this study was 0.521. This value is relatively fair and considered to be low to moderate in terms of reliability, but not unacceptable (37). The English version of the Mental Health Knowledge questionnaire, which was used for a randomized clinical trial on high school students, and which was the reference for the Indonesian questionnaire, demonstrated a Cronbach's alpha of 0.40 (preintervention) and 0.54 (postintervention) (23).

To acquire a higher Cronbach's alpha value, it has been suggested that providing a larger number of items in the scale could be taken into consideration. A low alpha value could be caused by the following reasons: fewer numbers of questions, poor inter-item correlation, or heterogeneous constructs of the items (38). Nevertheless, if the items targeting a unidimensional construct are parallel, one alternative method to enhance the alpha value is to develop a set of highly correlated items in the tool, without unduly increasing the number of items with a lack of inter-item correlation (39).

Future Appliance and Practical Settings

This validity and reliability study indicates that the translated mental health literacy questionnaires are reasonably valid and can be further applied for wider use in the setting of psychiatric services for adolescents. We also suggest that our study could be used to address the finding made by Vidourek et al., in which a lack of support and inadequate treatment from health professionals could increase the risk of students not finishing their studies, which subsequently leads to worsening mental health conditions (40).

Several studies have shown that students, who are in the period of transitioning from adolescents to young adults, tend to seek informal help from relatives or friends, rather than professional help when they encounter psychological distress. This lack of willingness to seek professional support results in higher stress levels and poor academic performance (41, 42). With the existence of these questionnaires in Indonesia, mental health professionals will be able to further understand and gain detailed insights into why individuals are hesitant to seek help.

Currently, according to the Indonesian 2020–2024's Plans of Actions in Mental Health and Psychoactive Drugs, one strategic issue faced by the government is a lack of funding, while not all district authorities have set policies to address mental health issues (43). This study suggests that the questionnaires could be utilized to create a hassle-free experience and are able to penetrate all layers of the community. In the future, these questionnaires could serve as tools for the government and relevant stakeholders to establish affordable mental health policies. Moreover, a study in Indonesia showed that despite the high number of university students suffering from anxiety (95.4%), ~90–96.4% exhibited positive coping strategies, and nearly 50% reported self-harming and had suicidal thoughts (21). We suggest that the tools examined in this study be applied by health facilities in universities to perform initial assessment of first-year students. In the long term, the data obtained can be used as a reference for more effective and accurately targeted interventions to address mental health issues in adolescents.

Study Limitations

There are several limitations to this study. The sample group for the study was adolescents in the transitional phase who are first-year medical students. In order to further examine the effectiveness of these questionnaires, the coverage could be expanded to include adolescents in different phases and from a general academic background. Furthermore, a test–retest study could also be considered to re-evaluate the set of questionnaires in the future.

Conclusion

The mental health and wellbeing of adolescents is a dynamic and growing issue that must be addressed properly in the field of psychiatry considering that literacy around the underlying causes, the existence of stigma, and sense of severity could impair an individual's quality of life. The availability of proper means of measurement by the existence of accessible yet valid tools is an integral part of addressing the issue. This is the first set of mental health literacy tools which have been translated and validated into the Indonesian language. From the results of our study, the Indonesian version of the Mental Health Literacy and Help-Seeking Behavior questionnaires is considered to be valid and reliable. These tools are easily understood by all levels of society and could be used to support future studies focusing on mental health literacy among adolescents.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by Health Research Ethics Committee, Faculty of Medicine, University of Indonesia. Written informed consent to participate in this study was provided by the participants' legal guardian/next of kin.

Author Contributions

FK, RI, and TW: designed the study and methods. SP, WI, HG, and VP: contributed to the design of the study and review the interpretation of the data. FK and CM: carried out the data collection. FK, CM, GN, and MA: analyzed the data and write the draft of the manuscript. FK, RI, TW, KM, and SP: contributed in finalizing the manuscript. TT, MK and AS: contributed to the interpretation of the data and critically reviewed and approved the final draft of the manuscript. All authors agree to accept responsibility for all aspects of the work.

Funding

The funding for this study was from the Program Pendanaan Perancangan dan Pengembangan Purwarupa (P5) grant which was granted by the Directorate of Innovation and Science Techno Park Universitas Indonesia.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We would like to acknowledge Stan Kutcher and Yifeng Wei for their permission and support to use the original questionnaires, and all the cross-cultural transition project group for their support to use these questionnaires among transitional age youth. We would also thank the participants of this study for their contribution.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2021.764666/full#supplementary-material

References

1. Remaja KJ. Kesehatan Jiwa Remaja. KEMAS J Kesehat Masy. (2009) 5:48–57.

Google Scholar

2. Wiguna T, Manengkei PSK, Pamela C, Rheza AM, Hapsari WA. Masalah Emosi dan Perilaku Pada Anak dan Remaja di Poliklinik Jiwa Anak dan Remaja RSUPN dr. Ciptomangunkusumo (RSCM), Jakarta. Sari Pediatr. (2016) 12:270–7. doi: 10.14238/sp12.4.2010.270-7

CrossRef Full Text | Google Scholar

3. Do R, Park J, Lee S, Cho M, Kim J, Shin M. Adolescents' attitudes and intentions toward help-seeking and computer-based treatment for depression. JAMA. (2019) 19:728–36. doi: 10.30773/pi.2019.07.17.4

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Gore FM, Bloem PJN, Patton GC, Ferguson J, Joseph V, Coffey C, et al. Global burden of disease in young people aged 10-24 years: a systematic analysis. Lancet. (2011) 11:6. doi: 10.1016/S0140-6736(11)60512-6

PubMed Abstract | CrossRef Full Text

5. Divin N, Harper P, Curran E, Corry D, Leavey G. Help-seeking measures and their use in adolescents: a systematic review. Adolesc Res Rev. (2018) 3:113–12. doi: 10.1007/s40894-017-0078-8

CrossRef Full Text | Google Scholar

6. Laski L. Realising the health and wellbeing of adolescents. BMJ. (2015) 15:4119. doi: 10.1136/bmj.h4119

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Davidson LL, Grigorenko EL, Boivin MJ, Rapa E, Stein A. A focus on adolescence to reduce neurological, mental health and substance-use disability. Nature. (2015) 527:S161–6. doi: 10.1038/nature16030

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Chan V, Moore J, Derenne J, Fuchs DC. transitional age youth and college mental health. Child Adolesc Psychiatr Clin N Am. (2019) 2:8.

PubMed Abstract | Google Scholar

9. Alonso J, Cuijpers P, Greif Green J, Hwang I, Liu H, Mortier P, et al. Mental disorders among college students in the who world mental health surveys. Psychol Med. (2016) 46:2955–70. doi: 10.1017/S0033291716001665

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Heinen I, Bullinger M, Kocalevent RD. perceived stress in first year medical students - associations with personal resources and emotional distress. BMC Med Educ [Internet]. (2017) 17:1–14. doi: 10.1186/S12909-016-0841-8

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Atkinson SR. Elevated psychological distress in undergraduate and graduate entry students entering first year medical school. PLoS ONE [Internet]. (2020) 15:1–11. doi: 10.1371/journal.pone.0237008

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Kaligis F. Life skills program for improving adolescent mental health in the af-termath mount merapi eruption, yogyakarta-indonesia. ASEAN J Community Engagem. (2017) 1:59. doi: 10.7454/ajce.v1i1.61

CrossRef Full Text | Google Scholar

13. Burns JR, Rapee RM. Adolescent mental health literacy: young people's knowledge of depression and help seeking. J Adolesc. (2006) 25:229–39. doi: 10.1016/j.adolescence.2005.05.004

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Lam LT. Mental health literacy and mental health status in adolescents: a population-based survey. Child Adolesc Psychiatry Ment Health. (2014) 8:1–8. doi: 10.1186/1753-2000-8-26

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Carr W, Wei Y, Kutcher S, Heffernan A. Preparing for the classroom: mental health knowledge improvement, stigma reduction and enhanced help-seeking efficacy in Canadian preservice teachers. Can J Sch Psychol. (2018) 33:314–26. doi: 10.1177/0829573516688596

CrossRef Full Text | Google Scholar

16. Burstein M, Ameli-Grillon L, Merikangas KR. Shyness versus social phobia in US youth. Pediatrics. (2011) 128:917–25. doi: 10.1542/peds.2011-1434

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Rowe SL, French RS, Henderson C, Ougrin D, Slade M, Moran P. Help-seeking behaviour and adolescent self-harm : a systematic review. (2014). 48:12. doi: 10.1177/0004867414555718

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Rasyida A, Psikologi F, Surabaya U. Faktor yang menjadi hambatan untuk mencari bantuan psikologis formal di kalangan mahasiswa. (2019). 8:193–207. doi: 10.30996/persona.v8i2.2586

CrossRef Full Text | Google Scholar

19. O'Connor PJ, Martin B, Weeks CS, Ong L. Factors that influence young people's mental health help-seeking behaviour: a study based on the health belief model. J Adv Nurs. (2014) 70:2577–87. doi: 10.1111/jan.12423

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Schneider M, Jaks R, Nowak-Flück D, Nicca D De Gani SM. Mental health literacy in zurich: a first measurement attempt using the general HLS-EU-Q47. Front Public Health. (2021) 9:723900. doi: 10.3389/fpubh.2021.723900

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Kaligis F, Ismail RI, Wiguna T, Prasetyo S, Indriatmi W, Gunardi T, et al. Mental health problems and needs among transitional-age youth in Indonesia. Int J Environ Res Public Health. (2021) 21:46. doi: 10.3390/ijerph18084046

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Sørensen K, Van den Broucke S, Pelikan JM, Fullam J, Doyle G, Slonska Z, et al. Measuring health literacy in populations: illuminating the design and development process of HLS-EU-Q. BMC Public Health. (2013) 128:917–25. doi: 10.1186/1471-2458-13-948

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Milin R, Kutcher S, Lewis SP, Walker S, Wei Y, Ferrill N, et al. Impact of a mental health curriculum for high school students on knowledge and stigma: a randomized controlled trial. J Am Acad Child Adolesc Psychiatry [Internet]. (2016) 2:18.

PubMed Abstract

24. Anthoine E, Moret L, Regnault A, Sbille V, Hardouin JB. Sample size used to validate a scale: a review of publications on newly-developed patient reported outcomes measures. Health Qual Life Outcomes. (2014) 2:176. doi: 10.1186/s12955-014-0176-2

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Daniel WW, Cross CL. Biostatistics: A Foundation of Analysis in the Health Sciences. Wiley Series in Probability and Statistics. New York: John Wiley & Sons; (2013). p. 192.

Google Scholar

26. Dhamayanti M. Masalah Mental Emosional Pada Remaja: Deteksi dan Intervensi. Sari Pediatr. (2011) 13:45–51.

27. Polit DF, Beck CT, Owen S V. Focus on research methods: is the CVI an acceptable indicator of content validity? appraisal and recommendations. Res Nurs Heal. (2007) 30:459–67. doi: 10.1002/nur.20199

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Torkian S, Shahesmaeili A, Malekmohammadi N, Khosravi V. Content validity and test-retest reliability of a questionnaire to measure virtual social network addiction among students. Int J High Risk Behav Addict. (2020) 9:1–5. doi: 10.5812/ijhrba.92353

CrossRef Full Text | Google Scholar

29. Western D, Rosenthal R. Quantifying construct validity: two simple measures. J Personal Soc Psychol. (2003) 84:608–18 doi: 10.1037/0022-3514.84.3.608

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Cohen J. “statistical power analysis for the behavioural science,” In: Statistical Power Anaylsis for the Behavioral Sciences (1988).

Google Scholar

31. Tsang S, Royse CF, Terkawi AS. Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine. Saudi Journal of Anaesthesia. (2017) 11:s80. doi: 10.4103/sja.SJA_203_17

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Ahrens R de B, Lirani L da S, de Francisco AC. Construct validity and reliability of the work environment assessment instrument WE-10. Int J Environ Res Public Health. (2020) 17:1–19. doi: 10.3390/ijerph17207364

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. (1951) 19:297–334. doi: 10.1007/BF02310555

CrossRef Full Text

34. Cohen RJ, Swerdlik ME, Sturman ED. Psychological Testing and Assessment: An Introduction to Tests and Measurement. New York: McGraw-Hill (2013).

Google Scholar

35. Soriano BJN, Duarte SG, Alonso AMF, Perales AB, Carreño TP. Validation of a questionnaire developed to evaluate a pediatric ehealth website for parents. Int J Environ Res Public Health. (2020) 17:2671. doi: 10.3390/ijerph17082671

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Wei Y, McGrath PJ, Hayden J, Kutcher S. Measurement properties of tools measuring mental health knowledge: a systematic review. BMC Psychiatry. (2016) 16:1–16. doi: 10.1186/s12888-016-1012-5

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Hinton PR, Brownlow C, McMurray I, Cozens B. SPSS Explained. New York, NY: Psychology Pres. (2004).

Google Scholar

38. Tavakol M, Dennick R. Making sense of Cronbach's Alpha. Int J Med Educ. (2011) 16:297–334. doi: 10.1080/14739879.2011.11494022

PubMed Abstract | CrossRef Full Text

39. Heo M, Kim N, Faith MS. Statistical power as a function of cronbach alpha of instrument questionnaire items. BMC Med Res Methodol. (2015) 70:6. doi: 10.1186/s12874-015-0070-6

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Vidourek RA, King KA, Nabors LA, Merianos AL. Students' benefits and barriers to mental health help-seeking. Health Psychol Behav Med Open Access J. (2014) 2:1009–22. doi: 10.1080/21642850.2014.963586

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Griffiths KM, Crisp DA, Barney L, Reid R. Seeking help for depression from family and friends: a qualitative analysis of perceived advantages and disadvantages. BMC Psychiatry. (2011) 11:196. doi: 10.1186/1471-244X-11-196

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Pedrelli P, Nyer M, Yeung A, Zulauf C, & Wilens T. College students: mental health problems and treatment considerations. Acad Psychiatry. (2015) 39:503–11. doi: 10.1007/s40596-014-0205-9

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Direktorat Pencegahan dan Pengendalian Masalah Kesehatan Jiwa dan NAPZA, Kementerian Kesehatan, RI. Rencana Aksi Kegiatan 2020-2024. [Internet]. (2020). Available online at: https://e-Renggar.Kemkes.Go.id/File2018/e-Performance/1-401733-4tahunan-440.pdf