Reliability and validity of the online Pittsburgh sleep quality index in college students from low-income regions

Objectives: We aimed to test the reliability and structural validity (also called dimensionality) of the online Pittsburgh Sleep Quality Index among college students from low-income regions.

Methods: We assessed 195 Brazilian college students from a low-income region (Gini index of 0.56), of whom 117 were reassessed to evaluate the reliability. We collected all data in a self-reported online twice, 2-week apart. We evaluated reliability and structural validity.

Results: All questionnaire components showed reliability, correlation coefficient ≥0.49. In the structural validity, the confirmatory analysis showed better global model adjustment for the one-factor (RMSEA = 0.019; SRMR = 0.041; CFI = 0.992; TLI = 0.986) solution compared with two-factor (RMSEA = 0.099; SRMR = 0.070; CFI = 0.764; TLI = 0.619) and three-factor (RMSEA = 0.108; SRMR = 0.066; CFI = 0.763; TLI = 0.548) solutions, respectively.

Discussion: The online questionnaire presents acceptable reliability and structural validity in Brazilian low-income regions.

1 Introduction

Sleep is crucial for optimal health. However, irregular work schedules, shift patterns, and poor sleep habits often lead to abnormal sleep quantity and quality in adults (1). In academia, the high prevalence of poor sleep quality among college students in low- and lower-middle-income countries represents a significant public health concern (2). Factors such as stress, insomnia, being in the second year, bedtime electronic device usage, and chronic illnesses are key determinants of sleep quality in this group (2). Monitoring sleep quality has the potential to enhance adult health, especially within the academic context.

Subjective measurement tools can be used in clinical practice for diagnostic purposes, for monitoring treatment responses, or in epidemiological studies to measure the quality of sleep on population level (3). These tools have the advantages of cost effectiveness, ease of administration and high patient compliance. In this context, scientists validated the Brazilian version of Pittsburgh Sleep Quality Index (PSQI-BR) questionnaire measured in person against overnight polysomnography (PSG) and concluded PSQI-BR is a valid and reliable instrument for the assessment of sleep quality in Brazilian populations (4). However, we emphasize that the evaluation was carried out with interviews in a clinical context, and when applying the same questionnaire in free-living population by self-report or in online formats, these results may be different (5). In addition, the PSQI dimensionality is much debated. A comprehensive systematic literature review indicated that one- to three-factor were the possible dimensionality for this subjective tool (6). In this regard, there is no consensus about the PSQI dimensionality in the literature. Mainly because the factor structure is easily influenced by sampled data (e.g., across different groups or over time); and, repetitive revalidation studies are needed to overcome the sampling bias and to confirm the latent variable structure (7). A psychometric evaluation comparing different dimensionalities (two- vs. three-model structure) in different timepoints identified two-factor as appropriated solution in Singaporean adults (8). Although there is no dimensionality assessment of the PSQI-BR in Brazilian adults, evidences in adolescents also indicated two-factor structure solution (9).

The surge in COVID-19-related research has witnessed the migration of numerous tools to online platforms (10). However, the literature is scarce on the psychometric properties of the online format of the PSQI, although the level of education of the participants and access to the internet may distort recruitment and the ability to respond to online tools, mainly in populations with marked socioeconomic and educational differences (10), which is the Brazilian case. Thus, when one wishes to apply these tools in other populations, proper cross-cultural adaptation, questionnaire administration and validation are required (11). Therefore, we hypothesized that i) the online PSQI-BR has acceptable reliability and structural validity in Brazilian low-income regions, as well as, ii) the scale presents dimensionality based on two-factor solution with a new dataset of college students.

2 Material and methods

2.1 Study design

The current project consists of a methodological study (11) belonging to the first stage of a multicenter longitudinal observational project entitled: 24-hour movement behavior and metabolic syndrome (24h-MESYN), a prospective cohort study on lifestyle and risk of developing metabolic syndrome in undergraduate students from low-income regions during a pandemic (12). In this phase of the study, the participants answered the same questionnaire twice, 2 weeks apart, in order to verify the reliability (temporal stability of the responses) and the construct (structural) validity (12). Data collection took place during the first half of 2021.

2.2 Ethical aspects

The project was approved by the Research Ethics Committee of Universidade Ceuma (ID: 4,055,604) and Universidade Federal do Tocantins (ID: 5,161,340), respectively. The project follows the ethical principles for research with human beings of the Declaration of Helsinki, revised in 2008, Seoul, Korea; the resolution of CNS 466/12; guidelines for conducting research activities during the pandemic caused by COVID-19 (available at: www.fo.usp.br/wp-content/uploads/2020/07/Orientações-conduza-de-pesquisa-etividades-CEP.pdf) and guidelines for research in a virtual environment (OFÍCIO CIRCULAR NO. 2/2021/CONEP/SECNS/MS). After the institution's written consent, the selected students received a formal and detailed invitation to participate about the study objectives, risks and benefits, so they had the opportunity to clarify any doubts and could voluntarily consent to collaborate with the research. In case of interest in participating in the study, the student was invited to sign an informed consent form (online).

2.3 Population and sample size

This project consisted of students enrolled in a higher education institution in the region of the state of Maranhão, Brazil, selected for convenience. The city has a Gini index of 0.56 (13). In 2020, the institution had 2,225 students enrolled in 9 undergraduate courses (Administration, Law, Physical Education, Nursing, Aesthetics and Cosmetics, Physiotherapy, Nutrition, Psychology and Social Work). The sample was calculated according to Nascimento-Ferreira's assumptions (14). The parameters used to calculate the sample size were: α of 0.05, β of 0.10 (or power of 90%), number of replicates of 2 and intraclass correlation coefficient of 0.65 (9). From these parameters, we estimated a sample of 41 individuals. Predicting 50% losses and rejections, in the first and second application, we established a minimum sample size of 62 participants to study the psychometric properties of the Pittsburgh Sleep Quality Index. However, the 24 h-MESYN project was designed to evaluate the psychometric properties of several epidemiological tools. Based on estimates of less robust psychometric properties, we invited 342 participants aiming to fulfill minimum aspects (α of 0.05 and β of 0.10) for all tested tools (12). The sample diversity was based on a stratified random distribution of students, according to sex (at least 60.0% for female biological sex), age (at least 25.0% for students up to 20 years old) and study program (at least 60.0% ongoing in the health area), based on previous representative cohort study (15).

2.4 Inclusion and exclusion criteria

All students had to enrolled at the Institution regularly, be at least 17 years of age, and provide signed an informed consent form were included in the study. Students aged 17 provided assent agreements, along with consent from their parent or legal guardian. The study exclusion criteria were: pregnancy, physical disability or failure to complete the questionnaires. In the case of pregnancy and physical disability, the students were evaluated, but excluded from the analyzes.

2.5 Study variables and instruments

The study variables were biological sex, age, nature of the course, academic period and sleep disorders. The information was accessed via an online form (available at: https://forms.gle/L92wXsVaxxfPNgpE8). We evaluated through self-reported questionnaires in Portuguese:

• Demographics and academics: biological sex, age, course and academic shift.

• Brazilian Portuguese version Pittsburgh Sleep Quality Index: this is a questionnaire with 19 self-reported questions that assess sleep quality over a period of one month (16). The same has its version translated into Portuguese being validated and reliable, in addition it comes down to 7 components, subjective sleep quality, sleep latency, sleep duration, habitual sleep efficiency, sleep disturbances, sleep medication use and daytime dysfunction (4). Thus, each item is evaluated separately and according to the participant's response, the score can range from 0 to 3, calculating at the end of the 7 components up to 21 points. In this logic, the higher the score, the worse the quality of sleep. At a global level, with a score above 5 points, the participant already has severe impairments in at least 2 items or moderate impairments in 3 items (4).

2.6 Data collection

The study was conducted by a multidisciplinary work team, composed of researchers who are undergraduates and graduates from courses in the health area. Previously, the team participated in training on the study protocol (approach, application and retrieval of online questionnaires) and sanitary measures. During the training, the team also reviewed the online version of the consent form and the questionnaire. We conducted the survey in three steps. In the first step, we invited students to participate in the study in person, respecting all sanitary and institutional norms. At this stage, we explain the project and send the link with the free and informed consent form (via the social network, e.g., WhatsApp). In the second step, after electronic confirmation of signing the consent form, the participants answered the questionnaire (Q1, first application). In the third step, 2 weeks after the first stage, we resent the link, and the participants answered the same questionnaire again (Q2, second application). In the last two steps, we contacted the participants via WhatsApp only.

2.7 Statistical analysis

We performed all statistical analyzes in Stata software (version 15.0, Stata Corporation, College Station, TX, USA). We established the criterion of statistical significance of 95% (p ≤ 0.05) in the hypothesis tests. The normality of the variables was evaluated with the Shapiro-Wilk test. In the sensitivity analysis, we applied the chi-square goodness of fit test. In the reliability analysis, we applied Spearman's correlation coefficient, with a cutoff point ≥0.30 (for acceptable reliability) (17). In the validity analysis, we assessed the dimensionality using confirmatory factor analysis. We employed structural equation modeling (SEM) to address the previously hypothesized dimensionality (6, 8), following the assumptions for developing and validating questionnaires (7). We evaluated the global model adjustment using the fit indices: root mean square error of approximation (RMSEA < 0.08), standardized root mean square residual (SRMR < 0.08), Comparative Fit Index (CFI > 0.90), Tucker-Lewis Index (TLI > 0.90) (18).

3 Results

Table 1 shows the sensitivity analysis, sociodemographic and university conditions of the research sample. In the Q1 and Q2, the sample consisted of 195 and 117 students, respectively. The largest share in both Q1 and Q2 was female, with 68.7% and 72.7%, respectively. The age ranged from 18 to over 36 years old, with the highest number between 21 and 25 years old, with 44.6% in the first application and 45.7% in the second. The courses varied between the areas of health and also included law and administration. Regarding sensitivity, no significant differences were found (p > 0.05).

Table 1

Table 1 Description of the sample and sensitivity analysis according to demographic and academic variables in the first and second application of the questionnaire.

Table 2 shows the reliability (temporal stability) assessment of the PSQI-BR. In this sense, we observed the Spearman correlation coefficient ≥0.30, considering the reliability as acceptable. Table 3 shows the construct validity (confirmatory factor analysis) of the PSQI-BR. The dimensionalities tested (path diagrams) by SEM are available in the Figure 1. All fit indices for the one-factor model showed acceptable adjustment. We observed better adjustment for the one-factor (RMSEA = 0.019; SRMR = 0.041; CFI = 0.992; TLI = 0.986; Figure 1A) solution compared with two-factor (RMSEA = 0.099; SRMR = 0.070; CFI = 0.764; TLI = 0.619; Figure 1B) and three-factor (RMSEA = 0.108; SRMR = 0.066; CFI = 0.763; TLI = 0.548; Figure 1C) solutions, respectively.

Table 2

Table 2 Descriptive results of Pittsburgh sleep quality Index (PSQI) scores and spearman correlation coefficient (rho) to estimate the reliability of the Pittsburgh sleep quality Index questionnaire in Brazilian adults.

Table 3

Table 3 Confirmatory factor analysis of the of the Pittsburgh sleep quality Index questionnaire (PSQI) in Brazilian college students based on factor solution model.

Figure 1

Figure 1 Coefficients for the one (A), two (B) and three-factor (C) model. DD, daily disturbance; SE; sleep efficiency; PSQ, perceived sleep quality; PSQI, Pittsburgh sleep quality index.

4 Discussion

The findings of our analysis suggest that online PSQI-BR questionnaire showed acceptable reliability and validity in college students from the low-income region. These results are very interesting for epidemiology, as they can help to identify the sleep quality in diverse and underrepresented populations in real-time data collection, even in conditions of restricted social contact, as experienced in a pandemic context.

Our study, conducted online among college students, revealed that a two-week interval offers acceptable temporal stability for the PSQI questionnaire. There is limited evidence on test-retest reliability in this population. As previously highlighted by Lash et al. (19), applying assessment methods in different populations and scenarios are essential steps to verify if the psychometric properties are maintained. However, the scientists identified acceptable temporal stability in Brazilian adolescents (9). This finding contributes to the ongoing debate regarding the reliability of subjective sleep quality reports. We offer potential explanations related to temporal stability. First, it is plausible that participants’ sleep patterns exhibit stability over a two-week period, contributing to the consistency observed in their responses. Additionally, the two-week interval may balance minimizing memory decay while capturing fluctuations in sleep patterns.

Our findings also indicated acceptable validity for the online version of the PSQI-BR in low-income college students. A comprehensive systematic review showed that the factor structure of PSQI was best explained by two-factor models (28.8% of the studies), followed by a one-factor model (20.0% of the studies) and three-factor models (17.8% of the studies) (6). Differently, our findings suggest the use of online PSQI-BR as a unitary construct in college students in contrast with studies that advocate the multifactorial application of this instrument (6, 8). We argue as a potential explanation that the observed construct of “sleep quality” covers a broad range of indicators relevant to sleep quality, as planned by PSQI developers (16). In the academic context, this is justified because longitudinal research during the pandemic shows that college students suffered from anxiety and poor sleep quality (20) mainly due to decreased social interactions (21), also they also developed “zoom fatigue” due to excessive hours of online classes (22). This is a reasonable explanation for why these constructs have presented higher loads in the analyzes we performed. For these reasons, and given the epidemiological evidence and usefulness backing up the unidimensional application of the PSQI in its ability to screen for sleep health, we advocate in favor of the global score of the PSQI as a potential marker of sleep quality in college students.

The present study is not without limitations, such as the sample selected by convenience, respondent bias (including social desirability) and the lack of epidemiological representativeness. Thus, the results should not be extended beyond the scope of the psychometric findings. Additionally, further studies should be conducted to confirm the validity of the PSQI in its online format across different population characteristics and backgrounds.

5 Conclusion

The online PSQI-BR has acceptable reliability and validity in Brazilian low-income regions. The questionnaire represents an easy and cost-effective way to measure remote sleep quality in college students from low-income regions.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by Centro Universitário do Maranhão (UNICEUMA). The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

ACFDM: Conceptualization, Writing – original draft. LCCdS: Conceptualization, Methodology, Writing – original draft. BSL: Conceptualization, Writing – original draft, Methodology. KAM: Investigation, Writing – review & editing. ETH: Investigation, Writing – review & editing. MVN-F: Investigation, Data curation, Supervision, Writing – review & editing.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article.

ACFDM was supported by UTHealth School of Public Health Start-up Funding. The 24h-MESYN study design, data collection and analysis were supported by the Brazilian Government from National Council for Scientific and Technological Development (CNPq; proc. 402391/2021-7) by the Federal University of Tocantins (PROPESQ Universal No. 088/2022). LC received a scientific initiation scholarship from the Instituto de Ensino Superior do Sul do Maranhão (IESMA/UNISULMA). BL received a scientific initiation scholarship from the National Council for Scientific and Technological Development through the Institutional Program of Scientific Initiation Scholarships of the Federal University of Tocantins (PIBIC/UFT).

Acknowledgments

All authors acknowledge the university dean/chair of both universities that agreed to participate in this observational study as well as the students for their voluntary participation in the 24h-MESYN (feasibility) study.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Gao C, Guo J, Gong TT, Lv JL, Li XY, Liu FH, et al. Sleep duration/quality with health outcomes: an umbrella review of meta-analyses of prospective studies. Front Med (Lausanne). (2021) 8:813943. doi: 10.3389/fmed.2021.813943

PubMed Abstract | Crossref Full Text | Google Scholar

2. Nakie G, Takelle GM, Rtbey G, Andualem F, Tinsae T, Kassa MA, et al. Sleep quality and associated factors among University students in Africa: a systematic review and meta-analysis study. Front Psychiatry. (2024) 15:1370757. doi: 10.3389/fpsyt.2024.1370757

PubMed Abstract | Crossref Full Text | Google Scholar

3. Nascimento-Ferreira MV, Collese TS, de Moraes AC, Rendo-Urteaga T, Moreno LA, Carvalho HB. Validity and reliability of sleep time questionnaires in children and adolescents: a systematic review and meta-analysis. Sleep Med Rev. (2016) 30:85–96. doi: 10.1016/j.smrv.2015.11.006

PubMed Abstract | Crossref Full Text | Google Scholar

4. Bertolazi AN, Fagondes SC, Hoff LS, Dartora EG, Miozzo IC, de Barba ME, et al. Validation of the Brazilian Portuguese version of the Pittsburgh sleep quality index. Sleep Med. (2011) 12(1):70–5. doi: 10.1016/j.sleep.2010.04.020

PubMed Abstract | Crossref Full Text | Google Scholar

5. Sperber AD. Translation and validation of study instruments for cross-cultural research. Gastroenterology. (2004) 126(1 Suppl 1):S124–8. doi: 10.1053/j.gastro.2003.10.016

PubMed Abstract | Crossref Full Text | Google Scholar

6. Manzar MD, BaHammam AS, Hameed UA, Spence DW, Pandi-Perumal SR, Moscovitch A, et al. Dimensionality of the Pittsburgh sleep quality index: a systematic review. Health Qual Life Outcomes. (2018) 16(1):89. doi: 10.1186/s12955-018-0915-x

PubMed Abstract | Crossref Full Text | Google Scholar

7. Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR, Young SL. Best practices for developing and validating scales for health, social, and behavioral research: a primer. Front Public Health. (2018) 6:149. doi: 10.3389/fpubh.2018.00149

PubMed Abstract | Crossref Full Text | Google Scholar

8. Dunleavy G, Bajpai R, Tonon AC, Chua AP, Cheung KL, Soh CK, et al. Examining the factor structure of the Pittsburgh sleep quality Index in a multi-ethnic working population in Singapore. Int J Environ Res Public Health. (2019) 16(23):4590. doi: 10.3390/ijerph16234590

PubMed Abstract | Crossref Full Text | Google Scholar

9. Passos MH, Silva HA, Pitangui AC, Oliveira VM, Lima AS, Araújo RC. Reliability and validity of the Brazilian version of the Pittsburgh sleep quality index in adolescents. J Pediatr (Rio J). (2017) 93(2):200–6. doi: 10.1016/j.jped.2016.06.006

PubMed Abstract | Crossref Full Text | Google Scholar

10. De Man J, Campbell L, Tabana H, Wouters E. The pandemic of online research in times of COVID-19. BMJ Open. (2021) 11(2):e043866. doi: 10.1136/bmjopen-2020-043866

PubMed Abstract | Crossref Full Text | Google Scholar

11. Szklo M, Nieto J. Epidemiology: Beyond the Basics. 3 ed Burlington, MA: Jones & Bartlett Learning (2014). p. 515.

Google Scholar

12. Nascimento-Ferreira M, Marin K, Abrão Ferreira R, Oliveira L, Bandeira A, Silva Sousa P, et al. 24 H movement behavior and metabolic syndrome study protocol: a prospective cohort study on lifestyle and risk of developing metabolic syndrome in undergraduate students from low-income regions during a pandemic. Front Epidemiol. 2:1010832. doi: 10.3389/fepid.2022.1010832

PubMed Abstract | Crossref Full Text | Google Scholar

13. Brasil. Índice de Gini da renda domiciliar per capita Maranhão (Período: 1991, 2000 e 2010) Ministério da Saúde. (2010). Available online at: http://tabnet.datasus.gov.br/cgi/ibge/censo/cnv/ginima.def

Google Scholar

14. Nascimento-Ferreira MV, De Moraes ACF, Toazza Oliveira PV, Rendo-Urteaga T, Gracia-Marco L, Forjaz CLM, et al. Assessment of physical activity intensity and duration in the paediatric population: evidence to support an a priori hypothesis and sample size in the agreement between subjective and objective methods. Obes Rev. (2018) 19(6):810–24. doi: 10.1111/obr.12676

PubMed Abstract | Crossref Full Text | Google Scholar

15. Barbosa JB, dos Santos AM, Barbosa MM, de Carvalho CA, Fonseca PC, Fonseca JM, et al. Metabolic syndrome, insulin resistance and other cardiovascular risk factors in university students. Cien Saude Colet. (2016) 21(4):1123–36. doi: 10.1590/1413-81232015214.10472015

PubMed Abstract | Crossref Full Text | Google Scholar

16. Buysse DJ, Reynolds CF, Monk TH, Berman SR, Kupfer DJ. The Pittsburgh sleep quality Index: a new instrument for psychiatric practice and research. Psychiatry Res. (1989) 28(2):193–213. doi: 10.1016/0165-1781(89)90047-4

PubMed Abstract | Crossref Full Text | Google Scholar

17. Strong WB, Malina RM, Blimkie CJ, Daniels SR, Dishman RK, Gutin B, et al. Evidence based physical activity for school-age youth. J Pediatr. (2005) 146(6):732–7. doi: 10.1016/j.jpeds.2005.01.055

PubMed Abstract | Crossref Full Text | Google Scholar

18. Hu L, Bentler P. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives, structural equation modeling. Struct Equ Modeling: A Multidiscip J. (2009) 6(1):1–5. doi: 10.1080/10705519909540118

Crossref Full Text | Google Scholar

19. Lash TL, Ahern TP, Collin LJ, Fox MP, MacLehose RF. Bias analysis gone bad. Am J Epidemiol. (2021) 190(8):1604–12. doi: 10.1093/aje/kwab072

PubMed Abstract | Crossref Full Text | Google Scholar

20. Lee H, Kim J, Moon J, Jung S, Jo Y, Kim B, et al. A study on the changes in life habits, mental health, and sleep quality of college students due to COVID-19. Work. (2022) 73(3):777–86. doi: 10.3233/WOR-220112

PubMed Abstract | Crossref Full Text | Google Scholar

21. David MCMM, Vieira GR, Leôncio LML, Neves LDS, Bezerra CG, Mattos MSB, et al. Predictors of stress in college students during the COVID-19 pandemic. J Affect Disord Rep. (2022) 10:100377. doi: 10.1016/j.jadr.2022.100377

PubMed Abstract | Crossref Full Text | Google Scholar

22. Nesher Shoshan H, Wehrt W. Understanding “zoom fatigue”: a mixed-method approach. Applied Psychology. (2022) 71(3):827–52. doi: 10.1111/apps.12360

Crossref Full Text | Google Scholar