Assessing Worry About Affording Healthcare in a General Population Sample

This study adapted a measure on worry about affording healthcare. The financial costs of healthcare are increasingly being shifted to patients. Financial burden from healthcare costs can be material (such as bankruptcy) or psychological. Psychological distress can be either worry about affording future care or distress due to material consequences and, despite evidence from clinical psychology that differentiates these types of emotional symptoms, this distinction has largely been ignored for financial burden in healthcare. We adapted a worry about affording healthcare scale for use in the general population (n = 398) to facilitate comparisons between disease groups and across countries. Participants completed a survey through an online platform. The worry about affording healthcare measure showed good reliability and validity through associations with quality of life (QOL) and measures of other types of financial burden. Worry about affording healthcare was also associated with cost-related non-adherence to medical care. Future research on patient QOL should consider worry about affording healthcare.

Introduction

Healthcare costs have been rising (Gordon N. et al., 2017). Theories and systematic reviews of financial burden in healthcare have distinguished material financial burden (bankruptcy, draining savings) due to healthcare costs from psychological financial burden (Altice et al., 2017; Gordon L.G. et al., 2017). Material financial burden is also called financial strain or objective financial burden due to the focus on tangible stressors such as not having enough money for medications. Psychological financial burden is also called subjective financial burden or financial stress due to the focus on emotions and perceptions related to financial health.

Psychological financial burden includes two distinct concepts: (1) the distress and depressive-type symptoms (such as feeling down or having little interest in activities) that result from the stress of material financial burden, and (2) worry and anxiety about affording future healthcare (Beck and Haigh, 2014). This distinction is particularly important as research has shown anxiety and depression are associated with different outcomes: depression and rumination is associated with insomnia, while anxiety and worry is associated with cortisol and heart rate variability (Carney et al., 2010; Ciesla et al., 2011; Aldao et al., 2013; Crowley et al., 2015; Lewis et al., 2017). Anxiety and worry need to reference a specific topic, rather than be general, to predict quality of life (QOL) and other outcomes (Jerndal et al., 2010; Buss et al., 2011; Reck et al., 2013). Previous work on financial burden has largely ignored the difference between different types of psychological financial burden (distress vs. general anxiety vs. worry specific to affording healthcare). Measures distinguishing worry about affording healthcare from other types of psychological financial burden, including general financial worry and anxiety, are needed.

The theory of conservation of resources (Hobfoll, 1989) is particularly useful for understanding the role of worry about affording healthcare. In the conservation of resources theory, people are motivated to prevent loss of resources such as money, and the threat of losing resources may be more motivating than actual loss (Halbesleben et al., 2014). For this reason, we elected to focus on worry rather than anxiety as it might better assess the perceived threat of resource loss, since worry is more cognitive while anxiety reflects emotions. Worry about affording healthcare would then represent a concern about a loss of resources that motivates cost-related non-adherence to medical treatments as a way to conserve resources. This could be motivating even for people without medical conditions as it could lead to less use of preventive care like cancer screenings, dental visits and optometry services.

To address the lack of a measure focusing on worry about affording healthcare measure, this study adapted the worry about affording healthcare scale (WAHS) for use in the general population. The measure was initially developed in Multiple Sclerosis patients living in the United States (Jones and Amtmann, 2014). The measure was specifically adapted to facilitate comparisons across countries and disease groups as well as comparisons between disease groups and healthy controls. Even in countries with nationalized healthcare, costs are increasingly being shifted to the patient (Hossein and Gerard, 2013). Countries with universal healthcare, such as the United Kingdom, Canada, and Australia, still have supplemental private insurance (International Health Care System Profiles, 2019) suggesting concerns about cost may still be an issue. To facilitate comparisons of how worry about affording healthcare may differentially affect QOL and healthcare use between countries and groups, we created a measure of worry about affording healthcare that could be used in nearly any setting or disease group.

Methods

Sample and Procedures

Our sample consisted of 398 adults recruited through Prolific Academic, a crowdsourcing website that helps people participate in online studies that has been used in numerous peer-reviewed publications (Palan and Schitter, 2018). Previous research has shown that the Prolific platform is likely less biased than alternatives such as in-person survey panels of students and Mechanical Turk (Peer et al., 2017). This is possibly because Prolific was originally created to support academic research whereas other platforms were created for private industry. Once a survey is posted to Prolific Academic, eligible participants either receive notification that a new study has been posted or they regularly check the website for new studies. Potential participants then click the survey link, read the consent form and, if they agree to the study, complete the survey. Participants completing the survey received USD$8. Participants had to be 18 years of age and able to read and write English. Participants also had to answer three of four attention check questions correctly. The survey was administered through Limesurvey on September 4, 2018. Informed consent was obtained through the online survey. We sought feedback from a paid patient consultant about changes to the worry about affording healthcare measure that would make the measure applicable outside the United States. Based on her feedback, we elected to test two versions of the wording for the WAHS. Limesurvey randomly assigned participants to receive one of the two versions of the survey wording: concern about affording healthcare or fearful about affording healthcare. This study was reviewed by the Fred Hutchinson Cancer Research Center institutional review board. The datasets for this manuscript are not publicly available because information on medical conditions was collected. Requests to access the datasets should be directed to the corresponding author.

Measures

Worry About Affording Healthcare Scale (WAHS)

The WAHS was originally created as a five-item scale to assess worry about affording healthcare in Multiple Sclerosis patients in the United States (Jones and Amtmann, 2014). However, due to substantial differences in the United States-based healthcare system compared to other countries, we edited the questions and had a patient consultant review the revised items. We edited the items so they would be applicable outside the United States and for other disease groups. We made the following changes for the general population version, we removed two questions about affording premiums or having insurance canceled. We changed one item on income going down and not being able to afford premiums, so it instead asked about income going down and not being able to afford healthcare services. We also added an item on worry about affording medical devices. Two questions on worry about affording healthcare services and prescriptions remained unchanged. The final scale had four items: income going down; affording healthcare services; affording prescriptions; and affording medical devices (Supplementary Materials). Each item is rated on a four-point scale (0 = “not at all concerned/fearful”, 1 = “not too concerned/fearful”, 2 = “Somewhat concerned/fearful,” and 3 = “Very concerned/fearful”). The total score is calculated by summing the items and ranges from 0 to 12.

General Financial Health

Participants completed two measures on general financial health for validation of the WAHS. First, the Financial Anxiety Scale (FAS) is a seven-item measure with items corresponding to the symptoms of Generalized Anxiety Disorder but specific to financial problems (American Psychiatric Association [APA], 2000) and items are rated on a 1–7 scale (Archuleta et al., 2013). The FAS has been shown to have good reliability and validity (Archuleta et al., 2013). Second, the Financial Well-being Scale (FWS) from the Consumer Financial Protection Bureau (CFPB) was administered. The FWS has ten items, each rated on a five-point scale, and five items are reverse-scored (Consumer Financial Protection Bureau, 2015). Raw sum scores are then converted to an item response theory (IRT) based scoring using a conversion table. For both the FAS and FWS, higher scores indicate better financial health.

General Anxiety

Also, for validation of the WAHS, participants completed the four-item version of the Patient-Reported Outcomes Measurement Information System (PROMIS) Anxiety Scale (Cella et al., 2010). Participants rate four symptoms of anxiety on a five-point scale. Raw sum scores are also converted to IRT-based t-scores (population mean of 50, standard deviation of 10) using a conversion table. Higher scores indicate more anxiety.

Quality of Life (QOL)

We assessed QOL using the two-item PROMIS measure for mental health (Hays et al., 2017). Each item is rated on a five-point scale. Raw scores are also converted to IRT-based t-scores. Higher scores indicate better QOL.

Medical Conditions and Demographics

Participants also reported their age, gender, education, marital status, region of current residence, and race/ethnicity. Participants also completed the Self-administered Comorbidity Questionnaire (SCQ) (Sangha et al., 2003). The SCQ has participants self-report diagnoses with 12 health conditions, whether they receive treatment and whether the condition limits their activities. Scores range from 0 to 36 on the SCQ with higher scores indicating more medical burden. The 12 medical conditions on the SCQ are: heart disease, high blood pressure (hypertension), lung disease, diabetes, ulcers or stomach disease, kidney disease, liver disease, anemia or blood disease, depression, osteoarthritis, and back pain or rheumatoid arthritis.

Cost-Related Non-adherence to Medical Care

Participants were also asked to respond to five questions about not adhering to medical recommendations due to costs. Items included in the measure were cutting back on prescriptions, not buying prescriptions, not making doctors’ appointments, not using medical services, and not having medical tests. Each item was rated on a three-point scale (never, sometimes, and often). If participants mark “sometimes” or “often” for any item, they are classified as experiencing cost-related non-adherence. This measure has previously been shown to have validity and reliability in those with cancer (Khera et al., 2014).

Psychometric Analyses

Reliability of the two versions of the WAHS was assessed using Cronbach’s alpha. We assessed validity of the two versions of the WAHS through Spearman’s correlations with the FAS, FWS, PROMIS-Anxiety Scale, PROMIS QOL Scales, and the SCQ. We also conducted two multiple linear regressions with the scores from the WAHS as criterion variables. The predictor variables were age, gender (male vs. female and non-binary), race/ethnicity (monoracial, non-Hispanic white vs. other groups), education (graduate degree vs. bachelor’s degree vs. some college, associate’s or technical degree vs. high school diploma or less), marital status (single vs. long-term vs. married), region (United States vs. Europe vs. other regions), and cost-related non-adherence. Gender, race/ethnicity, education, marital status, and region used dummy coded variables. We also compared the means between the two different wordings of the WAHS using a t-test, and we compared the specific items between the versions of the WAHS using differential item function (DIF). DIF uses IRT to determine whether the wording biased responses to specific items while controlling for the overall level of worry about affording healthcare. IRT creates a logistic model for each item using two parameter sets per item (Samejima, 1969). The first parameter estimates how accurately the item reflects the underlying construct, worry about affording healthcare in this case. The second parameter set estimates how much of the construct a person needs (or how worried someone needs to be) before they mark a particular response category for that item and are often called ability parameters. In DIF, separate IRT models are created for two groups (wording of the WAHS in this case) and compared for statistically significant differences in the accuracy and ability parameters while controlling for overall level of the construct. A lack of DIF would ensure that any differences seen between the two versions of the WAHS was due to the wording change (concerned vs. fearful) and not due to an interaction of a specific item with the wording. We also conducted a receiver operating curve (ROC) analysis for the WAHS with cost-related non-adherence as the outcome. IRT analyses were conducted using IRTPRO 4.2. All other analyses were conducted using SPSS 25 or 26.

Results

Sample Description

We originally recruited 400 participants but excluded two because they did not correctly answer three out of the four attention check questions. All other 398 participants correctly answered three or four of the four attention check questions. Descriptive statistics for the sample are reported in Table 1. Two-thirds of the sample (66.8%, n = 266) reported at least one of the 12 medical conditions on the survey. The mean score on the SCQ was 2.63 (SD = 2.88) with scores ranging from 0 to 18. The most common conditions were depression (44.7%, n = 178), back pain (38.4%, n = 153), high blood pressure (11.3%, n = 45), anemia (7.5%, n = 30), and osteoarthritis (7.5%, n = 30). In United States Dollars, the mean yearly income was $31,689 and the median was $23,377. Consistent with previous studies using Prolific Academic, participants were young and had some post high school education. The mean for the WAHS-concerned was 4.05 (SD = 3.74), and the WAHS-fearful was 4.04 (SD = 3.92); these were not significantly different (t(396) = 0.034, p = 0.973).

TABLE 1

Table 1. Sample description (n = 398) and standardized regression coefficients with WAHS from multiple linear regressions.

Psychometric Results

The reliability and validity for the different versions of the WAHS are reported in Table 2. Cronbach’s alpha was over 0.9 for both versions, and both versions were significantly correlated with FAS, FWS, PROMIS-Anxiety Scale, the PROMIS QOL scale, and SCQ in the hypothesized directions. The DIF analyses did not show any statistically significant bias from the different wording on the WAHS items (all p’s > 0.05; see Figure 1 for item characteristic curves and Figure 2 for the information curve). The DIF IRT model had a root mean square error of approximation (RMSEA, an indicator of model fit to the data) of 0.04, indicating a good fit for the model (RMSEA’s <0.1) (Browne and Cudeck, 1993). Results for the multiple linear regression are reported in Table 1. In multivariate analyses, the WAHS-concerned was only significantly associated with cost-related non-adherence. In contrast, the WAHS-fearful was significantly associated with cost-related non-adherence, region and education level. The coefficients for cost-related non-adherence were large (WAHS-concerned: 0.449; WAHS-Fearful: 0414). The statistically significant coefficients for region and education with the WAHS-Fearful were small (−0.184 to −0.224). The means for the European participants were still 3.48 on a 0–12 scale, compared to 5.29 for American participants, indicating that despite greater likelihood of nationalized healthcare coverage, participants in Europe reported some minimal worry about affording healthcare. Because the WAHS-fearful showed better validity than the WAHS-concerned, we only conducted the ROC analysis on the WAHS-fearful. The area under the curve was 0.766 (95% confidence interval: 0.698 and 0.834). A score of four or greater on the WAHS-fearful had the best balance of sensitivity and specificity for detecting cost-related non-adherence (see Figure 3).

TABLE 2

Table 2. Means, reliability, and validity for the worry about affording healthcare scale.

FIGURE 1

Figure 1. Item characteristic curves for the item response theory analyses. C, Category.

FIGURE 2

Figure 2. Information for the item response theory analyses.

FIGURE 3

Figure 3. Sensitivity, specificity and information for values of the WAHS-fearful. The criterion was any cost-related non-adherence to medical care. WAHS, worry about affording healthcare scale.

Discussion

This study reported on the adaptation and validation of the WAHS for use in the general population. Two different versions of the wording were tested; using “fearful” was associated with more socioeconomic indicators than “concerned.” Overall, the scale was reliable and showed validity through associations with cost-related non-adherence, medical diagnoses and other financial burden measures. It is important to note that the relationship to cost-related non-adherence remained in the multivariate analysis that controlled for potential confounds. Because the word “fearful” was associated with more socioeconomic indicators, future studies should use this wording. As a score of four or higher provided the best balance of sensitivity and specificity for detecting cost-related non-adherence, clinical use of the WAHS should use this cutoff score.

The study had several limitations. The sample size was relatively small although most measure validation studies are of comparable size (Williams et al., 2004). Because an online survey platform was used, there was likely a selection bias for people who could access the internet and would be willing to participate on such a platform. However, online survey platforms can overcome other selection biases as these allow people to participate who are not able to travel to participate in research, have to move frequently or might not be able to complete a telephone survey for economic or health reasons. As we were unable to ask detailed questions about location to control for cost of living, we could not use income as a variable. Future studies should consider subjective income questions such as those on the European Social Survey (ESS Round 8: European Social Survey, 2018). The sample was also predominantly Caucasian and had to read English, limiting generalizability. The low level of worry about affording healthcare in this sample also means the WAHS may not have sufficient reliability for higher levels of worry. Although the less than ideal sensitivity and specificity were noted, the outcome (cost-related non-adherence) was not an exact measure of the screening measure (the WAHS) so perfect concordance would not be expected. The study was also cross-sectional and a longitudinal validation study of the measure is warranted.

The WAHS has several potential uses in future studies and clinical care. First, it could be used to examine the role of worry about affording healthcare in motivating or demotivating healthcare use, such as cancer screening. Worry about affording healthcare could also be investigated as a reason for purchasing or not purchasing health insurance. Although our results showed some initial differences between the United States and Europe, more detailed and rigorous comparisons are warranted. Participants in Europe still reported slight worry about affording healthcare suggesting this might be an area for future research. People in the United States may also have different levels of worry about affording healthcare depending on the region and health insurance coverage. Different disease groups can also be compared to each other as well as people without medical conditions. As the WAHS is a relatively short measure (four items), it could be added to national or international surveys without substantially increasing participant burden or used as a screening measure in clinical care. Overall, worry about affording healthcare is a fruitful area of focus for future research.

Data Availability Statement

The datasets generated for this study are available on request to the corresponding author.

Ethics Statement

All procedures performed in this study were in accordance with the ethical standards of the Institutional Research Committee (the Fred Hutchinson Cancer Research Center, Human Subjects Division, #8703) and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Author Contributions

SJ: conceptualization of the study, data collection and analysis, drafted the manuscript, and interpretation of results. YD: conceptualization of analysis, drafted the manuscript, and interpretation of results. LP and NH: manuscript revisions and interpretation of results.

Conflict of Interest

YD was employed by Bayer Healthcare U.S. LLC.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2019.02622/full#supplementary-material

References

Aldao, A., Mennin, D. S., and McLaughlin, K. A. (2013). Differentiating worry and rumination: evidence from heart rate variability during spontaneous regulation. Cognit. Ther. Res. 37, 613–619. doi: 10.1007/s10608-012-9485-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Altice, C. K., Banegas, M. P., Tucker-Seeley, R. D., and Yabroff, K. R. (2017). Financial hardships experienced by cancer survivors: a systematic review. J. Natl. Cancer Inst. 109:djw205. doi: 10.1093/jnci/djw205

PubMed Abstract | CrossRef Full Text | Google Scholar

Archuleta, K. L., Dale, A., and Spann, S. M. (2013). College students and financial distress: exploring debt, financial satisfaction, and financial anxiety. J. Financ. Counsel. Plann. 24, 50–62.

Google Scholar

American Psychiatric Association [APA] (2000). Diagnostic and Statistical Manual of Mental Disorders: DSM-IV-TR, 4th Edn. Washington, DC: American Psychiatric Association.

Google Scholar

Beck, A. T., and Haigh, E. A. (2014). Advances in cognitive theory and therapy: the generic cognitive model. Annu. Rev. Clin. Psychol. 10, 1–24. doi: 10.1146/annurev-clinpsy-032813-153734

PubMed Abstract | CrossRef Full Text | Google Scholar

Browne, M., and Cudeck, R. (1993). Alternative Ways of Assessing Model Fit. London: Sage.

Google Scholar

Buss, C., Davis, E. P., Hobel, C. J., and Sandman, C. A. (2011). Maternal pregnancy-specific anxiety is associated with child executive function at 6-9 years age. Stress 14, 665–676. doi: 10.3109/10253890.2011.623250

PubMed Abstract | CrossRef Full Text | Google Scholar

Carney, C. E., Harris, A. L., Moss, T. G., and Edinger, J. D. (2010). Distinguishing rumination from worry in clinical insomnia. Behav. Res. Ther. 48, 540–546. doi: 10.1016/j.brat.2010.03.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., Yount, S., et al. (2010). The patient-reported outcomes measurement information system (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. J. Clin. Epidemiol. 63, 1179–1194. doi: 10.1016/j.jclinepi.2010.04.011

PubMed Abstract | CrossRef Full Text | Google Scholar

Ciesla, J. A., Dickson, K. S., Anderson, N. L., and Neal, D. J. (2011). Negative repetitive thought and college drinking: angry rumination, depressive rumination, co-rumination, and worry. Cogn. Ther. Res. 35, 142–150. doi: 10.1007/s10608-011-9355-1

CrossRef Full Text | Google Scholar

Consumer Financial Protection Bureau, (2015). Measuring Financial Well-Being: A Guide to Using the CFPB Financial Well-Being Scale. Washington, DC: Consumer Financial Protection Bureau.

Google Scholar

Crowley, M. J., Zullig, L. L., Shah, B. R., Shaw, R. J., Lindquist, J. H., Peterson, E. D., et al. (2015). Medication non-adherence after myocardial infarction: an exploration of modifying factors. J. Gen. Intern. Med. 30, 83–90. doi: 10.1007/s11606-014-3072-x

PubMed Abstract | CrossRef Full Text | Google Scholar

ESS Round 8: European Social Survey, (2018). ESS-8 2016 Documentation Report. Bergen: European Social Survey.

Google Scholar

Gordon, L. G., Merollini, K. M. D., Lowe, A., and Chan, R. J. (2017). A systematic review of financial toxicity among cancer survivors: we can’t pay the co-pay. Patient 10, 295–309. doi: 10.1007/s40271-016-0204-x

PubMed Abstract | CrossRef Full Text | Google Scholar

Gordon, N., Stemmer, S. M., Greenberg, D., and Goldstein, D. A. (2017). Trajectories of injectable cancer drug costs after launch in the United States. J. Clin. Oncol. 36, 319–325. doi: 10.1200/jco.2016.72.2124

PubMed Abstract | CrossRef Full Text | Google Scholar

Halbesleben, J. R. B., Neveu, J.-P., Paustian-Underdahl, S. C., and Westman, M. (2014). Getting to the “COR”: understanding the role of resources in conservation of resources theory. J. Manag. 40, 1334–1364. doi: 10.1177/0149206314527130

CrossRef Full Text | Google Scholar

Hays, R. D., Schalet, B. D., Spritzer, K. L., and Cella, D. (2017). Two-item PROMIS(R) global physical and mental health scales. J. Patient Rep. Outcomes 1:2. doi: 10.1186/s41687-017-0003-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Hobfoll, S. E. (1989). Conservation of resources. A new attempt at conceptualizing stress. Am. Psychol. 44, 513–524. doi: 10.1037//0003-066x.44.3.513

PubMed Abstract | CrossRef Full Text | Google Scholar

Hossein, Z., and Gerard, A. (2013). Trends in cost sharing among selected high income countries–2000-2010. Health Policy 112, 35–44. doi: 10.1016/j.healthpol.2013.05.020

PubMed Abstract | CrossRef Full Text | Google Scholar

International Health Care System Profiles (2019). International Health Care System Profiles. Available at: https://international.commonwealthfund.org/ (accessed March 18, 2019).

Google Scholar

Jerndal, P., Ringstrom, G., Agerforz, P., Karpefors, M., Akkermans, L. M., Bayati, A., et al. (2010). Gastrointestinal-specific anxiety: an important factor for severity of GI symptoms and quality of life in IBS. Neurogastroenterol. Motil. 22, 646–e179. doi: 10.1111/j.1365-2982.2010.01493.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Jones, S. M., and Amtmann, D. (2014). Health care worry is associated with worse outcomes in multiple sclerosis. Rehabil. Psychol. 59, 354–359. doi: 10.1037/a0037074

PubMed Abstract | CrossRef Full Text | Google Scholar

Khera, N., Chang, Y. H., Hashmi, S., Slack, J., Beebe, T., Roy, V., et al. (2014). Financial burden in recipients of allogeneic hematopoietic cell transplantation. Biol. Blood Marrow Transplant. 20, 1375–1381. doi: 10.1016/j.bbmt.2014.05.011

PubMed Abstract | CrossRef Full Text | Google Scholar

Lewis, E. J., Yoon, K. L., and Joormann, J. (2017). Emotion regulation and biological stress responding: associations with worry, rumination, and reappraisal. Cogn. Emot. 32, 1487–1498. doi: 10.1080/02699931.2017.1310088

PubMed Abstract | CrossRef Full Text | Google Scholar

Palan, S., and Schitter, C. (2018). Prolific.ac—A subject pool for online experiments. J. Behav. Exp. Finance 17, 22–27. doi: 10.1016/j.jbef.2017.12.004

CrossRef Full Text | Google Scholar

Peer, E., Brandimarte, L., Samat, S., and Acquisti, A. (2017). Beyond the turk: alternative platforms for crowdsourcing behavioral research. J. Exp. Soc. Psychol. 70, 153–163. doi: 10.1016/j.jesp.2017.01.006

CrossRef Full Text | Google Scholar

Reck, C., Zimmer, K., Dubber, S., Zipser, B., Schlehe, B., and Gawlik, S. (2013). The influence of general anxiety and childbirth-specific anxiety on birth outcome. Arch. Womens Ment. Health 16, 363–369. doi: 10.1007/s00737-013-0344-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Samejima, F. (1969). Estimation of Latent Ability Using a Response Pattern of Graded Scores. Richmond: Psychometric Society.

Google Scholar

Sangha, O., Stucki, G., Liang, M. H., Fossel, A. H., and Katz, J. N. (2003). The self-administered comorbidity questionnaire: a new method to assess comorbidity for clinical and health services research. Arthritis Rheum. 49, 156–163. doi: 10.1002/art.10993

PubMed Abstract | CrossRef Full Text | Google Scholar

Williams, J. W. Jr., Stellato, C. P., Cornell, J., and Barrett, J. E. (2004). The 13- and 20-item hopkins symptom checklist depression scale: psychometric properties in primary care patients with minor depression or dysthymia. Int. J. Psychiatry Med. 34, 37–50. doi: 10.2190/U1B0-NKWC-568V-4MAK

PubMed Abstract | CrossRef Full Text | Google Scholar