Family physicians overestimate diagnosis probabilities regardless of the test results

Introduction: As useful tools for clinical decision-making, diagnostic tests require careful interpretation in order to prevent underdiagnosis, overdiagnosis or misdiagnosis. The aim of this study was to explore primary care practitioners’ understanding and interpretation of the probability of disease before and after test results for six common clinical scenarios.

Methods: This cross-sectional study was conducted with 414 family physicians who were working at primary care in Istanbul via face-to-face interviews held between November 2021 and March 2022. The participants were asked to estimate the probability of diagnosis in six clinical scenarios provided to them. Clinical scenarios were about three cancer screening cases (breast, cervical and colorectal), and three infectious disease cases (pneumonia, urinary tract infection, and COVID-19). For each scenario participants estimated the probability of the diagnosis before application of a diagnostic test, after a positive test result, and after a negative test result. Their estimates were compared with the true answers derived from relevant guidelines.

Results: For all scenarios, physicians’ estimates were significantly higher than the scientific evidence range. The minimum overestimation was positive test result for COVID-19 and maximum was pre-test case for cervical cancer. In the hypothetical control question for prevalence and test accuracy, physicians estimated disease probability as 95.0% for a positive test result and 5.0% for a negative test result while the correct answers were 2.0 and 0%, respectively (p < 0.001).

Discussion: Comparing the scientific evidence, overestimation in all diagnostic scenarios, regardless of if the disease is an acute infection or a cancer, may indicate that the probabilistic approach is not conducted by the family physicians. To prevent inaccurate interpretation of the tests that may lead to incorrect or unnecessary treatments with adverse consequences, evidence-based decision-making capacity must be strengthened.

1 Introduction

Diagnostic tests are helpful tools to facilitate deciding the correct diagnosis in line with the medical history and symptoms of the patients in terms of clinical decision-making (1). Application of the principles of evidence-based medicine helps clinicians make better diagnostic and management decisions. All diagnostic procedures, including laboratory tests, are based on probability estimations that need careful interpretation. Unnecessary request or misinterpretation of diagnostic tests may lead to underdiagnosis, overdiagnosis or misdiagnosis (2). Misinterpretation can adversely affect treatment, recovery, and health expenditures (3).

The accuracy of the tests which can be estimated as sensitivity, specificity and predictive values is decisive when deciding on diagnosis and treatment. The prevalence of diseases is also a measure which strongly influences the positive predictive value: the lower the prevalence, the lower is the probability of being sick even after a positive test (4).

Methods regarding accuracy, reproducibility and probability estimations for diagnostic tests are provided to medical school students through clinical epidemiology and evidence-based medicine curriculum (5). However, studies show that physicians do not comprehensively understand and interpret the probabilities during their clinical practices (6). The main consequences of this phenomena is overestimation of both positive and negative test results by physicians (7–9). The same problem exists among primary care physicians who have a key role in screening programs and outpatient services (10).

In the Turkish healthcare system primary care and preventive services are provided by family physicians for the registered populations (11). Family physicians are general practitioners (GP) or specialists and have an important role in the screening in breast, cervical and colorectal cancers as a part of national control programs in addition to primary care practices (12). Respiratory system and urinary tract infections are among the most common causes of admission to their offices (13). Therefore, family physicians are expected to use and interpret the test results appropriately and estimate correct probabilities of these diseases. This issue has become more important during the COVID-19 pandemic when tests for infection detection were widely performed, and their correct interpretation was important (14).

In this study, our aim was to explore primary care practitioners’ understanding and interpretation of the probability of a disease before and after test results for six common clinical scenarios.

2 Materials and methods

This cross-sectional study was conducted among family physicians working at primary care services in Istanbul during November 2021 and March 2022. Sample size was calculated as 354 assuming a prevalence of risk overestimation of 50%, with 95% confidence intervals within ±5%. All primary care physicians in five geographically dispersed districts of İstanbul (Başakşehir, Eyüpsultan, Fatih, Sultanbeyli, Üsküdar) were included in the study without using any sampling method (n = 613). Among them, 414 physicians have participated in the study, with a 67.5% response rate. Data was collected during face-to-face interviews.

The questionnaire consisted of two sections. The first section contained seven questions regarding the participants’ sociodemographic and professional characteristics. The second section contained questions about the probability of diseases in six clinical scenarios (Appendix 1). These scenarios are adapted from the study of Morgan et al. (10) in consideration of the common health problems which are expected to be diagnosed and/or treated by family physicians in Türkiye. Scenarios included three cancer types (breast, cervical and colorectal) within scope of the national cancer control program, two frequent infectious diseases (pneumonia and urinary tract infection) in primary care, and ongoing COVID-19 pandemic. Cases in the cancer screening scenarios were asymptomatic while they were symptomatic in infectious disease scenarios. Each scenario was prepared in agreement with the recent literature. A hypothetical control question measuring the understanding of the participants on the accuracy of diagnostic tests was also included.

Mammography for breast cancer, pap smear for cervical cancer, stool occult blood test for colorectal cancer, chest radiography for pneumonia, complete urinalysis for urinary tract infection, and PCR test for COVID-19 were used as diagnostic tests and the participants were asked probability of the given disease in three conditions: (a) before performing the diagnostic test, (b) after a positive test result, and (c) after a negative test result. The participants’ responses were compared to the test accuracy values from existing evidence-based literature. True answers to the questions were determined considering the most relevant national and international guidelines for physicians in Türkiye (Appendix 1).

Python programming language was used for data analysis. Descriptive statistics to summarize the data were frequency, percentage, median, interquartile range (IQR) for non-normally distributed continuous variables, the mean, and the standard deviation for normally distributed continuous variables. The Wilcoxon signed-rank test was used to compare physicians’ estimates of disease probabilities and probabilities derived from the evidence. Type I error (α) level of 0.05 was used in the interpretation of the significant test results.

This study was conducted with the permission from the Ministry of Health of the Turkish Republic (07/10/2021) and was approved by the Ethics Committee of Non-Invasive Clinical Studies of Istanbul Medipol University (15/10/2021, No: 1021).

3 Results

414 family physicians participated in the study. 53.9% (n = 223) of them were male, 86.2% (n = 357) general practitioners and the mean age was 42.9 ± 10.3. The characteristics of the study sample are summarized in Table 1.

TABLE 1

Table 1. Descriptive characteristics of the participants.

The density distribution of physicians’ estimates of diagnostic probabilities for each scenario is presented in Figure 1. For all scenarios, physicians’ estimates were significantly higher than the scientific evidence range (Appendix 2; Supplementary Table S1).

FIGURE 1

Figure 1. Estimates of diagnostic probabilities for each case description* *Colored vertical bars indicate the evidence range.

For breast cancer, the median of physicians’ estimation was 10.0% (evidence was 0.2%) for pre-test disease probability (p < 0.001). Both for cervical and colorectal cancer, the median of physicians’ estimation was 5.0% (evidence was, respectively, 0.01 and 0.06%) for pre-test disease probability (for both comparisons, p < 0.001). After a positive test result, physicians estimated the disease probability as 50.0% both for breast cancer and cervical cancer while evidence was, respectively, 8.7 and 0.14% (for both comparisons, p < 0.001). After a positive test result, physicians estimated the colorectal cancer probability as 40.0% (evidence was 0.74%; p < 0.001). When test results were negative, physicians estimated the disease probability as 10.0% (evidence was 0%) for breast cancer, as 5.0% (evidence was 0.0032%) for cervical cancer and 5.0% (evidence was 0.02%) for colorectal cancer (for both comparisons, p < 0.001).

For pneumonia and urinary tract infection, physicians’ estimations were 20.0% and 40.0%, respectively, for pre-test disease probability, while the evidence was 5.0% and 1.0%. After a positive test result, these estimations increased to 85.0% and 90.0% respectively, while the evidence was less than 10.0%. When the test results were negative, physicians estimated the disease probability as 10.0% whereas the evidence was less than 5.0%.

For COVID-19, the median of physicians’ estimation was 80.0% (IQR 50.0–90.0%) for pre-test disease probability while evidence was 56.0% (p < 0.001). After a positive test result, physicians estimated the disease probability as 99.0% (IQR 90.0–100.0%) whereas evidence was 95.4% (p < 0.015). When test results were negative, physicians estimated the disease probability as 50.0% (IQR 25.0–70.0%) while evidence was 0.04% (p < 0.001).

When physicians were provided with prevalence and test accuracy information of the hypothetical control question, they estimated disease probability as 95.0% (IQR 95.0–100.0%) for a positive test while the true answer was 2.0% (p < 0.001) and 5.0% (IQR 5.0–10.0%) for a negative test whereas the true answer was 0% (p < 0.001).

Table 2 shows the breakdown of disease probability estimations by physician groups. Although specialists showed a slightly lower tendency toward overestimation than general practitioners in several scenarios, their estimates were not close to the evidence ranges. Subgroup analyses showed that female participants estimated probabilities higher than males with a median difference around the range of 2.0–15.0, but estimations of participants were consistent across districts (Appendix 2; Supplementary Tables S2–S3).

TABLE 2

Table 2. Estimated disease probabilities for each scenario by physician groups.

Physicians’ probability estimations were significantly different from the evidence values (Appendix 2; Supplementary Table S4). Further subgroup analyses by physician group, gender, and district were provided in Appendix 2; Supplementary Tables S5–S7.

4 Discussion

In this study, we investigated diagnostic probability estimations of the family physicians about different scenarios regarding six hypothetical clinical cases. Three of the cases were pneumonia, COVID-19, and UTI, which are frequently encountered infectious diseases in primary care; the other three were cervical cancer, breast cancer, and colorectal cancer, which are routinely screened in primary care within the scope of the national cancer control program. Most striking finding of the study was the overestimation of all diagnoses before and after test results for the given scenarios.

The overestimation in all scenarios shows that the underlying potential causes of overestimation are not specific to the clinical cases but represents a more general problem. It is noteworthy that the overestimation varies between 1.04 times (COVID-19, after positive test) and 1,250 times (cervical cancer, before test), and it is over 10 times in 13 of the 18 alternatives examined. As was found in Morgan et al.’s study, these results are related to the overestimates of pre-test probability (10).

Overestimated responses given to the negative test results in all cases show that physicians overemphasize the symptoms when deciding on the diagnosis. Besides, the overestimated answers given to the hypothetical control question indicate that the physicians do not have comprehensive knowledge of probability estimations. The fact that the overestimation in our study was similar to previous studies reveals the universality of the problem. In a review article investigating how healthcare professionals interpret the results of diagnostic tests, it was stated that the probability estimates were in the direction of overestimating, regardless of whether the test result being positive or negative, and it has been concluded that commonly used measures of test accuracy are poorly understood by health professionals (7).

Family physicians in primary care have a key role in the management of clinical cases examined in our study. They work as individual health consultants who deal with all the health problems of their enrollees, provide preventive services and who are expected to solve the handleable problems at the primary care level, or to refer the complicated, unresolved cases to further levels and then follow up (15). Therefore, their role is important not only in curative services but also in primary and secondary prevention such as cancer screening.

In our study, pretest probability was higher in cancer screening tests than in UTI and pneumonia, whereas it was higher in UTI and pneumonia than in breast cancer screening in Morgan et al.’s study. High levels of overestimation in presented cancer scenarios can be concluded as the general perception of physicians toward cancer screening tests. Comparable results in other studies on this matter have also shown that physicians tend to overestimate the risk of cancer (16, 17).

Probability of UTI was more overestimated than pneumonia in before and after negative test scenarios and this result may be regarded because of physicians’ prioritization of the patient’s symptoms compared to the test results while diagnosing UTI. According to the literature for UTI management, pretest probability, which is estimated with the patient’s current symptoms, can significantly affect the post-test probability and in clinically high-risk cases, the post-test probability is evaluated as high even if the test result is negative (18).

There was also an overestimation in COVID-19 case scenarios. However, the frequency of overestimated responses given before test and after positive test result (1.43 and 1.04, respectively) were lower than all other case scenarios. We conclude this finding as the result of availability of up-dated information for COVID-19 management prepared by the Ministry of Health (19).

Diagnostic tests are valuable tools when evaluated together with the patient’s symptoms and lead the physician to an exact diagnosis. Although some of them have successful diagnostic performance, some are not the gold standard and only used for screening which requires advanced procedures to confirm diagnosis (17). In fact, diagnosis ultimately depends on the physician’s decision. Symptoms, test results and consultations are important parts to decide an appropriate diagnosis and treatment. Current developments in medicine provide test alternatives to physicians, but it is criticized as the dependency on technological solutions puts the experiences of physicians on the shelf and they mostly rely on the test results rather than their own experience (20). From this perspective, while diagnostic tests have a noteworthy role, they may cause overestimation, as seen in our study.

Physicians’ decision-making is considered critical for patient safety, as diagnostic errors and inappropriate treatments can harm patients, fail to address actual problems and waste resources (21, 22). It was shown that if family physicians misdiagnose, they mostly misregulate the treatment (23). Misinterpreting test results during decision-making can have adverse effects on the patients. For instance, in cases such as pneumonia and UTI that may require antibiotic treatment, the overestimation can lead to unfavorable outcomes in terms of antibiotic resistance (24).

We found another important overestimation in cancer screening tests. These tests are not definitive diagnostic tools, and they only lead to further procedures to confirm diagnosis. In cancer screenings, it can be interpreted as favorable that physicians attribute more value to screening tests in order not to miss probable cases. However, the risk of labeling for false positive individuals and its social and psychological consequences should always be kept in mind (25).

Post-test probability is expected to be estimated by considering pre-test probability (i.e., prevalence) and test accuracy (i.e., sensitivity and specificity). The reasons for physicians’ poor performance in probability estimation are due to lack of knowledge or misunderstanding (26). They mostly do not use likelihood ratio and pre-test probability in their estimations, eventually, test results have been overestimated (7, 27). In a study evaluating physicians’ probabilistic approach to the test results, no significant difference was found according to the type of data (sensitivity-specificity vs. likelihood ratios) (28). In another study investigating the change between the pre-test and post-test estimations according to the type of data shared, the successful estimations were 8% for sensitivity-specificity, 34% for the likelihood ratio, and 73% for the graphic form. Researchers have emphasized that clinicians may have difficulties understanding values that require arithmetic calculation, but it is easier for them to understand with visual tools such as graphs (29). Although we did not inquire about the reasons in our study, the poor performance in the hypothetical question may have been caused by a misunderstanding of the question due to the technical terms. An accurate estimation of the probability of the test results is a fundamental competence during clinical decision-making (30), and the lack of knowledge on methodological topics can be eliminated with evidence-based medicine training (31).

When the sensitivity and specificity are constant, the predictive values change according to the prevalence while evaluating the test results (32). However, most physicians cannot consider the prevalence when estimating the probability of disease after a positive test result, regardless of clinical experience and the institution they work for (26, 29). On the other hand, it is necessary to know the prevalence of the disease in a country or region in order to estimate probabilities correctly during a clinical decision, and ignorance of the frequency is considered a bias (20). Yet, the updated prevalence of major diseases usually is not available for the participants of our study. They should be shared and updated regularly by the health authorities.

Our study has some limitations. Scenarios prepared in line with the current scientific guidelines were directed to physicians. However, we do not know the real situation as cases applying in real world experiences may not fit these ideal scenarios. A limitation is that the diagnostic criteria change in time and there is not any guide specific to the population in which the study was conducted. Except for the COVID-19, international data could be used for probability estimations in scenarios instead of national ones. Because the data were collected as face-to-face observation, the participants may have stated differently from the Hawthorne effect compared to their daily practice. Despite these limitations, the overestimation was excessive in all clinical cases, and cannot only be explained by confounding variables or misinterpretation of the questionnaire.

In conclusion, in the present study, primary care physicians consistently overestimated the actual risk of disease regardless of the results of diagnostic or screening tests, despite the relatively high frequency of such diagnoses, and the availability of well performing tests with their performance parameters. This problem makes it difficult to fulfill patients’ needs as it reduces the accurate decision-making by physicians when selecting diagnosis and treatment. The findings of this study indicate the extensiveness and the magnitude of the problem and warrant interventions to improve the quality of primary care. However, to determine the kind of intervention that addresses the issue best, there is a need for qualitative studies to illuminate why and how physicians overestimate the disease probabilities.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

This study involving humans were approved by Ethics Committee of Non-Invasive Clinical Studies of Istanbul Medipol University (15/10/2021, No: 1021). This study was conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

ÖA, HK, AF, AP, SÇ, MNA, MA, MS, YT, and OH: conception and design. MNA, MA, MS, and YT: acquisition of data. ÖA and HK: analysis. ÖA, HK, AF, AP, and SÇ: interpretation. ÖA, HK, AF, AP, SÇ, and OH: draft and revision.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fmed.2023.1123689/full#supplementary-material

References

1. Armstrong, KA, and Metlay, JP. Annals clinical decision making: using a diagnostic test. Ann Intern Med. (2020) 172:604–9. doi: 10.7326/M19-1940

CrossRef Full Text | Google Scholar

2. Zhi, M, Ding, EL, Theisen-Toupal, J, Whelan, J, and Arnaout, R. The landscape of inappropriate laboratory testing: a 15-year meta-analysis. PLoS One. (2013) 8:1–8. doi: 10.1371/journal.pone.0078962

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Lubin, IM, Astles, JR, Shahangian, S, Madison, B, Parry, R, Schmidt, RL, et al. Bringing the clinical laboratory into the strategy to advance diagnostic excellence. Diagnosi. (2021) 8:281–94. doi: 10.1515/dx-2020-0119

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Parikh, R, Mathai, A, Parikh, S, Chandra Sekhar, G, and Thomas, R. Understanding and using sensitivity, specificity and predictive values. Indian J Ophthalmol. (2008) 56:45–50. doi: 10.4103/0301-4738.37595

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Ismach, R. Teaching evidence-based medicine to medical students. Acad Emerg Med. (2004) 11:e6–e10. doi: 10.1197/j.aem.2004.08.037

CrossRef Full Text | Google Scholar

6. Steurer, J, Fischer, JE, Bachmann, LM, Koller, M, and Riet, G. Communicating accuracy of tests to general practitioners: a controlled study. BMJ. (2002) 324:1391a–a. doi: 10.1136/bmj.324.7350.1391/a

CrossRef Full Text | Google Scholar

7. Whiting, PF, Davenport, C, Jameson, C, Burke, M, Sterne, JA, Hyde, C, et al. How well do health professionals interpret diagnostic information? A systematic review. BMJ Open. (2015) 5:1–8. doi: 10.1136/bmjopen-2015-008155

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Manrai, AK, Bhatia, G, Strymish, J, Kohane, IS, and Jain, SH. Medicine’s uncomfortable relationship with math: calculating positive predictive value. JAMA Intern Med. (2014) 174:991–3. doi: 10.1001/jamainternmed.2014.1059

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Parker, VL, Ritchie, JE, Drake, TM, Hookham, J, and Balasubramanian, SP. A randomised assessment of trainee doctors’ understanding and interpretation of diagnostic test results. World J Surg. (2016) 40:21–8. doi: 10.1007/s00268-015-3214-2

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Morgan, DJ, Pineles, L, Owczarzak, J, Magder, L, Scherer, L, Brown, JP, et al. Accuracy of practitioner estimates of probability of diagnosis before and after testing. JAMA Intern Med. (2021) 181:747–55. doi: 10.1001/jamainternmed.2021.0269

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Tatar, M, Mollahaliloǧlu, S, Sahin, B, Aydin, S, Maresso, A, and Hernández-Quevedo, C. Turkey: health system review, 2011. (2011).

Google Scholar

12. General Directorate of Public Health. Turkey Cancer Control Programme (2021). Available at: https://hsgm.saglik.gov.tr/depo/birimler/kanser-db/yayinlar/raporlar/2021_Kanser_Kontrol_Programi_/Turkey_NCCP_18_April_2022.pdf.

Google Scholar

13. Fidancı, İ, Eren, ŞÜ, Arslan, İ, and Tekin, O. Retrospective assessment of family medicine office patients in last three years. Konuralp Med J. (2016) 8:151–7. doi: 10.18521/ktd.284556

CrossRef Full Text | Google Scholar

14. Zitek, T. The appropriate use of testing for Covid-19. West J Emerg Med. (2020) 21:470–2. doi: 10.5811/westjem.2020.4.47370

PubMed Abstract | CrossRef Full Text | Google Scholar

15. EURACT. The European definition of general practice/family medicine. Short Version. (2005). Available at: https://www.woncaeurope.org/file/bae1def1-c0b9-47ef-8a3f-5a9a334ba25a/DefinitionEURACTshortversion.pdf.

Google Scholar

16. Kostopoulou, O, Arora, K, and Pálfi, B. Using cancer risk algorithms to improve risk estimates and referral decisions. Commun Med. (2022) 2:2. doi: 10.1038/s43856-021-00069-1

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Kale, MS, and Korenstein, D. Overdiagnosis in primary care: framing the problem and finding solutions. BMJ. (2018) 362:362. doi: 10.1136/bmj.k2820

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Chu, CM, and Lowder, JL. Diagnosis and treatment of urinary tract infections across age groups. Am J Obstet Gynecol. (2018) 219:40–51. doi: 10.1016/j.ajog.2017.12.231

CrossRef Full Text | Google Scholar

19. Demirbilek, Y, Pehlivantürk, G, Özgüler, ZÖ, and Alp, ME. COVID-19 outbreak control, example of ministry of health of Turkey. Turk J Med Sci. (2020) 50:489–94. doi: 10.3906/sag-2004-187

PubMed Abstract | CrossRef Full Text | Google Scholar

20. O’Sullivan, ED, and Schofield, SJ. Cognitive bias clinical medicine. J R Coll Physicians Edinb. (2018) 48:225–32. doi: 10.4997/JRCPE.2018.306

CrossRef Full Text | Google Scholar

21. Royce, CS, Hayes, MM, and Schwartzstein, RM. Teaching critical thinking: a case for instruction in cognitive biases to reduce diagnostic errors and improve patient safety. Acad Med. (2019) 94:187–94. doi: 10.1097/ACM.0000000000002518

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Balogh, EP, Miller, BT, and Ball, JR. Improving diagnosis in health care. Washington, DC: National Academies Press (2015).

Google Scholar

23. Kostopoulou, O, Oudhoff, J, Nath, R, Delaney, BC, Munro, CW, Harries, C, et al. Predictors of diagnostic accuracy and safe management in difficult diagnostic problems in family medicine. Med Decis Mak. (2008) 28:668–80. doi: 10.1177/0272989X08319958

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Ventola, CL. The antibiotic resistance crisis: part 1: causes and threats. PT. (2015) 40:277–83.

PubMed Abstract | Google Scholar

25. Singh, H, Dickinson, JA, Thériault, G, Grad, R, Groulx, S, Wilson, BJ, et al. Overdiagnosis: causes and consequences in primary health care. Can Fam Physician. (2018) 64:654–9.

PubMed Abstract | Google Scholar

26. Agoritsas, T, Courvoisier, DS, Combescure, C, Deom, M, and Perneger, TV. Does prevalence matter to physicians in estimating post-test probability of disease? A randomized trial. J Gen Intern Med. (2011) 26:373–8. doi: 10.1007/s11606-010-1540-5

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Rottman, BM. Physician Bayesian updating from personal beliefs about the base rate and likelihood ratio. Mem Cogn. (2017) 45:270–80. doi: 10.3758/s13421-016-0658-z

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Puhan, MA, Steurer, J, Bachmann, LM, and Ter Riet, G. A randomized trial of ways to describe test accuracy: the effect on physicians’ post-test probability estimates. Ann Intern Med. (2005) 143:184–9. doi: 10.7326/0003-4819-143-3-200508020-00004

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Vermeersch, P, and Bossuyt, X. Comparative analysis of different approaches to report diagnostic accuracy. Arch Intern Med. (2010) 170:734–5. doi: 10.1001/archinternmed.2010.84

CrossRef Full Text | Google Scholar

30. Phillips, B, and Westwood, M. Testing our understanding of tests. Arch Dis Child. (2009) 94:178–9. doi: 10.1136/adc.2008.145961

CrossRef Full Text | Google Scholar

31. Argimon-Palls, JM, Flores-Mateo, G, Jiménez-Villa, J, and Pujol-Ribera, E. Effectiveness of a short-course in improving knowledge and skills on evidence-based practice. BMC Fam Pract. (2011) 12:12. doi: 10.1186/1471-2296-12-64

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Goroll, AH, and Mulley, AG. Principals of primary care: selection and interpretation of diagnostic tests In: AH Goroll and AG Mulley, editors. Primary care medicine: Office evaluation and Management of the Adult Patient. 8th ed. Philadelphia, PA: Lippincott Williams and Wilkins (2020)

Google Scholar