Ronda Lun1
Gregory B. Walker1,2
Adrien Guenego3Mohammed Kassab4,5Eduardo Portela5
Vignan Yogendrakumar1George Medvedev2Ken Wong6Michel Shamy1,7
Dar Dowlatshahi1,7
Robert Fahed1,5,7*- 1Ottawa Stroke Program, Department of Medicine (Neurology), University of Ottawa, Ottawa, ON, Canada
- 2Division of Neurology, Fraser Health Authority, Royal Columbian Hospital, New Westminster, BC, Canada
- 3Department of Interventional Neuroradiology, Erasme University Hospital, Brussels, Belgium
- 4Division of Neurosurgery, Department of Surgery, University of Ottawa, Ottawa, ON, Canada
- 5Interventional Neuroradiology, Department of Medical Imaging, University of Ottawa, Ottawa, ON, Canada
- 6Division of Medical Imaging, Fraser Health Authority, Royal Columbian Hospital, New Westminster, BC, Canada
- 7Ottawa Hospital Research Institute, Ottawa, ON, Canada
Background: Hemorrhagic transformation after acute ischemic stroke is a dreaded and severe complication of thrombolysis and thrombectomy. However, its detection on post-thrombectomy conventional non-contrast computed tomography (CT) scan can be complicated by the frequent (and sometimes concomitant) presence of contrast, resulting in changes in management.
Aims: Our objective was to assess the inter- and intra-rater reliability for the detection of blood and/or contrast on day-1 post-thrombectomy CT scans.
Methods: A total of 18 raters across 3 different specialties independently examined 30 post-thrombectomy CT scans selected from the Aspiration vs. STEnt-Retriever (ASTER) trial. They were asked to judge the presence of blood and contrast. Thirty days later, the same 18 raters again independently judged the 30 scans, in randomized order. Agreement was measured with Fleiss' and Cohen's K statistics.
Results: Overall agreement on blood and/ or contrast presence was only fair, k = 0.291 (95% CI = 0.273–0.309). There were 0 scans with consensus among the 18 readers on the presence of blood and/or contrast. However, intra-rater global agreement across all 18 physicians was relatively high, with a median kappa value of 0.675. This intra-rater consistency was seen across all specialties, regardless of level of training.
Conclusion: Physician judgment for the presence of blood and/or contrast on day-1 post-thrombectomy non-contrast CT scan shows limited inter-observer reliability. Advanced imaging modalities may then be warranted for challenging clinical cases.
Introduction
Endovascular thrombectomy (EVT) has become the standard of care for patients with acute ischemic stroke (AIS) secondary to large vessel occlusions (1). One of the major complications after AIS is hemorrhagic transformation (HT), reported to be up to 35% after EVT (2). However, arterial injection of iodine contrast during EVT may mimic the appearance of HT, due to its hyper-dense appearance on follow-up conventional non-contrast CT (NCCT) (3). Inaccurate identification of HT could delay necessary treatments such as antiplatelet or anticoagulant therapy, and potentially result in misdiagnosis of HT in future EVT trials. In this study, we aimed to evaluate inter-rater and intra-rater reliability for detection of HT and contrast staining (CS) on NCCT in EVT patients.
Methods
We analyzed imaging data from the “Aspiration vs. STEnt-Retriever” (ASTER) trial (4). Access to the data can be obtained through formal proposal to the authors of the study.
A total of 30 NCCT scans performed 24–36 h after EVT were selected from the ASTER database with a roughly equal distribution of scans with HT, scans with CS, scans with both HT and CS, and scans with no HT or CS (i.e., approximately 6–8 scans in each subcategory). The studies were identified as such in the core laboratory of the ASTER study, which is composed of four attending physicians with 5–20 years of experience in neuroradiology. All selected cases of HT were parenchymal; we did not assess for the detection of subarachnoid or extra-cranial hemorrhage. Additional clinical details of the study are outlined elsewhere (5).
Eighteen raters from 3 tertiary stroke centers were recruited for independent interpretation of studies: 6 stroke neurologists, 6 interventional neuroradiologists (INR), and 6 diagnostic neuroradiologists (DNR), as they are all involved in multidisciplinary decisions for stroke patients. Within each specialty, there were 3 junior level physicians and 3 senior level physicians. Junior physicians were defined as fellows with <2 years of experience, and senior physicians were defined as staff physicians with more than 5 years of independent practice experience. To evaluate inter-rater reliability, the raters were asked to independently evaluate each NCCT for (1) presence of hemorrhage and (2) presence of contrast. To evaluate intra-rater reliability, the same independent raters were then asked to repeat the study assessments 1 month later, with the study order randomized. The raters had access to basic clinical information, including basic demographics, treatment status with thrombolysis, final thrombolysis in cerebral infarction recanalization score, time from symptom onset to recanalization, and day-1 National Institutes of Health Stroke Scale score (5).
The raters' dichotomized (yes/no) answers were transformed into a “global judgment” score of whether there was blood, contrast, or a combination of both. Fleiss' kappa was run to determine if there was inter-rater agreement. Intra-rater reliability was also assessed with Cohen's unweighted kappa values. The median number of times a rater's answer changed between the two assessments was calculated. All statistics were performed using SPSS v26.0 (IBM, Armonk, NY). Graphs were generated using GraphPad Prism v8.3.1. Levels of agreement were defined according to Landis and Koch: slight (0.00–0.20), fair (0.20–0.40), moderate (0.40–0.60), substantial (0.60–0.80), and excellent (>0.80) (6).
Results
Inter-rater agreement on the detection of blood was moderate, k = 0.404 (95% CI = 0.375–0.432) (see Figure 1). There were 3/30 scans on which the 18 raters agreed upon the presence of blood, and 2/30 scans on which there was unanimous agreement on the absence of blood. Intra-rater agreement was at least substantial for the detection of blood across all specialties and levels of training, and 3 raters had perfect intra-rater agreement (median k value of 0.861) (Figure 2). The median number of changes of judgment between both readings was 2 (Table 1).
Figure 1. Interrater agreement displayed as kappa values across specialties for assessment of the presence vs. absence of blood, contrast, or presence of both (i.e., global assessment). SN, stroke neurologist; INR, interventional neuroradiologist; DNR, diagnostic neuroradiologist.
Figure 2. Intra-rater agreement displayed as kappa values for each independent rater. X-axis is displayed by type of assessment: global, blood, and contrast. SN, stroke neurologist; INR, interventional neuroradiologist; DNR, diagnostic neuroradiologist.
Table 1. Inter-rater agreement across specialties, displayed as Fleiss' Kappa values, with 95% confidence intervals.
The inter-rater agreement for the detection of CS was only fair, k = 0.269 (95% CI = 0.240–0.298) (Figure 1). There were 0 scans on which all raters agreed upon the presence of contrast, and only 2/30 scans on which they unanimously agreed on the absence of contrast. The INR physicians collectively had the highest level of agreement, which nevertheless remained moderate (k = 0.439, 95% CI = 0.346–0.531) (Table 1). Intra-rater agreement was below substantial for 9/18 raters, and no rater reached perfect intra-rater agreement (median k value of 0.598) (Figure 2). The median number of changes of judgment per physician was 4.5. Detailed kappa values for each rater are outlined in Table 2.
Table 2. Intra-rater agreements for 18 independent raters, displayed as Fleiss' Kappa values, with 95% confidence intervals.
Overall global judgment for the presence of blood and/or contrast across all specialties was fair, k = 0.291 (95% CI = 0.273–0.309) (Figure 1). There were 0 scans where the raters unanimously agreed on the presence of blood, contrast, neither, or both.
Discussion
Our study shows that the overall inter-rater agreement about the presence of hemorrhage and/ or contrast was limited across all specialties. While DNR physicians seemed to have the highest degree of inter-rater agreement on the presence of hemorrhage, a k value of 0.495 still only represents a “moderate” level of agreement, and is usually considered “weak” in the context of health care research (7). However, intra-rater agreement for the presence of blood was consistently high across all specialties and levels of training, including junior physicians with <2 years of experience. This is in contrast with agreement on CS, where inter-rater agreement was only fair, and 100% of the physicians disagreed with themselves on their second reading at least once. Even though INR physicians were the most consistent in their judgment of contrast, their agreement was only deemed “moderate.” The lack of consensus on HT has been previously reported, and affects even simple dichotomized classifications such as hemorrhagic infarction vs. parenchymal hematoma (5), which are often used as outcome measures in clinical trials settings. While there are newly proposed rigorous classification systems for grading HT after ischemic stroke/reperfusion therapy, their reliability has not been assessed and they fail to address the issue of distinguishing CS from HT (8). The overall unreliable interdisciplinary interpretation of scans therefore may be attributed to unclear diagnostic criteria for CS and lack of additional imaging techniques to differentiate concurrent presence of both (9).
It is well-established that the phenomenon of CS can be seen after EVT, and is thought to relate to disruption of blood–brain barrier integrity in established ischemic infarct (10). Factors such as prolonged procedure time and multiple passes in the same vessel have been associated with higher risk for CS (11). The incidence of cerebral hyper-dense lesions after revascularization is high, and has been reported to be between 23 and 84%, depending on the definitions used and timing of follow-up imaging (10, 12). CS itself has been postulated to be associated with increased risk for HT and symptomatic ICH, although this is likely confounded by similar risk factors, such as large infarct size (13). Unfortunately, inaccurate detection of hemorrhage can lead to delayed initiation of anti-thrombotics, erroneous prognostication, and unnecessary investigations (9). It may be necessary to perform advanced imaging such as dual-energy CT (DECT) or gradient-recalled echo (GRE) sequence MRI for definitive diagnosis of hemorrhage vs. contrast (9, 14). However, MRI may be inaccessible to many centers in a timely fashion, and can still lead to false positive hemorrhage detection or false negative contrast extravasation if performed too soon after administration of contrast (14). DECT utilizes two distinctive voltage acquisitions to discriminate between materials with various attenuation properties, such as iodine vs. calcium or hemorrhage, but its availability is currently limited across centers. Future studies may look at the combination of concurrent SWI with CT and compare them to plain CT images for reference.
Our study has important limitations. Imaging assessments were done in controlled settings with no time constraints and therefore results may differ from real-time clinical assessments. Accuracy analysis was not performed because of the extensive disagreements revealed between each rater, thereby defeating the relevance of such. While diagnostic accuracy was not the goal of the study, one potential way to address the lack of a “gold standard” would be the use of advanced imaging (i.e., MRI susceptibility based images or dual-energy CT scans). We recognize that the pragmatic approach to resolving disagreements and addressing uncertainty in imaging interpretation is effective communication between specialties. Lastly, this case series of patients was artificially constructed to minimize paradoxes of k statistics, and the exact results might not be reproducible in a different case series of patients.
Conclusion
There is a lack of agreement between physicians on the interpretation of post-EVT conventional CT scans for the presence/absence of both hemorrhage and contrast. Standardized definitions and clear diagnostic criteria for the two entities are warranted. Advanced imaging modalities such as DECT may be helpful in differentiating the two, if clinically indicated.
Data Availability Statement
Access to the data that support the findings of this study are available from the corresponding author upon reasonable request.
Ethics Statement
The studies involving human participants were reviewed and approved by Fondation Rothschild Ethics Committee. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.
Author Contributions
RL was responsible for performing the statistical analysis and writing the manuscript. RF was responsible for study design and data collection. All authors contributed to the writing and editing of the manuscript for intellectual content.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
DD has received grant funding from the Heart & Stroke foundation of Canada, and honoraria from Bayer and Apopharma unrelated to the submitted work. The corresponding author has full access to all the data in the study and has final responsibility for the decision to submit for publication.
References
1. Powers WJ, Rabinstein AA, Ackerson T, Adeoye OM, Bambakidis NC, Becker K. Guidelines for the early management of patients with acute ischemic stroke: 2019 update to the 2018 guidelines for the early management of acute ischemic stroke: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke. (2019) 50:e344–418. doi: 10.1161/STR.0000000000000211
2. Hao Y, Zhang Z, Zhang H, Xu L, Ye Z, Dai Q, et al. Risk of intracranial hemorrhage after endovascular treatment for acute ischemic stroke: systematic review and meta-analysis. Interv Neurol. (2017) 6:57–64. doi: 10.1159/000454721
3. Dekeyzer S, Nikoubashman O, Lutin B, De Groote J, Vancaester E, De Blauwe S, et al. Distinction between contrast staining and hemorrhage after endovascular stroke treatment: one CT is not enough. J NeuroInterventional Surg. (2017) 9:394–8. doi: 10.1136/neurintsurg-2016-012290
4. Lapergue B, Blanc R, Gory B, Labreuche J, Duhamel A, Marnat G, et al. Effect of endovascular contact aspiration vs. stent retriever on revascularization in patients with acute ischemic stroke and large vessel occlusion: the ASTER Randomized Clinical Trial. JAMA. (2017) 318:443–52. doi: 10.1001/jama.2017.9644
5. Guenego A, Lecler A, Raymond J, Sabben C, Khoury N, Premat K, et al. Hemorrhagic transformation after stroke: inter- and intrarater agreement. Eur J Neurol. (2019) 26:476–82. doi: 10.1111/ene.13859
6. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. (1977) 33:159–74.
8. von Kummer R, Broderick JP, Campbell Bruce CV, Demchuk A, Goyal M, Hill MD, et al. The Heidelberg bleeding classification: classification of bleeding events after ischemic stroke and reperfusion therapy. Stroke. (2015) 46:2981–6. doi: 10.1161/STROKEAHA.115.010049
9. Almqvist H, Holmin S, Mazya MV. Dual energy CT after stroke thrombectomy alters assessment of hemorrhagic complications. Neurology. (2019) 93:e1068–75. doi: 10.1212/WNL.0000000000008093
10. Puntonet J, Richard M-E, Edjlali M, Ben Hassen W, Legrand L, Benzakoun, et al. Imaging findings after mechanical thrombectomy in acute ischemic stroke. Stroke. (2019) 50:1618–25. doi: 10.1161/STROKEAHA.118.024754
11. Chen Z, Zhang Y, Su Y, Sun Y, He Y, Chen H. Contrast extravasation is predictive of poor clinical outcomes in patients undergoing endovascular therapy for acute ischemic stroke in the anterior circulation. J Stroke Cerebrovasc Dis. (2020) 29:104494. doi: 10.1016/j.jstrokecerebrovasdis.2019.104494
12. Parrilla G, García-Villalba B, de Rueda ME, Zamarro J, Carrión E, Hernández-Fernández F, et al. Hemorrhage/contrast staining areas after mechanical intra-arterial thrombectomy in acute ischemic stroke: imaging findings and clinical significance. AJNR Am J Neuroradiol. (2012) 33:1791–6. doi: 10.3174/ajnr.A3044
13. Boisseau W, Fahed R, Lapergue B, Desilles J-P, Zuber K, Khoury N, et al. Predictors of parenchymal hematoma after mechanical thrombectomy. Stroke. (2019) 50:2364–70. doi: 10.1161/STROKEAHA.118.024512
Keywords: stroke, ischemic stroke, thrombectomy, hemorrhage, cerebral, agreement, reliability
Citation: Lun R, Walker GB, Guenego A, Kassab M, Portela E, Yogendrakumar V, Medvedev G, Wong K, Shamy M, Dowlatshahi D and Fahed R (2020) Is This Contrast? Is This Blood? An Agreement Study on Post-thrombectomy Computed Tomography Scans. Front. Neurol. 11:593098. doi: 10.3389/fneur.2020.593098
Received: 09 August 2020; Accepted: 30 November 2020;
Published: 22 December 2020.
Edited by:
Bruce Campbell, The University of Melbourne, AustraliaReviewed by:
Johannes Pfaff, Heidelberg University Hospital, GermanyTeddy Y. Wu, Christchurch Hospital, New Zealand
Copyright © 2020 Lun, Walker, Guenego, Kassab, Portela, Yogendrakumar, Medvedev, Wong, Shamy, Dowlatshahi and Fahed. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Robert Fahed, cm9iZXJ0LmZhaGVkJiN4MDAwNDA7aG90bWFpbC5mcg==