Structured Reporting of Computed Tomography in the Staging of Neuroendocrine Neoplasms: A Delphi Consensus Proposal

Background: Structured reporting (SR) in radiology is becoming increasingly necessary and has been recognized recently by major scientific societies. This study aims to build structured CT-based reports in Neuroendocrine Neoplasms during the staging phase in order to improve communication between the radiologist and members of multidisciplinary teams.

Materials and Methods: A panel of expert radiologists, members of the Italian Society of Medical and Interventional Radiology, was established. A Modified Delphi process was used to develop the SR and to assess a level of agreement for all report sections. Cronbach’s alpha (Cα) correlation coefficient was used to assess internal consistency for each section and to measure quality analysis according to the average inter-item correlation.

Results: The final SR version was built by including n=16 items in the “Patient Clinical Data” section, n=13 items in the “Clinical Evaluation” section, n=8 items in the “Imaging Protocol” section, and n=17 items in the “Report” section. Overall, 54 items were included in the final version of the SR. Both in the first and second round, all sections received more than a good rating: a mean value of 4.7 and range of 4.2-5.0 in the first round and a mean value 4.9 and range of 4.9-5 in the second round. In the first round, the Cα correlation coefficient was a poor 0.57: the overall mean score of the experts and the sum of scores for the structured report were 4.7 (range 1-5) and 728 (mean value 52.00 and standard deviation 2.83), respectively. In the second round, the Cα correlation coefficient was a good 0.82: the overall mean score of the experts and the sum of scores for the structured report were 4.9 (range 4-5) and 760 (mean value 54.29 and standard deviation 1.64), respectively.

Conclusions: The present SR, based on a multi-round consensus-building Delphi exercise following in-depth discussion between expert radiologists in gastro-enteric and oncological imaging, derived from a multidisciplinary agreement between a radiologist, medical oncologist and surgeon in order to obtain the most appropriate communication tool for referring physicians.

Introduction

Neuroendocrine neoplasms (NENs) are heterogeneous with respect to their site of origin and metastatic behaviour. About 25% of NENs secrete hormones leading to specific clinical signs and symptoms, while the remaining approximate 75% are so-called non-secreting NENs, which are often diagnosed incidentally and 40–50% are already in a metastatic tumour stage (1–3). Functioning NENs usually show up relatively early, so it might be difficult to detect lesions on radiological imaging, since these are often too small to be seen (4). While the detection and follow-up of NENs still pose a diagnostic challenge, radiological imaging is essential for the assessment of metastatic lesions (especially in the liver) and of tumour response to treatment, playing a key role in guiding treatment planning (5).

In such a complex scenario, an effective communication of imaging data to referring physicians is crucial for patient care. Radiology reports are the gold standard as to comprehensiveness and accuracy, and they are traditionally created as non-structured free text reporting (FTR) written in a narrative language. However, inconsistencies regarding content, style, and presentation format can reduce the clarity of FTR and hinder the communication of key information to the referring physician, potentially leading to incorrect diagnosis, delayed initiation of adequate treatment, or adverse patient outcomes (6–9).

Radiological structured reporting (SR) has several advantages over FTR, including a higher standardization of reporting style and lexicon with the adherence to established practice guidelines and recommendations, greater consistency and reproducibility (10–12), possibility of data mining and integration with artificial intelligence systems (13), shorter reporting time, lower error rate, and better communication with referring clinicians and other radiologists (8, 14, 15). To the latter regard, oncologists tend to prefer SR over FTR due to its ability to share information more clearly and in a more standardised way (16–18). Moreover, in 2018 the European Neuroendocrine Tumor Society (ENETS) launched an initiative to provide guidance for synoptic reporting of computed tomography (CT) and magnetic resonance imaging (MRI) examinations for staging and follow-up of bronchial, pancreatic and gastrointestinal NENs, where members of the imaging group stated a strong preference for a combination of limited and standardised options by way of drop-down menus, wherever possible (19).

Despite its established advantages, SR has not yet become commonplace in the radiological routine due to several reasons, including the current paucity of usable templates and of commercially available SR software solutions (6). In this context, the Italian Society of Medical and Interventional Radiology (SIRM) has created an Italian warehouse of SR templates that can be freely accessed by all SIRM members [including MRI for primary staging and restaging of rectal cancer (7), and chest CT in the management of COVID-19 pneumonia (9)], thus facilitating their routine use in a clinical setting (20).

Our purpose is to devise and evaluate an SR template for CT examinations performed for primary staging of NENs, with the goal to improve the standardization of reporting based on current best practice guidelines, as well as the communication between radiologists and clinicians, and among radiologists themselves.

Materials and Methods

Expert Panel

Following extensive discussion between expert radiologists, a multi-round consensus-building Delphi exercise was performed to develop a comprehensive, focused SR template for CT at the staging phase of patients with NENs.

A SIRM radiologist with experience in abdominal imaging created the first draft of the SR with the collaboration of a surgeon and medical oncologist specialised in NENs.

A working team of 14 expert radiologists was set up, with members from the SIRM Chapters of Gastrointestinal and Abdominal Radiology and of Diagnostic Imaging in Oncology. Their aim was to revise the initial draft iteratively, with the objective of reaching a final consensus on the SR.

Selection of the Delphi Domains and Items

All panellists reviewed literature data on leading scientific databases (including PubMed, Scopus and Google Scholar), to assess papers on NENs, CT and structured radiology reports published from December 2000 to June 2021. The full text of the selected studies was reviewed by all members of the expert panel, and each of them developed and shared the list of Delphi items via email and/or teleconference.

The SR was divided into the following four sections: (a) Patient Clinical Data (including 16 items), (b) Clinical Evaluation (13 items), (c) Imaging Protocol (8 items) and (d) Report (17 items). A final section dedicated to key images was also included in the template.

Two Delphi rounds were performed. During the first round, each panellist independently contributed to refining the SR draft by means of online meetings or email exchanges. The level of panellists’ agreement for each SR model was tested in the second Delphi through a Google Form questionnaire shared by email. Each expert expressed individual comments for each specific template section by using a five-point Likert scale (1 = strongly disagree, 2 = slightly disagree, 3 = slightly agree; 4 = generally agree, 5 = strongly agree).

After the second Delphi round, the last version of the SR was generated on the dedicated RSNA website (www.radreport.org) by using a T-Rex template format, in line with IHE (Integrating Healthcare Enterprise) and MRRT (Management Of Radiology Report Templates) profiles and accessible as open-source software, with technical support by Exprivia (Exprivia SpA, Bari, Italy). Such profiles determine both the format of radiology report templates [using version 5 of Hypertext Markup Language (HTML5)] and the transporting mechanism to request, retrieve, and stock these schedules. The radiology report was structured by using a series of “codified queries” integrated in the T-Rex editor’s preselected sections (21).

Statistical Analysis

Answers from each panellist were exported into Microsoft Excel^® format for ease of data collection and statistical analysis.

All panellists’ ratings for each section were analysed, with descriptive statistics measuring the mean score, the standard deviation, and the sum of scores. An average mark of 3 was considered good, whereas 4 was considered excellent.

To measure the internal consistency of panellist ratings for each section of the report, a quality analysis based on the average inter-item correlation was performed with Cronbach’s alpha (Cα) correlation coefficient (22, 23). The Cα test provides a measure of the internal consistency of a test or scale; it is expressed as a number between 0 and 1. Internal consistency describes the extent to which all the items in a test measure the same concept. Cα was determined after each round.

The nearer the Cα coefficient to 1.0, the more accurate the internal consistency of the categories in the scale. An alpha coefficient (α) ≥0.9 was considered excellent, α ≥0.8 good, α ≥0.7 acceptable, α ≥0.6 questionable, α ≥0.5 poor, and α <0.5 unacceptable. However, an α = 0.8 was seen as an acceptable parameter in internal reliability during iterations.

Data analysis was performed using Matlab Statistic Toolbox (The MathWorks, Inc., Natick, MA, USA).

Results

Structured Report

The final SR (Appendix 1) template was built by including n=16 items in the “Patient Clinical Data” section, n=13 items in the “Clinical Evaluation” section, n=8 items in the “Imaging Protocol” section, and n=17 items in the “Report” section. Overall, 54 items were included in the final version of the SR.

The “Patient Clinical Data” section included patient clinical data, previous or family history of malignancies, risk factors, endocrine and or neuroendocrine neoplasms in young age, hereditary syndromes, and other genetic mutations. In this section, we included the item “Allergies” to the drug or no drug and contrast medium.

The “Clinical Evaluation” section collected previous examination results, a genetic panel, results of histopathological examination on biopsy specimen, Chromogranin A (CgA) level, Neuron-specific enolase (NSE) level, 5-hydroxyindoleacetic acid (5-HIAA) 24-h urine level, serum gastrin level, serum insulin level, serum glucagon level, serum VIP level, blood count, serum creatinine, liver function and clinical symptoms.

The “Imaging Protocol” section included data on the equipment used, the number of detectors and whether it was multidetector or dual energy, including data on the reconstruction algorithm and slice thickness. In addition, we collected data on contrast study protocol, including data on the contrast study phase, as well as data concerning the contrast medium, such as the active principle, commercial name, dosage, flow rate, concentration, and ongoing adverse events. In addition, in this section we included data about bowel preparation and contrast technique for gastro-enteric tract evaluation.

The “Report” section included data on lesion sites (primary tumour visible or not visible on CT imaging) and features such as number, site, size, and infiltration of neighbouring organs and/or structures for each site (e.g., lung, gastric, pancreatic, duodenal, small bowel, appendiceal and colorectal lesion). For small bowel lesions, we evaluated the desmoplastic reaction, while for pancreatic lesions assessment was made on the relationship with the pancreatic duct, with vessels and duodenum or ampulla, according to surgical planning. In addition, in this section we included tumour stage, node stage and metastases stage, as well as the presence of incidental radiological findings. To allow for maximum flexibility of SR use in different working scenarios, only the Report section fields are mandatory, whereas all fields from the other SR sections can be filled in upon user discretion.

Consensus Agreement

Tables 1, 2 show single scores and the sum of scores of the 14 panellists for the SR in the first and second rounds, respectively.

TABLE 1

Table 1 Single scores and sum of scores of 14 panellists for the structured report in the first round.

TABLE 2

Table 2 Single scores and sum of scores of 14 panellists for the structured report in the second round.

Both in the first and second round, all sections received more than a good rating, with a mean value of 4.7 (range 4.2-5.0) in the first round, and a mean value of 4.9 (range 4.9-5) in the second round.

In the first round, the Cα correlation coefficient was 0.57, with an overall mean score of experts and sum of scores for the SR being 4.7 (range 1-5) and 728 (mean value 52.00, standard deviation 2.83), respectively.

In the second round, the Cα correlation coefficient was 0.82, with an overall mean score of experts and sum of scores for the SR being 4.9 (range 4-5) and 760 (mean value 54.29, standard deviation 1.64), respectively.

Discussion

To the best of our knowledge, this is the first time that a panel of experts from a national radiological society has promoted the creation of a comprehensive SR template for the CT staging of NENs, systematically encompassing all steps of the radiological procedure (from registration of patient personal data to full details of CT examination and standardised reporting of relevant CT findings), along with the possibility of coupling together radiological and clinical data and in the perspective of integrating the SR template into the radiology workflow.

Our SR template was based on a multi-round consensus-building Delphi exercise following in-depth discussion between expert radiologists in gastrointestinal and oncological imaging. On a previous occasion, panellists had assessed and promoted a SR for MRI-based primary staging and restaging of rectal cancer (7). While such MRI templates were also based on a multi-round consensus-building Delphi exercise, in that earlier study the original draft derived from a single dedicated radiologist, without seeking any multidisciplinary agreement. On the other hand, in the present study the SR draft wa based on the multidisciplinary agreement of a radiologist, a medical oncologist and a surgeon, with the goal to achieve an optimal, all-around communication tool for referring physicians.

The approved SR template was divided into four sections (i.e., Patient Clinical Data, Clinical Evaluation, Imaging Protocol and Report), with a final section dedicated to key images. Although our template may appear somewhat long and complex (potentially slowing down the radiology workflow), it must be emphasised that only the Report section is mandatory, whereas other sections are optional. Furthermore, considering that not all data may be available to the radiologist, these open fields can also be filled in at a subsequent time. In addition, the possibility of connecting this template with the patient electronic health record allows for automatic import of available data.

All sections received more than a good rating in both Delphi rounds, with a mean value of 4.7 and 4.9 in the first and in the second round, respectively. In the first round, the Cα correlation coefficient was relatively poor (0.57), whereas in the second round, it was substantially improved (0.82). Moreover, the overall mean score of the experts in the second round was higher than that of the first round, with a lower standard deviation value being related to a greater agreement among experts for this SR. The sections with the lowest level of agreement were “Patient Clinical Data” and “Clinical Evaluation”, reflecting the opinion that these sections were too long and may slow down daily practice. However, following a conference call, all panellists expressed their agreement once the optional nature of the sections had been clarified, and the importance of collecting patient clinical data and history for big data creation and connecting radiological with clinical data was demonstrated as well.

The “Patient Clinical Data” section included patient data, previous or family history of malignancies, risk factors, endocrine and or neuroendocrine neoplasms in young age, hereditary syndromes, and other genetic mutations. The “Clinical Evaluation” section collected previous examinations findings, a genetic panel, results of histopathological examination on biopsy specimen, chromogranin A (CgA) level, neuron-specific enolase (NSE) level, 5-hydroxyindoleacetic acid (5-HIAA) 24-h urine level, serum gastrin level, serum insulin level, serum glucagon level, serum VIP level, blood count, serum creatinine, liver function and clinical symptoms. This data could serve as a basis for creating potentially large databases, allowing not only for epidemiological statistical analysis, but also for building a radiomics model through the combination of radiological features and clinical data (24–29). To this end, genomic data could also be leveraged to build a radiogenomics model, which may be useful in the upper levels of personalised risk stratification and advanced precision medicine for early cancer diagnosis, cancer therapy selection, prognosis prediction, and assessment of treatment response and resistance to therapy (30–34).

A “strong agreement” was the result for the “Imaging Protocol” and “Report” sections. The “Imaging Protocol” section included data on the CT equipment used and related technical parameters (e.g., number of detector rows, multidetector single-or dual-energy, reconstruction algorithm and slice thickness). In addition, this section included data regarding the CT acquisition protocol (including post-contrast phases) and contrast medium administration (such as the active principle, commercial name, volume, flow rate, concentration, and ongoing adverse events), as well as data about bowel preparation and contrast technique for the evaluation of the gastrointestinal tract. Sharing examination techniques (not only within one’s own department, but also with the radiology departments of other centres) allows for the standardisation and optimisation of study protocols. Indeed, during the follow-up phase, differences in acquisition parameters and segmentation algorithms are significant features that can lead to variability in volumetric assessment. Thus, slice thickness and other protocol-related factors (such as the reconstruction kernel and field of view) should remain constant for reliable measurements to be performed. In the protocol optimization stage, enhanced communication between different centres can lead to an overall quality improvement through optimization of radiation dose and contrast administration, higher patient safety and overall better diagnostic quality (35, 36). Moreover, such enhanced communication can facilitate comparing the results obtained from different studies, thus reducing the variability due to different imaging protocols (37, 38).

The “Report” section included data on primary tumour being visible or not on CT imaging, as well as features such as lesion site, number, size, and infiltration of adjacent organs and/or structures for each site (e.g., lung, gastric, pancreatic, duodenal, small bowel, appendiceal, and colorectal lesions). Desmoplastic reaction was evaluated for small bowel lesions, whereas for pancreatic lesions the relationship with the pancreatic duct, vessels and duodenum or ampulla was taken into account, according to prospective surgical planning. In addition, in this section we included tumour stage, node stage and metastases stage, as well as the presence of incidental radiological findings. The possibility of using a SR template to guide the radiology workflow allows describing all the main radiological features that could be omitted with FTR, e.g., by mere distraction. For example, in pancreatic NENs, a correct evaluation of the stage of the neoplasm and an assessment of the relationship with the pancreatic duct, vessels, duodenum or the infiltration of neighbouring structures, allows for a correct stratification of patients and can avoid unnecessary major surgery compared to tumour enucleation (39–42). Using a checklist and a systematic search pattern may help to prevent such diagnostic errors. Both radiologists and referring clinicians are keen to reduce the rate of diagnostic errors, which, for radiologists, accounts for as much as 4% of reports (43–47). A retrospective overview of 3,000 MRI examinations was useful in determining clinically relevant extraspinal results in 28.5% of patients, which were not present in initial, unstructured reports (48). Similarly, the use of a checklist-style SR template has been shown to improve the rate of diagnosis of fracture-unrelated findings on cervical CT (49). SRs have also been shown to enhance the clinical impact on tumour staging and surgical planning for pancreatic and rectal carcinoma (50–52). Brook et al. compared the results of structured versus nonstructured reporting of CT findings for the staging and assessment of resectability for pancreatic cancer, and they concluded that surgeons were more confident about tumour resectability using SR compared to FTR (50). Sahni et al. showed that the use of an MRI-based SR improved rectal cancer staging when compared to FTR (52).

According to Weiss et al. [who described three levels of SR (53)], our SR is based on standardised terminology and structure, which are features needed for adhering to diagnostic-therapeutic recommendations and enrolment in clinical trials. This also reduces any ambiguity that may arise from non-conventional language and enables better communication between radiologists and clinicians. In addition, this SR template has the advantage of having been validated by a multidisciplinary group, potentially helping radiologists provide referring clinicians with all data required for correct patient management. However, this study has several limitations. Firstly, panellists were of the same nationality, and the contribution of experts from multiple countries could have allowed for a broader sharing, potentially increasing the consistency of the SR. Secondly, our study was not aimed at assessing the impact of the SR on the clinical management of NEN patients.

Conclusion

We developed a SR template for primary CT staging of NENs based on a multi-round consensus-building Delphi exercise, following in-depth discussion between expert radiologists in gastrointestinal and oncological imaging and derived from the multidisciplinary agreement of a radiologists, a medical oncologist and a surgeon specialised in NENs. A widespread adoption of SR could improve the quality, clarity, and reproducibility of reports across hospital departments and locations, improving patient health care and fostering research development.

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary Material. Further inquiries can be directed to the corresponding author.

Ethics Statement

The studies involving human participants were reviewed and approved by Protocol number is: 13261–OSS, Azienda Ospedaliero - Universitaria Careggi, Florence, Italy. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work, and approved it for publication.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fendo.2021.748944/full#supplementary-material

References

1. Benedetti G, Mori M, Panzeri MM, Barbera M, Palumbo D, Sini C, et al. CT-Derived Radiomic Features to Discriminate Histologic Characteristics of Pancreatic Neuroendocrine Tumours. Radiol Med (2021) 126(6):745–60. doi: 10.1007/s11547-021-01333-z

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Grazzini G, Danti G, Cozzi D, Lanzetta MM, Addeo G, Falchini M, et al. Diagnostic Imaging of Gastrointestinal Neuroendocrine Tumours (GI-NETs): Relationship Between MDCT Features and 2010 WHO Classification. Radiol Med (2019) 124(2):94–102. doi: 10.1007/s11547-018-0946-8

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Danti G, Berti V, Abenavoli E, Briganti V, Linguanti F, Mungai F, et al. Diagnostic Imaging of Typical Lung Carcinoids: Relationship Between MDCT, 111in-Octreoscan and 18F-FDG-PET Imaging Features With Ki-67 Index. Radiol Med (2020) 125(8):715–29. doi: 10.1007/s11547-020-01172-4

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Granata V, Fusco R, Setola SV, Castelguidone ELD, Camera L, Tafuto S, et al. The Multidisciplinary Team for Gastroenteropancreatic Neuroendocrine Tumours: The Radiologist’s Challenge. Radiol Oncol (2019) 53(4):373–87. doi: 10.2478/raon-2019-0040

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Półtorak-Szymczak G, Budlewski T, Furmanek MI, Wierzba W, Sklinda K, Walecki J, et al. Radiological Imaging of Gastro-Entero-Pancreatic Neuroendocrine Tumors. The Review of Current Literature Emphasizing the Diagnostic Value of Chosen Imaging Methods. Front Oncol (2021) 11:670233. doi: 10.3389/fonc.2021.670233

PubMed Abstract | CrossRef Full Text | Google Scholar

6. European Society of Radiology (ESR). ESR Paper on Structured Reporting in Radiology. Insights Imaging (2018) 9(1):1–7. doi: 10.1007/s13244-017-0588-8

CrossRef Full Text | Google Scholar

7. Granata V, Caruso D, Grassi R, Cappabianca S, Reginelli A, Rizzati R, et al. Structured Reporting of Rectal Cancer Staging and Restaging: A Consensus Proposal. Cancers (Basel) (2021) 13(9):2135. doi: 10.3390/cancers13092135

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Faggioni L, Coppola F, Ferrari R, Neri E, Regge D. Usage of Structured Reporting in Radiological Practice: Results From an Italian Online Survey. Eur Radiol (2017) 27(5):1934–43. doi: 10.1007/s00330-016-4553-6

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Neri E, Coppola F, Larici AR, Sverzellati N, Mazzei MA, Sacco P, et al. Structured Reporting of Chest CT in COVID-19 Pneumonia: A Consensus Proposal. Insights Imaging (2020) 11(1):92. doi: 10.1186/s13244-020-00901-7

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Larson DB, Towbin AJ, Pryor RM, Donnelly LF. Improving Consistency in Radiology Reporting Through the Use of Department-Wide Standardized Structured Reporting. Radiology (2013) 267(1):240–50. doi: 10.1148/radiol.12121502

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Shea LAG, Towbin AJ. The State of Structured Reporting: The Nuance of Standardized Language. Pediatr Radiol (2019) 49(4):500–8. doi: 10.1007/s00247-019-04345-0

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Marcovici PA, Taylor GA. Journal Club: Structured Radiology Reports Are More Complete and More Effective Than Unstructured Reports. AJR Am J Roentgenol (2014) 203(6):1265–71. doi: 10.2214/AJR.14.12636

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Pinto Dos Santos D, Baeßler B. Big Data, Artificial Intelligence, and Structured Reporting. Eur Radiol Exp (2018) 2(1):42. doi: 10.1186/s41747-018-0071-4

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Ganeshan D, Duong PT, Probyn L, Lenchik L, McArthur TA, Retrouvey M, et al. Structured Reporting in Radiology. Acad Radiol (2018) 25(1):66–73. doi: 10.1016/j.acra.2017.08.005

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Goldberg-Stein S, Chernyak V. Adding Value in Radiology Reporting. J Am Coll Radiol (2019) 16(9 Pt B):1292–8. doi: 10.1016/j.jacr.2019.05.042

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Patel A, Rockall A, Guthrie A, Gleeson F, Worthy S, Grubnic S, et al. Can the Completeness of Radiological Cancer Staging Reports be Improved Using Proforma Reporting? A Prospective Multicentre Non-Blinded Interventional Study Across 21 Centres in the UK. BMJ Open (2018) 8(10):e018499. doi: 10.1136/bmjopen-2017-018499

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Brown PJ, Rossington H, Taylor J, Lambregts DMJ, Morris E, West NP, et al. Standardised Reports With a Template Format Are Superior to Free Text Reports: The Case for Rectal Cancer Reporting in Clinical Practice. Eur Radiol (2019) 29(9):5121–8. doi: 10.1007/s00330-019-06028-8

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Weber TF, Spurny M, Hasse FC, Sedlaczek O, Haag GM, Springfeld C, et al. Improving Radiologic Communication in Oncology: A Single-Centre Experience With Structured Reporting for Cancer Patients. Insights Imaging (2020) 11(1):106. doi: 10.1186/s13244-020-00907-1

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Dromain C, Vullierme M, Hicks R, Prasad V, O’Toole D, de Herder W, et al. ENETS Consensus Guidelines for Synoptic Reporting of Radiology Studies. J Neuroendocrinol (2021), e13044. doi: 10.1111/jne.13044

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Italian Society of Medical and Interventional Radiology . Available at: www.sirm.org (Accessed 8 October 2021).

Google Scholar

21. Kahn CE Jr, Genereaux B, Langlotz CP. Conversion of Radiology Reporting Templates to the MRRT Standard. J Digit Imaging (2015) 28(5):528–36. doi: 10.1007/s10278-015-9787-3

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Becker G. Creating Comparability Among Reliability Coefficients: The Case of Cronbach Alpha and Cohen Kappa. Psychol Rep (2000) 87(3 Pt 2):1171–82. doi: 10.2466/pr0.2000.87.3f.1171

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Cronbach LJ. Coefficient Alpha and the Internal Structure of Tests. Psychometrika (1951) 16:297–334. doi: 10.1007/BF02310555

CrossRef Full Text | Google Scholar

24. Kirienko M, Ninatti G, Cozzi L, Voulaz E, Gennaro N, Barajon I, et al. Computed Tomography (CT)-Derived Radiomic Features Differentiate Prevascular Mediastinum Masses as Thymic Neoplasms Versus Lymphomas. Radiol Med (2020) 125(10):951–60. doi: 10.1007/s11547-020-01188-w

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Hu HT, Shan QY, Chen SL, Li B, Feng ST, Xu EJ, et al. CT-Based Radiomics for Preoperative Prediction of Early Recurrent Hepatocellular Carcinoma: Technical Reproducibility of Acquisition and Scanners. Radiol Med (2020) 125(8):697–705. doi: 10.1007/s11547-020-01174-2

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Granata V, Grassi R, Fusco R, Galdiero R, Setola SV, Palaia R, et al. Pancreatic Cancer Detection and Characterization: State of the Art and Radiomics. Eur Rev Med Pharmacol Sci (2021) 25(10):3684–99. doi: 10.26355/eurrev_202105_25935

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Granata V, Fusco R, Barretta ML, Picone C, Avallone A, Belli A, et al. Radiomics in Hepatic Metastasis by Colorectal Cancer. Infect Agent Cancer (2021) 16(1):39. doi: 10.1186/s13027-021-00379-y

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Nazari M, Shiri I, Hajianfar G, Oveisi N, Abdollahi H, Deevband MR, et al. Noninvasive Fuhrman Grading of Clear Cell Renal Cell Carcinoma Using Computed Tomography Radiomic Features and Machine Learning. Radiol Med (2020) 125(8):754–62. doi: 10.1007/s11547-020-01169-z

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Fusco R, Granata V, Mazzei MA, Meglio ND, Roscio DD, Moroni C, et al. Quantitative Imaging Decision Support (QIDS™) Tool Consistency Evaluation and Radiomic Analysis by Means of 594 Metrics in Lung Carcinoma on Chest CT Scan. Cancer Control (2021) 28:1073274820985786. doi: 10.1177/1073274820985786

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Reinert CP, Krieg EM, Bösmüller H, Horger M. Mid-Term Response Assessment in Multiple Myeloma Using a Texture Analysis Approach on Dual Energy-CT-Derived Bone Marrow Images - A Proof of Principle Study. Eur J Radiol (2020) 131:109214. doi: 10.1016/j.ejrad.2020.109214

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Liu D, Zhang X, Zheng T, Shi Q, Cui Y, Wang Y, et al. Optimisation and Evaluation of the Random Forest Model in the Efficacy Prediction of Chemoradiotherapy for Advanced Cervical Cancer Based on Radiomics Signature From High-Resolution T2 Weighted Images. Arch Gynecol Obstet (2021) 303(3):811–20. doi: 10.1007/s00404-020-05908-5

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Farchione A, Larici AR, Masciocchi C, Cicchetti G, Congedo MT, Franchi P, et al. Exploring Technical Issues in Personalized Medicine: NSCLC Survival Prediction by Quantitative Image Analysis-Usefulness of Density Correction of Volumetric CT Data. Radiol Med (2020) 125(7):625–35. doi: 10.1007/s11547-020-01157-3

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Granata V, Fusco R, Avallone A, De Stefano A, Ottaiano A, Sbordone C, et al. Radiomics-Derived Data by Contrast Enhanced Magnetic Resonance in Ras Mutations Detection in Colorectal Liver Metastases. Cancers (Basel) (2021) 13(3):453. doi: 10.3390/cancers13030453

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Cui Y, Yang X, Shi Z, Yang Z, Du X, Zhao Z, et al. Radiomics Analysis of Multiparametric MRI for Prediction of Pathological Complete Response to Neoadjuvant Chemoradiotherapy in Locally Advanced Rectal Cancer. Eur Radiol (2019) 29(3):1211–20. doi: 10.1007/s00330-018-5683-9

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Reiner BI. Strategies for Radiology Reporting and Communication: Part 4: Quality Assurance and Education. J Digit Imaging (2014) 27(1):1–6. doi: 10.1007/s10278-013-9656-x

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Pfaff JAR, Füssel B, Harlan ME, Hubert A, Bendszus M. Variability of Acquisition Phase of Computed Tomography Angiography in Acute Ischemic Stroke in a Real-World Scenario. Eur Radiol (2021). doi: 10.1007/s00330-021-08084-5

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Virgolini I, Bahri S, Kjaer A, Gronbaek H, Iversen P, Carlsen EA, et al. A Randomised, Factorial Phase II Study to Determine the Optimal Dosing Regimen for ⁶⁸Ga-Satoreotide Trizoxetan as an Imaging Agent in Patients With Gastroenteropancreatic Neuroendocrine Tumours. J Nucl Med (2021). doi: 10.2967/jnumed.121.261936

CrossRef Full Text | Google Scholar

38. Ioannidis GS, Christensen S, Nikiforaki K, Trivizakis E, Perisinakis K, Hatzidakis A, et al. Cerebral CT Perfusion in Acute Stroke: The Effect of Lowering the Tube Load and Sampling Rate on the Reproducibility of Parametric Maps. Diagn (Basel) (2021) 11(6):1121. doi: 10.3390/diagnostics11061121

CrossRef Full Text | Google Scholar

39. Begum N, Maasberg S, Pascher A, Plöckinger U, Gress TM, Wurst C, et al. Long-Term Outcome of Surgical Resection in Patients With Gastroenteropancreatic Neuroendocrine Neoplasia: Results From a German Nation-Wide Multi-Centric Registry. Langenbecks Arch Surg (2020) 405(2):145–54. doi: 10.1007/s00423-020-01868-1

PubMed Abstract | CrossRef Full Text | Google Scholar

40. de Mestier L, Lepage C, Baudin E, Coriat R, Courbon F, Couvelard A, et al. Thésaurus National De Cancérologie Digestive (TNCD). Digestive Neuroendocrine Neoplasms (NEN): French Intergroup Clinical Practice Guidelines for Diagnosis, Treatment and Follow-Up (SNFGE, GTE, RENATEN, TENPATH, FFCD, GERCOR, UNICANCER, SFCD, SFED, SFRO, SFR). Dig Liver Dis (2020) 52(5):473–92. doi: 10.1016/j.dld.2020.02.011

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Merola E, Rinke A, Partelli S, Gress TM, Andreasi V, Kollár A, et al. Surgery With Radical Intent: Is There an Indication for G3 Neuroendocrine Neoplasms? Ann Surg Oncol (2020) 27(5):1348–55. doi: 10.1245/s10434-019-08049-5

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Zheng M, Li Y, Li T, Zhang L, Zhou L. Resection of the Primary Tumour Improves Survival in Patients With Gastro-Entero-Pancreatic Neuroendocrine Neoplasms With Liver Metastases: A SEER-Based Analysis. Cancer Med (2019) 8(11):5128–36. doi: 10.1002/cam4.2431

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Bender LC, Linnau KF, Meier EN, Anzai Y, Gunn ML. Interrater Agreement in the Evaluation of Discrepant Imaging Findings With the Radpeer System. AJR Am J Roentgenol (2012) 199:1320–7. doi: 10.2214/AJR.12.8972

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Borgstede JP, Lewis RS, Bhargavan M, Sunshine JH. RADPEER Quality Assurance Program: A Multifacility Study of Interpretive Disagreement Rates. J Am Coll Radiol: JACR (2004) 1:59–65. doi: 10.1016/S1546-1440(03)00002-4

CrossRef Full Text | Google Scholar

45. Donald JJ, Barnard SA. Common Patterns in 558 Diagnostic Radiology Errors. J Med Imaging Radiat Oncol (2012) 56:173–8. doi: 10.1111/j.1754-9485.2012.02348.x

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Hsu W, Han SX, Arnold CW, Bui AA, Enzmann DR. A Data-Driven Approach for Quality Assessment of Radiologic Interpretations. J Am Med Inform Assoc (2016) 23(e1):e152–6. doi: 10.1093/jamia/ocv161

PubMed Abstract | CrossRef Full Text | Google Scholar

47. McCreadie G, Oliver TB. Eight CT Lessons That We Learned the Hard Way: An Analysis of Current Patterns of Radiological Error and Discrepancy With Particular Emphasis on CT. Clin Radiol (2009) 64:491–9. doi: 10.1016/j.crad.2008.12.010

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Quattrocchi CC, Giona A, Di Martino AC, Errante Y, Scarciolla L, Mallio CA, et al. Extra-Spinal Incidental Findings at Lumbar Spine MRI in the General Population: A Large Cohort Study. Insights Imaging (2013) 4:301–8. doi: 10.1007/s13244-013-0234-z

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Lin E, Powell DK, Kagetsu NJ. Efficacy of a Checklist-Style Structured Radiology Reporting Template in Reducing Resident Misses on Cervical Spine Computed Tomography Examinations. J Digit Imaging (2014) 27:588–93. doi: 10.1007/s10278-014-9703-2

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Brook OR, Brook A, Vollmer CM, Kent TS, Sanchez N, Pedrosa I. Structured Reporting of Multiphasic CT for Pancreatic Cancer: Potential Effect on Staging and Surgical Planning. Radiology (2015) 274:464–72. doi: 10.1148/radiol.14140206

CrossRef Full Text | Google Scholar

51. Marcal LP, Fox PS, Evans DB, Fleming JB, Varadhachary GR, Katz MH, et al. Analysis of Free-Form Radiology Dictations for Completeness and Clarity for Pancreatic Cancer Staging. Abdom Imaging (2015) 40:2391–7. doi: 10.1007/s00261-015-0420-1

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Gupta NA, Mahajan S, Sumankumar A, Saklani A, Engineer R, Baheti AD. Impact of a Structured Report Template on the Quality of MRI Reports for Rectal Cancer Staging. AJR Am J Roentgenol (2015) 205:584–8. doi: 10.2214/AJR.14.14053

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Weiss DL, Bolos PR. “Reporting and Dictation”. In: Barton F, Branstetter IV, editors. Practical Imaging Informatics. Springer US: Foundations and applications for medical imaging (2021). doi: 10.1007/978-1-0716-1756-4

CrossRef Full Text | Google Scholar