Multimodal deep learning for predicting PD-L1 biomarker and clinical immunotherapy outcomes of esophageal cancer

Hui Liu ^1* Bin Wang ² Yinpu Bai ¹ Ruowei Bai ¹ Zhidong Wang ² Shi Yin ¹ Cheng Gong ²

• ¹ School of Computer Science and Technology, Nanjing Tech University, Nanjing, China
• ² Third Affiliated Hospital of Soochow University, Changzhou, Jiangsu, China

Although the immune checkpoint inhibitors (ICIs) have demonstrated remarkable anti-tumor efficacy in solid tumors, the proportion of ESCC patients who benefit from ICIs remains limited. Current biomarkers have assisted in identifying potential responders to immunotherapy, yet they all have inherent limitations. In this study, two ESCC cohorts were established from the Third Affiliated Hospital of Soochow University in China. One cohort included 220 patients with PD-L1 expression levels determined by immunohistochemistry, and the other cohort included 75 patients who underwent immunotherapy. For each patient in both cohorts, we curated multimodal data encompassing Hematoxylin and Eosin-stained pathology images, longitudinal computed tomography (CT) scans, and pertinent clinical variables. Next, we introduced a novel multimodal deep learning model that integrated pathological features, radiomic features, and clinical information to predict PD-L1 levels, immunotherapy response, and overall survival. Our model achieved an AUC value of 0.836 for PD-L1 biomarker prediction, and 0.809 for immunotherapy response prediction. Furthermore, our model also achieved an AUC value of 0.8 in predicting overall survival beyond one or three years. Our findings confirmed that the multimodal integration of pathological, radiomic, and clinical features offers a powerful means to predict PD-L1 biomarker levels and immunotherapy response in esophageal cancer.