A Multi-Modal Deep Learning Solution for Precise Pneumonia Diagnosis: The PneumoFusion-Net Model

Yujie Wang ^1,2 Can Liu ^1* Yinghan Fan ^1* Chenyue Niu ^1* Wanyun Huang ^1* Yixuan Pan ³ Jingze Li ^1,2 Yilin Wang ^1* Jun Li ^1,4,5*

• ¹ College of Information Engineering, Sichuan Agricultural University, Ya'an, Sichuan Province, China
• ² Deep Vision Agriculture Lab, Sichuan Agricultural University, Ya'an, China
• ³ College of Science, Sichuan Agricultural University, Ya an, China
• ⁴ Agricultural Information Engineering Higher Institution Key Laboratory of Sichuan Province, Ya'an, China
• ⁵ Ya’an Digital Agricultural Engineering Technology Research Center, Ya'an, China

Background: Pneumonia is considered one of the most important causes of morbidity and mortality in the world.Bacterial and viral pneumonia share many similar clinical features, thus making diagnosis a challenging task.Traditional diagnostic method developments mainly rely on radiological imaging and require a certain degree of consulting clinical experience, which can be inefficient and inconsistent. Deep learning for the classification of pneumonia in multiple modalities, especially integrating multiple data, has not been well explored.Methods: The study introduce the PneumoFusion-Net, a deep learning-based multimodal framework that incorporates CT images, clinical text, numerical lab test results, and radiology reports for improved diagnosis. In the experiments, a dataset of 10,095 pneumonia CT images was used-including associated clinical data-most of which was used for training and validation while keeping part of it for validation on a held-out test set. Five-fold cross-validation was considered in order to evaluate this model, calculating different metrics including accuracy and F1-Score.Results: PneumoFusion-Net, which achieved 98.96% classification accuracy with a 98% F1-score on the held-out test set, is highly effective in distinguishing bacterial from viral types of pneumonia. This has been highly beneficial for diagnosis, reducing misdiagnosis and further improving homogeneity across various data sets from multiple patients.Conclusions: PneumoFusion-Net offers an effective and efficient approach to pneumonia classification by integrating diverse data sources, resulting in high diagnostic accuracy. Its potential for clinical integration could significantly reduce the burden of pneumonia diagnosis by providing radiologists and clinicians with a robust, automated diagnostic tool.