Integration of Multimodal Imaging Data with Machine Learning for Improved Diagnosis and Prognosis in Neuroimaging

Saurabh Bhattacharya ¹ Dr. Sashikanta Prusty ² Dr. Sashikanta Prusty ³ Monali Gulhane ^4* Santosh H Lavate ⁵ Nitin Rakesh ⁴ Saravanan Veerasamy ^6*

• ¹ Galgotias University, Greater Noida, Uttar Pradesh, India
• ² Siksha O Anusandhan University, Bhubaneswar, Odisha, India
• ³ Yeshwantrao Chavan College of Engineering (YCCE), Nagpur, Maharashtra, India
• ⁴ Symbiosis Institute of Technology, Symbiosis International University, Pune, India
• ⁵ All India Shri Shivaji Memorial Society's College of Engineering, Pune, Maharashtra, India
• ⁶ Dambi Dollo University, Dambi Dollo, Ethiopia

Introduction The integration of multimodal imaging data, particularly structural MRI (sMRI) and functional MRI (fMRI), holds significant potential for improving the diagnosis and prognosis of neurological disorders (Alzheimer). However existing approaches often fail to effectively combine spatial and temporal patterns from these modalities, limiting their predictive power. This study proposes a novel hybrid deep learning(DL) framework that leverages CNN, GRU, and attention mechanisms for multimodal data fusion. This study introduces a hybrid deep learning framework combining CNN, GRU, and a novel Dynamic Cross-Modality Attention Module for effective integration of spatial and temporal neuroimaging features. The approach enhances diagnostic accuracy and interpretability, addressing limitations of existing multimodal fusion techniques.Methods The proposed framework extracts spatial features from sMRI using CNNs and models temporal dynamics from fMRI connectivity metrics using GRUs. An attention mechanism is incorporated to prioritize diagnostically relevant features, enabling robust multimodal integration. The model was trained and evaluated on the HCP dataset, which includes sMRI, fMRI, and behavioral data. Performance was assessed using metrics such as “accuracy, Recall, precision and F1-score.Results The hybrid framework achieved an accuracy of 96.79%. Compared to existing methods the proposed approach demonstrated superior performance in classifying neurological conditions.Discussion These findings emphasize the efficacy of the hybrid framework in leveraging complementary information from multimodal imaging. The use of attention mechanisms improved feature prioritization, enhancing both interpretability and diagnostic accuracy.ConclusionThe proposed framework sets a new benchmark for multimodal neuroimaging analysis, offering significant promise for clinical applications in neurological diagnosis and prognosis.