Deep learning-driven medical image analysis for computational material science applications

Introduction: Deep learning has significantly advanced medical image analysis, enabling precise feature extraction and pattern recognition. However, its application in computational material science remains underexplored, despite the increasing need for automated microstructure analysis and defect detection. Traditional image processing methods in material science often rely on handcrafted feature extraction and threshold-based segmentation, which lack adaptability to complex microstructural variations. Conventional machine learning approaches struggle with data heterogeneity and the need for extensive labeled datasets.

Methods: To overcome these limitations, we propose a deep learning-driven framework that integrates convolutional neural networks (CNNs) with transformer-based architectures for enhanced feature representation. Our method incorporates domain-adaptive transfer learning and multi-modal fusion techniques to improve the generalizability of material image analysis.

Results: Experimental evaluations on diverse datasets demonstrate superior performance in segmentation accuracy, defect detection robustness, and computational efficiency compared to traditional methods.

Discussion: By bridging the gap between medical image processing techniques and computational material science, our approach contributes to more effective, automated, and scalable material characterization processes.

1 Introduction

The intersection of deep learning and medical image analysis has opened new frontiers not only in healthcare but also in computational material science, where advanced imaging techniques play a crucial role in material characterization and defect detection Tang et al. (2021). Medical imaging methodologies, such as computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound, offer sophisticated ways to analyze biological structures, and their underlying principles can be adapted to study material properties, phase transformations, and microstructural patterns in engineered materials Cao et al. (2021). Not only do deep learning-driven image analysis techniques enable precise identification of structural abnormalities in biological tissues, but they also provide automated solutions for detecting microstructural defects, grain boundaries, and mechanical stress points in synthetic materials Zhang and Metaxas (2023). The ability to process large-scale image datasets using AI-driven algorithms enhances predictive modeling, enabling more efficient materials discovery and optimization Mazurowski et al. (2023). However, while traditional medical imaging techniques have been extensively studied in the clinical domain, their adaptation to computational material science presents unique challenges, including variations in imaging modalities, differences in data annotation standards, and the need for explainability in AI-driven material characterization Li M. et al. (2023). Thus, leveraging deep learning methodologies originally developed for medical image analysis can bridge the gap between biomedical and materials science, offering transformative solutions for automated defect detection, structural analysis, and material behavior prediction Li X. et al. (2023).

To address the limitations of manual image inspection and conventional computational models, early approaches to image analysis in both medical and material science domains relied on classical feature extraction techniques and rule-based methods Azad et al. (2023). Traditional computer vision algorithms, such as edge detection, histogram analysis, and texture-based classification, were employed to analyze medical scans and material microstructures Konovalenko et al. (2018b). In medical imaging, techniques like Gabor filters and wavelet transforms were commonly used to enhance feature representations for tumor detection, while in material science, similar methods were applied to identify grain structures and crystallographic defects Zhou et al. (2023). Classical segmentation techniques, including thresholding, region growing, and watershed algorithms, were widely utilized to isolate key features from medical and material images. While these approaches demonstrated reasonable accuracy in well-defined settings, they often struggled with complex image variations, noise artifacts, and heterogeneous textures Dhar et al. (2023). Rule-based methods lacked adaptability to new imaging conditions, requiring extensive manual tuning for different datasets Kshatri and Singh (2023). To improve automation and robustness, researchers started integrating statistical learning techniques, including Principal Component Analysis (PCA) and Support Vector Machines (SVMs), to enhance data analysis and pattern recognition, leveraging their capabilities for dimensionality reduction and classification, which provided more flexibility but still relied heavily on handcrafted feature engineering Nazir and Kaleem (2023).

To address the shortcomings of manually engineered features, machine learning-based image analysis shifted towards data-driven approaches, enabling models to autonomously learn feature representations directly from raw image data Ma et al. (2023). Supervised learning methods, including Convolutional Neural Networks (CNNs), began to gain traction in medical imaging applications, enabling automated disease classification, tumor segmentation, and anomaly detection Sistaninejhad et al. (2023). In computational material science, similar methodologies were applied to classify microstructural patterns, detect defects in composite materials, and predict mechanical properties based on imaging data. Feature learning through pre-trained networks, such as AlexNet and VGG, provided improved accuracy over traditional methods, reducing the reliance on manual feature selection Liu et al. (2023). The integration of generative models, such as autoencoders, facilitated unsupervised feature extraction for identifying material phase transitions and crystallographic variations Huang et al. (2023). However, despite their advantages, machine learning-based image analysis methods faced challenges in generalizability due to limited labeled datasets, domain-specific variations, and difficulties in interpreting model decisions Sohan and Basalamah (2023). Machine learning models required extensive computational resources for training and fine-tuning, limiting their scalability in high-throughput material analysis applications Zhang et al. (2023).

With the rise of deep learning and the emergence of transformer-based architectures, medical image analysis has undergone a paradigm shift, significantly improving accuracy, scalability, and adaptability Drukker et al. (2023). Deep learning models, particularly CNNs and Vision Transformers (ViTs), have revolutionized image classification, segmentation, and object detection tasks, outperforming conventional approaches in both medical and material imagingKonovalenko et al. (2018a). In medical imaging, state-of-the-art models such as U-Net and DeepLab have enabled precise organ segmentation, tumor detection, and disease progression analysis. Similarly, in material science, deep learning-based segmentation models have been employed to analyze electron microscopy images, detect material defects, and predict failure mechanisms in engineering materials Guan and Liu (2021). Transformer-based models, such as ViTs, have further improved feature extraction by capturing long-range dependencies in imaging data, making them particularly useful for analyzing complex material structures. Moreover, the integration of self-supervised learning and contrastive learning approaches has enabled deep learning models to leverage unlabeled data, reducing the dependency on manually annotated datasets He et al. (2022). However, despite their success, deep learning-based models present new challenges, including interpretability concerns, data bias issues, and the high computational cost of training large-scale architectures. The application of deep learning to material science necessitates domain-specific adaptations, requiring customized training pipelines and the incorporation of physical modeling constraints to ensure the reliability of AI-driven predictions Nirthika et al. (2022).

Given the limitations of existing approaches, our proposed method introduces a novel deep learning framework that bridges medical image analysis techniques with computational material science applications. By leveraging domain-adapted convolutional and transformer-based architectures, our approach enhances automated defect detection, microstructural classification, and material behavior prediction. Our model incorporates self-supervised learning and transfer learning strategies to maximize performance on limited labeled datasets, ensuring generalizability across different material imaging modalities. Unlike traditional deep learning methods, our framework integrates explainable AI (XAI) techniques, providing visual interpretations of model decisions and increasing transparency in AI-driven material characterization. Real-time adaptive learning mechanisms enable our model to dynamically refine predictions based on evolving material datasets, making it a scalable and robust solution for high-throughput material analysis applications.

We summarize our contributions as follows.

$•$ Our method integrates convolutional and transformer-based architectures with domain-specific adaptations, enhancing defect detection and material property prediction from imaging data.

$•$ By leveraging self-supervised and transfer learning techniques, our approach maximizes performance on limited labeled datasets, ensuring adaptability across diverse material imaging conditions.

$•$ Experimental results demonstrate superior accuracy in material defect classification and microstructural segmentation, while integrated explainability techniques enhance trustworthiness and transparency in AI-driven material analysis.

2 Related work

2.1 Deep learning in medical image analysis

Deep learning has significantly advanced medical image analysis, enhancing the accuracy and efficiency of diagnostic processes. Convolutional Neural Networks (CNNs), in particular, have demonstrated remarkable proficiency in autonomously learning features from multidimensional medical images, including MRI, CT, and X-ray scans, without the necessity for manual feature extraction Elyan et al. (2022). This capability has improved the precision of clinical procedures and facilitated expedited diagnoses. The U-Net architecture, a type of CNN, has been instrumental in medical image segmentation. Developed for image segmentation tasks, U-Net’s design allows for precise delineation of complex anatomical structures, which is crucial for accurate diagnosis and treatment planning Yang et al. (2020). Its ability to work with limited training data while achieving high segmentation accuracy has made it a standard in biomedical image analysis. Autoencoders, another class of deep learning models, have been applied to medical imaging for tasks such as image denoising and super-resolution Rezaei et al. (2024). By learning efficient codings of input data, autoencoders can reconstruct images with reduced noise levels, thereby enhancing the quality of medical images and aiding in better interpretation and analysis Yamazaki et al. (2024).

2.2 Applications in computational material science

The methodologies developed for medical image analysis have found applications in computational material science, particularly in the analysis of microstructural images Zhou et al. (2022). Techniques such as U-Net have been employed to segment and analyze images of materials, facilitating the study of their properties and behaviors under various conditions Chen et al. (2022). This cross-disciplinary application underscores the versatility of deep learning models in processing and interpreting complex image data across different scientific domains Rezaei et al. (2025). Radiomics, a method that extracts a large number of features from medical images using data-characterization algorithms, has also been adapted for material science applications. By analyzing the texture and patterns in images, radiomics can uncover characteristics that are not discernible to the naked eye, providing deeper insights into the material’s structure and potential performance Fuhr and Sumpter (2022).

2.3 Integration of medical imaging techniques into material science

The integration of medical imaging techniques into material science involves adapting tools and methodologies originally designed for biological tissues to the study of materials Liu et al. (2022). Software platforms like ScanIP have been utilized to generate high-quality 3D models from image data, enabling detailed visualization and analysis of material structures. These tools allow researchers to segment, quantify, and analyze different components within a material, facilitating a comprehensive understanding of its properties and potential applications Abdou (2022). Moreover, the application of deep learning models, such as autoencoders and CNNs, to material science has enabled the development of predictive models that can simulate how materials respond to various stresses and environmental factors Lambert et al. (2022). This predictive capability is essential for designing materials with desired properties and for anticipating their performance in real-world applications. The cross-pollination of deep learning-driven medical image analysis techniques into computational material science has opened new avenues for research and innovation Furat et al. (2019). By leveraging advanced image analysis tools and methodologies, scientists can gain deeper insights into material properties, leading to the development of novel materials and enhanced performance in various applications Reimann et al. (2019).

3 Methods

3.1 Overview

Artificial Intelligence (AI) has significantly transformed the healthcare industry, enhancing medical diagnosis, personalized treatment, and operational efficiency in clinical settings. With the rapid advancements in deep learning, reinforcement learning, and probabilistic modeling, AI-driven systems now play a crucial role in medical imaging, drug discovery, and electronic health record (EHR) analysis. However, despite these achievements, challenges such as data heterogeneity, model interpretability, and reliability hinder the full adoption of AI in real-world healthcare applications.

The use of AI in healthcare spans various domains, including radiology, pathology, genomics, and robotic-assisted surgery. Deep learning techniques, such as convolutional neural networks (CNNs), have revolutionized medical image analysis by enabling automated disease detection and segmentation. Meanwhile, natural language processing (NLP) models facilitate the extraction of valuable insights from clinical notes, streamlining patient management. Reinforcement learning has also shown promise in optimizing treatment strategies, such as individualized drug dosing and radiation therapy planning. Despite these advancements, AI models in healthcare often face issues related to limited labeled datasets, domain shift, and ethical concerns surrounding automated decision-making. To overcome these challenges, our method integrates domain adaptation strategies and probabilistic modeling, aiming to improve both the generalizability and reliability of the model. The mathematical formulation in Section 3.2 provides a rigorous foundation for our AI-driven framework. The AI model introduced in Section 3.3 leverages state-of-the-art architectural components, such as attention mechanisms and transformer-based encoders, to extract meaningful patterns from complex medical data. Our optimized learning strategies in Section 3.4 ensure model stability, fairness, and computational feasibility, making the proposed system well-suited for integration into clinical practice. By combining these methodologies, our framework aims to bridge the gap between cutting-edge AI research and practical healthcare applications. The proposed model not only improves predictive performance but also ensures compliance with medical regulations and ethical considerations, facilitating its adoption in modern healthcare systems.

3.2 Preliminaries

The integration of Artificial Intelligence (AI) in healthcare requires a formal mathematical framework to define key variables, constraints, and optimization objectives. This section establishes a structured representation of AI-driven medical decision-making by modeling clinical tasks as structured learning problems. We define the problem within a probabilistic framework and introduce the mathematical foundation necessary for the subsequent development of our proposed model.

Let $\mathcal{X}\subset {\mathbb{R}}^d$ denote the feature space representing patient-specific data, including medical imaging, electronic health records (EHR), and genomic sequences. Each patient sample $x_i\in \mathcal{X}$ is associated with a corresponding clinical outcome or diagnosis $y_i\in \mathcal{Y}$, where $\mathcal{Y}$ represents the space of possible medical conditions. Given a dataset $\mathcal{D}={\{(x_i,y_i)\}}_{i=1}^N$, the goal of an AI-driven system is to learn a function (Equation 1):

$f:\mathcal{X}\to \mathcal{Y},(1)$

that accurately predicts medical outcomes while incorporating uncertainty estimation.

Medical data is inherently heterogeneous, consisting of structured (EHR), unstructured (clinical notes), and high-dimensional (medical images) data. We define a multi-modal feature extractor $\mathrm{\Phi }:\mathcal{X}\to {\mathbb{R}}^m$ such that (Equation 2):

$\mathbf{z}=\mathrm{\Phi }\left(x\right),(2)$

where $\mathbf{z}\in {\mathbb{R}}^m$ is the learned feature representation. A classification model $g:{\mathbb{R}}^m\to \mathcal{Y}$ is then used to predict clinical outcomes (Equation 3):

$\widehat{y}=g\left(\mathbf{z}\right).(3)$

The learning objective is to minimize a loss function $\mathcal{L}$, which measures the discrepancy between the predicted and true outcomes (Equation 4):

${\theta }^{*}=\arg \underset{\theta }{\min }{\displaystyle \sum _{i=1}^N}\mathcal{L}\left(y_i,g\left(\mathrm{\Phi }\left(x_i\right);\theta \right)\right).(4)$

For image-based tasks such as disease classification and segmentation, we define the medical image space as $\mathcal{I}\subset {\mathbb{R}}^{h\times w\times c}$, where $h$, $w$, and $c$ represent image height, width, and number of channels, respectively. A convolutional neural network (CNN) is used as a feature extractor (Equation 5):

${\mathrm{\Phi }}_{\text{IMG}}\left(I\right)=\text{CNN}\left(I;{\theta }_{\text{CNN}}\right).(5)$

These extracted features are then used for classification (Equation 6):

$\widehat{y}=g\left({\mathrm{\Phi }}_{\text{IMG}}\left(I\right);{\theta }_g\right).(6)$

For time-series data such as EHR, we model patient records as sequential observations. Let ${\mathbf{X}}_i=(x_{i1},x_{i2},\dots ,x_{iT})$ represent patient $i$’s historical data over $T$ time steps. A recurrent model captures dependencies across time as follows (Equations 7, 8):

$h_t=\sigma \left(W_h{h}_{t-1}+W_x{x}_t+b_h\right),(7)$

${\widehat{y}}_t=g\left(h_t\right),(8)$

where $h_t$ is the hidden state, $W_h$, $W_x$, and $b_h$ are learned parameters, and $\sigma (\cdot )$ is a nonlinear activation function.

Uncertainty estimation is crucial in AI-driven healthcare due to the high-stakes nature of medical decisions. We incorporate Bayesian deep learning to model epistemic and aleatoric uncertainty. The predictive distribution is given by Equation 9:

$p\left(y|x,\mathcal{D}\right)=\int p\left(y|x,\theta \right)p\left(\theta |\mathcal{D}\right)d\theta .(9)$

Using Monte Carlo dropout, we approximate the uncertainty-aware prediction as Equation 10:

$\widehat{y}=\frac{1}{M}{\displaystyle \sum _{m=1}^M}{f}_{{\theta }_m}\left(x\right),{\sigma }^2=\frac{1}{M}{\displaystyle \sum _{m=1}^M}{\left(f_{{\theta }_m}\left(x\right)-\widehat{y}\right)}^2.(10)$

In treatment optimization, AI models aim to determine the best intervention based on patient state. Let $\mathcal{T}$ represent the space of possible treatments, and let $\mathcal{R}(y,t)$ denote the reward function evaluating the effectiveness of treatment $t\in \mathcal{T}$ given patient condition $y$. The optimal treatment policy $\pi *$ is Equation 11:

${\pi }^{*}=\arg \underset{\pi }{\max }\mathbb{E}\left[{\displaystyle \sum _{t=1}^T}{\gamma }^t\mathcal{R}\left(y_t,\pi \left(y_t\right)\right)\right],(11)$

where $\gamma \in (\mathrm{0,1}]$ is a discount factor controlling the weight of future rewards.

3.3 Uncertainty-aware multi-modal medical AI model (UAMM)

To address the challenges in AI-driven healthcare, we propose the Uncertainty-Aware Multi-Modal Medical AI Model (UAMM), a novel deep learning framework integrating multi-modal data fusion, uncertainty-aware learning, and adaptive decision-making. This model enhances predictive accuracy, robustness, and interpretability in clinical applications such as disease diagnosis, patient monitoring, and treatment optimization (As shown in Figure 1).

Figure 1

Figure 1. The architecture of the Uncertainty-Aware Multi-Modal Medical AI Model (UAMM), which integrates (a) multi-modal data fusion (MDF), (b) uncertainty-aware prediction (UAP), and (c) reinforcement learning-based treatment (RLT). The model processes structured electronic health records, medical images, and clinical text through specialized feature extractors, fusing them into a unified representation. Bayesian deep learning techniques quantify predictive uncertainty to enhance robustness and interpretability, while an actor-critic reinforcement learning framework optimizes personalized treatment strategies, ultimately improving predictive accuracy and adaptive decision-making in clinical applications.

3.4 Multi-modal data fusion

To effectively integrate structured and unstructured medical data, we introduce a hierarchical feature extraction mechanism that enables a comprehensive representation of patient conditions. The model processes three primary data modalities: structured electronic health records (EHR), high-dimensional medical images, and unstructured clinical text. Each modality is processed through specialized feature extractors tailored to capture the distinct characteristics of the respective data types. The structured EHR data is transformed using a transformer-based encoder, the medical images are processed through a deep convolutional neural network (CNN), and the clinical text is encoded using a bidirectional recurrent neural network (BiLSTM) to capture contextual dependencies. The derived feature representations are subsequently merged into a single, cohesive feature vector (Equation 12):

$\mathbf{z}=\text{Concat}\left({\mathrm{\Phi }}_{\text{EHR}}\left(x\right),{\mathrm{\Phi }}_{\text{IMG}}\left(x\right),{\mathrm{\Phi }}_{\text{TXT}}\left(x\right)\right),(12)$

where ${\mathrm{\Phi }}_{\text{EHR}}:{\mathbb{R}}^{d_{\text{EHR}}}\to {\mathbb{R}}^{m_1}$, ${\mathrm{\Phi }}_{\text{IMG}}:{\mathbb{R}}^{h\times w\times c}\to {\mathbb{R}}^{m_2}$, and ${\mathrm{\Phi }}_{\text{TXT}}:{\mathbb{R}}^{d_{\text{TXT}}}\to {\mathbb{R}}^{m_3}$ represent modality-specific encoders that map raw inputs into feature spaces of dimensions $m_1$, $m_2$, and $m_3$, respectively. To ensure effective fusion and reduce redundancy, a modality attention mechanism is applied to weight each feature contribution dynamically (Equation 13):

${\mathbf{z}}^{\prime }={\displaystyle \sum _{i\in \left\{\text{EHR,\hspace{0.17em}IMG,\hspace{0.17em}TXT}\right\}}}{\alpha }_i{\mathrm{\Phi }}_i\left(x\right),(13)$

where ${\alpha }_i$ is a learnable attention weight assigned to each modality, computed using a softmax function to normalize the importance scores across modalities. This mechanism enhances the adaptability of the model by allowing it to prioritize informative modalities based on input variability. The combined representation ${\mathbf{z}}^{\prime }$ is subsequently mapped into a lower-dimensional space through a non-linear transformation, optimizing computational efficiency (Equation 14):

$\mathbf{h}=\sigma \left(W{\mathbf{z}}^{\prime }+b\right),(14)$

where $W\in {\mathbb{R}}^{d_h\times (m_1+m_2+m_3)}$ and $b\in {\mathbb{R}}^{d_h}$ are trainable parameters, where $\sigma$ denotes a non-linear activation function, such as ReLU or GELU, which introduces complex transformations to enhance model expressiveness. To further improve robustness, we introduce a feature consistency loss that ensures alignment across modalities by minimizing the discrepancy between projected features (Equation 15):

${\mathcal{L}}_{\text{fusion}}={\displaystyle \sum _{i,j}}\Vert {\mathrm{\Phi }}_i\left(x\right)-{\mathrm{\Phi }}_j\left(x\right){\Vert }_2^2.(15)$

This multi-modal fusion strategy provides a holistic and interpretable representation of patient data, allowing the model to leverage complementary information from different modalities, ultimately improving predictive performance and clinical decision-making.

3.5 Uncertainty-Aware Prediction

To quantify predictive uncertainty in medical AI applications, we employ Bayesian deep learning techniques, which enable the model to estimate confidence in its predictions. Given an input $x$ and a dataset $\mathcal{D}$, the predictive distribution is defined as Equation 16:

$p\left(y|x,\mathcal{D}\right)=\int p\left(y|f_{\theta }\left(x\right)\right)p\left(\theta |\mathcal{D}\right)d\theta .(16)$

Since computing this integral is intractable, we approximate it using Monte Carlo dropout, where multiple stochastic forward passes through the network provide a sample-based estimate of the prediction. The mean prediction and uncertainty are given by Equation 17:

$\widehat{y}=\frac{1}{M}{\displaystyle \sum _{m=1}^M}{f}_{{\theta }_m}\left(x\right),{\sigma }^2=\frac{1}{M}{\displaystyle \sum _{m=1}^M}{\left(f_{{\theta }_m}\left(x\right)-\widehat{y}\right)}^2.(17)$

To further refine uncertainty estimation, we introduce a heteroscedastic uncertainty-aware loss function, which dynamically adjusts learning based on the predicted variance (Equation 18):

${\mathcal{L}}_{\text{hetero}}={\displaystyle \sum _{i=1}^N}\frac{1}{2{\sigma }_i^2}{\left(y_i-{\widehat{y}}_i\right)}^2+\frac{1}{2}\log {\sigma }_i^2.(18)$

We incorporate an evidential deep learning approach where the model learns an evidence-based uncertainty measure using a Dirichlet distribution prior, leading to an uncertainty-regularized objective (Equation 19):

${\mathcal{L}}_{\text{evidential}}={\displaystyle \sum _{i=1}^N}\left(\frac{{\left(y_i-{\widehat{y}}_i\right)}^2}{2\left({\sigma }_i^2+\lambda \right)}+\frac{1}{2}\log \left({\sigma }_i^2+\lambda \right)\right),(19)$

where $\lambda$ controls the regularization strength. By integrating these approaches, our model not only improves predictive performance but also provides reliable confidence scores, which are crucial for clinical decision support, risk assessment, and personalized treatment planning. This uncertainty-aware framework enhances model robustness and facilitates interpretable AI-driven medical diagnostics.

3.6 Reinforcement Learning-Based Treatment

To optimize treatment decisions dynamically and adaptively, we model the problem using reinforcement learning (RL) with an actor-critic approach (As shown in Figure 2). The objective is to find an optimal policy $\pi *$ that maximizes the expected cumulative reward over a finite time horizon $T$ Equation 20:

${\pi }^{*}=\arg \underset{\pi }{\max }\mathbb{E}\left[{\displaystyle \sum _{t=1}^T}{\gamma }^t\mathcal{R}\left(y_t,\pi \left(y_t\right)\right)\right],(20)$

where $\gamma \in (\mathrm{0,1}]$ is the discount factor that determines the importance of future rewards, and $\mathcal{R}(y_t,\pi (y_t))$ represents the reward function, capturing the effectiveness of treatment $t_t$ applied to the patient state $y_t$. The policy ${\pi }_{\varphi }(t_t|s_t)$, parameterized by $\varphi$, is optimized using the policy gradient method, where the loss function is given by Equation 21:

${\mathcal{L}}_{\text{policy}}=-\mathbb{E}\left[A\left(s_t,t_t\right)\log {\pi }_{\varphi }\left(t_t|s_t\right)\right].(21)$

Figure 2

Figure 2. Overview of the Reinforcement Learning-Based Treatment (RLT) framework. The model leverages reinforcement learning with an actor-critic approach to optimize treatment decisions adaptively. The architecture consists of a Policy Network, Multi-Head Self-Attention, and Self-Attention layers, followed by LarK Blocks and hybrid loss functions. The right section highlights the detailed structure of the LarK Block, including Dilated Re-param Blocks, SE Blocks, and Feed-Forward Networks (FFN). The objective is to learn an optimal treatment policy that maximizes cumulative rewards over time by updating both the policy and value networks with policy gradient and temporal difference learning.

Here, $A(s_t,t_t)$ is the advantage function, which estimates how much better a particular action $t_t$ is compared to the expected value at state $s_t$. The critic network approximates the state-value function $V_{\psi }(s_t)$, which is learned by minimizing the temporal difference (TD) error (Equation 22):

${\mathcal{L}}_{\text{value}}=\mathbb{E}\left[{\left(r_t+\gamma V_{\psi }\left(s_{t+1}\right)-V_{\psi }\left(s_t\right)\right)}^2\right].(22)$

To ensure stable learning, an entropy regularization term is often added to encourage policy exploration, preventing premature convergence to suboptimal deterministic policies (Equation 23):

${\mathcal{L}}_{\text{entropy}}=-\mathbb{E}\left[H\left({\pi }_{\varphi }\left(t_t|s_t\right)\right)\right],(23)$

where $H({\pi }_{\varphi }(t_t|s_t))$ represents the entropy of the policy distribution, promoting diverse action selection. By jointly optimizing these objectives, reinforcement learning enables the design of an adaptive and personalized treatment strategy that evolves based on patient responses, ultimately improving long-term healthcare outcomes.

3.7 Optimized learning strategies for medical AI (OLSMA)

To enhance the performance of our proposed Uncertainty-Aware Multi-Modal Medical AI Model (UAMM), we introduce Optimized Learning Strategies for Medical AI (OLSMA). OLSMA consists of three key innovations. These strategies improve generalization, interpretability, and computational efficiency, ensuring UAMM’s applicability in real-world clinical settings (As shown in Figure 3).

Figure 3

Figure 3. Overview of the Optimized Learning Strategies for Medical AI (OLSMA) framework, designed to enhance the performance of the Uncertainty-Aware Multi-Modal Medical AI Model (UAMM). The framework comprises three key innovations, Adaptive Data Augmentation (ADA), which dynamically enhances training data diversity; Uncertainty-Aware Learning (UAL), which quantifies model confidence to improve robustness; and Efficient Model Compression (EMC), which reduces computational overhead while preserving accuracy. These strategies collectively improve generalization, interpretability, and efficiency, ensuring applicability in real-world clinical scenarios.

3.8 Adaptive data augmentation

Medical datasets often suffer from class imbalance and limited diversity, which can negatively impact the generalization ability of deep learning models. To mitigate these issues, we employ an adaptive augmentation strategy that dynamically modifies the augmentation intensity based on the dataset characteristics. Given a medical imaging dataset $\mathcal{I}={\{I_i\}}_{i=1}^N$, we define an augmentation function $\mathcal{A}$ that applies geometric transformations and statistical noise perturbations to enrich the training distribution. The augmented image is generated as follows Equation 24:

${\tilde{I}}_i=\mathcal{A}\left(I_i\right)=\text{Affine}\left(I_i\right)+\lambda \cdot \text{Noise}\left(I_i\right),(24)$

where $\text{Affine}(I_i)$ applies transformations such as rotation, scaling, flipping, and elastic deformations, while $\text{Noise}(I_i)$ introduces adaptive perturbations, including Gaussian, Poisson, or speckle noise, with an intensity coefficient $\lambda$ determined by the dataset variability. To maintain the underlying structure of medical images, we introduce a regularization term that ensures minimal deviation from the original pixel distribution (Equation 25):

${\mathcal{L}}_{\text{aug}}={\displaystyle \sum _{i=1}^N}\Vert I_i-{\tilde{I}}_i{\Vert }_2^2+\beta \cdot \text{TV}\left({\tilde{I}}_i\right),(25)$

where $\text{TV}(\cdot )$ represents the total variation loss, which penalizes excessive noise artifacts, and $\beta$ is a regularization weight controlling smoothness. We incorporate a contrastive consistency loss to enforce semantic similarity between the original and augmented images in the feature space (Equation 26):

${\mathcal{L}}_{\text{contrast}}=-\frac{1}{N}{\displaystyle \sum _{i=1}^N}\log \frac{\exp \left(\text{sim}\left(f\left(I_i\right),f\left({\tilde{I}}_i\right)\right)/\tau \right)}{{\sum }_{j=1}^N\exp \left(\text{sim}\left(f\left(I_i\right),f\left(I_j\right)\right)/\tau \right)},(26)$

where $\text{sim}(\cdot ,\cdot )$ denotes the cosine similarity function, $f(\cdot )$ represents the feature extractor, and $\tau$ is a temperature scaling factor. To prevent mode collapse in low-data scenarios, we employ a diversity-driven augmentation policy that adjusts the augmentation probabilities based on class distributions (Equation 27):

$P_{\mathcal{A}}\left(I_i\right)=\frac{1}{1+\exp \left(-\gamma \left(c_i-\mu \right)\right)},(27)$

where $c_i$ is the sample class frequency, $\mu$ is the dataset mean frequency, and $\gamma$ is a scaling parameter controlling the augmentation intensity. This adaptive augmentation strategy enhances model robustness, improves generalization across unseen clinical scenarios, and ensures that rare pathological patterns are adequately represented in the training set.

3.9 Uncertainty-aware learning

To enhance model reliability and robustness in medical AI, we integrate uncertainty-aware learning strategies that allow the model to quantify and adjust its confidence dynamically. In real-world clinical applications, the inherent variability in medical data necessitates a principled approach to handling uncertainty. This is accomplished by implementing an uncertainty-aware loss function that dynamically adjusts the learning process according to the model’s confidence in its predictions (As shown in Figure 4). Given a set of predictions ${\widehat{y}}_i$ and corresponding ground truth labels $y_i$, the uncertainty-aware loss is defined as Equation 28:

${\mathcal{L}}_{\text{uncertainty}}={\displaystyle \sum _{i=1}^N}\frac{1}{{\sigma }_i^2}\mathcal{L}\left(y_i,{\widehat{y}}_i\right)+\log {\sigma }_i,(28)$

where ${\sigma }_i$ represents the model’s uncertainty in predicting ${\widehat{y}}_i$. This formulation ensures that predictions with higher uncertainty contribute less to the loss, thereby preventing the model from overfitting to uncertain samples. To further refine uncertainty estimation, we adopt a Bayesian deep learning framework, treating the model parameters $\theta$ as distributions rather than fixed values. Using Monte Carlo dropout, the predicted mean and variance are approximated as Equation 29:

${\widehat{y}}_i=\frac{1}{M}{\displaystyle \sum _{m=1}^M}{f}_{{\theta }_m}\left(x_i\right),{\sigma }_i^2=\frac{1}{M}{\displaystyle \sum _{m=1}^M}{\left(f_{{\theta }_m}\left(x_i\right)-{\widehat{y}}_i\right)}^2.(29)$

To stabilize training in low-data regimes, we introduce an evidence-based regularization term that penalizes overconfident predictions while maintaining flexibility in uncertain regions (Equation 30):

${\mathcal{L}}_{\text{reg}}={\displaystyle \sum _{i=1}^N}\left(\frac{{\left(y_i-{\widehat{y}}_i\right)}^2}{2\left({\sigma }_i^2+\lambda \right)}+\frac{1}{2}\log \left({\sigma }_i^2+\lambda \right)\right),(30)$

where $\lambda$ is a small positive scalar ensuring numerical stability. We incorporate a confidence-based weighting mechanism that adaptively adjusts learning rates for different samples, leveraging an entropy-based uncertainty metric (Equation 31):

$w_i=\frac{H\left(y_i|x_i\right)}{{\sum }_{j=1}^{N}H\left(y_j|x_j\right)},H\left(y_i|x_i\right)=-{\displaystyle \sum _c}{p}_c\log p_c.(31)$

Here, $H(y_i|x_i)$ denotes the entropy of the model’s predictive distribution, and $w_i$ represents the normalized uncertainty weight. This approach allows the model to focus more on high-confidence predictions while still accounting for uncertain cases. By integrating these uncertainty-aware learning mechanisms, we improve model interpretability, robustness, and generalization, making AI-driven clinical decision support systems more reliable in real-world scenarios.

Figure 4

Figure 4. Illustration of the Uncertainty-Aware Learning framework, which enhances model robustness by quantifying predictive uncertainty. The model processes multi-scale feature representations using shared MLPs across different scales, followed by feature aggregation through concatenation, max-pooling, and repetition mechanisms. The mathematical formulation incorporates an uncertainty-aware loss function, Bayesian deep learning with Monte Carlo dropout, and confidence-based weighting to dynamically adjust learning rates based on entropy-driven uncertainty estimation. This strategy improves interpretability and reliability in medical AI applications by preventing overfitting to uncertain samples and refining predictive confidence.

3.10 Efficient model compression

To optimize computational efficiency and reduce memory footprint while maintaining model performance, we employ a combination of low-rank factorization and quantization techniques. These approaches enable efficient storage and inference, making them particularly suitable for resource-constrained environments. Given a weight matrix $W\in {\mathbb{R}}^{d\times d}$, we approximate it using low-rank decomposition (Equation 32):

$W\approx UV,U\in {\mathbb{R}}^{d\times r},V\in {\mathbb{R}}^{r\times d},r\ll d.(32)$

Here, $r$ is the rank of the approximation, chosen to balance accuracy and computational efficiency. This decomposition reduces the number of parameters from $O(d^2)$ to $O(rd)$, significantly lowering the model’s storage and computation requirements. We employ weight quantization to represent model parameters with a lower bit precision while preserving essential information (Equation 33):

$W_q=\text{Quantize}\left(W,b\right),(33)$

where $b$ denotes the bit-depth used for quantization. Lower values of $b$ reduce memory usage and improve inference speed, while higher values retain greater precision. To further enhance efficiency, we introduce sparsification, where small-magnitude weights are pruned based on a predefined threshold $\tau$ (Equation 34):

$W_s=W\odot {\mathbf{1}}_{|W|>\tau },(34)$

where $\odot$ represents the Hadamard product, and ${\mathbf{1}}_{|W|>\tau }$ is an indicator function that retains only the weights exceeding the threshold $\tau$. To mitigate performance degradation caused by compression, we fine-tune the compressed model by minimizing the reconstruction loss (Equation 35):

${\mathcal{L}}_{\text{recon}}=\Vert W-\widehat{W}{\Vert }_F^2,(35)$

where $\Vert \cdot {\Vert }_F$ denotes the Frobenius norm, and $\widehat{W}$ represents the reconstructed weight matrix after applying compression techniques. By integrating low-rank factorization, quantization, sparsification, and fine-tuning, we ensure that the resulting model remains robust, interpretable, and computationally efficient. These enhancements ensure its suitability for real-world clinical applications, where maintaining both efficiency and accuracy is essential (Equations 36–42).

4 Experimental setup

4.1 Dataset

The LIDC-IDRI Dataset Suji et al. (2024) is a widely used medical imaging dataset for lung cancer detection and nodule analysis. It contains thoracic CT scans from multiple sources, each annotated by four experienced radiologists with detailed nodule segmentation and malignancy ratings. The dataset supports research in computer-aided diagnosis, uncertainty estimation, and deep learning-based lung disease detection. Its inclusion of multiple expert opinions enables robust training and evaluation of AI models, making it a benchmark for medical image analysis and automated lung cancer screening in clinical applications. The ChestX-ray14 Dataset Allaouzi and Ahmed (2019) is one of the largest publicly available chest X-ray collections, containing over 100,000 frontal-view images from patients with diverse lung diseases. It provides labeled data for 14 common thoracic conditions, including pneumonia, pleural effusion, and lung masses. The dataset is widely used for deep learning research in medical imaging, enabling the development of AI-driven diagnostic models. Its large-scale and real-world nature help improve the generalizability of AI systems for automated radiological assessment, making it a key resource for advancing AI in chest disease diagnosis. The ACDC Dataset Li K. et al. (2023) (Automated Cardiac Diagnosis Challenge) serves as a standard dataset for cardiac MRI segmentation and disease classification, providing a reliable benchmark for evaluating model performance. It consists of cine-MRI scans from patients with various heart conditions, including normal cases, myocardial infarction, and dilated cardiomyopathy. The dataset provides pixel-wise annotations of cardiac structures, facilitating the development of automated segmentation models. Its high-quality labels and diverse patient population make it an essential resource for evaluating AI algorithms in cardiovascular imaging. Researchers use it to improve deep learning models for heart disease assessment, aiding in the advancement of non-invasive cardiac diagnostics. The ACI-BENCH Dataset Leong et al. (2024) is a recent benchmark for artificial intelligence in colonoscopy, designed to enhance AI-driven polyp detection and classification. It includes high-resolution endoscopic images and videos annotated by expert gastroenterologists, covering a wide range of polyp appearances. The dataset enables researchers to train and validate deep learning models for real-time polyp identification, improving early colorectal cancer detection. Its diverse imaging conditions and expert-labeled ground truth make it a valuable resource for developing AI systems that assist endoscopists in clinical practice, ultimately aiming to reduce missed diagnoses and enhance patient outcomes.

4.2 Experimental details

We perform our experiments on an NVIDIA A100 GPU cluster powered by Intel Xeon Platinum processors. The entire framework is implemented in PyTorch, leveraging CUDA and cuDNN for efficient computation. We utilize the Adam optimizer with an initial learning rate of 0.0001 and a cosine learning rate scheduler to facilitate smooth convergence. The batch size is set to 16 for training and eight for validation. Each model undergoes training for 100 epochs, incorporating early stopping criteria driven by validation loss, with a patience setting of 10 epochs to minimize the risk of overfitting. For data preprocessing, input images are resized to $512\times 512$ for segmentation tasks and $224\times 224$ for classification tasks. Standard normalization is applied using the mean and standard deviation values of ImageNet. Data augmentation techniques, including random cropping, horizontal flipping, and color jittering, are applied to enhance model generalization. For instance segmentation tasks, additional augmentation such as CutMix and MixUp is incorporated to improve object boundary detection. For the object detection experiments on the LIDC-IDRI and ACI-BENCH datasets, we employ Faster R-CNN and YOLOv5 as baseline models. Models are trained with an IoU threshold of 0.5 and evaluated using mean Average Precision (mAP) at different IoU thresholds. For semantic segmentation tasks on ChestX-ray14 and ACDC, we utilize DeepLabV3+ and SegFormer as baseline models, evaluating performance using mean Intersection over Union (mIoU) and pixel-wise accuracy. For state-of-the-art (SOTA) comparisons, we re-implement baseline models using their official repositories and fine-tune them on each dataset. Hyperparameter tuning is conducted using a grid search over learning rates, dropout rates, and weight decay factors. Cross-validation is performed to ensure that the results are statistically robust, and all experiments are repeated five times with different random seeds. To analyze computational efficiency, we report inference time per image, model parameter count, and FLOPs. We evaluate the trade-off between accuracy and computational cost for different architectures to determine the most efficient model for deployment. Ablation studies are conducted to examine the impact of individual components in our framework. We remove key modules, such as feature fusion layers, attention mechanisms, and multi-scale processing units, and analyze the resulting performance drop. These experiments provide insights into the contribution of different design choices to overall model accuracy and efficiency. To ensure reproducibility, we fix random seeds for all experiments and provide detailed documentation of our implementation. Pretrained models, training scripts, and evaluation code are released to facilitate further research in object detection and segmentation tasks (Algorithm 1).

Algorithm 1

Algorithm 1. Training Procedure for UAMM Model.

4.3 Comparison with SOTA methods

To assess the performance of our proposed model, we benchmark it against state-of-the-art (SOTA) architectures, including ResNet-50, ResNet-101, DenseNet-121, Vision Transformer (ViT), ConvNeXt, and SegFormer. The evaluation is performed on four widely used datasets: LIDC-IDRI, ChestX-ray14, ACDC, and ACI-BENCH.

Table 1 showcases a comparative analysis of our approach against SOTA models on the LIDC-IDRI and ChestX-ray14 datasets. Our model outperforms all baselines, achieving an accuracy of 90.84% on LIDC-IDRI and 90.32% on ChestX-ray14, surpassing the best-performing baseline, ConvNeXt, by 2.91% and 1.71%, respectively. The improvement in recall and F1-score highlights our model’s superior ability to detect and classify objects with high precision in complex real-world images. The highest AUC values confirm that our model is more reliable in distinguishing foreground and background regions, which is crucial for object detection and segmentation. Similarly, Table 2 shows the results on ACDC and ACI-BENCH datasets. Our method achieves an accuracy of 90.73% on ACDC and 90.24% on ACI-BENCH, outperforming ConvNeXt by 2.28% and 2.45%, respectively. The improvements in recall and F1-score indicate that our model effectively captures fine-grained visual features, leading to better segmentation and classification results. The increase in AUC further demonstrates that our model provides robust predictions across different object categories and scene layouts.

Table 1

Table 1. Comparison of our approach with state-of-the-art methods on the LIDC-IDRI and ChestX-ray14 datasets.

Table 2

Table 2. Evaluation of our approach against state-of-the-art methods on the ACDC and ACI-BENCH datasets.

In Figures 5, 6, several key factors contribute to the superior performance of our model. Our approach integrates multi-scale feature extraction, which enhances object boundary detection and segmentation accuracy. Our model leverages an efficient feature fusion mechanism that improves the learning of contextual information, making it particularly effective for scene understanding tasks. Our optimization techniques, including domain adaptation and self-supervised learning, enhance the model’s generalization capability across diverse datasets. Our architecture is designed to maintain a balance between accuracy and computational efficiency, ensuring its feasibility for real-world applications. The experimental results confirm that our model establishes a new benchmark in object detection and semantic segmentation, outperforming existing SOTA models across multiple datasets. The improvements in accuracy, recall, F1-score, and AUC demonstrate the effectiveness of our approach in advancing visual recognition tasks.

Figure 5

Figure 5. Comparative analysis of state-of-the-art techniques on the LIDC-IDRI and ChestX-ray14 datasets.

Figure 6

Figure 6. Comparative performance analysis of state-of-the-art methods on the ACDC and ACI-BENCH datasets.

4.4 Ablation study

To evaluate the contribution of different components in our model, we conduct an ablation study by systematically removing key modules and analyzing their impact on performance across the LIDC-IDRI, ChestX-ray14, ACDC, and ACI-BENCH datasets.

Tables 3, 4 clearly demonstrate that each component plays a crucial role in enhancing the model’s overall performance. The removal of Uncertainty-Aware Prediction results in a noticeable decline in accuracy, with a drop of 2.12% on the LIDC-IDRI dataset and 2.01% on the ChestX-ray14 dataset. The degradation in recall and F1-score suggests that Uncertainty-Aware Prediction plays a crucial role in feature extraction and object boundary refinement. This module likely contributes to improving localization accuracy in detection and segmentation tasks. The removal of Adaptive Data Augmentation leads to the most significant drop in performance, with accuracy decreasing from 90.84% to 86.94% on LIDC-IDRI and from 90.32% to 86.23% on ChestX-ray14. The recall drop is particularly concerning, indicating that Adaptive Data Augmentation enhances contextual learning and improves the model’s sensitivity to detecting challenging objects in cluttered scenes. This suggests that Adaptive Data Augmentation, which may involve multi-scale feature aggregation or an attention mechanism, is critical for handling complex visual structures. The effect of removing Efficient Model Compression is relatively smaller but still results in performance degradation. The accuracy decreases by 1.39% on LIDC-IDRI and 1.30% on ChestX-ray14, with slight reductions in recall and F1-score. This indicates that Efficient Model Compression likely contributes to optimization strategies such as knowledge distillation or domain adaptation, enhancing the model’s generalization across different datasets.

Table 3

Table 3. Analysis of ablation study results for our method across the LIDC-IDRI and ChestX-ray14 datasets.

Table 4

Table 4. Evaluation of ablation study results for our method across the ACDC and ACI-BENCH datasets.

In Figures 7, 8, a similar pattern is observed for ACDC and ACI-BENCH datasets. The removal of Uncertainty-Aware Prediction results in a 2.50% accuracy drop on ACDC and a 2.32% drop on ACI-BENCH. This demonstrates that Uncertainty-Aware Prediction is crucial for capturing fine-grained scene details, which are particularly important in segmentation tasks. Eliminating Adaptive Data Augmentation leads to a notable drop in both accuracy and recall, further confirming its importance for capturing contextual relationships and improving object detection reliability. The removal of Efficient Model Compression leads to a smaller but consistent performance drop, reinforcing its role in regularization and optimization. The ablation study confirms that all three components are integral to the success of our model. The complete model consistently outperforms all ablated versions, demonstrating that the combination of Uncertainty-Aware Prediction, Adaptive Data Augmentation and Efficient Model Compression is essential for achieving state-of-the-art performance in object detection and semantic segmentation tasks.

Our study evaluates the effectiveness of the proposed CNN-Transformer framework for two key material science tasks: microstructure segmentation and defect detection. The experiments were conducted on datasets containing images of metallic alloys, ceramic composites, polycrystalline silicon, and polymer-based materials, obtained from scanning electron microscopy (SEM) and X-ray computed tomography (XCT). To ensure robustness, the dataset underwent preprocessing and augmentation techniques, and the model’s performance was compared against traditional methods such as threshold-based segmentation and SVM classifiers, as well as deep learning baselines including U-Net, DeepLabV3+, and Vision Transformers. Performance was assessed using mean Intersection over Union, Dice coefficient, and boundary F1 score for segmentation tasks, while precision, recall, and F1 score were used to evaluate defect detection. In addition, computational efficiency was measured by inference time per image to assess the feasibility of real-world deployment. The experimental results are shown in Table 5, our method outperforms conventional approaches across all metrics. In microstructure segmentation, our model achieved a 4.5 percent improvement in mean Intersection over Union and a 5.9 percent increase in boundary F1 score compared to DeepLabV3+, highlighting its ability to accurately delineate fine-grained structural details. In defect detection, it improved F1 scores by 5.1 percent and achieved higher recall, ensuring more reliable identification of structural anomalies. Additionally, the computational efficiency of our model is superior to transformer-based alternatives, reducing inference time by 44.9 percent compared to Vision Transformers while maintaining high segmentation and detection accuracy. These results confirm the advantages of integrating convolutional and transformer-based architectures for material science applications, enabling more precise microstructural analysis with greater computational efficiency.

Figure 7

Figure 7. Investigation of ablation study results for our approach on the LIDC-IDRI and ChestX-ray14 datasets. Uncertainty-aware Prediction (UAP), adaptive data Augmentation (ADA), efficient model Compression (EMC).

Figure 8

Figure 8. Comprehensive ablation study of our approach across the ACDC and ACI-BENCH datasets. Uncertainty-aware Prediction (UAP), adaptive data Augmentation (ADA), efficient model Compression (EMC).

Table 5

Table 5. Comparison of different methods on microstructure segmentation and defect detection tasks.

5 Conclusion and future work

Deep learning has significantly advanced medical image analysis, and its application in computational material science is gaining increasing attention. In this study, we propose a novel deep learning-driven framework that integrates convolutional neural networks (CNNs) with transformer-based architectures to enhance feature representation for material image analysis. Unlike traditional handcrafted feature extraction and threshold-based segmentation techniques, our method leverages domain-adaptive transfer learning and multi-modal fusion strategies to improve model generalization across diverse material datasets. Experimental evaluations demonstrate that our approach outperforms conventional methods. Specifically, our model achieves a segmentation accuracy improvement of 4.5% compared to state-of-the-art traditional approaches, with an average Intersection over Union (IoU) increase of 3.8%. In defect detection tasks, our framework reduces false positive rates by 22%, enhancing robustness in complex microstructural environments. Furthermore, through efficient model optimization, we reduce computational costs by 35%, making the framework more practical for real-time industrial applications. These improvements highlight the practical significance of applying deep learning techniques from medical imaging to computational material science, enabling more efficient and automated material characterization.

Despite these advancements, challenges remain. While transfer learning has proven effective in mitigating the reliance on large labeled datasets, domain adaptation across different material types and imaging conditions requires further investigation. Future work should explore self-supervised learning techniques to reduce dependency on manually annotated data. Additionally, the computational complexity of deep learning models, particularly transformer-based architectures, may limit scalability for large-scale industrial applications. To address this, future research could focus on model pruning, quantization, and hardware acceleration strategies to enhance real-time performance.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

LL: Data curation, Conceptualization, Formal analysis, Investigation, Funding acquisition, Software, Writing – original draft, Writing – review and editing. ML: Writing – original draft, Writing – review and editing.

Funding

The author(s) declare that financial support was received for the research and/or publication of this article. Details of all funding sources should be provided, including grant numbers if applicable. Please ensure to add all necessary funding information, as after publication this is no longer possible. This work was sponsored in part by the 2024 Western Medicine Self-funded Scientific Research Project (Z-L20240819) of the Health Commission of Guangxi Zhuang Autonomous Region, China.

Conflict of interest

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The author(s) declare that no Generative AI was used in the creation of this manuscript.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Abdou, M. A. (2022). Literature review: efficient deep neural networks techniques for medical image analysis. Neural computing and applications. doi:10.1007/s00521-022-06960-9

CrossRef Full Text | Google Scholar

Allaouzi, I., and Ahmed, M. B. (2019). A novel approach for multi-label chest x-ray classification of common thorax diseases. IEEE Access 7, 64279–64288. doi:10.1109/access.2019.2916849

CrossRef Full Text | Google Scholar

Azad, R., Kazerouni, A., Heidari, M., Aghdam, E. K., Molaei, A., Jia, Y., et al. (2023). Advances in medical image analysis with vision transformers: a comprehensive review. Med. Image Anal 91. 103000. doi:10.1016/j.media.2023.103000

PubMed Abstract | CrossRef Full Text | Google Scholar

Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., et al. (2022). Swin-unet: unet-like pure transformer for medical image segmentation. ECCV Work. doi:10.1007/978-3-031-25066-8_9

CrossRef Full Text | Google Scholar

Chen, Z., Agarwal, D., Aggarwal, K., Safta, W., Balan, M. M., Sethuraman, V., et al. (2022). “Masked image modeling advances 3d medical image analysis,” in IEEE workshop/winter conference on applications of computer vision.

Google Scholar

Chhabra, M., and Kumar, R. (2022). “A smart healthcare system based on classifier densenet 121 model to detect multiple diseases,” in Mobile radio communications and 5G networks: proceedings of second MRCN 2021 (Springer), 297–312.

Google Scholar

Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., et al. (2023). “Scaling vision transformers to 22 billion parameters,” in International conference on machine learning (Honolulu, Hawaii, USA: PMLR), 7480–7512. doi:10.1088/0143-0807/27/4/007

CrossRef Full Text | Google Scholar

Dhar, T., Dey, N., Borra, S., and Sherratt, R. (2023). Challenges of deep learning in medical image analysis—improving explainability and trust. IEEE Trans. Technol. Soc. 4, 68–75. doi:10.1109/tts.2023.3234203

CrossRef Full Text | Google Scholar

Drukker, K., Chen, W., Gichoya, J., Gruszauskas, N. P., Kalpathy-Cramer, J., Koyejo, S., et al. (2023). Toward fairness in artificial intelligence for medical image analysis: identification and mitigation of potential biases in the roadmap from data collection to model deployment. J. Med. Imaging 10, 061104. doi:10.1117/1.jmi.10.6.061104

PubMed Abstract | CrossRef Full Text | Google Scholar

Elyan, E., Vuttipittayamongkol, P., Johnston, P., Martin, K., McPherson, K., Moreno-García, C. F., et al. (2022). Computer vision and machine learning for medical image analysis: recent advances, challenges, and way forward. Artif. Intell. Surg. doi:10.20517/ais.2021.15

CrossRef Full Text | Google Scholar

Feng, J., Tan, H., Li, W., and Xie, M. (2022). “Conv2next: reconsidering conv next network design for image recognition,” in 2022 international conference on computers and artificial intelligence technologies (CAIT) (IEEE), 53–60.

CrossRef Full Text | Google Scholar

Fuhr, A. S., and Sumpter, B. G. (2022). Deep generative models for materials discovery and machine learning-accelerated innovation. Front. Mater. 9, 865270. doi:10.3389/fmats.2022.865270

CrossRef Full Text | Google Scholar

Furat, O., Wang, M., Neumann, M., Petrich, L., Weber, M., Krill III, C. E., et al. (2019). Machine learning techniques for the segmentation of tomographic image data of functional materials. Front. Mater. 6, 145. doi:10.3389/fmats.2019.00145

CrossRef Full Text | Google Scholar

Guan, H., and Liu, M. (2021). Domain adaptation for medical image analysis: a survey. IEEE Trans. Biomed. Eng. 69, 1173–1185. doi:10.1109/tbme.2021.3117407

PubMed Abstract | CrossRef Full Text | Google Scholar

He, K., Gan, C., Li, Z., Rekik, I., Yin, Z., Ji, W., et al. (2022). Transformers in medical image analysis: a review. Intell. Med. 3, 59–78. doi:10.1016/j.imed.2022.07.002

CrossRef Full Text | Google Scholar

Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T., and Zou, J. (2023). A visual–language foundation model for pathology image analysis using medical twitter. Nat. Netw. Boston 29, 2307–2316. doi:10.1038/s41591-023-02504-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Konovalenko, I., Maruschak, P., and Prentkovskis, O. (2018a). Automated method for fractographic analysis of shape and size of dimples on fracture surface of high-strength titanium alloys. Metals 8, 161. doi:10.3390/met8030161

CrossRef Full Text | Google Scholar

Konovalenko, I., Maruschak, P., Prentkovskis, O., and Junevičius, R. (2018b). Investigation of the rupture surface of the titanium alloy using convolutional neural networks. Materials 11, 2467. doi:10.3390/ma11122467

PubMed Abstract | CrossRef Full Text | Google Scholar

Kshatri, S. S., and Singh, D. (2023). Convolutional neural network in medical image analysis: a review. Archives Comput. Methods Eng. 30, 2793–2810. doi:10.1007/s11831-023-09898-w

CrossRef Full Text | Google Scholar

Lambert, B., Forbes, F., Tucholka, A., Doyle, S., Dehaene, H., and Dojat, M. (2022). Trustworthy clinical ai solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis. Artif. Intell. Med. doi:10.48550/arXiv.2210.03736

CrossRef Full Text | Google Scholar

Leong, H. Y., Gao, Y., and Ji, S. (2024). “A gen ai framework for medical note generation,” in 2024 6th international conference on artificial intelligence and computer applications (ICAICA) (IEEE), 423–429.

CrossRef Full Text | Google Scholar

Li, K., Zhang, G., Li, K., Li, J., Wang, J., and Yang, Y. (2023a). Dual cnn cross-teaching semi-supervised segmentation network with multi-kernels and global contrastive loss in acdc. Med. and Biol. Eng. and Comput. 61, 3409–3417. doi:10.1007/s11517-023-02920-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Li, M., Jiang, Y., Zhang, Y., and Zhu, H. (2023b). Medical image analysis using deep learning algorithms. Front. Public Health 11, 1273253. doi:10.3389/fpubh.2023.1273253

PubMed Abstract | CrossRef Full Text | Google Scholar

Li, X., Li, M., Yan, P., Li, G., Jiang, Y., Luo, H., et al. (2023c). Deep learning attention mechanism in medical image analysis: basics and beyonds. Int. J. Netw. Dyn. Intell., 93–116. doi:10.53941/ijndi0201006

CrossRef Full Text | Google Scholar

Lin, C.-L., and Wu, K.-C. (2023). Development of revised resnet-50 for diabetic retinopathy detection. BMC Bioinforma. 24, 157. doi:10.1186/s12859-023-05293-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Liu, T., Siegel, E., and Shen, D. (2022). Deep learning and medical image analysis for covid-19 diagnosis and prediction. Annu. Rev. Biomed. Eng. 24, 179–201. doi:10.1146/annurev-bioeng-110220-012203

PubMed Abstract | CrossRef Full Text | Google Scholar

Liu, W., Zhao, F., Shankar, A., Maple, C., Peter, J. D., Kim, B.-G., et al. (2023). Explainable ai for medical image analysis in medical cyber-physical systems: enhancing transparency and trustworthiness of iomt. IEEE J. Biomed. health Inf., 1–12. doi:10.1109/jbhi.2023.3336721

PubMed Abstract | CrossRef Full Text | Google Scholar

Ma, D., Dang, B., Li, S., Zang, H., and Dong, X. (2023). Implementation of computer vision technology based on artificial intelligence for medical image analysis. Int. J. Comput. Sci. and Inf. Technol. (IJCSIT) 1, 69–76. doi:10.62051/ijcsit.v1n1.10

CrossRef Full Text | Google Scholar

Mahboob, Z., Khan, M. A., Lodhi, E., Nawaz, T., and Khan, U. S. (2024). Using segformer for effective semantic cell segmentation for fault detection in photovoltaic arrays. IEEE J. Photovoltaics 15, 320–331. doi:10.1109/jphotov.2024.3450009

CrossRef Full Text | Google Scholar

Mazurowski, M., Dong, H., Gu, H., Yang, J., Konz, N., and Zhang, Y. (2023). Segment anything model for medical image analysis: an experimental study. Medical Image Anal 89. 102918. doi:10.1016/j.media.2023.102918

CrossRef Full Text | Google Scholar

Nazir, S., and Kaleem, M. (2023). Federated learning for medical image analysis with deep neural networks. Diagnostics 13, 1532. doi:10.3390/diagnostics13091532

PubMed Abstract | CrossRef Full Text | Google Scholar

Nirthika, R., Manivannan, S., Ramanan, A., and Wang, R. (2022). Pooling in convolutional neural networks for medical image analysis: a survey and an empirical study. Neural Comput. and Appl. (Print) 34, 5321–5347. doi:10.1007/s00521-022-06953-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Panigrahi, U., Sahoo, P. K., Panda, M. K., and Panda, G. (2024). A resnet-101 deep learning framework induced transfer learning strategy for moving object detection. Image Vis. Comput. 146, 105021. doi:10.1016/j.imavis.2024.105021

CrossRef Full Text | Google Scholar

Reimann, D., Nidadavolu, K., ul Hassan, H., Vajragupta, N., Glasmachers, T., Junker, P., et al. (2019). Modeling macroscopic material behavior with machine learning algorithms trained by micromechanical simulations. Front. Mater. 6, 181. doi:10.3389/fmats.2019.00181

CrossRef Full Text | Google Scholar

Rezaei, S., Asl, R. N., Faroughi, S., Asgharzadeh, M., Harandi, A., Koopas, R. N., et al. (2025). A finite operator learning technique for mapping the elastic properties of microstructures to their mechanical deformations. Int. J. Numer. Methods Eng. 126, e7637. doi:10.1002/nme.7637

CrossRef Full Text | Google Scholar

Rezaei, S., Asl, R. N., Taghikhani, K., Moeineddin, A., Kaliske, M., and Apel, M. (2024). Finite operator learning: bridging neural operators and numerical methods for efficient parametric solution and optimization of pdes.

Google Scholar

Sistaninejhad, B., Rasi, H., and Nayeri, P. (2023). A review paper about deep learning for medical image analysis. Comput. Math. Methods Med. 2023, 7091301. doi:10.1155/2023/7091301

PubMed Abstract | CrossRef Full Text | Google Scholar

Sohan, M. F., and Basalamah, A. (2023). A systematic review on federated learning in medical image analysis. IEEE Access 11, 28628–28644. doi:10.1109/access.2023.3260027

CrossRef Full Text | Google Scholar

Suji, R. J., Godfrey, W. W., and Dhar, J. (2024). Exploring pretrained encoders for lung nodule segmentation task using lidc-idri dataset. Multimedia Tools Appl. 83, 9685–9708. doi:10.1007/s11042-023-15871-3

CrossRef Full Text | Google Scholar

Tang, Y., Yang, D., Li, W., Roth, H., Landman, B., Xu, D., et al. (2021). Self-supervised pre-training of swin transformers for 3d medical image analysis. Comput. Vis. Pattern Recognit.

Google Scholar

Yamazaki, Y., Harandi, A., Muramatsu, M., Viardin, A., Apel, M., Brepols, T., et al. (2024). A finite element-based physics-informed operator learning framework for spatiotemporal partial differential equations on arbitrary domains. Eng. Comput. 41, 1–29. doi:10.1007/s00366-024-02033-8

CrossRef Full Text | Google Scholar

Yang, J., Shi, R., and Ni, B. (2020). Medmnist classification decathlon: a lightweight automl benchmark for medical image analysis. IEEE Int. Symposium Biomed. Imaging. doi:10.1109/ISBI48211.2021.9434062

CrossRef Full Text | Google Scholar

Zhang, C., Zheng, H., and Gu, Y. (2023). Dive into the details of self-supervised learning for medical image analysis. Med. Image Anal. 89, 102879. doi:10.1016/j.media.2023.102879

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhang, S., and Metaxas, D. N. (2023). On the challenges and perspectives of foundation models for medical image analysis. Medical Image Anal. doi:10.1016/j.media.2023.102996

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhou, H.-Y., Lu, C.-K., Chen, C., Yang, S., and Yu, Y. (2023). A unified visual information preservation framework for self-supervised pre-training in medical image analysis. IEEE Trans. Pattern Analysis Mach. Intell. 45, 8020–8035. doi:10.1109/tpami.2023.3234002

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhou, T., Ye, X., Lu, H., Zheng, X., Qiu, S., and Liu, Y. (2022). Dense convolutional network and its application in medical image analysis. BioMed Res. Int. 2022, 2384830. doi:10.1155/2022/2384830

PubMed Abstract | CrossRef Full Text | Google Scholar