- 1School of Information Engineering, Hubei University of Economics, Wuhan, China
- 2CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, and the SIAT Branch, Shenzhen Institute of Artificial Intelligence and Robotics for Society, Shenzhen, China
- 3School of Mathematics and Computer Science, Wuhan Polytechnic University, Wuhan, China
- 4Guangdong-Hong Kong-Macau Joint Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen, China
- 5State Key Laboratory of Petroleum Resource and Prospecting, and Unconventional Petroleum Research Institute, China University of Petroleum, Beijing, China
- 6State Key Laboratory of Geomechanics and Geotechnical Engineering, Institute of Rock and Soil Mechanics, Chinese Academy of Sciences, Wuhan, China
Introduction: The brain functional network can describe the spontaneous activity of nerve cells and reveal the subtle abnormal changes associated with brain disease. It has been widely used for analyzing early Alzheimer's disease (AD) and exploring pathological mechanisms. However, the current methods of constructing functional connectivity networks from functional magnetic resonance imaging (fMRI) heavily depend on the software toolboxes, which may lead to errors in connection strength estimation and bad performance in disease analysis because of many subjective settings.
Methods: To solve this problem, in this paper, a novel Adversarial Temporal-Spatial Aligned Transformer (ATAT) model is proposed to automatically map 4D fMRI into functional connectivity network for early AD analysis. By incorporating the volume and location of anatomical brain regions, the region-guided feature learning network can roughly focus on local features for each brain region. Also, the spatial-temporal aligned transformer network is developed to adaptively adjust boundary features of adjacent regions and capture global functional connectivity patterns of distant regions. Furthermore, a multi-channel temporal discriminator is devised to distinguish the joint distributions of the multi-region time series from the generator and the real sample.
Results: Experimental results on the Alzheimer's Disease Neuroimaging Initiative (ADNI) proved the effectiveness and superior performance of the proposed model in early AD prediction and progression analysis.
Discussion: To verify the reliability of the proposed model, the detected important ROIs are compared with clinical studies and show partial consistency. Furthermore, the most significant altered connectivity reflects the main characteristics associated with AD.
Conclusion: Generally, the proposed ATAT provides a new perspective in constructing functional connectivity networks and is able to evaluate the disease-related changing characteristics at different stages for neuroscience exploration and clinical disease analysis.
1. Introduction
Early Alzheimer's disease (AD) includes the following three successive stages: significant memory concern (SMC), early mild cognitive impairment (EMCI), and late mild cognitive impairment (LMCI). AD is a common long-term neurological disorder in the elderly, which is generally connected with the gradual decline in understanding, judgment, memory, and executive ability until complete loss. AD is known as the leading cause of death among old people worldwide (Zhang et al., 2022), and its great harmfulness brings heavy psychological pressure and economic burden to the families of patients. According to literature (Derby, 2020), the number of people suffering from AD and other dementias in the world currently exceeds 50 million, and the aging population further aggravates the rise of the patient population. However, there is no consensus on the pathological mechanism (Yuzwa et al., 2008; Diplas et al., 2018), and many pharmaceutical companies had tried and failed to develop effective drugs to cure AD. Therefore, early detection and timely intervention for AD are the only possible way in slowing down or preventing disease deterioration (Jack et al., 2013). The development of neuroimaging has made the use of non-invasive AD study become the mainstream of current research because of no side effects on patients (Wang et al., 2018c; Grassi et al., 2019; Yu et al., 2021; Alvi et al., 2022; Lei et al., 2022; You et al., 2022). It is very promising for the scientific community to develop effective methods to detect brain disease and assist clinical treatment from medical imaging data (Wang et al., 2022).
The brain functional network (BFN) derived from functional Magnetic Resonance Imaging (fMRI) describes the functional interactions among spatially distributed brain regions. Brain science indicates that abnormal functional connectivity always appears at the early stage of AD (Berron et al., 2020). The BFN can give a universal understanding of neurological symptoms and unravel the pathogenesis of cognitive diseases. As mentioned in Yu et al. (2020) and Zuo et al. (2021a), the whole brain is divided into several Region-of-Interests (ROIs) according to the anatomical template. The BFN is modeled as a graph, where each node represents the ROI and each edge represents the functional connection strength between paired ROIs. The conventional method is to use a software toolbox to construct functional connectivity (FC) and then extract effective features for disease diagnosis. For example, Kabbara et al. (2018) investigated the abnormal hub patterns associated with patients' cognitive performance by applying graph-theory analysis on the constructed functional connectivity. This work preserved the topological structure and gained better evaluation performance than the feature extraction algorithm (Wang et al., 2017; Zuo et al., 2021b; Yu et al., 2022) in Euclidean space. Considering the complexity of brain neural activities and noisy data preprocessed from the raw fMRI, it is significant for clinicians to investigate more advanced methods for modeling effective BFNs in early AD analysis.
Brain functional network construction by using time series can be divided into two classes: static-based method, and dynamic-based method. The former utilized the whole brain time series of fMRI to bridge links between ROIs for AD analysis. The direct way of constructing a brain functional network is to compute the person's correlation (PC) between any paired brain regions (Wang et al., 2007). To reduce the possible impact of adjacent ROIs, Fransson and Marrelec (2008) employed partial correlations to handle this problem and achieved good performance in characterizing the changes of the default mode network associated with the disease. But the calculation of an inverse matrix usually comes up with multiple solutions, so researchers adopted certain constraints on the partial correlation estimation for a stable solution. For example, the matrix-regularized network was encoded as modularity prior to optimizing sparse brain network and they (Qiao et al., 2016) discovered potential biomarkers for personalized diagnosis. The latter method benefits the temporal changes of brain functional connectivity for capturing subtle transient neural abnormalities and has recently been a hot spot in neurological disease analysis. The direct approach is to generate a sequence of functional networks and designed a fused learning algorithm to jointly estimate the temporal network for early MCI detection (Wee et al., 2016). Furthermore, the work in Gong et al. (2022) treated the functional time series and functional connectivity as the node features and edges respectively, and developed a graph convolutional network (GCN) based model to generate multiple brain networks for characterizing brain temporal community by setting six-time sliding steps. To address the noisy problem of limited volumes in a sliding window, Zhou et al. (2018) proposed a matrix-regularized learning framework to learn sparse and modular high-order connectivity features for MCI classification. Although many studies have been conducted in BFN construction, they mainly rely on some specific preprocessing in the software toolboxes to obtain temporal features of each ROI. The drawbacks lie in two fields: one is that the multiple parameter settings may lead to different errors from person to person, and another is that a series of processes can consume much time and fall far away from the goal of clinical application.
Recently, data-driven models are capable of mining effective common characteristics from noisy data. It has been widely applied in various fields of medical image analysis, such as disease severe assessment (Wang et al., 2020c), lesion area segmentation (Hong et al., 2022b), health assessment (Wang et al., 2018b), disease detection (Wang et al., 2018a; Yang et al., 2022), image reconstruction (Hu et al., 2020b). To improve disease analysis performance, many advanced machine learning algorithms are designed to extract discriminative and robust features (Zeng et al., 2017; Lei et al., 2018; Hong et al., 2019; Wang et al., 2020b). Compared with the classification performance of traditional Convolutional Neural Networks (CNN) (Wang S.-Q. et al., 2015), the 3D Convolutional Neural Network (C3D) is good at capturing the local spatial features in a three-dimensional volume and has been successfully applied on the cross-modal image synthesis (Hu et al., 2020a) and disease recognition (Wang et al., 2020a). Moreover, the transformer network (Jiang et al., 2021) can model the global relationship between distant sub-patch regions. The ROI-based features can be learned by C3D and transformer in sequence from 4D fMRI data. Besides, Generative adversarial networks (GANs) are regarded as a special case of variational inference (Mo and Wang, 2009; Wang, 2009) and demonstrates impressive performance in matching generated data distributions. The obvious evidence is the success in generating cross-modal medical images (Hu et al., 2019, 2021) and domain adaptation segmentation (Hong et al., 2022a). It can be used as a regularizer to constrain the representation learning for stable and generalizable disease analysis.
Inspired by the above observations, in this paper, a novel Adversarial Temporal-Spatial Aligned Transformer (ATAT) model is proposed to automatically learn brain functional networks from 4D fMRI for detecting early AD. The constructed brain functional networks are also analyzed to identify important ROIs and abnormal connections. The main contributions of this work are as follows: (1) The region-sequence aligned generator (RAG) is developed to first learn rough ROI-based features by incorporating the brain anatomical information, then finely adjust the boundary features of adjacent ROIs to generate ROI time series and connectivity features. It greatly enhances the ROI time series learning and fully explores the spatial-temporal characteristics and connectivity information among the whole brain. (2) The multi-channel temporal discriminator is designed to constrain the learned ROI time series with the empirical samples. It regularizes the generator optimization and makes the connectivity feature more robust. (3) Experimental classification results prove the effectiveness of our model, and the discovered important ROIs and abnormal connections may be potential biomarkers for early AD diagnosis or treatment.
The rest of this article is organized as follows. Section 2 describes the novel proposed ATAT model for brain functional network construction. The experimental settings and prediction results with competing methods are presented in Section 3. In the Section 4, the reliability and limitations of this work are discussed. Finally, the Section 5 summarizes the main remarks of this paper.
2. Materials and methods
The proposed model includes three main parts, such as (1) data preprocessing, (2) architecture of the proposed model, and (3) objective functions for optimization.
2.1. Data description and preprocessing
The experimental data comes from the public Alzheimer's Disease Neuroimaging Initiative (ADNI-3). A total of 330 subjects with functional Magnetic resonance (fMRI) were downloaded from the website1, including 86 Normal Control (NC), and three successive stages of early AD (i.e., 82 SMC, 86 EMCI, 76 LMCI). The fMRI data is acquired under the 3.0 Tesla machine. The detailed scanning parameters for fMRI are as follows: the imaging resolution ranges from 2.5 to 3.75 mm along X and Y dimensional direction, the imaging slice thickness is between 2.5 and 3.4 mm; the time of repetition (TR) ranges from 0.607 to 3.0 s, and the time of echo (TE) value is in the range of 30 to 32 ms. The recording time length is about 10 min. The mean age of NC, SMC, EMCI, and LMCI is 74.4, 76.1, 75.7, and 75.8, respectively. The gender is roughly the same in each category.
The fMRI data is preprocessed by the software toolbox GRETNA (Wang J. et al., 2015), which contains about six procedures for constructing ROI-based time series. Each 4D fMRI data is processed by balancing magnetization equilibrium, removing head-motion artifacts, normalizing spatial space, smoothing, and filtering (0.01Hz ≤ f ≤ 0.08Hz). Finally, the automated anatomical labeling (AAL) atlas (Tzourio-Mazoyer et al., 2002) warps the preprocessed image to 90 non-overlapping spatial ROIs, and the final functional features with the size 90 × 187 are obtained as the truth samples. Meanwhile, the empirical functional connectivity is estimated by calculating the Pearson correlation coefficients between paired ROI time series, and this procedure can generate a 90 × 90 correlation matrix for each subject.
2.2. Architecture
The architecture of the proposed ATAT is shown in Figure 1. It contains three parts: the region-sequence aligned generator (RAG), the multi-channel temporal discriminator (MTD), and the global-local connectivity classifier (GCC). The RAG includes a region-guided feature learning network(RFLNet), and a spatial-temporal aligned transformer (SAT), which transforms the 4D fMRI into ROI time series and brain functional network. Firstly, the raw fMRI data is first sent to the RFLNet for rough ROI-based feature extraction, and the SAT is utilized to finely adjust the feature for adjacent ROIs and align the global temporal correlation between any paired ROIs. Meanwhile, the obtained ROI time series is linearly transformed into brain functional networks through the connectivity learning (CL) network. After that, the generated ROI time series is constrained with the real sample distribution by the MTD. Finally, both ROI time series and brain functional networks are sent to the GCC for disease prediction. There are five objective functions in the model's optimization, including generator loss, discriminator loss, reconstruction loss, classifier loss, and regularized loss.
Figure 1. The framework of the proposed model. It consists of three parts: generator, discriminator, and classifier. The input is a four-dimensional fMRI, and the output is a brain functional network.
2.2.1. Region-sequence aligned generator
2.2.1.1. Region-guided feature learning network
As illustrated in Figure 2. This network learns a rough mapping from the raw 4D fMRI to ROI-based time series by introducing the position and volume of the brain anatomical regions. The size of input data X is 64 × 64 × 48 × 187. It first passes through four blocks with three successive layers: 3 × 3 × 3 convolutional layers with 1-stride, 2 × 2 × 2 average pooling layer with 2-stride, and a combination layer of batch normalization(BN)+ReLu activation. The channel number of the above four convolutional layers are 8, 16, 32, 64. Then one 1 × 1 × 1 convolutional layer with 1-stride is used to increase the channels for matching the N ROIs, followed by a sigmoid activation function.
Figure 2. The detailed structure of the RFLNet. The input is an example of three-dimensional fMRI volume with the size 64 × 64 × 48, and the ROI information of the anatomical atlas with the size N × 4. The FRFNet outputs the initial feature for each ROI.
Next, we normalize the central location (x, y, z) and volume (v) of N anatomical ROIs to constrain the brain region information in the range 0 − 1. Finally, the (x, y, z, v) of N ROIs are treated as ROI embeddings, which are concatenated with the flattened feature of sigmoid layer output, which is sent to a one-layer linear projection (LP) layer for generating rough ROI features. The rough ROI feature can be expressed as:
here, the X is the four dimensional volume data fMRI; x, y, z, v∈ℝN×1; , the RFLNet is a combination of several convolution and pooling operations.
2.2.1.2. Spatial-temporal aligned transformer
To learn more fined ROI temporal features, the spatial-temporal aligned transformer module is designed to recalibrate boundary ROI features and time sequence variations. It splits into two parts: the spatial multi-head central attention (SMCA) and the temporal aligned feed-forward (TAFF). Every ROI is regarded as a token. The rough ROI feature is first sent to three parallel LP layers to get query (Q), key (K), and value (V). Note that, the calculation of K and V needs to consider the ROI embeddings. The formulas can be defined as:
where, || means the concatenation operation. Then Q, K, V∈ℝN×q are separated into h heads. Each head of token (i.e., Qi, Ki, Vi) has the dimensional size q/h. Taking one head as an example, the central attention (CA) can be expressed:
here, i means the index of h heads. The output of the spatial multi-head central-attention module is the concatenation of all heads and then with an LP layer (including residual mapping and layer normalization). It can be defined as:
The SMCA has the size N×q.
Next, the TAFF module adjusts the temporal characteristics through the down mapping (DM) and up mapping (UM) layers and reduces the potential noise effect. The DM layer reduces the dimensional of SMCA from q to q/2, and the UM layer recovers the feature's dimension. Finally, the output of the TAFF module can be defined as:
where, Fg is the generated ROI time series with the size N×q.
To learn an effective brain functional network Ag, we first compute the Euclidean distance between any pair of ROI features and then apply a mapping matrix to it for similarity adjustment. Finally, a Gaussian kernel is introduced to learn non-linear projection for precise connectivity estimation. The formula can be defined as:
here, Ag(i, j) represents the functional connectivity between pairwise ROIs. means the ith ROI time series. W∈ℝq×q is time series transformable matrix. σ is the bandwidth of the Gaussian kernel, controlling the sparsity with the default value 2.
2.2.2. Multi-channel temporal discriminator
As shown in Figure 3, the multi-channel temporal discriminator (MTD) is used to constrain the generated functional time series (Fg) distribution consistent with the empirical functional time series (Fe). The Fe is computed from the software toolbox, which is treated as the true sample. The structure of MTD consists of N parallel networks, containing three linear projections with q/2, q, and q/2 neurons. Each MTD accepts i-th ROI time series and outputs one discriminate value. Averaging all the discriminate values is the final discriminate result.
Figure 3. The structure of the multi-channel temporal discriminator. It accepts the empirical time series or the generated time series, and outputs the average discriminant result for distribution constraints of all ROI time series.
2.2.3. Global-local connectivity classifier
The structure of the global-local connectivity classifier (GCC) is illustrated in Figure 4. It accepts both functional time series (i.e., Fe or Fg) and functional network (i.e., Ae or Ag), outputs the disease label. A total of 5 layers are designed in the GCC, including three graph convolutional layers, one graph pooling layer, and one three-layer perceptron. It is based on the graph convolutional network. The first three layers (i.e., Gconv1, Gconv2, Gconv3) are utilized to diffuse global features and reduce the ROI feature dimension. The graph pooling layer (Gpool) is utilized to average features along the ROI feature dimension and get one value for each ROI. And the MLP layer learns a linear mapping to recognize the disease.
2.3. Objective functions
In this section, the five loss functions defined below are utilized to optimize the model for disease prediction and analysis. The reconstruction loss Lrec can constrain the generator and retain the empirical features Fe, the generate loss Lg and discriminate loss Ld are combined to optimize the generator and discriminator, the classification loss Lcls and regularized loss Lreg are utilized to upgrade the parameters of CL network and GCC network. For the convenience of explanation, we make the following simplification: G means all the operations in the Region-sequence aligned generator, D means the multi-channel temporal discriminator, and C is the global-local connectivity classifier. The raw fMRI data X follows the distribution PfMRI, the PFe and PAe represent the empirical functional time series Fe and empirical BFN Ae distribution, respectively. Y is the truth label. These loss functions are defined as follows:
The hybrid cost of the proposed model is:
3. Experiments and results
3.1. Experimental setup
There are six binary classification tasks for the evaluation of the proposed model, including NC vs. SMC, NC vs. EMCI, NC vs. LMCI, SMC vs. EMCI, SMC vs. LMCI, and EMCI vs. LMCI. The evaluation metrics are Accuracy (ACC), Sensitive (SEN), Specificity (SPE), and F1-score. We repeated the 10 times experiment using the five-fold cross-validation on each binary classification and utilized the mean value metrics for the final prediction. To demonstrate our model's good ability in FBN construction, we introduce two classifiers [i.e., SVM (Suthaharan, 2016) and GCN (Kipf and Welling, 2016)] to compare the BFN constructed by ATAT and Empirical.
Our proposed model was implemented with the TensorFlow framework on Ubuntu18.04 and the GPU of NVIDIA GeForce RTX 3080 Ti. The parameters in the experiments are defined as follows: N = 90, q = 187, h = 11, m = 3, λ = 10−5. During the training, we first update the weights in the generator and the discriminator, then fix part of the generator and optimize the network of CL and GCC. The learning rate of the generator and the classifier were set to 0.0001, and for the discriminator, the learning rate is set to 0.0004. The Adam was adopted for training the proposed model with batch size 2.
3.2. Prediction results
This section demonstrates the good performance of BFN constructed by the proposed model. As shown in Figure 5, the upper row shows the four stages of empirical FBN derived from the GRETNA, while the lower row displays the corresponding FBNs by the proposed model. The main connectivity patterns have been preserved and dense connections become sparse by comparing the empirical and ours. Figure 6 gives the classification result comparison in terms of three scenarios tasks. For the GCN classifier, the BFNs constructed by ours achieve the best prediction results with a mean ACC of 87.50%, a mean SEN of 84.26%, a mean SPE of 90.58% and a mean F1 of 86.81% in NC vs. SMC task; the mean values of SMC vs. EMCI are 90.47, 91.86, 89.02, and 90.80%; in EMCI vs. LMCI task, the predicted results are 85.61, 84.86, 86.27, and 84.70%. The standard error also shows the superior stability of the proposed model.
Figure 5. Display of brain functional network examples at different disease stages. The BFNs in the upper row are generated by the GRETNA toolbox, and the BFNs in the lower row are generated by the proposed model.
Figure 6. Prediction results of three scenarios tasks using (A) the SVM classifier and (B) the GCN classifier.
To investigate the potential AD-related ROIs, we shield one brain region and calculate the classification ACC as the effect of this ROI on AD progression. After sorting the ACCs in ascending order, the top 10 values are the most important ROI in the classification evaluation. As is shown in Figure 7, the spatial distribution of 10 important AD-related ROIs is displayed in lateral, medial, and dorsal views using the BrainNet Viewer (Xia et al., 2013). Specifically, the top 10 related ROIs are IFGoperc.L, MTG.R, PCL.L, PUT.R, CUN.L, SMA.R, LING.L, DCG.R, PCUN.R, DCG.L in NC vs. SMC classification scenario; The ten ROIs, including PCL.R, CAL.L, CUN.R, HIP.R, CAL.R, TPOsup.L, SFGdor.L, ACG.R, CAU.R, PCL.L, are important for NC vs. EMCI; also, the top 10 ROIs of NC vs. LMCI are SOG.L, ORBsup.L, REC.L, PUT.L, PCG.L, ITG.L, PCUN.R, MTG.R, PUT.L, ORBsupmed.L; For SMC vs. EMCI and SMC vs. LMCI classification, the important ROIs are OLF.L, CUN.R, PCUN.L, CAL.R, CAU.R, LING.L, ACG.R, CAL.L, PCL.L, DCG.R, and PCUN.R, PUT.L, PUT.R, PCL.L, SMA.R, ORBsup.L, LING.L, ANG.L, HIP.R, ACG.R, respectively; For EMCI vs. LMCI, the important ROIs are PCUN.L, ORBsupmed.R, THA.L, ORBsupmed.L, ORBsup.R, CAU.R, CAL.L, PUT.L, REC.L, ORBsup.L.
Figure 7. Spatial visualization of top 10 brain regions in the six classification scenarios. (A) NC vs. SMC. (B) NC vs. EMCI. (C) NC vs. LMCI. (D) SMC vs. EMCI. (E) SMC vs. LMCI. (F) EMCI vs. LMCI.
3.3. Brain network analysis
Besides the prediction of different early AD stages, the other major purpose is to analyze the learned FBNs. After applying the ATAT model to each subject, we can obtain the mean FBN for each group of patients (i.e., NC, SMC, EMCI, and LMCI). To investigate the altered connectivity of FBN between different groups, we compute the difference of six paired scenarios as shown in Figure 8. In each subplot, reduced and increased connectivity can be observed between different paired groups. To analyze the significant connections, we set the 90% quantile value of the altered connectivity strength as the threshold. The pictures in the lower row of each subplot are the corresponding connectivity matrices by setting the threshold value. Figure 9 shows these significant connections in a circular graph. The number of reduced connections are 219, 263, 235, 251, 222, and 163 for NC vs. SMC, NC vs. EMCI, NC vs. LMCI, SMC vs. EMCI, SMC vs. LMCI, EMCI vs.LMCI, respectively; the corresponding number of increased connections are 183, 139, 166, 150, 179, 239. To show the main connectivity patterns in different classification scenarios, we select the top 2% largest altered connections (i.e., reduced, and increased). As shown in Figure 10, different connectivity patterns can be seen in different classification scenarios. Figure 11 depicts the top 5 reduced and top 5 increased connections in the axial and coronal view direction. The connectivity-related ROIs are listed in Tables 1, 2.
Figure 8. (A–F) The results of the altered functional connectivity estimated from the averaged BFNs between different groups (i.e., NC vs. SMC, NC vs. EMCI, NC vs. LMCI, SMC vs. EMCI, SMC vs. LMCI, EMCI vs. LMCI). In each subfigure, the upper row means the reduced and increased connections, the lower row shows the altered connections selected from the upper row with a threshold of 90% quantile value.
Figure 9. Circular graph of altered functional connectivity in MCI patients among 90 Anatomical Automatic Labeling (AAL) atlas regions. (A) From NC to SMC. (B) From NC to EMCI. (C) From NC to LMCI. (D) From SMC to EMCI. (E) From SMC to LMCI. (F) From EMCI to LMCI.
Figure 10. Top 2% altered functional connections in strength evaluation in the six classification scenarios. Each subfigure shares the same color bar, which means the absolute connection strength. (A) NC vs. SMC. (B) NC vs. EMCI. (C) NC vs. LMCI. (D) SMC vs. EMCI. (E) SMC vs. LMCI. (F) EMCI vs. LMCI.
Figure 11. The most significant 5 reduced connections and 5 increased connections mapped on the AAL 90 template using the BrainNet Viewer software package. Blue color means the ROIs, red color means reduced connections, and green color means increased connections. (A) From NC to SMC. (B) From NC to EMCI. (C) From NC to LMCI. (D) From SMC to EMCI. (E) From SMC to LMCI. (F) From EMCI to LMCI.
Table 1. The top 10 significant altered connections estimated from the generated FBNs in NC vs. SMC, NC vs. EMCI, NC vs. LMCI using AAL90 template (− means reduced connections, + means increased connections).
Table 2. The top 10 significant altered connections estimated from the generated FBNs in SMC vs. EMCI, SMC vs. LMCI, EMCI vs. LMCI using AAL90 template (− means reduced connections, + means increased connections).
4. Discussion
4.1. Effect of the generator
The main goal of the proposed model is to generate BFNs from 4D fMRI data. The modules in the generator play an important role in disease prediction and brain network analysis. To investigate the influence of the generator structure on the classification performance (i.e., NC vs. LMCI), we replace the RFLNet and the SAT modules with traditional C3D (Hong et al., 2020) and transformer (Jiang et al., 2021), respectively. In both cases, the anatomical ROI information is not included in the module. Figure 12 shows that either the C3D or Transformer network can degrade the prediction performance, and the traditional transformer network has a worse influence on classification than the C3D network. It may indicate the proposed RFLNet learns rough ROI-based features with a litter effect on the results, and the SAT network finely adjusts the adjacent ROI-based temporal features which may greatly influence the classification performance. Furthermore, the reconstructed error of the ROI time series is measured by the mean absolute error (MAE) metric. As shown in Figure 13, the divergence of the MAE for each disease (i.e., NC, SMC, EMCI, and LMCI) demonstrates the reliable results of the designed generator.
Figure 13. Mean absolute error between the empirical and generated time series over the training processes.
4.2. Comparison with related works
In the six classification scenarios, there are eight ROIs that overlap more than three times in the identified brain regions. These important ROIs are the orbital part of the superior frontal gyrus (ORBsup.L), the anterior cingulate and paracingulate gyri (ACG.R), the calcarine fissure and surrounding cortex (CAL.L), the lingual gyrus (LING.L), the precuneus (PCUN.R), the paracentral lobule (PCL.L), the caudate nucleus (CAU.R), the lenticular nucleus putamen (PUT.L). Most of these ROIs are consistent with the previous studies (Li et al., 2017; Yu et al., 2017; Ye et al., 2019), which demonstrates the strongly correlation with AD. In addition, the most significant altered connectivity (also called abnormal connections) related ROIs contain the identified eight brain regions. These listed ROIs in the table are mainly distributed in the frontal lobe, temporal lobe, and occipital lobe. The frontal lobe is located in the most anterior part of the cerebral hemispheres, accounting for the first 1/3 of the surface of the cerebral hemispheres. It is mainly related to higher mental activity, including physical activity control, language speaking, self-awareness, and emotional expression. The identified ROIs (i.e., SFGdor, ORBsup, IFGoperc, IFGtriang) associated with the superior frontal gyrus can be founded in Whitwell et al. (2007). The visual and language information is memorized by the temporal lobe, in which patients with AD showed abnormal levels of tau protein in the inferior temporal gyrus (Mormino et al., 2016). The occipital lobe participates in visual processing, for example, the lingual gyrus shows altered functional connectivity in AD patients (Skouras et al., 2019). In general, the derived important ROIs and abnormal connections by the proposed model can reflect the main characteristics associated with AD.
4.3. Limitations and future directions
Although the proposed model in the experiment has achieved good classification results and reliable connectivity analysis, there are two deficiencies in this work. One is that it only considers the binary classification tasks, which may not capture the continuously changing characteristics during the disease progression. We will conduct multi-class classification experiments to investigate the common changes at different stages of AD. Another limitation is that the dataset used in this study is relatively small. In the future study, we will increase the amount of data to validate the proposed model for brain disorder analysis by using other larger datasets [UK biobank (Sudlow et al., 2015), ABIDE (Heinsfeld et al., 2018)].
5. Conclusion
In this paper, we proposed a novel ATAT model to construct brain functional networks for early AD diagnosis and analysis. The three-player generative adversarial network is alternatively optimized and can learn effective functional connectivity features from 4D fMRI. By incorporating the brain anatomical information, the rough ROI features can be extracted by focusing on the local spatial information of individual brain region. Furthermore, the SAT module explores the temporal characteristics and connectivity information for finely adjusting the boundary features of adjacent ROIs. Meanwhile, the generated features from the region-sequence aligned generator are constrained by the adversarial loss and reconstruction loss. Compared to the empirical method, the brain functional networks constructed by the proposed model achieve higher classification performance. The identified important ROIs and abnormal connections may be the potential biomarkers for early AD diagnosis. Generally, our proposed model has the potential in constructing complex functional connectivity features and exploring abnormal functional connections for neurodegenerative diseases study.
Data availability statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author/s.
Author contributions
QZ designed the methodology, conducted the experiment, and wrote the manuscript. LW was the head of the funds of the paper and supervised and verified the reliability of the experimental results. LL provided suggestions for result visualization. JZ and TO reviewed and edited the manuscript. All authors read and approved the final manuscript.
Funding
This work was supported in part by National Key R&D Program of China under grant 2020YFC2005803, International Partnership Program of Chinese Academy of Sciences under grant GJHZ2021132, and NSFC 62003331.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Footnotes
References
Alvi, A. M., Siuly, S., Wang, H., Wang, K., and Whittaker, F. (2022). A deep learning based framework for diagnosis of mild cognitive impairment. Knowledge Based Syst. 248, 108815. doi: 10.1016/j.knosys.2022.108815
Berron, D., van Westen, D., Ossenkoppele, R., Strandberg, O., and Hansson, O. (2020). Medial temporal lobe connectivity and its associations with cognition in early Alzheimer's disease. Brain 143, 1233–1248. doi: 10.1093/brain/awaa068
Derby, C. A. (2020). Trends in the public health significance, definitions of disease, and implications for prevention of Alzheimer's disease. Curr. Epidemiol. Rep. 7, 68–76. doi: 10.1007/s40471-020-00231-8
Diplas, B. H., He, X., Brosnan-Cashman, J. A., Liu, H., Chen, L. H., Wang, Z., et al. (2018). The genomic landscape of tert promoter wildtype-IDH wildtype glioblastoma. Nat. Commun. 9, 1–11. doi: 10.1038/s41467-018-04448-6
Fransson, P., and Marrelec, G. (2008). The precuneus/posterior cingulate cortex plays a pivotal role in the default mode network: evidence from a partial correlation network analysis. Neuroimage 42, 1178–1184. doi: 10.1016/j.neuroimage.2008.05.059
Gong, C., Xue, B., Jing, C., He, C.-H., Wu, G.-C., Lei, B., et al. (2022). Time-sequential graph adversarial learning for brain modularity community detection. Math. Biosci. Eng. 19, 13276–13293. doi: 10.3934/mbe.2022621
Grassi, M., Rouleaux, N., Caldirola, D., Loewenstein, D., Schruers, K., Perna, G., et al. (2019). A novel ensemble-based machine learning algorithm to predict the conversion from mild cognitive impairment to Alzheimer's disease using socio-demographic characteristics, clinical information, and neuropsychological measures. Front. Neurol. 10, 756. doi: 10.3389/fneur.2019.00756
Heinsfeld, A. S., Franco, A. R., Craddock, R. C., Buchweitz, A., and Meneguzzi, F. (2018). Identification of autism spectrum disorder using deep learning and the abide dataset. Neuroimage Clin. 17, 16–23. doi: 10.1016/j.nicl.2017.08.017
Hong, J., Cheng, H., Wang, S.-H., and Liu, J. (2019). Improvement of cerebral microbleeds detection based on discriminative feature learning. Fund. Inform. 168, 231–248. doi: 10.3233/FI-2019-1830
Hong, J., Feng, Z., Wang, S.-H., Peet, A., Zhang, Y.-D., Sun, Y., et al. (2020). Brain age prediction of children using routine brain MR images via deep learning. Front. Neurol. 11, 584682. doi: 10.3389/fneur.2020.584682
Hong, J., Yu, S. C.-H., and Chen, W. (2022a). Unsupervised domain adaptation for cross-modality liver segmentation via joint adversarial learning and self-learning. Appl. Soft Comput. 121, 108729. doi: 10.1016/j.asoc.2022.108729
Hong, J., Zhang, Y.-D., and Chen, W. (2022b). Source-free unsupervised domain adaptation for cross-modality abdominal multi-organ segmentation. Knowledge Based Syst. 250, 109155. doi: 10.1016/j.knosys.2022.109155
Hu, S., Lei, B., Wang, S., Wang, Y., Feng, Z., and Shen, Y. (2021). Bidirectional mapping generative adversarial networks for brain MR to PET synthesis. IEEE Trans. Med. Imaging 41, 145–157. doi: 10.1109/TMI.2021.3107013
Hu, S., Shen, Y., Wang, S., and Lei, B. (2020a). “Brain MR to PET synthesis via bidirectional generative adversarial network,” in International Conference on Medical Image Computing and Computer-Assisted Intervention (Cham: Springer), 698–707. doi: 10.1007/978-3-030-59713-9_67
Hu, S., Yu, W., Chen, Z., and Wang, S. (2020b). “Medical image reconstruction using generative adversarial network for Alzheimer disease assessment with class-imbalance problem,” in 2020 IEEE 6th International Conference on Computer and Communications (ICCC) (IEEE), 1323–1327. doi: 10.1109/ICCC51575.2020.9344912
Hu, S., Yuan, J., and Wang, S. (2019). “Cross-modality synthesis from MRI to PET using adversarial U-Net with different normalization,” in 2019 International Conference on Medical Imaging Physics and Engineering (ICMIPE) (Shenzhen: IEEE), 1–5. doi: 10.1109/ICMIPE47306.2019.9098219
Jack, C. R. Jr., Knopman, D. S., Jagust, W. J., Petersen, R. C., Weiner, M. W., Aisen, P. S., et al. (2013). Tracking pathophysiological processes in Alzheimer's disease: an updated hypothetical model of dynamic biomarkers. Lancet Neurol. 12, 207–216. doi: 10.1016/S1474-4422(12)70291-0
Jiang, Y., Chang, S., and Wang, Z. (2021). “TransGAN: two pure transformers can make one strong GAN, and that can scale up,” in Advances in Neural Information Processing Systems (Montreal: MIT Press). 34, 14745–14758.
Kabbara, A., Eid, H., El Falou, W., Khalil, M., Wendling, F., and Hassan, M. (2018). Reduced integration and improved segregation of functional brain networks in Alzheimer's disease. J. Neural Eng. 15, 026023. doi: 10.1088/1741-2552/aaaa76
Kipf, T. N., and Welling, M. (2016). Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907. doi: 10.48550/arXiv.1609.02907
Lei, B., Huang, S., Li, R., Bian, C., Li, H., Chou, Y.-H., et al. (2018). Segmentation of breast anatomy for automated whole breast ultrasound images with boundary regularized convolutional encoder-decoder network. Neurocomputing 321, 178–186. doi: 10.1016/j.neucom.2018.09.043
Lei, B., Liang, E., Yang, M., Yang, P., Zhou, F., Tan, E.-L., et al. (2022). Predicting clinical scores for Alzheimer's disease based on joint and deep learning. Expert Syst. Appl. 187, 115966. doi: 10.1016/j.eswa.2021.115966
Li, X., Steffens, D. C., Potter, G. G., Guo, H., Song, S., and Wang, L. (2017). Decreased between-hemisphere connectivity strength and network efficiency in geriatric depression. Hum. Brain Mapp. 38, 53–67. doi: 10.1002/hbm.23343
Mo, L.-F., and Wang, S.-Q. (2009). A variational approach to nonlinear two-point boundary value problems. Nonlinear Anal. Theory Methods Appl. 71, e834–e838. doi: 10.1016/j.na.2008.12.006
Mormino, E. C., Papp, K. V., Rentz, D. M., Schultz, A. P., LaPoint, M., Amariglio, R., et al. (2016). Heterogeneity in suspected non-Alzheimer disease pathophysiology among clinically normal older individuals. JAMA Neurol. 73, 1185–1191. doi: 10.1001/jamaneurol.2016.2237
Qiao, L., Zhang, H., Kim, M., Teng, S., Zhang, L., and Shen, D. (2016). Estimating functional brain networks by incorporating a modularity prior. Neuroimage 141, 399–407. doi: 10.1016/j.neuroimage.2016.07.058
Skouras, S., Falcon, C., Tucholka, A., Rami, L., Sanchez-Valle, R., Lladó, A., et al. (2019). Mechanisms of functional compensation, delineated by eigenvector centrality mapping, across the pathophysiological continuum of Alzheimer's disease. Neuroimage Clin. 22, 101777. doi: 10.1016/j.nicl.2019.101777
Sudlow, C., Gallacher, J., Allen, N., Beral, V., Burton, P., Danesh, J., et al. (2015). UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med. 12, e1001779. doi: 10.1371/journal.pmed.1001779
Suthaharan, S. (2016). “Support vector machine,” in Machine Learning Models and Algorithms for Big Data Classification, eds R. Sharda and S. Voß (Boston, MA: Springer), 207–235. doi: 10.1007/978-1-4899-7641-3_9
Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., Crivello, F., Etard, O., Delcroix, N., et al. (2002). Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. Neuroimage 15, 273–289. doi: 10.1006/nimg.2001.0978
Wang, J., Wang, X., Xia, M., Liao, X., Evans, A., and He, Y. (2015). GRETNA: a graph theoretical network analysis toolbox for imaging connectomics. Front. Hum. Neurosci. 9, 386. doi: 10.3389/fnhum.2015.00386
Wang, K., Liang, M., Wang, L., Tian, L., Zhang, X., Li, K., et al. (2007). Altered functional connectivity in early Alzheimer's disease: a resting-state fMRI study. Hum. Brain Mapp. 28, 967–978. doi: 10.1002/hbm.20324
Wang, S., Chen, Z., You, S., Wang, B., Shen, Y., and Lei, B. (2022). Brain stroke lesion segmentation using consistent perception generative adversarial network. Neural Comput. Appl. 34, 8657–8669. doi: 10.1007/s00521-021-06816-8
Wang, S., Shen, Y., Chen, W., Xiao, T., and Hu, J. (2017). “Automatic recognition of mild cognitive impairment from MRI images using expedited convolutional neural networks,” in International Conference on Artificial Neural Networks (Alghero: Springer), 373–380. doi: 10.1007/978-3-319-68600-4_43
Wang, S., Hu, Y., Shen, Y., and Li, H. (2018a). Classification of diffusion tensor metrics for the diagnosis of a myelopathic cord using machine learning. Int. J. Neural Syst. 28, 1750036. doi: 10.1142/S0129065717500368
Wang, S., Shen, Y., Zeng, D., and Hu, Y. (2018b). “Bone age assessment using convolutional neural networks,” in 2018 International Conference on Artificial Intelligence and Big Data (ICAIBD) (Chengdu: IEEE), 175–178. doi: 10.1109/ICAIBD.2018.8396189
Wang, S., Wang, H., Shen, Y., and Wang, X. (2018c). “Automatic recognition of mild cognitive impairment and Alzheimer's disease using ensemble based 3D densely connected convolutional networks,” in 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), eds M. Wani, M. Kantardzic, M. Sayed-Mouchaweh, (Orlando, FL: IEEE), 517–523. doi: 10.1109/ICMLA.2018.00083
Wang, S., Wang, H., Cheung, A. C., Shen, Y., and Gan, M. (2020a). “Ensemble of 3D densely connected convolutional network for diagnosis of mild cognitive impairment and Alzheimer's disease,” in Deep Learning Applications (Singapore: Springer), 53–73. doi: 10.1007/978-981-15-1816-4_4
Wang, S., Wang, X., Hu, Y., Shen, Y., Yang, Z., Gan, M., et al. (2020b). Diabetic retinopathy diagnosis using multichannel generative adversarial network with semisupervision. IEEE Trans. Automat. Sci. Eng. 18, 574–585. doi: 10.1109/TASE.2020.2981637
Wang, S., Wang, X., Shen, Y., He, B., Zhao, X., Cheung, P. W.-H., et al. (2020c). An ensemble-based densely-connected deep learning system for assessment of skeletal maturity. IEEE Trans. Syst. Man Cybernet. Syst. 52, 426–437. doi: 10.1109/TSMC.2020.2997852
Wang, S.-Q. (2009). A variational approach to nonlinear two-point boundary value problems. Comput. Math. Appl. 58, 2452–2455. doi: 10.1016/j.camwa.2009.03.050
Wang, S.-Q., Li, X., Cui, J.-L., Li, H.-X., Luk, K. D., and Hu, Y. (2015). Prediction of myelopathic level in cervical spondylotic myelopathy using diffusion tensor imaging. J. Magnet. Reson. Imaging 41, 1682–1688. doi: 10.1002/jMRI.24709
Wee, C.-Y., Yang, S., Yap, P.-T., and Shen, D. (2016). Sparse temporally dynamic resting-state functional connectivity networks for early MCI identification. Brain Imaging Behav. 10, 342–356. doi: 10.1007/s11682-015-9408-2
Whitwell, J. L., Przybelski, S. A., Weigand, S. D., Knopman, D. S., Boeve, B. F., Petersen, R. C., et al. (2007). 3D maps from multiple MRI illustrate changing atrophy patterns as subjects progress from mild cognitive impairment to Alzheimer's disease. Brain 130, 1777–1786. doi: 10.1093/brain/awm112
Xia, M., Wang, J., and He, Y. (2013). BrainNet viewer: a network visualization tool for human brain connectomics. PLoS ONE 8, e68910. doi: 10.1371/journal.pone.0068910
Yang, P., Zhao, C., Yang, Q., Wei, Z., Xiao, X., Shen, L., et al. (2022). Diagnosis of obsessive-compulsive disorder via spatial similarity-aware learning and fused deep polynomial network. Med. Image Anal. 75, 102244. doi: 10.1016/j.media.2021.102244
Ye, C., Mori, S., Chan, P., and Ma, T. (2019). Connectome-wide network analysis of white matter connectivity in Alzheimer's disease. Neuroimage Clin. 22, 101690. doi: 10.1016/j.nicl.2019.101690
You, S., Lei, B., Wang, S., Chui, C. K., Cheung, A. C., Liu, Y., et al. (2022). Fine perceptive GANs for brain MR image super-resolution in wavelet domain. IEEE Trans. Neural Netw. Learn. Syst. 1–13. doi: 10.1109/TNNLS.2022.3153088
Yu, M., Engels, M. M., Hillebrand, A., Van Straaten, E. C., Gouw, A. A., Teunissen, C., et al. (2017). Selective impairment of hippocampus and posterior hub areas in Alzheimer's disease: an MEG-based multiplex network study. Brain 140, 1466–1485. doi: 10.1093/brain/awx050
Yu, S., Wang, S., Xiao, X., Cao, J., Yue, G., Liu, D., et al. (2020). “Multi-scale enhanced graph convolutional network for early mild cognitive impairment detection,” in International Conference on Medical Image Computing and Computer-Assisted Intervention (Lima: Springer), 228–237. doi: 10.1007/978-3-030-59728-3_23
Yu, W., Lei, B., Ng, M. K., Cheung, A. C., Shen, Y., and Wang, S. (2021). Tensorizing GAN with high-order pooling for Alzheimer's disease assessment. IEEE Trans. Neural Netw. Learn. Syst. 33, 4945–4959. doi: 10.1109/TNNLS.2021.3063516
Yu, W., Lei, B., Wang, S., Liu, Y., Feng, Z., Hu, Y., et al. (2022). Morphological feature visualization of Alzheimer's disease via multidirectional perception GAN. IEEE Trans. Neural Netw. Learn. Syst. 1–15. doi: 10.1109/TNNLS.2021.3118369
Yuzwa, S. A., Macauley, M. S., Heinonen, J. E., Shan, X., Dennis, R. J., He, Y., et al. (2008). A potent mechanism-inspired o-glcnacase inhibitor that blocks phosphorylation of tau in vivo. Nat. Chem. Biol. 4, 483–490. doi: 10.1038/nchembio.96
Zeng, D., Wang, S., Shen, Y., and Shi, C. (2017). A GA-based feature selection and parameter optimization for support tucker machine. Proc. Comput. Sci. 111, 17–23. doi: 10.1016/j.procs.2017.06.004
Zhang, H., Wang, Y., Wang, Y., Li, X., Wang, S., and Wang, Z. (2022). Recent advance on carbamate-based cholinesterase inhibitors as potential multifunctional agents against Alzheimer's disease. Eur. J. Med. Chem. 240, 114606. doi: 10.1016/j.ejmech.2022.114606
Zhou, Y., Zhang, L., Teng, S., Qiao, L., and Shen, D. (2018). Improving sparsity and modularity of high-order functional connectivity networks for mci and asd identification. Front. Neurosci. 12, 959. doi: 10.3389/fnins.2018.00959
Zuo, Q., Lei, B., Shen, Y., Liu, Y., Feng, Z., and Wang, S. (2021a). “Multimodal representations learning and adversarial hypergraph fusion for early Alzheimer's disease prediction,” in Chinese Conference on Pattern Recognition and Computer Vision (PRCV) (Zhuhai: Springer), 479–490. doi: 10.1007/978-3-030-88010-1_40
Keywords: functional brain connectivity, temporal-spatial transformer alignment, generative adversarial learning, graph convolutional network, early Alzheimer's disease
Citation: Zuo Q, Lu L, Wang L, Zuo J and Ouyang T (2022) Constructing brain functional network by Adversarial Temporal-Spatial Aligned Transformer for early AD analysis. Front. Neurosci. 16:1087176. doi: 10.3389/fnins.2022.1087176
Received: 02 November 2022; Accepted: 10 November 2022;
Published: 28 November 2022.
Edited by:
Fei He, Coventry University, United KingdomReviewed by:
Jin Hong, Zhongshan Institute of Changchun University of Science and Technology, ChinaHongfei Wang, The University of Hong Kong, Hong Kong SAR, China
Copyright © 2022 Zuo, Lu, Wang, Zuo and Ouyang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Lin Wang, bGluLndhbmcxJiN4MDAwNDA7c2lhdC5hYy5jbg==