A hybrid neural architecture search for hyperspectral image classification

Abstract

Convolution neural network (CNN)is widely used in hyperspectral image (HSI) classification. However, the network architecture of CNNs is usually designed manually, which requires careful fine-tuning. Recently, many technologies for neural architecture search (NAS) have been proposed to automatically design networks, further improving the accuracy of HSI classification to a new level. This paper proposes a circular kernel convolution-β-decay regulation NAS-confident learning rate (CK-βNAS-CLR) framework to automatically design the neural network structure for HSI classification. First, this paper constructs a hybrid search space with 12 kinds of operation, which considers the difference between enhanced circular kernel convolution and square kernel convolution in feature acquisition, so as to improve the sensitivity of the network to hyperspectral information features. Then, the β-decay regulation scheme is introduced to enhance the robustness of differential architecture search (DARTS) and reduce the discretization differences in architecture search. Finally, we combined the confidence learning rate strategy to alleviate the problem of performance collapse. The experimental results on public HSI datasets (Indian Pines, Pavia University) show that the proposed NAS method achieves impressive classification performance and effectively improves classification accuracy.

1 Introduction

Hyperspectral sensing images (HSIs) collect rich spatial–spectral information in hundreds of spectral bands, which can be used to effectively distinguish ground cover. HSI classification is based on pixel level, and many traditional methods based on machine learning have been used, such as the K-nearest neighbor (KNN) [1] and support vector machine (SVM) [2]. The HSI classification method based on deep learning can effectively extract the robust features to obtain better classification performance [3–5].

Limited by the cost of computing resources and the workload of parameter adjustment, it is inevitable to promote the development of automatic design efficient neural network technology [6]. The goal of NAS (neural architecture search)is to select and combine different neural operations from predefined search spaces and to automate the construction of high-performance neural network structures. Traditional NAS work uses the reinforcement learning algorithm (RL) [7], evolutionary algorithm (EA) [8], and the gradient-based method to conduct architecture search.

In order to reduce resource consumption, one-shot NAS methods based on supernet are developed [9]. DARTS is a one-shot NAS method with a distinguishable search strategy [10]. By introducing Softmax function, it expands the discrete space into a continuous search optimization process. Specifically, it can reduce the workload of network architecture design and reduce the process of a large number of verification experiments [9].

The method based on the automatic design of convolutional neural network for hyperspectral image classification (CNAS) introduces DARTS into the HSI classification task for the first time. The method uses point-by-point convolution to compress the spectral dimensions of HSI into dozens of dimensions and then uses DARTS to search for neural network architecture suitable for the HSI dataset [11]. Subsequently, based on the method of 3D asymmetric neural architecture search (3D-ANAS), a classification framework from pixel to pixel was designed, and the redundant operation problem was solved by using the 3D asymmetric CNN, which significantly improved the calculation speed of the model [12].

2 Materials and methods

As shown in Figure 1, the NAS framework for HSI proposed is described, called as CK-βNAS-CLR. Compared with other HSI classification methods, this method aims to alleviate the shortcomings of traditional microNAS methods from three aspects of search space, search strategy, and architecture resource optimization and effectively improve the classification accuracy.

FIGURE 1

DARTS is a basic framework which adopts weight sharing and combines hypernetwork training with the search of the best candidate architecture to effectively reduce the waste of computing resources. First, the hyperspectral image is clipped into patch by sliding window as input. Then, the hybrid search space of CK convolution and attention mechanism is constructed, and the operation search between nodes is carried out in the hybrid search space to effectively improve the feature acquisition ability of the receptive field. At the same time, the architecture parameter set β, which represents the importance of the operator, is attenuated and regularized, effectively strengthening the robustness of DARTS and reducing the discretization differences in the architecture search process. After the search is completed, the algorithm stacks multiple normal cells and reduction cells to form the optimal neural structure, and then the classification results are obtained through Softmax operation. In addition, CLR is proposed to stack decay regularization to alleviate the performance crash of DARTS, improve memory efficiency, and reduce architecture search time.

2.1 The proposed NAS framework for HSI classification

2.1.1 Integrating circular kernel to convolution

The circular kernel is isotropic and can be realized from all directions. In addition, symmetric circular nuclei can ensure rotation invariance, which uses bilinear interpolation to approximate the traditional square convolution kernel to a circular convolution kernel, and uses matrix transformation to reparametrize the weight matrix, replacing the original matrix with the changed matrix to realize the offset of receptive field reception. Without considering the loss, the expression of receptive field H of standard 3 × 3 square convolution kernel with a dilation of 1 is written as follows:

where represents the offset set of the neighborhood convolved on the center pixel. By convolution, the feature map is and kernel is . The output feature map can be obtained, and the coordinates of each position are shown in formula (2).

So, we get , where represents the classical convolution operation used by the CNN. Therefore, the change of receptive field of nucleus circularis 3 × 3 is shown in formula(3).

For the sampling problem of circular convolution kernels, we selected the offset () of for different discrete kernel positions and resampled the offset to input to obtain circular receptive fields. Because the sampling receptive field of circular nucleus has a fraction, we use bilinear interpolation to approximate the sampling value of the receptive field.

where represents the grid position in the circular receptive field and represents all grid positions in the square receptive field, which is the kernel of two-dimensional bilinear interpolation. According to the bilinear interpolation, can be divided into two one-dimensional cores.

Therefore, and only correspond to the corresponding grid of receptive field with grid location . Then, we let and represent the adjusted receptive field centered on position and nucleus, respectively. Generally, the standard convolution can be defined as shown in formula (8), so after replacing the circular convolution kernel, the circular convolution can be located as shown in formula (9).

where is a fixed sparse coefficient matrix, so let , , and be the input feature map, output feature map, and kernel, respectively, so the corresponding definition of formula (9) can be written as formula (10).

where is the convolution process of changing the square receptive field into a circular receptive field. Thus, we can calculate the core weight to achieve operation . This calculation method can effectively avoid calculating the offset of multiple convolutions and reduce the cost of core operation. Next, we summarize the analysis of the actual effect of the transformation matrix. We let , and the value of a change on the output is shown in formula (11). The squared value of a change on the output is shown in formula (12).

In contrast, of the traditional convolution layer is defined by . Therefore, it can be concluded that the transformation matrix caused by the circular core can provide a better choice for the gradient descent path of DARTS.

2.1.2 β-decay regularization scheme

In order to alleviate unfair competition in DARTS, we introduced the β-decay regulation scheme [14] so as to improve its robustness and generalization ability and effectively reduce the search memory and the search cost to find the best architecture as shown in Figure 2.

FIGURE 2

converts the discretization operation of the optional operation set in the search space into an operable continuous set. After implementing Softmax operation, the architecture parameter set is obtained, which is attenuated and regularized.

where is the combination of architecture parameters between node m and node n and is the number of optional operations. Each cell can have up to N nodes, and represents the architecture parameters. A special coefficient for each candidate operation is defined.

Starting from the default setting of regularization, consider the one-step update of architecture parameter , where represents the learning rate a of architecture parameters.

For the special gradient descent algorithm of DARTS, these regularized gradients need to be normalized () through the sum size and to realize the average distribution of the total gradient without normalization.

In the DARTS search process, the architecture parameter set, β, is used to express the importance of all operators. The research on explicit regularization of β can more clearly standardize the optimization of architecture parameters so as to improve the robustness and architecture universality of DARTS. We use the function with as the independent variable to express the total impact of attenuation regularization.

where the function ( is the independent variable) represents the overall influence of β attenuation regularization and is the mapping function. We can iterate for dividing the single-step update parameter value and parameter value weighted sum of β.

It can be found that mapping function determines the impact of on β. To avoid excessive regularization and optimization difficulties, Softmax is used to normalize .

We can obtain the impact and effect of our method.

2.1.3 Confident learning rate strategy

When the NAS method is used to classify hyperspectral datasets, a large number of parameters will be generated. When the training samples are limited, the performance of the network may be reduced due to the over-fitting phenomenon, which will lead to low memory utilization during the training process. CLR is used to alleviate these two problems [15].

After applying the Softmax operation, the structure is relaxed. The gradient descent algorithm is used to optimize the matrix, and the original weight of the network is called . Then, the cross-entropy formula is used to calculate the loss value in the training stage and the parameters and are updated.

In order to enable both to achieve the optimization strategy at the same time, it is necessary to fix the value of matrix on the training set, update using the gradient descent algorithm, fix the value on the verification set, update the value using the gradient descent algorithm, and obtain the best parameter value repeatedly. Stop the optimization after finding the best architecture neural architecture and minimize the verification loss .

NAS architecture parameters will be over-parameterized with the increase of training time. Therefore, the gradient confidence obtained from the parameterized DARTS should increase with the training time of the architecture weight update.

where represents the number of epochs currently trained, represents the preset total epochs, and is the confidence factor of CLR. Through the update of the confidence learning rate, the network obtains and uses it for gradient update.

The confidence learning rate is established in the process of architecture gradient update.

3 Results

Our experiments are conducted using Intel (R) Xeon (R) 4208 CPU@2.10GHz Processor and Nvidia GeForce RTX 2080Ti graphics card. We selected the average of 10 experiments to compute the overall accuracy (OA), average accuracy (AA), and Kappa coefficient (K).

3.1 Comparison with state-of-the-art methods

In this section, we select some advanced methods to make comparison so as to evaluate the classification performance, which include extended morphological profile combined with support vector machine (EMP-SVM) [16], spectral spatial residual network (SSRN) [17], residual network (ResNet) [18], pyramid residual network (PyResNet) [19], multi-layer perceptron mixer (MLP Mixer) [20], CNAS [11], and efficient convolutional neural architecture search (ANAS-CPA-LS) [21]. All experimental results are shown in Tables 1, 2. The sample is clipped by using the sliding window strategy size of 32 × 32, and the overlap rate is set to 50%. We randomly selected 30 samples as the training dataset and 20 samples as the validation dataset. The training time is set to 200, and the learning rate of the three data sets is set to 0.001.

In Table 1, compared with EMP-SVM, SSRN, ResNet, PyResNet, CNAS, MLP Mixer, and ANAS-CPA-LS, OA obtained by our proposed method is increased by 16.26%, 4.32%, 3.37%, 2.95%, 2.9%, 1.95%, and 1.33%, respectively, on the Indian Pines dataset. Figures 3, 4 shows the classification diagram obtained from a visual perspective. By comparing the classification diagrams obtained, we can draw a conclusion that our algorithm achieves better performance. Compared with CNAS, our method uses a hybrid search space, which can effectively expand the receptive field acquired by pixels, improve the flexibility of different convolution kernel operations to process spectrum and space, and achieve higher classification accuracy.

FIGURE 3

FIGURE 4

4 Discussion

The ablation study results are provided in Table 3. When CNAS is combined with hybrid search space, OA increases by 0.70%, 0.35%, and 0.54%, which proves that the hybrid search space can improve the sensitivity of the network to hyperspectral information features and slightly improve the classification performance of the model. Compared with CNAS, CK-NAS has no time change in the search time on the three datasets but has achieved better classification accuracy. CK-βNAS-CLR search gets better results with fewer parameters and involves less computational complexity.

5 Conclusion

In this paper, the neural network structure CK-βNAS-CLR is proposed. First of all, we introduce a hybrid search space with circular kernel convolution, which can not only enhance the robustness of the model and the ability of receptive field acquisition but also achieve a better role in optimizing the path. Second, we quoted the β-decay regulation scheme, which reduced the discretization difference and the search time. Finally, the confidence learning rate strategy is introduced to improve the accuracy of model classification and reduce computational complexity. The experiment was conducted on two HSI datasets, and CK-βNAS-CLR is compared with seven methods, and the experimental results show that our method achieves the most advanced performance while using less computing resources. In future, we will use an adaptive subset of the data even when training the final architecture, which may lead to faster runtime and lower regularization term.

References

• 1.

  ChandraBSharmaRK. On improving recurrent neural network for image classification. In: Proceeding of the International Joint Conference on Neural Networks. Alaska, USA: IJCNN (2017). p. 1904–7. 10.1109/IJCNN.2017.7966083

  • CrossRef
  • Google Scholar

• 2.

  SamadzadeganFHasaniHSchenkT. Simultaneous feature selection and SVM parameter determination in classification of hyperspectral imagery using Ant Colony Optimization. Remote Sens (2012) 38:139–56. 10.5589/m12-022

  • CrossRef
  • Google Scholar

• 3.

  HuWHuangYWeiLZhangFLiH. Deep convolutional neural networks for hyperspectral image classification. J Sensors (2015) 2015:1–12. 10.1155/2015/258619

  • CrossRef
  • Google Scholar

• 4.

  MakantasisKKarantzalosKDoulamisADoulamisN. Deep supervised learning for hyperspectral data classification through convolutional neural networks. In: 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS). Milan, Italy: IGARSS (2015). p. 4959–62.

  • Google Scholar

• 5.

  LiWWuGZhangFDuQ. Hyperspectral image classification using deep pixel-pair features. IEEE Trans Geosci Remote Sensing (2017) 55(2):844–53. 10.1109/tgrs.2016.2616355

  • CrossRef
  • Google Scholar

• 6.

  KaifengBLingxiXChenXLonghuiWTianQ. GOLD-NAS: Gradual,one-level, differentiable (2020). p. 03331. arXiv:2007.

  • Google Scholar

• 7.

  RealEAggarwalAHuangYLeQV, “Regularized evolution for image classifier architecture search,” in Proc. AAAI Conf. Artif. Intell (2019) 33, 4780–9. 10.1609/aaai.v33i01.33014780

  • CrossRef
  • Google Scholar

• 8.

  TanMChenBPangRV asudevanVSandlerMHowardAet alMnasNet: Platform-aware neural architecture search for mobile. Long Beach, CA, USA: CVPR (2019). 2820–8.

  • Google Scholar

• 9.

  LiangH. DARTS+: Improved differentiable architecture search with early stopping (2020). arXiv:1909.06035. [Online]. Available: https://arxiv.org/abs/1909.06035 (Accessed October 20, 2020).

  • Google Scholar

• 10.

  GuoZ, “Single path one-shot neural architecture search with uniform sampling,” in Proc. IEEE Eur. Conf. Comput. Vis (2020), 544–60.

  • Google Scholar

• 11.

  ChenYZhuKZhuLHeXGhamisiPBenediktssonJA. Automatic design of convolutional neural network for hyperspectral image classification. IEEE Trans Geosci Remote Sensing (2019) 57(9):7048–66. 10.1109/tgrs.2019.2910603

  • CrossRef
  • Google Scholar

• 12.

  ZhangHGongCBaiYBaiZLiY. 3d-anas: 3d asymmetric neural architecture search for fast hyperspectral image classification (2021). arXiv preprint arXiv:2101.04287, 2021.[Online]. Available: https://arxiv.org/abs/2101.04287 (Accessed Janunary 12, 2021).

  • Google Scholar

• 13.

  LiGQianGDelgadilloICMullerMThabetAGhanemB. Sgas: Sequential greedy architecture search. In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit (2020). p. 1620–30.

  • Google Scholar

• 14.

  YePLiBLiYChenTFanJOuyangWet albeta$-DARTS: Beta-Decay regularization for differentiable architecture search, in Proceeding of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA (2022), 10864–73.

  • Google Scholar

• 15.

  DingZChenYLiNZhaoD. BNAS-v2: Memory-Efficient and performance-collapse-prevented broad neural architecture search. IEEE Trans Syst Man, Cybernetics: Syst (2022) 52(10):6259–72. 10.1109/TSMC.2022.3143201

  • CrossRef
  • Google Scholar

• 16.

  MelganiFBruzzoneL. Classification of hyperspectral remote sensing images with support vector machines. IEEE Trans Geosci Remote Sensing (2004) 42(8):1778–90. 10.1109/tgrs.2004.831865

  • CrossRef
  • Google Scholar

• 17.

  ZhongZLiJLuoZChapmanM. Spectral–spatial residual network for hyperspectral image classification: A 3-D deep learning framework. IEEE Trans Geosci Remote Sensing (2018) 56(2):847–58. 10.1109/tgrs.2017.2755542

  • CrossRef
  • Google Scholar

• 18.

  LiuXMengYFuM, "Classification research based on residual network for hyperspectral image,"In, Proceeding of the 2019 IEEE 4th International Conference on Signal and Image Processing (IC) (2019). 911–5.

  • Google Scholar

• 19.

  PaolettiMEHautJMFernandez-BeltranRPlazaJPlazaAJPlaF. Deep pyramidal residual networks for spectral–spatial hyperspectral image classification. IEEE Trans Geosci Remote Sensing (2019) 57(2):740–54. 10.1109/TGRS.2018.2860125

  • CrossRef
  • Google Scholar

• 20.

  HeXChenY. Modifications of the multi-layer Perceptron for hyperspectral image classification. Remote Sensing (2021) 13(17):3547. 10.3390/rs13173547

  • CrossRef
  • Google Scholar

• 21.

  WangAXueDWuHGuY. Efficient convolutional neural architecture search for LiDAR DSM classification. IEEE Trans Geosci Remote Sensing (2022) 60:1–17. Art no. 5703317. 10.1109/TGRS.2022.3171520

  • CrossRef
  • Google Scholar