Enhancing the depth perception of DSA images with 2D–3D registration

Objective: Today, cerebrovascular disease has become an important health hazard. Therefore, it is necessary to perform a more accurate and less time-consuming registration of preoperative three-dimensional (3D) images and intraoperative two-dimensional (2D) projection images which is very important for conducting cerebrovascular disease interventions. The 2D–3D registration method proposed in this study is designed to solve the problems of long registration time and large registration errors in 3D computed tomography angiography (CTA) images and 2D digital subtraction angiography (DSA) images.

Methods: To make a more comprehensive and active diagnosis, treatment and surgery plan for patients with cerebrovascular diseases, we propose a weighted similarity measure function, the normalized mutual information-gradient difference (NMG), which can evaluate the 2D–3D registration results. Then, using a multi-resolution fusion optimization strategy, the multi-resolution fused regular step gradient descent optimization (MR-RSGD) method is presented to attain the optimal value of the registration results in the process of the optimization algorithm.

Result: In this study, we adopt two datasets of the brain vessels to validate and obtain similarity metric values which are 0.0037 and 0.0003, respectively. Using the registration method proposed in this study, the time taken for the experiment was calculated to be 56.55s and 50.8070s, respectively, for the two sets of data. The results show that the registration methods proposed in this study are both better than the Normalized Mutual (NM) and Normalized Mutual Information (NMI).

Conclusion: The experimental results in this study show that in the 2D–3D registration process, to evaluate the registration results more accurately, we can use the similarity metric function containing the image gray information and spatial information. To improve the efficiency of the registration process, we can choose the algorithm with gradient optimization strategy. Our method has great potential to be applied in practical interventional treatment for intuitive 3D navigation.

1. Introduction

In the surgical navigation treatment of cerebrovascular diseases, an accurate identification of the lesion location is one of the important factors that determine the operation plan. Improving the level of multimodal data analysis and image processing technology is an important step in surgical navigation treatment.

Preoperative 3D scanning images can visualize the lesion and provide 3D data information, but they cannot be obtained in real time during surgery, while 2D images cannot provide an accurate 3D navigation information. If both 2D and 3D images are aligned, it can provide an accurate navigation information and improve the success rate of surgery. Multimodal 2D–3D image registration is the key technology. Image registration is the integration of image information gathered from different times or conditions. The goal of image registration is to find the conversion relationship between different images at the same location to obtain an integrated information of two images (1–3).

Multimodal image registration plays an important role in clinical nursing, mainly by including computed tomography angiography (CTA) and digital subtraction angiography (DSA), shown as the CTA image in Figure 1A and DSA image in Figure 1B. In the whole 2D–3D registration process, two types of images, namely, the floating image and the reference image, are required as data sources (4), and the spatial position transformation operation of floating image is required to align it with the reference image to complete the registration (5). However, the registration accuracy of both CTA images and DSA images is low at present, while the time spent on registration is long, which may reduce the surgical efficiency.

FIGURE 1

Figure 1. A comparison of three-dimensional computed tomography angiography (3D CTA) image and two-dimensional digital subtraction angiography (2D DSA) image presentation, where (A) 3D CTA image is a floating image, and (B) 2D DSA image is the reference image.

At present, for evaluation of the results of the 2D–3D image registration, the selected similarity measurement function includes only the gray-level information of a single image and lacks the spatial information of the image as an aid, resulting in large errors in the results of the final similarity evaluation, such as Normalized Mutual (NM) and Normalized Mutual Information (NMI) (6, 7). In the 2D–3D registration process, the selected optimization algorithm considers only the optimal value and it does not consider improving the registration efficiency, such as Powell and Gradient Descent (GD) (8, 9).

We propose a weighting function with normalized mutual information and gradient difference, considering the grayscale and spatial information of the image, and applying Regular Step Gradient Descent (RSGD) as a means to find the optimal value of the optimization function, reducing the step size by changing the direction of each iteration, avoiding the local optimal value, and improving the speed of registration. The multi-resolution strategy used in this study is the Gaussian pyramid convolution algorithm (10). After the input image is convolved by a layer of convolution, it is sent to the Gaussian pyramid for multi-resolution sampling processing. The pyramid algorithm samples according to the resolution of the input image, and divides the whole sampling process into coarse resolution and fine resolution. The image resolution is the degree of quantization of the image in the horizontal and vertical directions, refers to the degree of image detail that can be shown (11–14). Low resolution image is used to analyze larger structures or the overall content, while high resolution image is used to analyze detailed characteristics. A multi-scale strategy is used to observe continuous images at different scales which helps to understand the image content (15, 16). By applying a Gaussian kernel for multi-scale changes (17), the multi-resolution strategy proposed in this study can enhance the robustness of the algorithm and improve the registration efficiency.

This study makes the following contributions.

1. The 2D–3D registration method proposed in this study optimizes the efficiency of the registration process and the accuracy of the registration results compared to those of the traditional registration methods.

2. We propose normalized mutual information-gradient difference (NMG) as a weighting similarity metric function for the registration results, which includes both grayscale and spatial information of the images.

3. The proposed method chooses multi-resolution fused regular step gradient descent (MR-RSGD) as an optimization algorithm strategy to avoid inefficiency and the local optimal value of the registration result. The proposed method divides the coarse-resolution and fine-resolution images into two sampling processes. Regular step gradient descent (RSGD) can find the optimal value to avoid falling into local extremes, and the result of the objective function is kept constant as a condition for stopping the iterative optimization.

2. Related work

In multimodal image registration, the manual manipulation of 2D–3D registration is often limited by the amount of time and expertise required to register segmented data to an image. To be able to reduce the errors introduced by subjectivity, a new intensity-based 2D–3D registration method is proposed, which can exploit all forms of visual information, including not only variations in image intensity but also the apparent contours of structures. However, it uses only information about the grayscale values of pixels without taking into account their surface characteristics (18). A method is proposed that allows to pre-compute all time-intensive steps, entwining spatial information from 3D volumes and 2D projections.

The main work in the actual registration task is reduced to a simple resampling of pre-computed values, which can be performed on the graphics processing unit (GPU), but requires a 3D Radon transformation of the input image, which leads to an increase in computational effort, is also susceptible to image distortion, and can only be applied if the object undergoes a planar to flat transformation (19). A globally optimal method is proposed that iteratively searches the transformation space, constraining the objective function at each stage and discarding parts of the transformation space for which no solution is possible, but only for points, lines, or a combination of both (20).

A new probabilistic 2D–3D vessel registration method that extends the Gaussian Mixture Model (GMM) to 2D–3D registration and integrates orientation information throughout the registration framework is proposed. The final evaluation obtained is based on the 2D projection error, but the out-of-plane orientation error estimate is much larger due to the planar setting (21). A comprehensive landmark-based 2D–3D registration method that uses X-ray images of the object and a 3D reference model to reconstruct the object's 3D pose is proposed. The aforementioned method incorporates an automated 3D landmark extraction technique and a deep neural network for 2D landmark detection, but considers only the limb of one animal sample, ignoring the presence of muscle and other types of soft tissue in other real animal models, complicating the 2D landmark detection, performing noise simulations that do not take into account the model of the X-ray source the response of the detector (22).

The 2D–3D registration based on a convex optimization procedure is applied to a 3D central model of a coronary artery with a pair of perspective images, and the proposed optimization procedure jointly optimizes the correspondence between points and projections, as well as the relative transformations. However, the ability to include translations through centrality is limited, the design of a fast solver for the convex procedure is not considered, and the exact recovery is guaranteed only when there is no noise in the image (23). Biplane 2D–3D registration was used to measure the 3D joint motion, relying primarily on an automated 2D–3D registration procedure based on computed tomography (CT) and magnetic resonance imaging (MRI) images used together to quantify joint motion, but the study was limited to passive shoulder motion and the accuracy of the registration method was limited by the bone quality, speed of motion, and shoulder orientation in the biplane fluoroscopic field of view, and therefore could not be generalized to motion with significant deviations in this area (24).

The use of 2D–3D catheter-based continuous wayfinding fusion of preoperative or intraoperative images in liver surgery can add relevant information to navigated procedures, and the proposed real-time registration of 3D rotational angiographic images in the operative period with intraoperative single-plane 2D fluoroscopic images can improve the guidance of surgical interventions; however, this method deals only with 2D image catheters and the accuracy of the results is low (25).

The choice of the similarity measure function plays an important role in the accuracy of the final result. The statistical method mutual information (MI) (26, 27), which uses the joint probability distribution images to measure the strength of statistical relationships between images, is well adapted to images of different modalities and is therefore used widely in multimodal medical registration. When using mutual information as a similarity metric in the registration framework, mutual information can show a high performance if the recovered displacement is small and it is difficult to obtain a larger displacement, which highlights the advantages of fast global optimization of mutual information. To enhance the fast computation of mutual information and its robustness, many improved algorithms based on the MI have been proposed, introducing normalized mutual information processing (NMI) for sensitivity overlap (28, 29), optimizing the computation of joint distribution, and combining with other intensity-based similarity metrics to form a new metric (30). However, the mutual information calculates only the gray value of each pixel of the image and does not take the spatial features into account, so the spatial information is missing, resulting in a slightly lower registration efficiency.

After choosing the similarity metric function, the choice of optimization algorithm also affects the results of final registration. Powell is mostly used for a local search of optimal values, which has the features of simple calculation, fast convergence, and high accuracy, but it is easy to fall into the local optimal point when searching for the registration parameters, resulting in an insufficient accuracy of the registration. GD can reduce the iterative optimization time to iterate in the direction of the fastest change of the objective function value (31), which can improve the speed of the algorithm, but it does not guarantee to find the global optimal solution that meets the requirements because of the limitation of each iteration step (32–34).

3. Methods

The essence of 2D–3D registration is the mathematical problem of iteratively solving the optimal value of an objective function, first adjusting the spatial position of the 3D floating image, then projecting the 3D floating image to generate the 2D projection image, then sampling the projection image and the reference image by multi-resolution strategy, and relying on the optimization algorithm to iteratively find the optimal value, so that the similarity between the two images is maximized, as shown in the 2D–3D registration flowchart in Figure 2.

FIGURE 2

Figure 2. The framework of the proposed two-dimensional three-dimensional (2D–3D) registration method, from the very first input 3D image and 2D image, 3D image is projected to generate 2D image digitally reconstructed radiograph (DRR), 2D image and DRR are processed by pyramidal multi-resolution, then the optimization algorithm regular step gradient descent (RSGD) finds the optimal value of similarity measure function to attain the best registration effect.

In the process of 2D–3D registration, we need to first solve the problem of different dimensions of the registration image, using a projection algorithm to project the 3D preoperative image to produce a 2D projection image, the process is the digitally reconstructed radiograph (DRR) algorithm, so that the whole 2D–3D registration is converted to 2D–2D registration. Unlike the shadow formed by ordinary light, the DRR technology is a rendering operation for 3D images, which are converted according to the formation principle of X-ray images. By performing the DRR operation on a 3D object, the corresponding projection image of the object can be obtained, also called the DRR image. The process of DRR generation is shown in Figure 2.

3.1. The normalized mutual information-gradient difference

In the 2D–3D image registration process, different similarity metric functions have different abilities to capture image information, and the registration results are dependent on factors such as image morphology, tissue class, and feature information. The proposed function in this experiment is a weighted normalized mutual information-gradient difference metric function.

To improve the accuracy of the registration, this study improves on the two aforementioned similarity metric functions by assigning a coefficient to each of them and combining them to form a new similarity metric function called the NMG, which combines the information of NMI and GD, so that it can take advantage of the normalized mutual information in multimodal image registration and complement the lack of gradient information. Its definition is shown in Equation (1) in Appendix.

The α₁ and α₂ of Equation (1) in Appendix are the weights of NMI and GD, respectively, and their sum is 1. In this experiment, α₁ and α₂ are both taken as 0.5. Both mutual information (MI) and normalized mutual information (NMI) are based on image information, thus building statistical models based on discrete random variables, and MI represents the information common between two variables. It evolved from Shannon's entropy (Shannon's entropy) theorem (35, 36), while normalized mutual information is a further improvement over the former. Mutual information is widely used in image registration, especially in multimodal image registration. The Shannon entropy formula for image A is shown in Equation (2) in Appendix.

The joint entropy formula for image A and image B is shown in Equation (3) in Appendix.

The A and B in all Appendix Equation are pixels in the two images, the P(A_i) and P(B_j) of Equation (2) in Appendix are edge probability distributions, and the p(A_i, B_j) of Equation (3) in Appendix is a joint probability distribution. The mutual information is expressed, as shown in Equation (4) in Appendix.

The normalized mutual information makes a change based on mutual information by removing the joint entropy of the images by the sum of the Shannon entropy of the two images. It is defined as shown in Equation (5) in Appendix.

Gradient difference (GD) is to first obtain the gradient images of two images separately, then perform the gradient image subtraction operation to obtain the difference image, and use the difference image to measure the degree of similarity. GD judges the final registration result by measuring the change of low-frequency gray value due to gradient reduction. The formula is shown in Equations (5)–(7) in Appendix.

In Equations (5)–(7) in Appendix, the $\frac{dA}{dm}$ and $\frac{dB}{dm}$ represent the gradient images in the horizontal direction of the two images, $\frac{dA}{dn}$ and $\frac{dB}{dn}$ represent the gradient images in the vertical direction of the two images, I_dV is the difference between the horizontal gradient images, I_dH is the difference between the vertical gradient images, s is the scale constraint factor, σ_v and σ_h are constants.

3.2. Multi-resolution fusion optimization strategy

The 2D–3D registration requires adjusting the position of the 3D image to generate the corresponding projection image in each iteration, which leads to an excessive computation of the registration. To reduce the time consumed in the whole registration process, a multi-resolution strategy can be incorporated into the registration process. The multi-resolution strategy divides the registration process into two, to realize the process from coarse to fine registration, which can improve the probability of finding the optimal solution in the optimization iteration, accelerate the convergence speed of the algorithm, and reduce the registration time loss.

The basic idea of multi-resolution is that the multi-resolution strategy arranges the images in order of their resolutions and aligns them from the lowest to the highest resolution, and the result of the last registration is used as the initial value of the next one. This bottom-up registration method can render the low-resolution images smooth during the registration process, thus improving the robustness of the images, avoiding the algorithm from converging badly in the iterative process and falling into the local optimum, which greatly improves the efficiency and stability of the registration algorithm, speeds up the registration optimization speed, and improves the capture range. In this study, the Gaussian pyramidal convolution algorithm is used for the multi-resolution processing of images, so that the fused multi-resolution pyramidal convolution algorithm improves the stability of registration and registration efficiency. The principle behind the algorithm is that the layer g_k image is convolved from the layer g_k−1 image through two convolutional layers characterized by different Gaussian kernels. The computational formula is given in Equation (9) in Appendix.

In Equation (9) in Appendix, g_k(i, j) denotes the layer k image, i and j denote the number of rows and columns of the image, and W(m, n) is a window function.

After the similarity metric function of the registration is determined, a suitable optimization algorithm is needed to solve the optimal value of the objective function, so the selection of the optimization algorithm affects the final result of the registration directly. Compared with the traditional gradient descent algorithm, in this experiment, the RSGD algorithm is used to reduce the step size when the gradient direction is changed to update the parameter values each time, it can prevent the generation of excessive step sizes during the gradient descent process, and the solution along the gradient descent direction can improve the registration speed. In each iteration, RSGD takes a fixed step along the gradient direction of the metric and solves for the parameter values within this gradient range for that step value condition. The optimization algorithm formula is given in Equation (10) in Appendix. The optimization process is shown in Figure 3.

FIGURE 3

Figure 3. Multi-resolution fusion optimization strategy samples the input image, according to the different image resolutions, and the registration process is divided into two types from coarse resolution to fine resolution. The regular step gradient descent (RSGD) algorithm performs iterative optimization of the registration through the registration results and continuously adjusts the computed tomography (CT) image position parameters to match the optimal projection position, so as to obtain the optimal value of the registration result.

In Equation (10) in Appendix, θ_k denotes the parameter value updated in the iteration of k times, θ_k+1 denotes the parameter value updated in the iteration of k + 1 times, k denotes the number of iteration times, α_k denotes the step size chosen in the gradient direction in the iteration of k times, and d_k times represent the gradient searched in the iteration of k times.

In the whole optimization algorithm, the process of finding the optimal value of the similarity measure function, the optimization method adopted is the multi-resolution strategy fused with the conventional stepwise gradient descent algorithm to solve, the pyramid convolution algorithm first samples the coarse-resolution image and the fine-resolution image in two processes from coarse to fine, so as to speed up the alignment efficiency. The similarity metric function finds the metric value and the spatial location parameter, so as to analyze whether the obtained value is the optimal value. Then, the similarity metric function is used to obtain the metric values and spatial location parameters to determine whether the acquired value is the optimal value, if not, the target value is iteratively obtained using the conventional stepwise gradient descent algorithm to speed up the registration, and the optimal value is approximated finally. The algorithm code about this process is shown in Algorithm 1.

ALGORITHM 1

Algorithm 1. Pyramid convolutional fusion regular step gradient descent (RSGD) optimization process.

4. Results

In the whole registration process, first, the 3D floating image entering the registration is spatially transformed and its position parameters are continuously adjusted to get the best projection position, the projection image is projected to get the qualified projection image, and the 2D reference image is pre-processed at the same time. Second, the Gaussian pyramid convolution algorithm is used for multi-resolution processing, and both the projection image and reference image are sampled from coarse to fine resolution. This process not only improves the signal-to-noise ratio of the sampled signals and the robustness of the algorithm but also increases the speed of registration by compressing the sampling information at each point. The similarity of the sampled projection and reference images is evaluated by the similarity metric function, the optimization algorithm iteratively solves the optimal value of the objective function, until the global optimum is obtained, and the registration process is completed. The first set of data used in this experiment pertains to blood vessel data in a human body, and the data sampled from the CT images and DRR images are shown in Table 1.

TABLE 1

Table 1. The parameters associated with the first set of cerebrovascular data, the size and resolution of the input three-dimensional computed tomography angiography (3D-CTA) images and two-dimensional digitally reconstructed radiograph (2D DRR) images.

As regards the translation of the initial position in CT, rotation values are specifically taken as 0 in the X-axis rotation angle, 0 in the Y-axis rotation angle, 0 in the Z-axis rotation angle, 10 mm for the initial value of translation in the X-axis direction, –50 mm for the initial value of translation in the Y-axis direction, and –70 mm for the initial value of translation in the Z-axis direction, r_x = 0, r_y = 0, r_z = 0, t_x = 10 mm, t_y = –50 mm, and t_z = –70 mm. Table 2 shows the solutions using the first set of data under the same NMG with the application of three different optimization algorithms, Powell, GD, and RSGD. From Table 2, we can see that the registration result error and registration time of the Powell algorithm are larger than those of the other two algorithms, and the registration result error of both GD and RSGD is 0.

TABLE 2

Table 2. Angle deviation, position deviation, and metric value of the image registration result of blood vessel data.

The experimental results indicate that the accuracy of the registration result can be improved by applying the gradient optimization algorithm to the improved similarity measure function, until the registration error of the spatial parameters tends to 0. Moreover, the time used by RSGD is significantly smaller than that of than the other two optimization algorithms. In summary, RSGD is significantly better than the other two optimization algorithms in iterative optimization of the similarity measure function to find the optimal value. The final experimental results show that the iterative optimization of NMG by applying the RSGD algorithm improved the efficiency and time of registration, and improved the stability, robustness, and accuracy of registration. For two sets of test data, the registration values obtained under different optimization algorithms are shown in Figure 4.

FIGURE 4

Figure 4. Five sets of controlled experiments were done for the first set of vascular data under three optimization algorithms, Powell, gradient descent (GD), and regular step gradient descent (RSGD). Change the five times position space parameters of the input image, respectively. The first column represents the input digital subtraction angiography (DSA) image, the second column represents the digitally reconstructed radiograph (DRR) image generated by the Powell algorithm, the third column represents the DRR and DSA registration generated by the Powell algorithm, the fourth column represents the DRR image generated by the GD algorithm projection, the fifth column represents the DRR and DSA registration maps generated by the GD algorithm projection, and the results show that some error still seems to be present, the sixth column shows the DRR image generated by the RSGD algorithm, and the seventh column shows the DRR and DSA registration maps generated by the RSGD algorithm, which shows that the registration error is reduced and the registration accuracy is improved compared with the previous two algorithms. The registration results of vessel obtained by Powell in the third column and by GD in the fifth column have errors between the floating image and the reference image, as indicated by the blue arrows.

The three sets of similarity metric values under the three different optimization algorithms are shown in Table 3. It can be visualized that under the RSGD algorithm optimization, not only is the registration time the fastest, but also the final similarity metric value is also the smallest, and the registration accuracy is the highest. To verify that the RSGD based on the multi-resolution strategy proposed in this study to find the optimal value of NMG has better registration efficiency and stronger robustness, controlled trials were carried out by changing the parameters of the initial position, and the experimental results are shown in Figure 5.

TABLE 3

Table 3. The parameters associated with the second set of brain data, the size and resolution of the input three-dimensional computed tomography angiography (3D-CTA) images and two-dimensional digitally reconstructed radiograph (2D-DRR) images.

FIGURE 5

Figure 5. Five controlled trials of three optimization algorithms on blood vessel data and brain data. The labels in row (A–E) represent the tests conducted under different initial positions of 3D CTA images. The three initial rotation position parameters of the first group are 0, 0, 0, and the three translation position parameters are 10, −50, −70. Then the remaining five groups can change the measurement value of 20 each time based on the data of the first group. The first column represents the original DSA image, the second, fourth and sixth columns represent the DRR image generated under different optimization algorithms, and the third, fifth and seventh columns represent the registration results of the generated DRR image and DSA image.

To verify the advantages of the weighting function and the optimization algorithm proposed in this study, two controlled experiments of three optimization algorithms (Powell, GD, and RSGD) under three similarity metric functions (MI, NMI, and NMG) were conducted on two sets of data, the blood vessels and the brain, and the comparative experimental results are shown in Figure 4, analyzed by the qualitative and quantitative results obtained from the aforementioned sets of experiments, and the registration method proposed in this study shows certain advantages in registration efficiency and registration accuracy.

To verify that the registration method proposed in this experiment has stronger robustness and can adapt to images of different parts of the human body and different image styles, the second set of brain data was selected for the experiment, and the relevant data are shown in Table 3.

After conducting the second set of data experiments, it can be seen from Table 4 that the conclusions drawn are consistent with the conclusions of the first set of data experiments, the RSGD algorithm for iterative optimization of NMG, the accuracy of the experimental results obtained, and the registration time spent is the best performing set. The results of the Powell algorithm for registration have larger errors in rotation angle and translation distance compared to the other two optimization algorithms, and the registration time used is larger than those of the other two optimization algorithms. The values of the rotation angle error and translation distance error for both the GD and RSGD algorithms are zero, and the registration time used for the RSGD algorithm is smaller than that of the GD algorithm. The RSGD proposed in this experiment is applied to the NMG registration method, and the results of the registration experiments are good, with the distance error of each group being 0, the angle error being 0, the least time consumed, and the error does not increase due to the selection of different parts and different image styles, indicating that the algorithm proposed in this experiment has better generalization applicability and stronger robustness.

TABLE 4

Table 4. Angle deviation, position deviation, and metric value of image registration result of blood vessel data.

The results of the second set of data experiments are shown in Figure 6. The registration similarity metric values for the second set of data are shown in the Figure 6, and according to the experimental results, it can be seen that the registration results under the RSGD optimization algorithm are more accurate than the other two. The same transformation of the initial position parameters is performed for the second set of data to verify the universality and stability of the registration framework in this article.

FIGURE 6

Figure 6. Five sets of controlled trials were done for the second group of brain data under three optimization algorithms, Powell, gradient descent, and regular step gradient descent (RSGD). Change the five times position space parameters of the input image, respectively. The first column represents the input digital subtraction angiograph (DGA) image, the second column represents the digitally reconstructed radiograph (DRR) image generated by the Powell algorithm, the third column represents the DRR and DSA registration generated by the Powell algorithm, the fourth column represents the DRR image generated by GD algorithm projection, the fifth column represents the DRR and DSA registration maps generated by the GD algorithm projection, and the results show that some error still seems to be present, the sixth column shows the DRR image generated by the RSGD algorithm, and the seventh column shows the DRR and DSA registration maps generated by the RSGD algorithm, which shows that the registration error is reduced and the registration accuracy is improved compared with the previous two algorithms, the registration efficiency is also improved. The registration results of the brain obtained by Powell in the third column and by GD in the fifth column have errors between the floating image and the reference image, as indicated by the blue arrows. The labels in row (A–E) represent the tests conducted under different initial positions of 3D CTA images. The three initial rotation position parameters of the first group are 0, 0, 0, and the three translation position parameters are 10, −50, −70. Then the remaining five groups can change the measurement value of 20 each time based on the data of the first group.

To verify the advantages of the weighting function and the optimization algorithm proposed in this study, two controlled experiments of three optimization algorithms (Powell, GD, and RSGD) under three similarity metric functions (MI, NMI, and NMG) were conducted on two sets of data, the blood vessels and the brain, and the comparative experimental results are shown in Figures 7, 8, analyzed by the qualitative and quantitative results obtained from the aforementioned sets of experiments, and the registration method proposed in this study shows certain advantages in registration efficiency and registration accuracy.

FIGURE 7

Figure 7. Under the condition of different similarity measure functions, the measure values of the three optimization algorithms.

FIGURE 8

Figure 8. The registration time spent by the three optimization algorithms under the conditions of different similarity measure functions.

5. Discussion

During the intervention of cerebrovascular disease, accurately finding the vascular morphology of the target region will greatly improve the success rate of the operation. In the field of 2D–3D medical image registration, it is important to improve the registration accuracy and reduce the registration time to locate the lesion more accurately in surgical procedures. When multi-resolution completes sampling, the similarity is obtained by NMG and the optimal value is iteratively obtained by the RSGD algorithm, so that our method can achieve the effect of optimal registration, and the spatial position parameters of the CT image are continuously adjusted in this process, so that the projected image can be accurately aligned with the reference image.

According to the preliminary experiments and controlled experiments done on two groups of human data, the finally presented experimental data and registration results show that the registration method proposed in this experiment has a better performance for 2D–3D registration. The errors of spatial position parameters of the images in both sets of experiments are 0, and the spent registration time is also the smallest. The experimental results obtained from the control experiments after the initial spatial parameters are changed also meet the requirements. The traditional similarity metric functions, MI and NMI, contain only the grayscale information of the image, which leads to an increase in the error of the final registration results.

The fusion with the gradient difference function makes up for the lack of spatial feature information of the original algorithm in the image, and the NMG function proposed in this experiment can capture both grayscale and spatial information of the image. The multi-resolution strategy selects the pyramid convolution algorithm, which adds another convolution layer before each original pyramid layer. The Strategy can improve the sampling signal ability of the pyramid and play the role of suppressing image noise while improving the robustness of the algorithm. The optimization algorithm selects the RSGD, which avoids local extremes by changing the step length of gradient descent in each iteration process, and which can speed up the registration speed.

Based on the aforementioned analysis, the results and registration efficiency of the optimized NMG registration method using RSGD based on a multi-resolution sampling strategy for multimodal image registration are excellent compared with the other two optimization algorithms; however, this experiment can align only single images at present and cannot perform batch processing, which is the limitation of this experiment.

6. Conclusion

This study introduces the RSGD-optimized NMG registration method based on the multi-resolution strategy, while experiments show that the registration performance has improved greatly compared with the traditional registration method. NMG makes the aligned image contain grayscale information and spatial gradient information, while the multi-resolution strategy improves the image signal sampling ability and enhances the stability and robustness. RSGD can avoid local extremes to approach the optimal solution of the objective function faster. Future research prospects can look into non-rigid 2D–3D alignment algorithms and how to efficiently process large batches of aligned images.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by Shenzhen Second People's Hospital. The patients/participants provided their written informed consent to participate in this study.

Author contributions

XZ: design and program implementation of experimental ideas and methods. YD: preparation of experimental materials. CT: visualization of experimental results. SC: arrangement of experimental data. YW: verification of experimental design. MZ: experimental data analysis. QW: review and revision of papers. XL: experimental supervision and leadership. WS: review and revision of the first draft of the papers. All authors contributed to the article and approved the submitted version.

Funding

This study was supported by the Key-Area Research and Development Program of Guangdong Province, China (No. 2020B010165004), the Major Science and Technology Scheme under Key Medical Research Project of Shanxi Province (2021XM04), the National Natural Science Foundation of China (U22A2034, 62072452, and 61902386), the Shenzhen a Fundamental Research Program (JCYJ20200109110420626, JCYJ20200109110208764, and JCYJ20200109115201707), and the Natural Science Foundation of Guangdong (2021A1515012604).

Acknowledgments

We gratefully thank the reviewers for their constructive comments.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Schulz CJ, Böckler D, Krisam J, Geisbüsch P. Two-dimensional-three-dimensional registration for fusion imaging is noninferior to three-dimensional-three-dimensional registration in infrarenal endovascular aneurysm repair. J Vascular Surg. (2019) 70:2005–13. doi: 10.1016/j.jvs.2019.02.027

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Paudel DP, Habed A, Demonceaux C, Vasseur P. Robust and optimal registration of image sets and structured scenes via sum-of-squares polynomials. Int J Comput Vis. (2019) 127:415–36. doi: 10.1007/s11263-018-1114-2

CrossRef Full Text | Google Scholar

3. Sui Rd, Wang Cg, Han Dw, Zhang Xq, Li Q, Xu Cf, et al. Application of computed tomography angiography for evaluating clinical morphology in intracranial aneurysms-monocentric study. J Int Med Res. (2020) 48:0300060519894790. doi: 10.1177/0300060519894790

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Chen B, Jungstedt E. Fast and large-converge-radius inverse compositional Levenberg-Marquardt algorithm for digital image correlation: principle, validation, and open-source toolbox. Optics Lasers Eng. (2022) 151:106930. doi: 10.1016/j.optlaseng.2021.106930

CrossRef Full Text | Google Scholar

5. Ou C, Qian Y, Chong W, Hou X, Zhang M, Zhang X, et al. A deep learning-based automatic system for intracranial aneurysms diagnosis on three-dimensional digital subtraction angiographic images. Med Phys. (2022) 49:7038–53. doi: 10.1002/mp.15846

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Hirotaki K, Moriya S, Akita T, Yokoyama K, Sakae T. Image preprocessing to improve the accuracy and robustness of mutual-information-based automatic image registration in proton therapy. Phys Med. (2022) 101:95–103. doi: 10.1016/j.ejmp.2022.08.005

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Lu G, Zhou L. Localization of prostatic tumor's infection based on normalized mutual information MRI image segmentation. J Infect Public Health. (2021) 14:432–6. doi: 10.1016/j.jiph.2019.08.011

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Zhang C, Yao M, Chen W, Zhang S, Chen D, Wu Y. Gradient descent optimization in deep learning model training based on multistage and method combination strategy. Security Commun Netw. (2021) 2021:9956773. doi: 10.1155/2021/9956773

CrossRef Full Text | Google Scholar

9. Chen J, Lu M, Chen X, Chen J, Chen L. A spectral gradient difference based approach for land cover change detection. ISPRS J Photogram Remote Sens. (2013) 85:1–12. doi: 10.1016/j.isprsjprs.2013.07.009

CrossRef Full Text | Google Scholar

10. Guo C, Wang H, Jian T, He Y, Zhang X. Radar target recognition based on feature Pyramid fusion lightweight CNN. IEEE Access. (2019) 7:51140–9. doi: 10.1109/ACCESS.2019.2909348

CrossRef Full Text | Google Scholar

11. Bashir I, Ahmed F, Ahmad J, Boulila W, Alharbi N. A secure and robust image hashing scheme using Gaussian pyramids. Entropy. (2019) 21:1132. doi: 10.3390/e21111132

CrossRef Full Text | Google Scholar

12. Li S, Hao Q, Kang X, Benediktsson JA. Gaussian pyramid based multiscale feature fusion for hyperspectral image classification. IEEE J Select Top Appl Earth Observat Remote Sens. (2018) 11:3312–24. doi: 10.1109/JSTARS.2018.2856741

CrossRef Full Text | Google Scholar

13. Ravisankar P, Sree Sharmila T, Rajendran V. Acoustic image enhancement using Gaussian and laplacian pyramid-a multiresolution based technique. Multimedia Tools Appl. (2018) 77:5547–61. doi: 10.1007/s11042-017-4466-7

CrossRef Full Text | Google Scholar

14. Suarez M, Brea VM, Fernandez-Berni J, Carmona-Galan R, Cabello D, Rodriguez-Vazquez A. Low-power CMOS vision sensor for Gaussian pyramid extraction. IEEE J Solid State Circ. (2016) 52:483–95. doi: 10.1109/JSSC.2016.2610580

CrossRef Full Text | Google Scholar

15. Nasihatkon B, Kahl F. Multiresolution search of the rigid motion space for intensity-based registration. IEEE Trans Pattern Anal Mach Intell. (2017) 40:179–91. doi: 10.1109/TPAMI.2017.2654245

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Spahr N, Thoduka S, Abolmaali N, Kikinis R, Schenk A. Multimodal image registration for liver radioembolization planning and patient assessment. Int J Comput Assist Radiol Surg. (2019) 14:215–25. doi: 10.1007/s11548-018-1877-5

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Tu B, Li N, Fang L, He D, Ghamisi P. Hyperspectral image classification with multi-scale feature extraction. Remote Sens. (2019) 11:534. doi: 10.3390/rs11050534

CrossRef Full Text | Google Scholar

18. Postolka B, List R, Thelen B, Schütz P, Taylor WR, Zheng G. Evaluation of an intensity-based algorithm for 2D/3D registration of natural knee videofluoroscopy data. Med Eng Phys. (2020) 77:107–13. doi: 10.1016/j.medengphy.2020.01.002

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Frysch R, Pfeiffer T, Rose G. A novel approach to 2D/3D registration of X-ray images using Grangeat's relation. Med Image Anal. (2021) 67:101815. doi: 10.1016/j.media.2020.101815

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Brown M, Windridge D, Guillemaut JY. A family of globally optimal branch-and-bound algorithms for 2D-3D correspondence-free registration. Pattern Recogn. (2019) 93:36–54. doi: 10.1016/j.patcog.2019.04.002

CrossRef Full Text | Google Scholar

21. Baka N, Metz C, Schultz CJ, van Geuns RJ, Niessen WJ, van Walsum T. Oriented Gaussian mixture models for nonrigid 2D/3D coronary artery registration. IEEE Trans Med Imaging. (2014) 33:1023–34. doi: 10.1109/TMI.2014.2300117

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Nguyen V, Alves Pereira LF, Liang Z, Mielke F, Van Houtte J, Sijbers J, et al. Automatic landmark detection and mapping for 2D/3D registration with BoneNet. Front Vet Sci. (2022) 9:923449. doi: 10.3389/fvets.2022.923449

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Khoo Y, Kapoor A. Non-iterative rigid 2D/3D point-set registration using semidefinite programming. IEEE Trans Image Process. (2016) 25:2956–70. doi: 10.1109/TIP.2016.2540810

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Akbari-Shandiz M, Lawrence RL, Ellingson AM, Johnson CP, Zhao KD, Ludewig PM. MRI vs CT-based 2D-3D auto-registration accuracy for quantifying shoulder motion using biplane video-radiography. J Biomech. (2019) 82:375–80. doi: 10.1016/j.jbiomech.2018.09.019

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Ambrosini P, Ruijters D, Niessen WJ, Moelker A, van Walsum T. Continuous roadmapping in liver TACE procedures using 2D-3D catheter-based registration. Int J Comput Assist Radiol Surg. (2015) 10:1357–70. doi: 10.1007/s11548-015-1218-x

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Meng C, Wang Q, Guan S, Sun K, Liu B. 2D-3D registration with weighted local mutual information in vascular interventions. IEEE Access. (2019) 7:162629–38. doi: 10.1109/ACCESS.2019.2905345

CrossRef Full Text | Google Scholar

27. Song S, Herrmann JM, Si B, Liu K, Feng X. Two-dimensional forward-looking sonar image registration by maximization of peripheral mutual information. Int J Adv Rob Syst. (2017) 14:1729881417746270. doi: 10.1177/1729881417746270

CrossRef Full Text | Google Scholar

28. Yang T, Tang Q, Li L, Song J, Zhu C, Tang L. Nonrigid registration of medical image based on adaptive local structure tensor and normalized mutual information. J Appl Clin Med Phys. (2019) 20:99–110. doi: 10.1002/acm2.12612

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Lee S, Park YT, d'Auriol BJ, et al. A novel feature selection method based on normalized mutual information. Appl Intell. (2012) 37:100–20. doi: 10.1007/s10489-011-0315-y

CrossRef Full Text | Google Scholar

30. Wang Y, Ji J, Liang P. Feature selection of fMRI data based on normalized mutual information and fisher discriminant ratio. J X-ray Sci Technol. (2016) 24:467–75. doi: 10.3233/XST-160565

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Taş E, Memmedl I M. Near optimal step size and momentum in gradient descent for quadratic functions. Turkish J Math. (2017) 41:110–21. doi: 10.3906/mat-1411-51

CrossRef Full Text | Google Scholar

32. Qiao Y, Lelieveldt BP, Staring M. An efficient preconditioner for stochastic gradient descent optimization of image registration. IEEE Trans Med Imaging. (2019) 38:2314–25. doi: 10.1109/TMI.2019.2897943

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Audrito G, Damiani F, Viroli M. Optimal single-path information propagation in gradient-based algorithms. Sci Comput Program. (2018) 166:146–66. doi: 10.1016/j.scico.2018.06.002

CrossRef Full Text | Google Scholar

34. Kamsing P, Torteeka P, Yooyen S. An enhanced learning algorithm with a particle filter-based gradient descent optimizer method. Neural Comput Appl. (2020) 32:12789–800. doi: 10.1007/s00521-020-04726-9

CrossRef Full Text | Google Scholar

35. Hu Q, Guo M, Yu D, Liu J. Information entropy for ordinal classification. Sci China Inf Sci. (2010) 53:1188–200. doi: 10.1007/s11432-010-3117-7

CrossRef Full Text | Google Scholar

36. Zhang Z, Henderson T, Karaman S, Sze V. FSMI: Fast computation of Shannon mutual information for information-theoretic mapping. Int J Rob Res. (2020) 39:1155–77. doi: 10.1177/0278364920921941

CrossRef Full Text | Google Scholar

Appendix

$\begin{array}{ll}NMG(A,B)={\alpha }_1·I_{NMI}(A,B)+{\alpha }_2·I_{GD}(A,B)& (1)\end{array}$

$\begin{array}{ll}H(A)=-{\displaystyle \sum _{i=1}^N}p(A_i){log}_p(A_i)& (2)\end{array}$

$\begin{array}{ll}H(A,B)=-{\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{j=1}^N}p(A_i,B_j){log}_p(A_i,B_j)& (3)\end{array}$

$\begin{array}{ll}MI(A,B)=H(A)+H(B)-H(A,B)& (4)\end{array}$

$\begin{array}{ll}NMI(A,B)=\frac{H(A)+H(B)}{H(A,B)}=1+\frac{MI(A,B)}{H(A,B)}& (5)\end{array}$

$\begin{array}{ll}GD(A,B)={\displaystyle \sum _{m,n}}\frac{{\sigma }_v}{{\sigma }_v+{(I_{dV}(m,n))}^2}+{\displaystyle \sum _{m,n}}\frac{{\sigma }_h}{{\sigma }_h+{(I_{dH}(m,n))}^2}& (6)\end{array}$

$\begin{array}{ll}{I}_{dV}(m,n)=\frac{dA}{dm}-s\ast \frac{dB}{dm}& (7)\end{array}$

$\begin{array}{ll}{I}_{dH}(m,n)=\frac{dA}{dn}-s\ast \frac{dB}{dn}& (8)\end{array}$

$\begin{array}{ll}{g}_k(i,j)={\displaystyle \sum _m}{\displaystyle \sum _n}(W(m,n)\times g_{k-1}(2i+m,2j+n))& (9)\end{array}$

$\begin{array}{ll}{\theta }_{k+1}={\theta }_k-{\alpha }_k{d}_k& (10)\end{array}$