Fast color point cloud registration based on virtual viewpoint image

With the increase of point cloud scale, the time required by traditional ICP-related point cloud registration methods increases dramatically, which cannot meet the registration requirements of large-scale point clouds. In this paper, a fast registration technique for large scale point clouds based on virtual viewpoint image generation is studied. Firstly, the projection image of color point cloud is generated by virtual viewpoint. Then, the feature is extracted based on ORB and the rotation and translation matrix is calculated. The experimental results show that the registration time of the proposed method is about 1s when the size of the point cloud is from 300,000 to 2 million, which is improved by 17–258 times compared with the traditional ICP registration method, and the registration error is reduced by 80% from ICP 5.0 to 1.0. This paper provides a new idea and method for large-scale color point cloud registration.

Introduction

The optical 3D sensor based on structured light illumination is affected by the limitation of the depth of field and the range of field of view of the lens as well as the self-occlusion of the object. It needs to conduct multi-angle point cloud imaging of the measured object, and then unify the point cloud data to the global coordinate system through point cloud registration [1, 2], so as to obtain the complete 3D point cloud data of the target object.

There are four main methods for point cloud registration: direct registration, feature-based, deep learning and image-based. 1) Based on point cloud direct registration, there are mainly ICP [3, 4], GO-ICP [5, 6] and other algorithms. Theoretically, the accuracy is high, but when the scale of the point cloud is large, the computation is too large, and the time is too long. 2) Feature-based point cloud registration mainly includes PFH, FPFH, 4PCS and super-4PCS [7]. It is mainly limited by the fact that the feature extraction algorithm cannot completely and accurately describe the point cloud features, and the high computational amount and complexity of some feature extraction and description algorithms affect the practical application of the registration algorithm. 3) Point cloud registration based on deep learning [8–10] mainly includes PointNet, PointNet+, PointNet++, PointNetLk [8] and other network structures. Limited by the point cloud disorder and no spatial structure, it cannot be sampled on regular grids and processed as image pixels, and lacks color information, deep learning can generally only be used to process the relative positions of points, which limits the application of this technology. 4) Based on the point cloud registration of intermediate media such as image [3, 4, 11–13], the projection image, optical image or depth map is used to convert the 3D point cloud registration problem to the 2D image matching problem, and the transformation parameters between point clouds are indirectly calculated by using the 2D image registration algorithm. Such algorithms often have high timeliness and matching accuracy. However, most applications do not have corresponding optical images or intensity images, so there are certain limitations.

In this paper, large-scale color point cloud registration is realized based on ORB feature extraction [14] and matching of virtual viewpoint projection images. Firstly, the color image is obtained by the color point cloud projection based on the virtual viewpoint. Then, the ORB is used to extract the image features and registration, and the rotation and translation matrix between the virtual viewpoint images is solved. The rotation and translation matrix are used to register the point cloud. Experimental results show that, compared with ICP method, the registration can be achieved with high precision without the limitation of initial pose and rotation at any Angle. At the same time, different from the ICP method, the registration time increases with the increase of the size of the point cloud, the registration time of the proposed method is about 1s, and the registration accuracy is 20% of the ICP registration accuracy when the scale of the point cloud is 300,000 ∼2 million. The experimental results confirm that the virtual viewpoint based large-scale point cloud registration technology proposed in this paper can achieve high speed and high precision point cloud registration. It provides a new idea for the registration of large-scale point clouds.

ORB color point cloud fast registration technology based on virtual viewpoint

Rigid point cloud registration is essentially for the transformation matrix between the two group of point cloud clusters [R|t], in which R is rotation matrix, t is translation matrix. This paper proposes a rapid image registration technique based on virtual view images, will first XYZ color point cloud images projected into 2d images in the Z axis direction, and then use the ORB in the 2D image feature extraction algorithm for matching feature points, then according to the matching feature points by using SVD decomposition or least square method to solve R, t matrix. Finally using the transformation matrix [R|t] to coordinate transformation of color point cloud, so as to realize two clusters of color of point cloud registration.

Figures 1A,B show two clusters of colored point clouds before registration, and the scale of point clouds is about 500,000 points. The point cloud of Figure 1B is obtained by rotation and translation of the point cloud of Figure 1A. Two group of point cloud clusters to get Z axis direction projection direction of the virtual view image (Figures 2A,B), according to the virtual view + Z direction projection image, respectively, using the ORB to feature extraction and feature matching to get the rotation of the virtual view image translation matrix [R|t], registration results as shown in Figure 2C.

FIGURE 1

FIGURE 1. Registration of the first two clusters of point clouds.

FIGURE 2

FIGURE 2. Virtual view imaging and feature extraction and matching of two cluster point clouds. (A) the virtual view image of Figure 1A, (B) the virtual view image of Figure 1B.

The virtual view image to extract the [R|t] is applied to the second point cloud clusters, the second point cloud clusters rotating shift to make it and the first bunch of point cloud registration and two clusters of point cloud registration after image, the Figure 3A is the difference in two clusters of point cloud registration after color figure, one orange said first point cloud clusters, green said the second point cloud clusters. Figure 3B is the color result after the fusion of two clusters of point clouds. Direct observation can show that the point cloud registration method based on virtual viewpoint proposed in this paper can realize the registration of two clusters of point clouds.

FIGURE 3

FIGURE 3. Registration results of two clusters of point clouds (A) registration results of point clouds (B) fusion results of two clusters of point clouds.

Comparison of ICP and ORB registration results and registration time under different angle transformation

Taking the collected color point cloud as the reference point cloud data A, the point cloud contains A total of 464,966 color points, and each data packet contains three-dimensional spatial coordinates XYZ and color data RGB. The reference point cloud data were rotated 30° counterclockwise around the Z-axis, and the random translation of XYZ was increased within the range of ±5 mm to generate color point cloud B to be registered. The reference point cloud data A was rotated 12 times in total, with the rotation Angle ranging from 30° to 360°, and 12 groups of color point clouds to be matched were generated. The transformation parameters of 12 groups of point clouds to be registered are shown in the following Table 1 and in Figure 4.

TABLE 1

TABLE 1. Rotation and translation parameters of point cloud B.

FIGURE 4

FIGURE 4. Comparison of results: (A) point cloud registration time,(B) registration error.

Will then respectively stay 12 groups B using traditional ICP registration color point cloud point cloud registration algorithm and proposed in this paper, based on the virtual view of the ORB image feature point cloud registration algorithm with A benchmark color point cloud registration, and calculate the registration after the color point cloud B′ and benchmark color point cloud registration results: the mean absolute error of specific formula is as follows:

$\mathrm{\Delta }=\frac{1}{N}{\displaystyle \sum }_{i=1}^N\left(\left|x_i-x_0\right|+\left|y_i-y_0\right|+\left|z_i-z_0\right|\right)$

Where xi, yi, zi is the spatial coordinates of the color point cloud after registration. and x0, y0, z0 is the base color point cloud space coordinate. N = 464,966 indicates the total number of point clouds in a point cloud cluster.

The registration time and error of 12 groups of color point cloud B to be registered and reference color point cloud A are shown in Table 2.

TABLE 2

TABLE 2. Comparison of registration time and error of different rotation and translation matrices.

Compared with the traditional ICP method, the registration time of the proposed method is about 3% of the traditional ICP method, and the registration time is about 1s for the color point cloud with a scale of 500,000 points. In terms of registration accuracy, the error of the traditional ICP registration method increases with the increase of the point cloud offset Angle. When the two-point clouds rotate 180°, the error reaches the maximum, which is about 4 times of the error when the two group point clouds rotate 30°. However, the registration error of the proposed method is less affected by the rotation Angle, and the average error is 2 mm at most, which is 2% of the traditional ICP registration error. The registration result is shown in Figure 5, where Figure 5A is the registration result of traditional ICP, and Figure 5B is the registration result of the proposed method.

FIGURE 5

FIGURE 5. Registration results of two cluster point clouds, (A) ICP (B) Our method.

Registration results and registration times of ICP and ORB point clouds of different sizes

A cluster of color point cloud with 2.04 million size was collected by the color three-dimensional imaging equipment with high precision acquisition mode, and four groups of reference color point cloud A of different sizes were obtained after simplification. By rotating 3° counterclockwise and adding random displacements within a range of ±5 mm in the XYZ direction, four groups of color point cloud data B with different sizes to be registered were obtained. The proposed method and ICP are used for point cloud registration test, and the test results are shown in Table 3.

TABLE 3

TABLE 3. Comparison of registration time and error of different scales.

The registration results of four different scale point clouds are shown in Figure 6, where Figure 6A is the registration results of the proposed method, and Figure 6B is the registration results of ICP. From the comparison results, it can be seen that with the increase of the scale of point clouds, the time required for the registration of traditional ICP point clouds is increasing continuously, from 18.82 s for 300,000 point clouds to 227.65 s for 2 million point clouds. However, the registration time of the proposed method has little correlation with the scale of the point cloud, and the registration time is about 1s. From the perspective of registration error, for small Angle point cloud rotation of 3°, ICP has a high accuracy, which is not correlated with the scale of point cloud, and the overall error is about 5 mm. However, the registration accuracy of the method proposed in this paper is also not correlated with the scale of the point cloud, and the overall error is about 1mm, which is 20% of the traditional ICP.

FIGURE 6

FIGURE 6. Registration results of different scale point clouds, (A) Our method, (B) ICP.

Discussion

Limited by the field of view of 3D optical imaging equipment and the self-occlusion of the target to be measured, it is necessary to image the target several times, and then achieve coordinate unification through point cloud registration. As the precision of data becomes higher and higher, the scale of point cloud becomes larger and larger. With the increase of the scale of point cloud, the registration time of existing global registration and semi-global registration algorithms becomes longer and longer, which limits the rapid development and application of optical 3D imaging technology.

In this paper, a large-scale point cloud registration algorithm is proposed based on ORB feature extraction and matching of virtual viewpoint projection images. Firstly, the color image is obtained by the color point cloud projection based on the virtual viewpoint. Then, the ORB is used to extract the image features and registration, and the rotation and translation matrix between the virtual viewpoint images is solved. The rotation and translation matrix is used to register the point cloud. Compared with ICP method, it is not limited by the initial pose and can achieve high precision registration with any rotation Angle. At the same time, the registration time of the proposed method is about 1 s, and the registration accuracy is 20% of the ICP registration accuracy when the scale of point clouds ranges from 300,000 to 2 million. The experimental results confirm that the virtual viewpoint based large-scale point cloud registration technology proposed in this paper can achieve high speed and high precision point cloud registration. It provides a new idea for the registration of large-scale point clouds and expands the application scope and application scenarios of large-scale point clouds.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

Funding

S&T Program of Hebei (20371802D).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Li J, Zhan J. Review on 3D point cloud registration method. JOURNAL IMAGE GRAPHICS (2022) 27(02):349–67. doi:10.11834/jig.210243

CrossRef Full Text | Google Scholar

2. Sui Y-L. Research on large-scale 3D point cloud registration technology. Zheng Zhou: PLA Strategic Support Force Information Engineering University (2020).

Google Scholar

3. Su B-Y, Han W, Peng Y-S, Sheng M. 4D-ICP point cloud registration method for RGB-D data. JOURNAL NANJING UNIVERSITY (NATURAL SCIENCE) (2018) 54(04):829–37. doi:10.13232/j.cnki.jnju.2018.04.019

CrossRef Full Text | Google Scholar

4. Men H, Gebre B, Pochiraju K. Color point cloud registration with 4D ICP algorithm. In: 2011 IEEE International Conference on Robotics and Automation; 09-13 May 2011; Shanghai, China (2011). doi:10.1109/ICRA.2011.5980407

CrossRef Full Text | Google Scholar

5. Lu W, Wan G, Zhou Y, Fu X, Yuan P, Song S. DeepICP: An end-to-end deep neural network for 3D point cloud registration. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV); 27 October 2019 - 02 November 2019; Seoul, Korea (South) (2019). doi:10.1109/ICCV.2019.00010

CrossRef Full Text | Google Scholar

6. Yang J, Li H, Campbell D, Jia Y. Go-ICP: A globally optimal solution to 3D ICP point-set registration. IEEE Trans Pattern Anal Mach Intell (2015) 38(11):2241–54. doi:10.1109/TPAMI.2015.2513405

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Mellado N, Mitra N, Aiger D. Super4pcs: Fast global pointcloud registration via smart indexing. COMPUT GRAPH FORUM (2014) 5(33):205–15. doi:10.1111/cgf.12446

CrossRef Full Text | Google Scholar

8. Aoki Y, Goforth H, Srivatsan RA. PointNetLK: Robust & efficient point cloud registration using PointNet (2019). Available at: https://arxiv.org/abs/1903.05711.

Google Scholar

9. Wang Y, Solomon JM. PRNet: Self-Supervised learning for partial-to-partial registration (2019). Available at: https://arxiv.org/abs/1910.12240.

Google Scholar

10. Pais GD, Ramalingam S, Govindu VM, Nascimento JC, Chellappa R, Miraldo P. 3DRegNet: A deep neural network for 3D point registration (2020).

Google Scholar

11. Bao Z, Xie L, Lu N-N, et al. A point cloud registration algorithm using depth image. NAVIGATION OF CHINA (2019) 42(02):8–11.

Google Scholar

12. Zhong S, Li Y, Liu Z, Xie Z, Chen J, Wang W, et al. Robust rigid point cloud registration via RGB-D images. J Computer-Aided Comput Graphics (2022) 34(01):25–35. doi:10.3724/SP.J.1089.2022.18823

CrossRef Full Text | Google Scholar

13. Sheng M, Peng Y-S, Su B-Y, et al. RGBD point cloud registration based on feature similarity. JOURNAL GRAPHICS (2019) 40(05):829–34.

Google Scholar

14. Li Y-C. Research on 3D point cloud registration algorithm based on image features and ICP. Gui Yang: GuiZhou University (2020).

Google Scholar