A New Calibration Method of Stereo Line-Scan Cameras for 2D Coordinate Measurement

Camera calibration plays an important role in various applications including machine vision and optical measurement. In this paper, we propose a new calibration method of stereo line-scan cameras for 2D coordinate measurement using a simple calibration target rather than requiring complex calibration patterns or an auxiliary area-scan camera. Considering the viewing planes associated with the stereo line-scan cameras are coplanar, an imaging model combining perspective projection and lens distortion are established and can depict the relationship between the 2D world coordinate and its corresponding 1D pixel coordinates of the image pairs. A two-step calibration algorithm is proposed to obtain the optimal values of intrinsic, extrinsic and distortion parameters of stereo line-scan cameras. The performance of the proposed camera calibration method is evaluated on 2D coordinate measurement and the experimental results demonstrate that the proposed method is more effective and accurate than the existing method.

Introduction

Line-scan (also called linear or 1D) cameras are becoming widely used to various applications including machine vision and optical measurement, such as agricultural robot navigation [1], railway catenary inspection [2], surface defect detection [3], 2D coordinate measurement [4] and 3D shape measurement [5–7]. There are two types of line-scan cameras: single-line camera and multiline camera [8], which can be considered as a particular area-scan (also called matrix or 2D) camera consisting of a single row or multiple rows of pixel elements [9]. Compared with area-scan cameras, line-scan cameras can provide higher spatial resolution, higher imaging rates (up to 16384 pixels at 300 kHz) and lower manufacturing cost [9, 10].

In the camera-based applications, camera calibration plays an important role and is a necessary step in order to extract metric information from 2D images [8, 11]. The key of calibration is to find out the initial values of intrinsic and extrinsic camera parameters, which demonstrate the mapping relationship between the 2D coordinates in the image and the 3D coordinates in the object world [12] and will be refined by a non-linear optimization technique to obtain more accurate calibration parameters. Conventional calibration methods for area-scan cameras [11, 13] are not applicable to the line-scan cameras because of their different imaging principles [8]. For line-scan cameras, the calibration process is more complicated and several calibration methods have been presented in the last decades. The authors have performed the investigation and reviewed on line-scan camera calibration methods [9]. Generally, these methods can be classified into two groups: static imaging calibration and dynamic imaging calibration [8, 14].

Static imaging calibration employs the patterns on the calibration target comprising several feature lines. The 3D coordinates of intersection points that can be found by solving the viewing plane equation of the line-scan camera and the feature lines equations are applied for calibration. It is worth noting that each row data of the captured image is almost the same because there is no relative motion between the camera and the calibration target. Horaud et al. [15] proposed a multiline calibration method where cross-ratio invariance is used to compute the intersection points. However, it is necessary to move the calibration target along the $Y_w$ and/or $Z_w$ axises with known increments. In order to relax these constraints, Luna et al. [16] presented a novel calibration method using a 3D pattern consisting of two parallel planes with the lines like in [15]. Lilienblum et al. [17] designed a more complex 3D pattern with vertical and diagonal straight lines taken together in several coded targets on two levels for camera calibration. Li et al. [18] gave a alterative 3D pattern comprising two orthogonal planes where there are two groups of parallel feature lines; this pattern was introduced by Su et al. [1] to calibrate the hyperspectral line-scan camera in agricultural robots. Song et al. [19] designed a new calibration target with a coded eight trigrams pattern which has a circular symmetry structure to get more effective coded information from different directions. Niu et al. [20] presented another 3D right-angle stereo target based on concentric rings. Liao et al. [14] designed a stereo target with hollow stripes to solve the eccentricity error problem. These methods [1, 14, 16–20] do not require the precise target translation but need high-precision 3D calibration patterns. Instead of using the 3D patterns, Yao et al. [21] proposed an innovative calibration method using a stationary planar pattern which consists of repeated vertical and slanted lines and constructed a 2D calibration framework by projecting original 1D data onto its orthogonal direction. Li et al. [10] presented a flexible calibration technique where the line-scan camera is rigidly coupled to a calibrated area-scan camera to establish a pair of stereo cameras. The stereo cameras are required to observe a specially designed planar pattern shown at a few different orientations; thus the 3D feature points for line-scan camera calibration can be obtained by the extrinsic parameters of area-scan camera. Similarly, Sun et al. [22] also used a customized planar pattern and a precalibrated area-scan camera to get the 3D coordinates of the feature points. The area-scan cameras in [10, 22] remove the necessity of precise pattern movement; however, it greatly increases the cost of the calibration set-up.

Dynamic imaging calibration estimates the line-scan camera parameters by a series of scanning operations. More generally, a linear relative motion with constant velocity between the camera and the calibration target is typically considered. Draréni et al. [23] utilized a controllable linear stage to translate the line-scan camera along the $Y_c$ axis, when the camera is watching a planar checkerboard pattern; hence a 2D scan image is captured. If the planar pattern is almost parallel to the image plane, however, this method can not work since it entails dividing by elements of rotation matrices. In order to avoid this problem, Donné et al. [24] improved the method [23] by dividing the homography $H$ by its end element. Similar to [23], Hui et al. [25, 26] also fixed the line-scan camera to a programmable linear stage. The camera scans a 3D calibration pattern along an arbitrary direction whose three motion parameters are added to the dynamic imaging model and does not require strict motion along the $Y_c$ axis like in [23]. Furthermore, Hui et al. [27] used an auxiliary area-scan camera and a 2D checkerboard pattern to calibrate the line-scan camera instead of using the linear stage and 3D pattern. Steger and Ulrich [28] proposed a versatile camera model for line-scan cameras with telecentric lenses using a planar calibration target. The model takes into account the telecentric lens distortions and supports arbitrary positions of the linear sensor with respect to the optical axis.

As for the stereo line-scan cameras, 2D coordinates or 3D shape can be calculated by the well-known triangulation method when they are properly calibrated. A few stereo line-scan camera systems for industrial applications have been reported in the literature. Ma et al. [4] presented a 2D coordinate measurement system based on stereo line-scan cameras which are calibrated by using several rods perpendicular to a flat plate at some special locations. The calibration is easy to implement due to its simple target, but its major defects are quite obvious: time consuming and low accuracy caused by occlusion imaging. Sun et al. [5] developed a high-speed 3D shape measurement system for moving objects. The stereo line-scan cameras can be calibrated by their static imaging calibration method [22] or [6]. Lilienblum et al. [29] extended the calibration method [17] for 3D measurement system consisting of stereo line-scan cameras. Zhan et al. [2] proposed an accurate and efficient measurement approach for railway catenary geometry parameters by using stereo line-scan cameras. Liao et al. [7] gave a dense 3D point cloud measurement system comprising dual line-scan cameras and a matching strategy to expand the depth measurement range. Most recently, Steger and Ulrich [30] introduced their line-scan camera model [28] to the multi-view case with an arbitrary number of cameras for 3D surface reconstruction.

In this paper, based on the work of Ma et al. [4] and our previous work [9], we propose a new calibration method of stereo line-scan cameras for 2D coordinate measurement. The rest of this paper is organized as follows. In section 2, we establish an imaging model of stereo line-scan cameras that serves as the basis for our method. In section 3, the proposed calibration method for intrinsic, extrinsic and distortion parameters is described in detail. Experimental results on 2D coordinate measurement are performed and discussed in section 4. Finally, we conclude our method in section 5.

Our main contributions are summarized as follows:

1) We propose a static imaging calibration method of stereo line-scan cameras using a simple calibration target. Complex calibration patterns/targets are not required to calibrate cameras; auxiliary imaging devices, such as area-scan camera, do not need to aid the calibration.

2) We establish an imaging model for stereo line-scan cameras that are coplanar aligned. Lens distortion is taken into consideration to improve the model performance.

3) We design a calibration target with simple structure and low cost. The target comprises a group of magnetic rods perpendicular to a flat iron plate with a coordinate map which can give the world coordinates.

4) The time required for the calibration process is greatly reduced with the help of the simple target, translation and a two-step algorithm that determines the camera parameters using the images of a group of rods.

Imaging Model of Stereo Line-Scan Cameras

Stereo Line-Scan Camera System

The stereo line-scan camera system has various arrangement modes [7]. For our system, the side-by-side mode is chosen. As depicted in Figure 1A, the system mainly includes two line-scan cameras (Dalsa, spL2048-140km) mounted lens (Nikon, AF Nikkor 14 mm f/2.8D), two line lasers, a industrial computer with two frame grabbers (Dalsa, OR-X4C0-XPF00), several mechanical and calibration devices. Like most stereo line-scan camera systems [2, 4–7, 29], our system also requires the two line-scan cameras to be coplanar aligned manually using a suitable mechanical device so that the two viewing planes determined by the projection center and the pixel elements are approximately coplanar. Consequently, the measurement field is the overlapping area of the two planes.

FIGURE 1

FIGURE 1. Stereo line-scan camera system and imaging model. (A) Stereo line-scan camera system and (B) Imaging model of stereo line-scan cameras.

Imaging Model

It is necessary for an imaging model to define the coordinate systems in order to describe the intrinsic and extrinsic parameters. As shown in Figure 1B, the world coordinate system, the two camera coordinate systems and the two pixel coordinate systems are denoted by $O_w-X_w{Y}_w{Z}_w$, $O_{c1}-X_{c1}{Y}_{c1}{Z}_{c1},O_{c2}-X_{c2}{Y}_{c2}{Z}_{c2},o_1-u_1,o_2-u_2$, respectively. $\left[R,T\right]$ are extrinsic parameters between the camera 1 and 2 coordinate systems, and $\left[R_1,T_1\right]$ and $\left[R_2,T_2\right]$ are extrinsic parameters between the world coordinate system and the camera coordinate system, where $R,R_1,R_2$ and $T,T_1,T_2$ are the rotation and translation matrixes, respectively.

For the line-scan camera, perspective projection is only satisfied along the pixel elements direction, i.e., $u_1$ and $u_2$. According to perspective projection [2, 11], the relationship between a point $P_{ck}\left(k=\mathrm{1,2}\right)$ on the viewing plane and its image points $p_k\left(u_k\right)$ can be described as

$Z_{ck}\left[\begin{array}{c}{u}_k\\ 1\end{array}\right]=\left[\begin{array}{ccc}{\alpha }_k& u_{k0}& 0\\ 0& 1& 0\end{array}\right]\left[\begin{array}{c}{X}_{ck}\\ Z_{ck}\\ 1\end{array}\right](1)$

where ${\alpha }_k$ and $u_{k0}$ are the intrinsic parameters and denote the focal length in pixel units and the coordinate of the principle point of the camera k, respectively; $\left(X_{ck},Z_{ck}\right)$ are the coordinates of $P_{ck}$ reference to the camera k coordinate systems, respectively.

Because a world point $P_w$ only appears on the viewing planes can be imaged by the stereo line-scan cameras, for convenience, the $O_w-X_w{Y}_w$ is required to coincide with the viewing planes. Therefore, $\left(X_w,Y_w\right)$ denote the 2D coordinates in the measurement field. Furthermore, the world coordinate system and the camera k coordinate system can be related in the following ways

$\begin{array}{c}\left[\begin{array}{c}{X}_{ck}\\ Y_{ck}\\ Z_{ck}\\ 1\end{array}\right]=\left[\begin{array}{cc}{R}_k& T_k\\ \mathbf{0}& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ Z_w\\ 1\end{array}\right]=\left[\begin{array}{cccc}-\sin {\theta }_k& \cos {\theta }_k& 0& t_{kx}\\ 0& 0& {\left(-1\right)}^{k+1}& 0\\ {\left(-1\right)}^{k+1}\cos {\theta }_k& {\left(-1\right)}^{k+1}\sin {\theta }_k& 0& t_{kz}\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ 0\\ 1\end{array}\right]\\ \Rightarrow \left[\begin{array}{c}{X}_{ck}\\ Z_{ck}\\ 1\end{array}\right]=\left[\begin{array}{ccc}-\sin {\theta }_k& \cos {\theta }_k& t_{kx}\\ {\left(-1\right)}^{k+1}\cos {\theta }_k& {\left(-1\right)}^{k+1}\sin {\theta }_k& t_{kz}\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ 1\end{array}\right]\end{array}(2)$

where ${\theta }_k$ is a rotation angle and ${\left[t_{kx}{t}_{kz}\right]}^T$ is a translation vector.

Taking Eq. 2 into Eq. 1, we can get

$\{\begin{array}{l}{Z}_{c1}\left[\begin{array}{c}{u}_1\\ 1\end{array}\right]=\left[\begin{array}{ccc}-{\alpha }_1\sin {\theta }_1+u_{10}\cos {\theta }_1& {\alpha }_1\cos {\theta }_1+u_{10}\sin {\theta }_1& {\alpha }_1{t}_{1x}+u_{10}{t}_{1z}\\ \cos {\theta }_1& \sin {\theta }_1& t_{1z}\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ 1\end{array}\right]=\left[\begin{array}{ccc}{h^{\prime }}_{11}& {h^{\prime }}_{12}& {h^{\prime }}_{13}\\ {h^{\prime }}_{21}& {h^{\prime }}_{22}& {h^{\prime }}_{23}\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ 1\end{array}\right],\\ Z_{c2}\left[\begin{array}{c}{u}_2\\ 1\end{array}\right]=\left[\begin{array}{ccc}-{\alpha }_2\sin {\theta }_2-u_{20}\cos {\theta }_2& {\alpha }_2\cos {\theta }_2-u_{20}\sin {\theta }_2& {\alpha }_2{t}_{2x}+u_{20}{t}_{2z}\\ -\cos {\theta }_2& -\sin {\theta }_2& t_{2z}\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ 1\end{array}\right]=\left[\begin{array}{ccc}{h^{″}}_{11}& {h^{″}}_{12}& {h^{″}}_{13}\\ {h^{″}}_{21}& {h^{″}}_{22}& {h^{″}}_{23}\end{array}\right]\left[\begin{array}{c}{X}_w\\ Y_w\\ 1\end{array}\right]\end{array}(3)$

By substituting the second equation into the first equation of (3) and substituting the fourth equation into the third equation of (3), Eq. 3 can be transformed into the following descriptions

$\{\begin{array}{l}{X}_w{h^{\prime }}_{11}+Y_w{h^{\prime }}_{12}+{h^{\prime }}_{13}-u_1{X}_w{h^{\prime }}_{21}-u_1{Y}_w{h^{\prime }}_{22}-u_1{h^{\prime }}_{23}=0,\\ X_w{h^{″}}_{11}+Y_w{h^{″}}_{12}+{h^{″}}_{13}-u_2{X}_w{h^{″}}_{21}-u_2{Y}_w{h^{″}}_{22}-u_2{h^{″}}_{23}=0\end{array}(4)$

Obviously, the imaging model (3) or (4) can depict the relationship between the 2D world coordinate $\left(X_w,Y_w\right)$ and its corresponding 1D pixel coordinates ($u_1$ and $u_2$) of the image pairs.

Lens Distortion Model

It is easy to see that the imaging model (3) or (4) is linear. However, the actual imaging is a nonlinear process inevitably affected by different kinds of lens distortions [2]. To improve the model performance and measurement accuracy, lens distortion is taken into consideration.

For the area-scan cameras, the mathematical expressions for three types of lens distortions are as follows [9, 31].

$\{\begin{array}{ll}{\tilde{x}}^{\prime }=\tilde{x}+\tilde{x}\left(k_1{\tilde{r}}^2+k_2{\tilde{r}}^4+k_3{\tilde{r}}^6\right)+2p_1\tilde{x}\tilde{y}+p_2\left(3{\tilde{x}}^2+{\tilde{y}}^2\right)+s_1\left({\tilde{x}}^2+{\tilde{y}}^2\right),& {\tilde{y}}^{\prime }=\tilde{y}+\tilde{y}\left(k_1{\tilde{r}}^2+k_2{\tilde{r}}^4+k_3{\tilde{r}}^6\right)+p_1\left({\tilde{x}}^2+3{\tilde{y}}^2\right)+2p_2\tilde{x}\tilde{y}+s_2\left({\tilde{x}}^2+{\tilde{y}}^2\right)\end{array}(5)$

where $\tilde{r}=\sqrt{{\tilde{x}}^2+{\tilde{y}}^2};\left(\tilde{x},\tilde{y}\right)=\left(X_c/Z_c,Y_c/Z_c\right)$ is a distortion-free normalized image coordinates; $\left({\tilde{x}}^{\prime },{\tilde{y}}^{\prime }\right)$ is the corresponding distorted normalized image coordinates; $\left(k_1,k_2,k_3\right),\left(p_1,p_2\right),\left(s_1,s_2\right)$ are the coefficients of radial, decentering and thin prism distortions, respectively.

Considering the specialty of line-scan camera [32], $\tilde{y}\approx 0$ and thus lens distortion model (5) can be simplified as

${\tilde{x}}^{\prime }=\tilde{x}+\left(s_1+3p_2\right){\tilde{x}}^2+k_1{\tilde{x}}^3+k_2{\tilde{x}}^5+k_3{\tilde{x}}^7(6)$

If we ignore the third radial distortion and set $k_0=s_1+3p_2$, Eq. 6 can be furthermore simplified to

${{\tilde{x}}^{\prime }}_k={\tilde{x}}_k+k_{k0}{\tilde{x}}_k^2+k_{k1}{\tilde{x}}_k^3+k_{k2}{\tilde{x}}_k^5(7)$

Considering the lens distortion, Eq. 1 can be rewritten as

$\left[\begin{array}{c}{u}_k\\ 1\end{array}\right]=\left[\begin{array}{cc}{\alpha }_k& u_{k0}\\ 0& 1\end{array}\right]\left[\begin{array}{c}\frac{X_{ck}}{Z_{ck}}\\ 1\end{array}\right]=\left[\begin{array}{cc}{\alpha }_k& u_{k0}\\ 0& 1\end{array}\right]\left[\begin{array}{c}{\tilde{x}}_k\\ 1\end{array}\right]\Rightarrow \left[\begin{array}{c}{u^{\prime }}_k\\ 1\end{array}\right]=\left[\begin{array}{cc}{\alpha }_k& u_{k0}\\ 0& 1\end{array}\right]\left[\begin{array}{c}{{\tilde{x}}^{\prime }}_k\\ 1\end{array}\right](8)$

Substituting Eq. 8 into Eq. 4, we can derive the real distorted imaging model

$\{\begin{array}{ll}{X}_w{h^{\prime }}_{11}+Y_w{h^{\prime }}_{12}+{h^{\prime }}_{13}-\left(u_1\mathrm{\prime }-{\alpha }_1{k}_{10}{\tilde{x}}_1^2-{\alpha }_1{k}_{11}{\tilde{x}}_1^3-{\alpha }_1{k}_{12}{\tilde{x}}_1^5\right)\left(X_w{h^{\prime }}_{21}+Y_w{h^{\prime }}_{22}+{h^{\prime }}_{23}\right)=0,& X_w{h^{″}}_{11}+Y_w{h^{″}}_{12}+{h^{″}}_{13}-\left(u_2\mathrm{\prime }-{\alpha }_2{k}_{20}{\tilde{x}}_2^2-{\alpha }_2{k}_{21}{\tilde{x}}_2^3-{\alpha }_2{k}_{22}{\tilde{x}}_2^5\right)\left(X_w{h^{″}}_{21}+Y_w{h^{″}}_{22}+{h^{″}}_{23}\right)=0\end{array}(9)$

Calibration Method

Calibration Target

As shown in Figure 2A, our designed calibration target comprises a group of magnetic rods (8 collinear feature points with 100 mm spacing) perpendicular to a flat iron plate. The plate is adjusted to be parallel to the viewing plane and pasted by a coordinate map which can give the 2D world coordinates $\left(X_w,Y_w\right)$ when the target translates. It can be seen from Figure 2A that the calibration target has a simple structure and auxiliary devices, such as linear stage and area-scan camera, do not need to assist the calibration. Therefore our target is low cost.

FIGURE 2

FIGURE 2. Calibration target and sub-pixel location. (A) Calibration target with a group of magnetic rods (B) Sub-pixel locations of rod centers in the image pairs.

Computing the Initial Values

A two-step algorithm is proposed to determine the optimal values of intrinsic, extrinsic and distortion parameters. Firstly, we will compute the initial values of intrinsic and extrinsic camera parameters by solving the imaging model (4) with given feature points. Secondly, the initial values will be refined by a non-linear optimization technique to greatly improve the calibration accuracy.

Given $n\left(n\ge 2\right)$ translations of the calibration target, the coefficients of $H_1={\left[{h^{\prime }}_{11},{h^{\prime }}_{12},{h^{\prime }}_{13},{h^{\prime }}_{21},{h^{\prime }}_{22},{h^{\prime }}_{23}\right]}^T$ and $H_2={\left[{h^{″}}_{11},{h^{″}}_{12},{h^{″}}_{13},{h^{″}}_{21},{h^{″}}_{22},{h^{″}}_{23}\right]}^T$ of Eq. 4 can be solved by the system of homogeneous equations

$W_k{H}_k={\mathbf{0}}^{\left(8n\right)}(10)$

with

$W_k=\left[\begin{array}{cccccc}{X}_w^{\left(1\right)}& Y_w^{\left(1\right)}& 1& -u_k^{\left(1\right)}{X}_w^{\left(1\right)}& -u_k^{\left(1\right)}{Y}_w^{\left(1\right)}& -u_k^{\left(1\right)}\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ X_w^{\left(8\right)}& Y_w^{\left(8\right)}& 1& -u_k^{\left(8\right)}{X}_w^{\left(8\right)}& -u_k^{\left(8\right)}{Y}_w^{\left(8\right)}& -u_k^{\left(8\right)}\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ X_w^{\left(8n\right)}& Y_w^{\left(8n\right)}& 1& -u_k^{\left(8n\right)}{X}_w^{\left(8n\right)}& -u_k^{\left(8n\right)}{Y}_w^{\left(8n\right)}& -u_k^{\left(8n\right)}\end{array}\right](11)$

where $u_k^{\left(1\right)},\cdots ,u_k^{\left(8\right)},\cdots ,u_k^{\left(8n\right)}$ can be obtained by a sub-pixel location algorithm that can detect the centers of 8 rods in $n$ captured images.

The matrix $W_k$ can be decomposed by using singular value decomposition (SVD). The solution to Eq. 10 is the eigenvector of $W_k^T{W}_k$ with the smallest eigenvalue. Since $H_k$ is only defined up to a scale factor, we can get a solution ${\tilde{H}}_k=H_k/H_k^{end}={\left[{\tilde{h}}_{k1},{\tilde{h}}_{k2},{\tilde{h}}_{k3},{\tilde{h}}_{k4},{\tilde{h}}_{k5},1\right]}^T$, where $H_k^{end}$ is the end element of $H_k$. The following expressions can be given by recalling that Eq. 3

$t_{kz}\left[\begin{array}{ccc}{\tilde{h}}_{k1}& {\tilde{h}}_{k2}& {\tilde{h}}_{k3}\\ {\tilde{h}}_{k4}& {\tilde{h}}_{k5}& 1\end{array}\right]=\left[\begin{array}{ccc}-{\alpha }_k\sin {\theta }_k+{\left(-1\right)}^{k+1}{u}_{k0}\cos {\theta }_k& {\alpha }_k\cos {\theta }_k+{\left(-1\right)}^{k+1}{u}_{k0}\sin {\theta }_k& {\alpha }_k{t}_{kx}+u_{k0}{t}_{kz}\\ {\left(-1\right)}^{k+1}\cos {\theta }_k& {\left(-1\right)}^{k+1}\sin {\theta }_k& t_{kz}\end{array}\right](12)$

Solving this equation, the intrinsic and extrinsic parameters can be obtained

$\{\begin{array}{c}\begin{array}{l}{t}_{kz}^2{\tilde{h}}_{k4}^2+t_{kz}^2{\tilde{h}}_{k5}^2=1\Rightarrow t_{kz}=\pm 1/\sqrt{{\tilde{h}}_{k4}^2+{\tilde{h}}_{k5}^2},\\ {\alpha }_k={\left(-1\right)}^k{t}_{kz}^2\left({\tilde{h}}_{k1}{\tilde{h}}_{k5}-{\tilde{h}}_{k2}{\tilde{h}}_{k4}\right),\\ u_{k0}=t_{kz}^2\left({\tilde{h}}_{k1}{\tilde{h}}_{k4}+{\tilde{h}}_{k2}{\tilde{h}}_{k5}\right),\\ {\theta }_k={\tan }^{-1}\left({\tilde{h}}_{k5}/{\tilde{h}}_{k4}\right),\\ t_{kz}{\tilde{h}}_{k3}={\alpha }_k{t}_{kx}+u_{k0}{t}_{kz}\Rightarrow t_{kx}=t_{kz}\left({\tilde{h}}_{k3}-u_{k0}\right)/{\alpha }_k,\end{array}\\ R_k=\left[\begin{array}{ccc}{\left(-1\right)}^k{t}_{kz}{\tilde{h}}_{k5}& {\left(-1\right)}^{k+1}{t}_{kz}{\tilde{h}}_{k4}& 0\\ 0& 0& {\left(-1\right)}^{k+1}\\ t_{kz}{\tilde{h}}_{k4}& t_{kz}{\tilde{h}}_{k5}& 0\end{array}\right],T_k=\left[\begin{array}{c}{t}_{kx}\\ 0\\ t_{kz}\end{array}\right]\end{array}(13)$

Obviously, according to the coordinate systems defined in Figure 1B, we can get $t_{kz}>0$.

Non-linear Optimization

In the initial calculation of the parameters, the distortion is neglected. In order to obtain more accurate calibration parameters, a non-linear optimization technique such as Levenberg-Marquardt algorithm is employed to refine the camera parameters. The goal is to find out the optimal intrinsic, extrinsic and distortion parameters which minimize the sum of squares of reprojection errors, i.e.,

$\underset{{\alpha }_k,u_{k0},R_k,T_k,k_{k0},k_{k1},k_{k2}}{\min }{\displaystyle \sum _{j=1}^{8n}{\Vert {u^{\prime }}_k^{\left(j\right)}-{\widehat{u^{\prime }}}_k^{\left(j\right)}\left(X_w^{\left(j\right)},Y_w^{\left(j\right)}\right)\Vert }^2}(14)$

where ${\widehat{u^{\prime }}}_k^j\left(X_w^j,Y_w^j\right)$ is a function which projects the world points $\left(X_w^j,Y_w^j\right)$ into the image points according to the real distorted imaging model (9). For the initial values of distortion parameters, we may take $k_{k0}=k_{k1}=k_{k2}=0$ since the lens distortion is usually small and close to zero.

Once the parameters of two line-scan cameras are respectively refined, we can furthermore minimize the sum of squares of reprojection errors for each camera, i.e.,

$\underset{{\alpha }_k,u_{k0},R_k,T_k,R,T,k_{k0},k_{k1},k_{k2}}{\min }{\displaystyle \sum _{k=1}^2{\displaystyle \sum _{j=1}^{8n}{\Vert {u^{\prime }}_k^{\left(j\right)}-{\widehat{u^{\prime }}}_k^{\left(j\right)}\left(X_w^{\left(j\right)},Y_w^{\left(j\right)}\right)\Vert }^2}}(15)$

to obtain the final accurate parameters of stereo line-scan cameras, where the initial values of $R,T$ can be calculated by $R=R_2{R}_1^{-1}=R_2{R}_1^T,T=T_2-R{T}_1$.

Experimental Results and Discussion

Calibration Experiment

To evaluate the effectiveness of the proposed method, we calibrate our stereo line-scan camera system. It can be observed from Figures 1A and 2A, when the calibration target is translated on the flat plate, the coordinate map can give the 2D world coordinates $\left(X_w,Y_w\right)$ for camera calibration. In our experiment, $n=7$, that is to say we translate target seven times, a total of $56\left(8\times 7\right)$ feature points. Figure 2B exhibits two of seven image pairs of 8 magnetic rods illuminated by the line lasers and the sub-pixel locations of rod centers. The first and third images are captured by camera 1 while the second and fourth images are captured by camera 2. We set the line-scan cameras to 300 rows per frame, so the image resolution is $2048\times 300$.

Table 1 shows the calibration results of stereo line-scan cameras. We implement our method in Matlab. The root mean square (RMS) of reprojection errors of two camera is less than 0.40 pixel. It is well mentioning that the time of our calibration process is no more than 5 min for both cameras while the time of the method [4] for each camera is about 30 min. The time required for our calibration process is greatly reduced mainly due to the help of the simple target and simple translation operation. Therefore it can be said that our proposed method is significantly effective.

TABLE 1

TABLE 1. Calibration results of stereo line-scan cameras.

2D Coordinate Measurement

Furthermore, our calibration parameters are verified by measuring the 2D coordinates in the measurement field with a range of $1000\text{mm}\times 800\text{mm}$.

The 2D coordinates can be triangulated by solving Eq. 4, then

$\left[\begin{array}{c}{X}_w\\ Y_w\end{array}\right]=\frac{\left[\begin{array}{cc}{u}_2{h^{″}}_{22}-{h^{″}}_{12}& -u_1{h^{\prime }}_{22}+{h^{\prime }}_{12}\\ -u_2{h^{″}}_{21}+{h^{″}}_{11}& u_1{h^{\prime }}_{21}-{h^{\prime }}_{11}\end{array}\right]\times \left[\begin{array}{c}{h^{\prime }}_{13}-u_1{h^{\prime }}_{23}\\ {h^{″}}_{13}-u_2{h^{″}}_{23}\end{array}\right]}{\left(u_1{h^{\prime }}_{21}-{h^{\prime }}_{11}\right)\left(u_2{h^{″}}_{22}-{h^{″}}_{12}\right)-\left(u_1{h^{\prime }}_{22}-{h^{\prime }}_{12}\right)\left(u_2{h^{″}}_{21}-{h^{″}}_{11}\right)}(16)$

where $u_k$ is distortion-free pixel coordinates which can be related to ${\tilde{x}}_k$ by Eq. 8. It is noticed that ${\tilde{x}}_k$ should be solved through Eq. 7 using an iterative method since ${{\tilde{x}}^{\prime }}_k$ can be obtained.

A comparison between our and [4]’s measurement results is shown in Table 2. We test a total of 21 2D points similar to [4]. According to the results, it can be seen obviously that the maximum errors of our method for $X_w$ and $Y_w$ are 0.84 and 0.67 mm respectively, while the maximum errors of [4]’s method for $X_w$ and $Y_w$ are 1.41 and 1.07 mm, respectively. The RMS errors of $X_w$ and $Y_w$ of our method are 0.449 and 0.373 mm respectively, which are also less than [4]’s method. Generally, these errors can be more or less affected by the accuracy of coordinate map, the planeness of iron plate, the precision of calibration target, the parallelism of two viewing planes and iron plate, and the sub-pixel location algorithm.

TABLE 2

TABLE 2. Comparisons of methods for 2D coordinate measurement.

In conclusion, the experimental results demonstrate that our proposed method can provide satisfactory 2D coordinate measurement and has much more superiority both in the calibration time and accuracy compared with the [4]’s method.

Conclusion

We have proposed a calibration method for the stereo line-scan camera system. We firstly established the real distorted imaging model by combining a perspective projection and three types of lens distortions. On the basis of the model which can depict the relationship between the 2D world coordinate and its corresponding 1D pixel coordinates of the image pairs, we have proposed a two-step algorithm to find out the optimal parameters of stereo line-scan cameras. The initial values of intrinsic and extrinsic parameters were computed by solving the system of homogeneous equations derived by the image model and feature points. We employed the Levenberg-Marquardt optimization algorithm to refine the intrinsic, extrinsic and distortion parameters by minimizing the reprojection errors. Finally, the experimental results verified that our proposed method can provide satisfactory 2D coordinate measurement with less calibration time and a higher accuracy. Further study of line-scan camera calibration includes the flexible calibration target and non-coplanar alignment.

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Author Contributions

All authors contributed to the research work. GW proposed the method and designed the experiments; LZ and HZ performed the experiments and analyzed the data; GW wrote the manuscript.

Funding

This research is funded by Natural Science Basic Research Program of Shaanxi (Program No. 2022JM-318) and Comprehensive Reform Research and Practice Project of Graduate Education in Shaanxi Province (Program No. YJSZG2020074).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The work is partly done during the first author’s visit at Nanyang Technological University, Singapore, with a support by Kemao Qian. We gratefully acknowledge Dr. Kemao Qian for his very valuable help. We would also like to thank the reviewers for the valuable and constructive comments that helped us improve the presentation.

References

1. Su D, Bender A, Sukkarieh S. Improved Cross-Ratio Invariant-Based Intrinsic Calibration of A Hyperspectral Line-Scan Camera. Sensors (2018) 18:1885. doi:10.3390/s18061885

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Zhan D, Jing D, Wu M, Zhang D, Yu L, Chen T. An Accurate and Efficient Vision Measurement Approach for Railway Catenary Geometry Parameters. IEEE Trans Instrum Meas (2018) 67:2841–53. doi:10.1109/TIM.2018.2830862

CrossRef Full Text | Google Scholar

3. Liu Z, Wu S, Wu Q, Quan C, Ren Y. A Novel Stereo Vision Measurement System Using Both Line Scan Camera and Frame Camera. IEEE Trans Instrum Meas (2019) 68:3563–75. doi:10.1109/TIM.2018.2880080

CrossRef Full Text | Google Scholar

4. Ma W, Dong T, Tian H, Ni J. Line-Scan CCD Camera Calibration in 2D Coordinate Measurement. Optik (2014) 125:4795–8. doi:10.1016/j.ijleo.2014.04.057

CrossRef Full Text | Google Scholar

5. Sun B, Zhu J, Yang L, Yang S, Guo Y. Sensor for In-Motion Continuous 3D Shape Measurement Based on Dual Line-Scan Cameras. Sensors (2016) 16:1949. doi:10.3390/s16111949

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Sun B, Zhu J, Yang L, Guo Y, Lin J. Stereo Line-Scan Sensor Calibration for 3D Shape Measurement. Appl Opt (2017) 56:7905–14. doi:10.1364/AO.56.007905

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Liao R, Yang L, Ma L, Yang J, Zhu J. A Dense 3-D Point Cloud Measurement Based on 1-D Background-Normalized Fourier Transform. IEEE Trans Instrum Meas (2021) 70:5014412. doi:10.1109/TIM.2021.3075740

CrossRef Full Text | Google Scholar

8. Yuan G, Zheng L, Ding Y, Zhang H, Zhang X, Liu X, et al. A Precise Calibration Method for Line Scan Cameras. IEEE Trans Instrum Meas (2021) 70:5013709. doi:10.1109/TIM.2021.3090157

CrossRef Full Text | Google Scholar

9. Wang G, Qian K. Review on Line-Scan Camera Calibration Methods. Acta Optica Sinica (2020) 40:0111011. doi:10.3788/AOS202040.0111011

CrossRef Full Text | Google Scholar

10. Li D, Wen G, Qiu S. Cross-Ratio-Based Line Scan Camera Calibration Using A Planar Pattern. Opt Eng (2016) 55:014104. doi:10.1117/1.OE.55.1.014104

CrossRef Full Text | Google Scholar

11. Zhang Z. A Flexible New Technique for Camera Calibration. IEEE Trans Pattern Anal Machine Intell (2000) 22:1330–4. doi:10.1109/34.888718

CrossRef Full Text | Google Scholar

12. Wang G, Zheng H, Zhang X. A Robust Checkerboard Corner Detection Method for Camera Calibration Based on Improved YOLOX. Front Phys (2022) 9:819019. doi:10.3389/fphy.2021.819019

CrossRef Full Text | Google Scholar

13. Abdel-Aziz YI, Karara HM. Direct Linear Transformation from Comparator Coordinates into Object Space Coordinates in Close-Range Photogrammetry. Photogramm Eng Remote Sensing (2015) 81:103–7. doi:10.14358/PERS.81.2.103

CrossRef Full Text | Google Scholar

14. Liao R, Zhu J, Yang L, Lin J, Sun B, Yang J. Flexible Calibration Method for Line-Scan Cameras Using A Stereo Target with Hollow Stripes. Opt Lasers Eng (2019) 113:6–13. doi:10.1016/j.optlaseng.2018.09.014

CrossRef Full Text | Google Scholar

15. Horaud R, Mohr R, Lorecki B. On Single-Scanline Camera Calibration. IEEE Trans Robot Automat (1993) 9:71–5. doi:10.1109/70.210796

CrossRef Full Text | Google Scholar

16. Luna CA, Mazo M, Lázaro JL, Vázquez JF. Calibration of Line-Scan Cameras. IEEE Trans Instrum Meas (2010) 59:2185–90. doi:10.1109/TIM.2009.2031344

CrossRef Full Text | Google Scholar

17. Lilienblum E, Al-Hamadi A, Michaelis B. A Coded 3d Calibration Method for Line-Scan Cameras. In: J Weickert, M Hein, and B Schiele, editors. Proceedings of the 35th German Conference on Pattern Recognition. Berlin, Heidelberg: Springer (2013). p. 81–90. doi:10.1007/978-3-642-40602-7_9

CrossRef Full Text | Google Scholar

18. Li D, Wen G, Hui BW, Qiu S, Wang W. Cross-Ratio Invariant Based Line Scan Camera Geometric Calibration with Static Linear Data. Opt Lasers Eng (2014) 62:119–25. doi:10.1016/j.optlaseng.2014.03.004

CrossRef Full Text | Google Scholar

19. Song K, Hou B, Niu M, Wen X, Yan Y. Flexible Line-Scan Camera Calibration Method Using A Coded Eight Trigrams Pattern. Opt Lasers Eng (2018) 110:296–307. doi:10.1016/j.optlaseng.2018.06.014

CrossRef Full Text | Google Scholar

20. Niu M, Song K, Wen X, Zhang D, Yan Y. The Line Scan Camera Calibration Based on Space Rings Group. IEEE Access (2018) 6:23711–21. doi:10.1109/ACCESS.2018.2817629

CrossRef Full Text | Google Scholar

21. Yao M, Zhao Z, Xu B. Geometric Calibration of Line-Scan Camera Using a Planar Pattern. J Electron Imaging (2014) 23:013028. doi:10.1117/1.JEI.23.1.013028

CrossRef Full Text | Google Scholar

22. Sun B, Zhu J, Yang L, Yang S, Niu Z. Calibration of Line-Scan Cameras for Precision Measurement. Appl Opt (2016) 55:6836–43. doi:10.1364/AO.55.006836

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Draréni J, Roy S, Sturm P. Plane-Based Calibration for Linear Cameras. Int J Comput Vis (2011) 91:146–56. doi:10.1007/s11263-010-0349-3

CrossRef Full Text | Google Scholar

24. Donné S, Luong H, Dhondt S, Wuyts N, Inzé D, Goossens B, et al. Robust Plane-Based Calibration for Linear Cameras. In: Proceedings of the IEEE International Conference on Image Processing; Septempber 2017; Beijing, China (2017). p. 36–40. doi:10.1109/ICIP.2017.8296238

CrossRef Full Text | Google Scholar

25. Hui B, Wen G, Zhao Z, Li D. Line-Scan Camera Calibration in Close-Range Photogrammetry. Opt Eng (2012) 51:053602. doi:10.1117/1.OE.51.5.053602

CrossRef Full Text | Google Scholar

26. Hui B, Zhong J, Wen G, Li D. Determination of Line Scan Camera Parameters via the Direct Linear Transformation. Opt Eng (2012) 51:113201. doi:10.1117/1.OE.51.11.113201

CrossRef Full Text | Google Scholar

27. Hui B, Wen G, Zhang P, Li D. A Novel Line Scan Camera Calibration Technique with an Auxiliary Frame Camera. IEEE Trans Instrum Meas (2013) 62:2567–75. doi:10.1109/TIM.2013.2256815

CrossRef Full Text | Google Scholar

28. Steger C, Ulrich M. A Camera Model for Line-Scan Cameras with Telecentric Lenses. Int J Comput Vis (2021) 129:80–99. doi:10.1007/s11263-020-01358-3

CrossRef Full Text | Google Scholar

29. Lilienblum E, Handrich S, Al-Hamadi A. Low Cost Calibration of Stereo Line Scan Camera Systems. In: Proceedings of the 15th IAPR International Conference on Machine Vision Applications; 8–12 May 2017; Nagoya, Japan (2017). p. 322–5. doi:10.23919/MVA.2017.7986866

CrossRef Full Text | Google Scholar

30. Steger C, Ulrich M. A Multi-View Camera Model for Line-Scan Cameras with Telecentric Lenses. J Math Imaging Vis (2022) 64:105–30. doi:10.1007/s10851-021-01055-x

CrossRef Full Text | Google Scholar

31. Wang J, Shi F, Zhang J, Liu Y. A New Calibration Model of Camera Lens Distortion. Pattern Recognition (2008) 41:607–15. doi:10.1016/j.patcog.2007.06.012

CrossRef Full Text | Google Scholar

32. Fang S, Xia X, Xiao Y. A Calibration Method of Lens Distortion for Line Scan Cameras. Optik (2013) 124:6749–51. doi:10.1016/j.ijleo.2013.05.084

CrossRef Full Text | Google Scholar