A data-driven acceleration-level scheme for image-based visual servoing of manipulators with unknown structure

The research on acceleration-level visual servoing of manipulators is crucial yet insufficient, which restricts the potential application range of visual servoing. To address this issue, this paper proposes a quadratic programming-based acceleration-level image-based visual servoing (AIVS) scheme, which considers joint constraints. Besides, aiming to address the unknown problems in visual servoing systems, a data-driven learning algorithm is proposed to facilitate estimating structural information. Building upon this foundation, a data-driven acceleration-level image-based visual servoing (DAIVS) scheme is proposed, integrating learning and control capabilities. Subsequently, a recurrent neural network (RNN) is developed to tackle the DAIVS scheme, followed by theoretical analyses substantiating its stability. Afterwards, simulations and experiments on a Franka Emika Panda manipulator with eye-in-hand structure and comparisons among the existing methods are provided. The obtained results demonstrate the feasibility and practicality of the proposed schemes and highlight the superior learning and control ability of the proposed RNN. This method is particularly well-suited for visual servoing applications of manipulators with unknown structure.

1 Introduction

Robots can accurately perform complex tasks and have become a vital driving force in industrial production (Agarwal and Akella, 2024). Among industrial robots, redundant robots, equipped with multiple degrees of freedom (DOFs), have gained significant recognition and favor due to their exceptional flexibility and automation capabilities (Tang and Zhang, 2022; Zheng et al., 2024). Therefore, numerous control schemes are designed to extend the application range of redundant robots, such as medical services (Zeng et al., 2024) and visual navigation (Wang et al., 2023). Furthermore, in these application scenarios, information on the external environment and the robot's status is acquired from various sensors, especially for the image capture of visual information (Jin et al., 2023). Therefore, unknown situations inevitably exist caused by sensor limitations, environmental variability, and robot modification, which hinder the evolution of robot applications. To address this issue, intelligent algorithms based on data-driven technology are exploited to process the acquired information and convert it into knowledge to drive the regular operation of the robot system (Na et al., 2021; Xie et al., 2022). Yang et al. (2019) construct a robot learning system by improving the adaptive ability of a robot with the information interaction between the robot and environment, which enhances the safety and reliability of robot applications in reality Peng et al. (2023). Li et al. (2019) investigate a model-free control method to cope with the unknown Jacobian problems inside the robot system. On this basis, the dynamic estimation method of robot parameters is researched in the study by Xie and Jin (2023). However, the aforementioned methods primarily operate at the joint velocity level and cannot directly applicable to robots driven by joint acceleration.

As a crucial robot application, visual servoing simulates the bionic system of human eyes, which can obtain information about real objects through optical devices, thus dynamically responding to a visible object. The fundamental task of visual servoing is to impose the error between the corresponding image feature and the desired static reference to approach zero (Zhu et al., 2022). According to the spatial position or image characteristics of the robot, the visual servoing system can be categorized into two types: position-based visual servoing (PBVS) system (Park et al., 2012), which utilizes 3-D position and orientation information to adjust the robot's state, and image-based visual servoing (IBVS) system, which utilizes 2-D image information for guidance (Van et al., 2018). Recently, the research on visual servoing has achieved many unexpected results (Hashimoto et al., 1991; Malis et al., 2010; Zhang et al., 2017; Liang et al., 2018). For instance, visual servoing is applied to bioinspired soft robots in the underwater environment with an adaptive control method, which extends the scope of visual servoing (Xu et al., 2019). Based on the neural network method, a resolution scheme for IBVS is developed at the velocity level. This enables the manipulator to accurately track fixed desired pixels, resulting in fast convergence (Zhang and Li, 2018). However, the aforementioned methods are difficult to deal with the emergence of unknown conditions, such as focal length change, robot abrasion, or parameter variation. This is because these methods rely on accurate structural information of the robot vision system. To tackle this challenge, this study focuses on data-driven control of visual servoing for robots with an unknown Jacobian matrix.

Neural networks have gained significant recognition as powerful tools for solving challenging problems, such as automatic drive (Jin et al., 2024), mechanism control (Xu et al., 2023), and mathematical calculation (Zeng et al., 2003; Stanimirovic et al., 2015). In robot redundancy analysis, neural networks have shown superior performance. In recent decades, numerous control laws based on neural networks have been developed to harness the potential of redundant manipulators (Zhang and Li, 2023). One specific application of the neural network approach addresses the IBVS problem. In this context, the IBVS problem is formulated as a quadratic programming scheme and tackled using a recurrent neural network (RNN). The RNN drives the robot vision system's feature to rapidly converge toward the desired point (Zhang et al., 2017). Additionally, Li et al. (2020) investigate an inverse-free neural network technique to deal with the IBVS task, ensuring that the error approaches zero within a finite time while considering the manipulator's physical constraints.

Most control schemes accomplish the given task at the velocity level, especially for visual servoing applications (Hashimoto et al., 1991; Malis et al., 2010; Zhang et al., 2017; Liang et al., 2018; Van et al., 2018; Zhang and Li, 2018; Xu et al., 2019; Li et al., 2020). These velocity-level schemes control redundant robots via joint velocities. However, when confronted with acceleration or torque-driven robots, the velocity-level schemes exhibit limitations and cannot provide precise control. Furthermore, the velocity-level scheme may yield abrupt joint velocities that are impractical in real-world applications. Consequently, research on acceleration-level visual servoing for robot manipulators has become crucial (Keshmiri et al., 2014; Anwar et al., 2019). Motivated by the issues above, this study investigates the application of visual servoing in robots at the acceleration level. The technical route of this study is shown in Figure 1. As illustrated, the contributions of this study are shown as follows:

• An acceleration-level image-based visual servoing (AIVS) scheme is designed, taking into account multiple joint constraints.

• Considering potential unknown factors in the visual servoing system, a data-driven acceleration-level image-based visual servoing (DAIVS) scheme is developed, enabling simultaneous learning and control.

• RNNs are proposed to solve the AIVS scheme and DAIVS scheme, enabling visual servoing control of the manipulator. Theoretical analyses guarantee the stability of the RNNs.

Figure 1

Figure 1. Technical route of this study.

In addition, the feasibility of the proposed schemes is demonstrated through simulative and experimental results conducted on a Franka Emika Panda manipulator with an eye-in-hand structure.

Before concluding this section, the remaining sections of the study are shown as follows. Section 2 presents the robot kinematics of visual servoing and introduces the data-driven learning algorithm, formulating the problem at the acceleration level. Section 3 constructs an AIVS scheme with the relevant RNN. Subsequently, considering the unknown factors, a DAIVS scheme and corresponding RNN are proposed, and theoretical analyses proved the learning and control ability of the RNN, as shown in Section 4. Section 5 provides abundant simulations and performance comparisons, embodying the proposed method's validity and superiority. Section 6 displays physical experiments on a real manipulator. Finally, Section 7 briefly concludes this study.

2 Preliminaries

In this section, the robot visual servoing kinematics and data-driven learning algorithm are introduced as the preliminaries. Note that this study specifically tackles the problem at the acceleration level.

2.1 Robot visual servoing kinematics

The forward kinematics, which contains the transformation between the joint angle ϕ(t)∈ℝ^m of a robot and the end-effector position and posture s(t)∈ℝ⁶, can be expressed as follows:

$\begin{array}{ll}f(\varphi (t))=\text{s}(t),& (1)\end{array}$

where f(·) is the non-linear mapping related to the structure of the robot. In view of strongly non-linear and redundant characteristics of f(·), it is difficult to obtain the desired angle information directly from the desired end-effector information s_d(t), i.e., s(t) = s_d(t). By taking the time derivative of both sides of Equation (1), one can deduce

$\begin{array}{ll}{J}_{\text{ro}}\dot{\varphi }(t)=\dot{\text{s}}(t),& (2)\end{array}$

where $\dot{\varphi }(t)$ denotes the joint velocity; $\dot{\text{s}}(t)$ covers the joint velocity and translational velocity of the end-effector; $J_{\mathrm{\text{ro}}}=\partial f(\varphi (t))/\partial \varphi (t)\in {ℝ}^{6\times m}$ stands for the robot Jacobian matrix. Owing to the physical properties of manipulators, output control signals based on design formulas and intelligent calculations may not be suitable for the normal operation of real robots. Therefore, to ensure the protection of the robot, it is crucial to take into account the following joint restrictions:

$\begin{array}{l}{\varphi }^{-}\le \varphi \le {\varphi }^{+}\\ {\dot{\varphi }}^{-}\le \dot{\varphi }\le {\dot{\varphi }}^{+}\\ {\ddot{\varphi }}^{-}\le \ddot{\varphi }\le {\ddot{\varphi }}^{+},\end{array}$

where ϕ⁻, ${\dot{\varphi }}^{-}$, and ${\ddot{\varphi }}^{-}$ signify the lower bounds of joint angle, joint velocity, and joint acceleration, respectively; ϕ⁺, ${\dot{\varphi }}^{+}$ and ${\ddot{\varphi }}^{+}$ denote the upper bounds of joint angle, joint velocity, and joint acceleration, respectively. Utilizing the special conversion techniques (Zhang and Zhang, 2012; Xie et al., 2022), the joint restrictions would be integrated into the acceleration level as $\ddot{\varphi }\in \gamma ,$ where γ = {g∈ℝ^m, γ⁻ ≤ g ≤ γ⁺} is the safe range of joints with γ⁻ and γ⁺ denoting the lower bound and upper bound of γ, respectively. In detail, the i-th elements of γ⁻ and γ⁺ are designed as

$\begin{array}{l}\begin{array}{c}{\gamma }_i^{-}=\text{max}\left\{\mu ({\varphi }_i^{-}+{\theta }_i-{\varphi }_i),\nu ({\dot{\varphi }}_i^{-}-{\dot{\varphi }}_i),{\ddot{\varphi }}_i^{-}\right\}\\ {\gamma }_i^{+}=\text{min}\left\{\mu ({\varphi }_i^{+}-{\theta }_i-{\varphi }_i),\nu ({\dot{\varphi }}_i^{+}-{\dot{\varphi }}_i),{\ddot{\varphi }}_i^{+}\right\},\end{array}\end{array}$

where i = 1, 2, 3, ⋯ , m; μ>0 and ν>0 are designed to select the feasible region for different levels; θ_i is the margin to ensure that the acceleration has a sufficiently large feasible region (Xie et al., 2022). Then, a brief introduction to the visual servoing system is presented as follows. Regarding visual servoing tasks, the number of features determines the complexity of a visual servoing system. Simply considering a visual servoing system with one feature, a miniature camera is mounted on the end-effector of the manipulator and moves with the end-effector. Figure 2 illustrates the geometric transformation in different coordinate systems. Three-dimensional space with O_ca as the original point and [X, Y, Z] as the coordinate axis is called the camera system with the internal coordinate point q = [x, y, z]^T. Relatively, with O_im as the center point, the image system is the two-dimensional space with the projection pixel point of q being ${\left[p_x,p_y\right]}^{\mathrm{\text{T}}}$ and the pixel coordinate being $\text{p}={\left[p_u,p_v\right]}^{\mathrm{\text{T}}}\in {ℝ}^2$. According to the similar triangle, it can be readily obtained in the study by Zhang et al. (2017) and Zhang and Li (2018):

$\begin{array}{ll}\left[\begin{array}{c}{p}_x\\ p_y\end{array}\right]=\frac{l}{z}\left[\begin{array}{c}x\\ y\end{array}\right]& (3)\end{array}$

and

$\begin{array}{ll}{p}_u=u_0+a_x{p}_x& (4)\end{array}$

$\begin{array}{l}{p}_v=v_0+a_y{p}_y,\end{array}$

Figure 2

Figure 2. Geometric schematic of the camera system.

with l standing for the focal length of the camera; u₀ and v₀ denoting the pixel coordinate of principle point; and ${\left[a_x,a_y\right]}^{\mathrm{\text{T}}}$ standing for the conversion scale. Based on Equations (3, 4), the image Jacobian matrix $J_{\mathrm{\text{im}}}(\text{p},z)\in {ℝ}^{2\times 6}$ is defined using the following relationship (Liang et al., 2018):

$\begin{array}{ll}{J}_{\text{im}}(\text{p},z)\dot{\text{s}}=\dot{\text{p}},& (5)\end{array}$

where $\dot{\text{p}}$ stands for the movement velocity of the pixel coordinate and

$\begin{array}{l}{J}_{\text{im}}(\text{p},z)=H\left[\begin{array}{cccccc}-\frac{l}{z}& 0& \frac{l{p}_x}{z}& \frac{p_x{p}_y}{l}& -\frac{p_x^2+l^2}{l}& p_y\\ 0& -\frac{l}{z}& \frac{p_y}{z}& -\frac{p_y^2+l^2}{l}& -\frac{p_x{p}_y}{l}& -p_x\end{array}\right],\end{array}$

with

$\begin{array}{l}{p}_x=\frac{p_u-u_0}{a_x},p_y=\frac{p_v-v_0}{a_y},H=\left[\begin{array}{cc}{a}_x& 0\\ 0& a_y\end{array}\right].\end{array}$

For the sake of convenience, Equations (2, 5) can be combined as follows:

$\begin{array}{ll}\mathcal{J}\dot{\varphi }=\dot{\text{p}},& (6)\end{array}$

with $\mathcal{J}=J_{\mathrm{\text{im}}}(\text{p},z)J_{\mathrm{\text{ro}}}\in {ℝ}^{2\times m}$ defined as the visual Jacobian matrix. The relationship between joint space and image space is established directly by Equation (6) at the velocity level. Taking the time derivatives of both sides of Equation (6) generates

$\begin{array}{ll}\dot{\mathcal{J}}\dot{\varphi }+\mathcal{J}\ddot{\varphi }=\ddot{\text{p}},& (7)\end{array}$

where $\dot{\mathcal{J}}$ is the time derivative of 𝒥; $\ddot{\varphi }$ denotes the joint acceleration; and $\ddot{\text{p}}$ stands for the movement acceleration of the pixel coordinate. When it comes to a complicated situation with more features, the above analyses still hold under the requirements of appropriate dimensions. It is worth noting that a single feature is analyzed as an example for simple illustration. When the number of features increases, the principle of coordinate transformation remains unchanged along with the increase in dimension.

2.2 Data-driven learning algorithm

However, unknown conditions may exist in the robot visual servoing system, such as focal length changes or robot modifications. In this regard, it could not control the robot accurately to execute the IBVS task based on $\dot{\mathcal{J}}$. Hence, motivated by this issue, a data-driven learning algorithm is designed as follows. To begin with, a virtual IBVS system is established, incorporating the virtual visual Jacobian matrix $\bar{\mathcal{J}}\in {ℝ}^{2\times m}$ and the following relationship:

$\begin{array}{l}\bar{\mathcal{J}}\dot{\varphi }=\overline{\dot{\text{p}}},\end{array}$

where $\overline{\dot{\text{p}}}\in {ℝ}^2$ is the virtual pixel velocity determined by the virtual robot and $\dot{\varphi }$ is the joint velocity measured in real time from the robot. Beyond dispute, the goal of the data-driven learning algorithm is to guarantee that $\overline{\dot{\text{p}}}$ can rapidly converge to the real pixel velocity $\dot{\text{p}}$. Thereout, an error function is devised as $\ell =\left|\right|\overline{\dot{\text{p}}}-\dot{\text{p}}|{|}_2^2/2$, where ||·||₂ is the Euclidean norm of a vector. On the basis of the gradient descent method (Stanimirovic et al., 2015) to minimize the error function along the negative gradient direction, one can get

$\begin{array}{ll}\dot{\bar{\mathcal{J}}}=-\delta \frac{\partial \ell }{\partial \bar{\mathcal{J}}}=-\delta (\bar{\mathcal{J}}\dot{\varphi }-\dot{\text{p}}){\dot{\varphi }}^{\text{T}},& (8)\end{array}$

where $\dot{\bar{\mathcal{J}}}$ is the time derivation of $\bar{\mathcal{J}}$; δ>0 denotes the coefficient that controls the convergence rate. Hereinafter, $\dot{\bar{\mathcal{J}}}$ and $\bar{\mathcal{J}}$ are used to replace the calibrated parameter $\dot{\mathcal{J}}$ and 𝒥 to deal with the unknown situations. This method directly explores the relationship between joint space and image space without the utilization of $\dot{\mathcal{J}}$ and 𝒥. It is worth highlighting that Equation (8) does not involve real structural information and estimates structural information from the joint velocity $\dot{\varphi }$ and velocity of the pixel coordinate $\dot{\text{p}}$ measured by sensors, which belongs to the core idea of the data-driven learning algorithm.

3 Acceleration-level IBVS solution

In this section, an AIVS scheme is proposed with joint constraints considered. Subsequently, we propose a corresponding RNN and provide theoretical analyses. Note that the presented method requires an accurate visual Jacobian matrix.

3.1 AIVS scheme

It is worth pointing out that there are few acceleration-level robot control schemes for dealing with IBVS problems. None of the existing acceleration-level solutions take joint constraints into account (Keshmiri et al., 2014; Anwar et al., 2019). In this regard, considering joint constraints, acceleration control, and visual servoing kinematics, the AIVS scheme is constructed as a quadratic programming problem, taking the following form:

$\begin{array}{ll}\text{minimize }\frac{1}{2}{\ddot{\varphi }}^{\text{T}}\ddot{\varphi }& (\mathrm{9a})\end{array}$

$\begin{array}{ll}\text{subject to }\ddot{\text{p}}=\mathcal{J}\ddot{\varphi }+\dot{\mathcal{J}}\dot{\varphi }& (\mathrm{9b})\end{array}$

$\begin{array}{ll}\text{p}={\text{p}}_{\text{d}}& (\mathrm{9c})\end{array}$

$\begin{array}{ll}\ddot{\varphi }\in \gamma ,& (\mathrm{9d})\end{array}$

where p_d denotes the desired pixel coordinate. As a result, the goal of AIVS scheme (9) is to make the end-effector track the desired pixel point. In addition, according to robot Jacobian matrix J_ro and the image Jacobian matrix J_im, the visual Jacobian matrix 𝒥 and its time derivative $\dot{\mathcal{J}}$ are determined by the structure and parameters of the robot and the parameter settings inside the camera. Hence, if there are any changes in the internal parameters or structures, leading to an unknown state, the accuracy of 𝒥 and $\dot{\mathcal{J}}$ may be compromised, potentially leading to a decline in performance. In contrast to velocity-level visual servoing schemes (Hashimoto et al., 1991; Malis et al., 2010; Zhang et al., 2017; Liang et al., 2018; Van et al., 2018; Zhang and Li, 2018; Xu et al., 2019; Li et al., 2020), the proposed AIVS scheme (9) offers two advantages. First, it utilizes joint acceleration as the control signal, resulting in continuous joint velocities. This helps mitigate the issues associated with excessive and discontinuous joint velocities. Second, AIVS scheme (9) takes into account the equality and inequality constraints at the acceleration level. This allows for a more comprehensive consideration of constraints, expanding the range of applications.

3.2 RNN solution and theoretical analysis

For the AIVS scheme (9), the pseudoinverse method is applied to generate the relevant RNN solution (Cigliano et al., 2015; Li et al., 2020). Primarily, as reported in the study by Zhang and Zhang (2012) and Xie et al. (2022), one can readily extend pixel coordinate error p−p_d into the acceleration level by neural dynamics method (Liufu et al., 2024) as

$\begin{array}{ll}\ddot{\text{p}}-{\ddot{\text{p}}}_{\text{d}}=-\alpha (\dot{\text{p}}-{\dot{\text{p}}}_{\text{d}})-\beta (\text{p}-{\text{p}}_{\text{d}}),& (10)\end{array}$

where the design parameter α>0 and β>0; ${\dot{\text{p}}}_{\mathrm{\text{d}}}$ and ${\ddot{\text{p}}}_{\mathrm{\text{d}}}$ are the desired velocity and the desired acceleration of the pixel coordinates, respectively. It is worth pointing out that the desired pixel coordinates p_d is a constant, thus ${\dot{\text{p}}}_{\mathrm{\text{d}}}={\ddot{\text{p}}}_{\mathrm{\text{d}}}=0$. As a result, Equation (10) can be rearranged as

$\begin{array}{ll}\ddot{\text{p}}=-\alpha \dot{\text{p}}-\beta (\text{p}-{\text{p}}_{\text{d}}).& (11)\end{array}$

Substituting Equation (11) into Equation (9b), it could be obtained:

$\begin{array}{l}\dot{\mathcal{J}}\dot{\varphi }+\mathcal{J}\ddot{\varphi }=-\alpha \dot{\text{p}}-\beta (\text{p}-{\text{p}}_{\text{d}}).\end{array}$

In light of the pseudoinverse method, the joint acceleration can be minimized with the following formula:

$\begin{array}{ll}\ddot{\varphi }={\mathcal{J}}^{†}(-\alpha \dot{\text{p}}-\beta (\text{p}-{\text{p}}_{\text{d}})-\dot{\mathcal{J}}\dot{\varphi }),& (12)\end{array}$

where superscript ^† denotes the pseudoinverse operation of a matrix with 𝒥^† = 𝒥^T(𝒥𝒥^T)−1. It is deserved to note that Equation (12) is employed in the study by Keshmiri et al. (2014) and Anwar et al. (2019) to generate the acceleration command for a manipulator. However, the research in the study by Keshmiri et al. (2014) and Anwar et al. (2019) does not consider joint constraints of the manipulator. To address this problem, the RNN corresponding to the AIVS scheme (9) is derived as

$\begin{array}{ll}\ddot{\varphi }={\mathcal{p}}_{\gamma }({\mathcal{J}}^{†}(-\alpha \dot{\text{p}}-\beta (\text{p}-{\text{p}}_{\text{d}})-\dot{\mathcal{J}}\dot{\varphi })),& (13)\end{array}$

where projection function 𝒫_γ(c) = argmin_b∈γ||b−c||₂. Furthermore, theoretical analyses regarding the convergence of RNN (13) are presented as follows.

Theorem 1: The pixel error ξ = p−p_d driven by AIVS scheme (9) assisted with RNN (13) globally converges to a zero vector.

Proof: According to Equations (7, 13), one has

$\begin{array}{l}\ddot{\text{p}}=\mathcal{J}\ddot{\varphi }+\dot{\mathcal{J}}\dot{\varphi }=\mathcal{J}{\mathcal{P}}_{\gamma }({\mathcal{J}}^{†}(-\alpha \dot{\text{p}}-\beta (\text{p}-{\text{p}}_{\text{d}})-\dot{\mathcal{J}}\dot{\varphi }))+\dot{\mathcal{J}}\dot{\varphi }.\end{array}$

Due to the fact that p_d is a fixed feature, error function $\ddot{\xi }$ can be readily derived as

$\begin{array}{l}\ddot{\xi }=\mathcal{J}{\mathcal{P}}_{\gamma }({\mathcal{J}}^{†}(-\alpha \dot{\xi }-\beta \xi -\dot{\mathcal{J}}\dot{\varphi }))+\dot{\mathcal{J}}\dot{\varphi }.\end{array}$

By considering the projection function, a substitution matrix $\mathcal{H}$ is designed to replace 𝒫_γ(·), leading to

$\begin{array}{ll}\ddot{\xi }=\mathcal{J}\mathcal{H}{\mathcal{J}}^{†}(-\alpha \dot{\xi }-\beta \xi -\dot{\mathcal{J}}\dot{\varphi })+\dot{\mathcal{J}}\dot{\varphi },& (14)\end{array}$

of which

$\begin{array}{l}\mathcal{H}=\left[\begin{array}{cccc}{h}_1& 0& \cdots & 0\\ 0& h_2& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & h_m\end{array}\right]\in {ℝ}^{m\times m}\end{array}$

and

$\begin{array}{l}{h}_i=\frac{{({\mathcal{P}}_{\gamma }({\mathcal{J}}^{†}(-\alpha \dot{\text{p}}-\beta (\text{p}-{\text{p}}_{\text{d}})-\dot{\mathcal{J}}\dot{\varphi })))}_i}{{({\mathcal{J}}^{†}(-\alpha \dot{\text{p}}-\beta (\text{p}-{\text{p}}_{\text{d}})-\dot{\mathcal{J}}\dot{\varphi }))}_i}\in (0,1].\end{array}$

By matrix decomposition, structural analyses of matrix $\mathcal{J}\mathscr{H}{\mathcal{J}}^{†}=[a_{11},a_{12};a_{21},a_{22}\in {ℝ}^{2\times 2}$ are given as follows:

$\begin{array}{l}\mathcal{J}\mathcal{H}{\mathcal{J}}^{†}=\mathcal{J}L{L}^{\text{T}}{\mathcal{J}}^{\text{T}}{(\mathcal{J}{\mathcal{J}}^{\text{T}})}^{-1},\end{array}$

where $L=\sqrt{\mathcal{H}}$. In this regard, matrix 𝒥$\mathcal{H}$𝒥^† can be viewed as the product of two positive definite matrices. It is evident that the eigenvalues of 𝒥$\mathcal{H}$𝒥^† are greater than zero and det(𝒥$\mathcal{H}$𝒥^†) = det(𝒥LL^T𝒥^T)det((𝒥𝒥^T)−1)>0 with det(·), denoting the determinant of a matrix. According to the properties of the diagonal elements of the matrix, it can be concluded that the diagonal elements of 𝒥$\mathcal{H}$𝒥^† are greater than zero (a₁₁>0, a₂₂>0). Furthermore, Equation (12) can be rewritten as

$\begin{array}{l}\left[\begin{array}{c}{\ddot{\xi }}_1\\ {\ddot{\xi }}_2\end{array}\right]=\left[\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right]\left[\begin{array}{c}-\alpha {\dot{\xi }}_1-\beta \delta {\xi }_1-{(\dot{\mathcal{J}}\dot{\varphi })}_1\\ -\alpha {\dot{\xi }}_2-\beta {\xi }_2-{(\dot{\mathcal{J}}\dot{\varphi })}_2\end{array}\right]+\left[\begin{array}{c}{(\dot{\mathcal{J}}\dot{\varphi })}_1\\ {(\dot{\mathcal{J}}\dot{\varphi })}_2\end{array}\right],\end{array}$

and further we get

$\begin{array}{l}{\ddot{\xi }}_1+a_{11}\alpha {\dot{\xi }}_1+a_{11}\beta {\xi }_1=-a_{12}(\alpha {\dot{\xi }}_2+\beta {\xi }_2+{(\dot{\mathcal{J}}\dot{\varphi })}_2)\hfill \\ +(1-a_{11}){(\dot{\mathcal{J}}\dot{\varphi })}_1\hfill \end{array}$

and

$\begin{array}{l}{\ddot{\xi }}_2+a_{22}\alpha {\dot{\xi }}_2+a_{22}\beta {\xi }_2=-a_{21}(\alpha {\dot{\xi }}_1+\beta {\xi }_1+{(\dot{\mathcal{J}}\dot{\varphi })}_1)\hfill \\ +(1-a_{22}){(\dot{\mathcal{J}}\dot{\varphi })}_2\hfill \end{array}$

which can be regarded as a perturbed second-order constant coefficient differential equation with respect to ξ. In conclusion, pixel error ξ is able to converge exponentially. To illustrate the steady state of the system (Equation 14), further derivations continue to be given. As the pixel error decreases, all joint properties return to the interior of joint constraints. In this sense, joint properties, i.e., $\ddot{\varphi }$, $\dot{\varphi }$ and ϕ, are all inside the joint limits with h_i = 1. Therefore, Equation (14) can be reorganized as

$\begin{array}{ll}\ddot{\xi }(t)+\alpha \dot{\xi }(t)+\beta \xi (t)=0.& (15)\end{array}$

It is worth mentioning that Equation (15) can be regarded as a second-order constant coefficient differential equation with regard to ξ. Moreover, the solutions of Equation (15) can be segmented into three subcases on account of different settings of α and β, given the original state ξ(0) = p−p_d.

Subcase I: As for α²−4β>0, the characteristic roots could be obtained simply as ${\mathcal{R}}_1=(-\alpha +\sqrt{{\alpha }^2-4\beta })/2$ and ${\mathcal{R}}_2=(-\alpha -\sqrt{{\alpha }^2-4\beta })/2$ with real number $\mathcal{R}$₁≠$\mathcal{R}$₂. Therefore, one can readily deduce

$\begin{array}{l}\xi (t)=\xi (0)(D_1\text{exp}({\mathcal{R}}_{1}t)+D_2\text{exp}({\mathcal{R}}_{2}t)),\end{array}$

with $D_1=\alpha /(2\sqrt{{\alpha }^2-4\beta })+1/2$ and $D_2=1/2-\alpha /(2\sqrt{{\alpha }^2-4\beta })$

Subcase II: As to α²−4β = 0, calculating characteristic roots generates $\mathcal{R}$₁ = $\mathcal{R}$₂ = −α/2. Hence, it can be readily obtained:

$\begin{array}{l}\xi (t)=\xi (0)\text{exp}(-\alpha /2t)(1+\alpha /2t).\end{array}$

Subcase III: As to α²−4β < 0, we get two complex number roots as $\mathcal{R}$₁ = ζ+iη and $\mathcal{R}$₂ = ζ−iη. Accordingly, it is evident that

$\begin{array}{l}\xi (t)=\xi (0)\text{exp}(-\zeta t)(\text{cos}(\eta t)-\zeta \text{sin}(\eta t)/\eta ).\end{array}$

The above three subcases indicate that the pixel error ξ = p−p_d converges to zero over time globally. The proof is complete.

4 DAIVS solution

The existing IBVS schemes, including the AIVS scheme (9), often require a detailed knowledge of the robot visual servoing system. However, in a non-ideal state, many unknown cases often exist, which can disturb the precise control of the robot, thus resulting in large errors. Recalling the data-driven learning algorithm (Equation 8), virtual visual Jacobian matrix $\bar{\mathcal{J}}$ is exploited to solve this issue.

4.1 DAIVS scheme and RNN solution

Based on the virtual visual Jacobian matrix, a DAIVS scheme (8) would be designed as

$\begin{array}{l}\text{minimize }\frac{1}{2}{\ddot{\varphi }}^{\text{T}}\ddot{\varphi }\\ \text{subject to }\ddot{\text{p}}=\bar{\mathcal{J}}\ddot{\varphi }+\dot{\bar{\mathcal{J}}}\dot{\varphi }\\ \text{p}={\text{p}}_{\text{d}}\\ \ddot{\varphi }\in \gamma .\end{array}$

It is a remarkable fact that the DAIVS scheme does not involve the visual structure of the real robot. Instead, the virtual visual Jacobian matrix $\bar{\mathcal{J}}$ conveys the transformation relationship between the joint space and image space to deal with possible unknowns in the structure of the robot system. Compared with acceleration-level visual servoing schemes (Keshmiri et al., 2014; Anwar et al., 2019), the proposed DAIVS scheme offers two distinct advantages. First, it prioritizes the safety aspect by considering joint limits. Second, the DAIVS scheme takes into account the uncertainty of the robot vision system and employs the virtual visual Jacobian matrix for robot control, enhancing the fault tolerance ability. The existing acceleration-level visual servoing schemes (Keshmiri et al., 2014; Anwar et al., 2019) cannot accurately implement visual servoing tasks when the Jacobian matrix lacks precision. Furthermore, combining Equations (8, 13) generates

$\begin{array}{ll}\ddot{\varphi }={\mathcal{P}}_{\gamma }({\bar{\mathcal{J}}}^{†}(-\alpha \dot{\text{p}}-\beta (\text{p}-{\text{p}}_{\text{d}})-\dot{\bar{\mathcal{J}}}\dot{\varphi }))& (\mathrm{16a})\end{array}$

$\begin{array}{ll}\dot{\bar{\mathcal{J}}}=-\delta (\bar{\mathcal{J}}\dot{\varphi }-\dot{\text{p}}){\dot{\varphi }}^{\text{T}}.& (\mathrm{16b})\end{array}$

It is worth pointing out that the RNN (16) is divided into the inner cycle and outer cycle, i.e., the learning cycle and control cycle. Subsystem (Equation 16a), which can be viewed as the outer cycle, mainly generates the control signal to adjust the joint properties via virtual visual Jacobian matrix $\bar{\mathcal{J}}$. In return, inner cycle (Equation 16b) with learning ability can explore the relationship between end-effector motion and joint motion, thus producing virtual visual Jacobian matrix $\bar{\mathcal{J}}$ to simulate the movement process of real robots. From a control point of view, the inner cycle (Equation 16b) must converge faster than the outer cycle (Equation 16a). In this sense, δ≫α is a necessary condition for the normal operation of the system.

Note that both RNN (13) and RNN (16) involve the use of pseudo-inverse operations. As a result, various existing methods can be employed to mitigate singularity issues, such as the damped least squares method. Specifically, 𝒥^† can be calculated via 𝒥^† = 𝒥^T(𝒥𝒥^T+h$\mathcal{I}$)−1 with h being a tiny constant and $\mathcal{I}$ being an identity matrix. The additional item h$\mathcal{I}$ ensures that all eigenvalues of 𝒥𝒥^T+h$\mathcal{I}$ are never zero during the inversion process, thereby preventing singular issues. In addition, RNN (16) relies on the virtual visual Jacobian matrix and estimates the real Jacobian matrix using Equation (16b). This enables a robust handling of the visual system's uncertainty. However, RNN (13) relies on the real visual Jacobian matrix, leading to potential inaccuracies in the robot control process.

4.2 Stability analyses of RNN

The learning and control performance of the proposed DAIVS scheme aided with RNN (16) are proved by the following theorem.

Theorem 2: The Jacobian matrix error $E=\bar{\mathcal{J}}-\mathcal{J}$ and pixel error ξ = p−p_d produced by RNN (16) converges to zero, given a large enough δ.

Proof: The proof is segmented into two parts: (1) proving learning convergence; (2) proving control convergence.

Part 1: Proving learning convergence. Design the i-th system of Jacobian matrix error as $E_i={\bar{\mathcal{J}}}_i-{\mathcal{J}}_i$ (i = 1, 2) where ${\bar{\mathcal{J}}}_i$ and 𝒥_i denote the i-th row of $\bar{\mathcal{J}}$ and 𝒥 and set the Lyapunov candidate ${\mathcal{V}}_i=({\bar{\mathcal{J}}}_i-{\mathcal{J}}_i){({\bar{\mathcal{J}}}_i-{\mathcal{J}}_i)}^{\mathrm{\text{T}}}$. Calculating the time derivative of 𝒱_i leads to

$\begin{array}{l}{\stackrel{.}{V}}_i=({\stackrel{.}{\overline{\mathcal{J}}}}_i-{\stackrel{.}{\mathcal{J}}}_i)({\overline{\mathcal{J}}}_i-{\mathcal{J}}_i{)}^{\text{T}}\\ =-\delta ({\overline{\mathcal{J}}}_i\dot{\varphi }-{\stackrel{.}{p}}_i){\dot{\varphi }}^{\text{T}}({\overline{\mathcal{J}}}_i-{\mathcal{J}}_i{)}^{\text{T}}-{\stackrel{.}{\mathcal{J}}}_i({\overline{\mathcal{J}}}_i-{\mathcal{J}}_i{)}^{\text{T}}\\ =-\delta ({\overline{\mathcal{J}}}_i\dot{\varphi }-{\mathcal{J}}_i\dot{\varphi }){\dot{\varphi }}^{\text{T}}({\overline{\mathcal{J}}}_i-{\mathcal{J}}_i{)}^{\text{T}}-{\stackrel{.}{\mathcal{J}}}_i({\overline{\mathcal{J}}}_i-{\mathcal{J}}_i{)}^{\text{T}}\\ \le -\delta \Pi (\dot{\varphi }{\dot{\varphi }}^{\text{T}})({\overline{\mathcal{J}}}_i-{\mathcal{J}}_i)({\overline{\mathcal{J}}}_i-{\mathcal{J}}_i{)}^{\text{T}}-{\stackrel{.}{\mathcal{J}}}_i({\overline{\mathcal{J}}}_i-{\mathcal{J}}_i{)}^{\text{T}},\end{array}$

where ${\dot{\text{p}}}_i$ represents the i-th element of $\dot{\text{p}}$, and $\Pi (\dot{\varphi }{\dot{\varphi }}^{\mathrm{\text{T}}})$ denotes the least eigenvalue of matrix $\dot{\varphi }{\dot{\varphi }}^{\mathrm{\text{T}}}$. When the manipulator is tracking the feature, the value of $\Pi (\dot{\varphi }{\dot{\varphi }}^{\mathrm{\text{T}}})$ is always greater than zero. In this case, we substitute $E_i={\bar{\mathcal{J}}}_i-{\mathcal{J}}_i$ into the above equation, resulting in the following expression:

$\begin{array}{l}{\stackrel{.}{V}}_i\le -\delta \Pi (\dot{\varphi }{\dot{\varphi }}^{\text{T}})E_i{E}_i^{\text{T}}-{\stackrel{.}{\mathcal{J}}}_i{E}_i^{\text{T}}\\ \le -\delta \Pi (\dot{\varphi }{\dot{\varphi }}^{\text{T}})||E_i|{|}_2^2+||{\stackrel{.}{\mathcal{J}}}_i|{|}_2||E_i|{|}_2\\ =||E_i|{|}_2(||{\stackrel{.}{\mathcal{J}}}_i|{|}_2-\delta \Pi (\dot{\varphi }{\dot{\varphi }}^{\text{T}})||E_i|{|}_2).\end{array}$

For further analysis, we consider three cases based on the above equation:

• If $||E_i|{|}_2>||{\dot{\mathcal{J}}}_i|{|}_2/\delta \Pi (\dot{\varphi }{\dot{\varphi }}^{\mathrm{\text{T}}})$, we observe ${\dot{\mathcal{V}}}_i<0$ and 𝒱_i>0. This indicates that in this case, E_i converges until $||E_i|{|}_2=||{\dot{\mathcal{J}}}_i|{|}_2/\delta \Pi (\dot{\varphi }{\dot{\varphi }}^{\mathrm{\text{T}}})$.

• If $||E_i|{|}_2=||{\dot{\mathcal{J}}}_i|{|}_2/\delta \Pi (\dot{\varphi }{\dot{\varphi }}^{\mathrm{\text{T}}})$, we find ${\dot{\mathcal{V}}}_i\le 0$ and 𝒱_i>0. This implies that E_i will continue to converge or remain at the state with $||E_i|{|}_2=||{\dot{\mathcal{J}}}_i|{|}_2/\delta \Pi (\dot{\varphi }{\dot{\varphi }}^{\mathrm{\text{T}}})$.

• If $||E_i|{|}_2<||{\dot{\mathcal{J}}}_i|{|}_2/\delta \Pi (\dot{\varphi }{\dot{\varphi }}^{\mathrm{\text{T}}})$, we have two possibilities: either ${\dot{\mathcal{V}}}_i>0$ and 𝒱_i> 0, or ${\dot{\mathcal{V}}}_i\le 0$ and 𝒱_i> 0. In the former possibility, the error will increase until ||E_i||₂= $||{\dot{\mathcal{J}}}_i|{|}_2/\delta \Pi (\dot{\varphi }{\dot{\varphi }}^{\mathrm{\text{T}}})$. In the latter possibility, the error will continue to converge or remain constant.

Combining the above three cases, it can be summarized that $\underset{t\to +\infty }{lim}||E_i|{|}_2\le ||{\stackrel{°}{\mathcal{J}}}_i|{|}_2/\delta \Pi (\stackrel{°}{\varphi }{\stackrel{°}{\varphi }}^{\mathrm{\text{T}}})$. Furthermore, it can be deduced that the Jacobian matrix error $E=\bar{\mathcal{J}}-\mathcal{J}$ produced by RNN (16a) globally approach zero, given a sufficiently large value of δ.

Part 2: Proving control convergence.

According to the proof in Part 1, we take advantage of the LaSalle's invariant principle (Khalil, 2001) again to conduct the convergence proof on Equation (16b). In other words, the following formula is provided by replacing $\bar{\mathcal{J}}$ and $\dot{\bar{\mathcal{J}}}$ with 𝒥 and $\dot{\mathcal{J}}$:

$\begin{array}{ll}\ddot{\varphi }={\mathcal{P}}_{\gamma }({\mathcal{J}}^{†}(-\alpha \dot{\text{p}}-\beta (\text{p}-{\text{p}}_{\text{d}})-\dot{\mathcal{J}}\dot{\varphi })),& (17)\end{array}$

which is equivalent to Equation (13). In consequence, the proof on the convergence of the pixel error p−p_d in Equation (17) has been discussed in Theorem 1 and thus omitted here. The proof is complete.

5 Simulation verifications

In this section, simulations are conducted on a Franka Emika Panda manipulator with 7 DOFs for completing a visual servoing task, which are synthesized by the proposed AIVS scheme (9) and the proposed DAIVS scheme . Note that the AIVS scheme (9) is able to drive the redundant manipulator to perform the visual servoing task with a given visual Jacobian matrix, and that, the DAIVS scheme can deal with the unknown situation in the robot system dynamically in the absence of the visual Jacobian matrix. For the simulations, this study utilizes a computer with an Intel Core i7-12700 processor and 32 GB RAM. The simulations are performed using MATLAB/Simulink software version R2022a.

First, some necessary information and parameter settings about the manipulator and camera structure are given below. The Franka Emika Panda manipulator is a 7-DOF redundant manipulator (Gaz et al., 2019), with a camera mounted on its end-effector. In addition, we set l = 8 × 10⁻³ m, u₀ = v₀ = 256 pixel, $a_x=a_y=8\times 10^4\mathrm{\text{pixel/m}}$, and design μ = ν = 20 with z = 2, task execution time T = 20 s and ${\text{p}}_{\mathrm{\text{d}}}={\left[256,256\right]}^{\mathrm{\text{T}}}$ pixel. In addition, the joint limits are set as ${\ddot{\varphi }}^{+}=-{\ddot{\varphi }}^{-}={\left[2\right]}_{7\times 1}$ rad/s², ${\dot{\varphi }}^{+}=-{\dot{\varphi }}^{-}={\left[0.6\right]}_{7\times 1}$ rad/s, ${\varphi }^{+}=-{\varphi }^{-}={\left[2.5\right]}_{7\times 1}$ rad and θ = [0.076]_{7 × 1} rad. It is noteworthy that the parameters can be divided into two categories: structural parameters and convergence parameters. Structural parameters, such as l, u₀, v₀, a_x, and a_y, are dependent on the configuration of the visual servo system. On the other hand, the convergence parameters, namely, μ, ν, α, β, and δ, play a vital role in adjusting the convergence behavior of RNN (16). These convergence parameters are set to values greater than zero, and their specific values can be determined through the trial and error method.

5.1 Simulation of AIVS scheme

In this subsection, in order to prove the feasibility of the AIVS scheme (9), four simulations with different initial position states of the Franka Emika Panda manipulator are conducted to trace one desired feature with results shown in Figure 3. Simply design α = 10 and β = 10. It would be readily discovered from Figure 3A that four test examples from four different directions are straightforward to successfully pursue the desired pixel. With test 4 as an example, detailed joint data and pixel errors are shown in Figure 3B through Figure 3F, which illustrate that the joint angle, joint velocity, and joint acceleration are all kept inside the joint limit and that the pixel error can converge to zero within 5 s. The above descriptions well verify the validity of the proposed AIVS scheme (9) in the case of the known visual servoing Jacobian matrix to solve the visual servoing problem at the acceleration level.

Figure 3

Figure 3. Simulations on a Franka Emika Panda manipulator carrying out IBVS task synthesized by the AIVS scheme (9) assisted by RNN (13) with four test examples. (A) Profiles of feature trajectories and desired pixel point in four tests. (B) Profiles of joint acceleration in test 4. (C) Profiles of joint velocity in test 4. (D) Profiles of joint angle in test 4. (E) Profiles of pixel error in test 4. (F) Profiles of Euclidean norm of pixel error in test 4.

5.2 Simulation of DAIVS scheme

This subsection indicates the feasibility and capability of the pixel error convergence of the DAIVS scheme aided with the RNN (16) by providing simulation results, as shown in Figure 4. Furthermore, we choose δ = 2 × 10⁴, α = 10 and β = 40. Notably, the virtual visual Jacobian matrix is exploited with random initial values, instead of the real visual Jacobian matrix to facilitate system operation. The end-effector of the robotic arm is oriented toward the object, as shown in Figure 4A. In addition, the joint acceleration is shown in Figure 4B, which is confined to the joint limit and maintain the normal operation. As shown in Figure 4C, the Franka Emika Panda manipulator successfully traces the desired feature with pixel error converging to zero and maintaining the order of 10⁻² pixel. As for the learning ability, Figure 4D illustrates that the virtual robot manipulator can learn the movement of the real robot manipulator with the learning error approaching to zero in 0.05 s and maintaining the order of 10⁻⁴ pixel/s. In short, the simulation results in Figure 4 highlight the simultaneous learning and control ability of RNN (16).

Figure 4

Figure 4. Simulation results on a Franka Emika Panda manipulator with unknown structure carrying out the IBVS task synthesized by the DAIVS scheme assisted with RNN (16). (A) Profiles of the movement process. (B) Profiles of the joint acceleration. (C) Profiles of the pixel error. (D) Profiles of the Euclidean norm of learning error.

5.3 Comparisons of proposed schemes

This subsection offers simulation comparison results between the proposed schemes aided with the corresponding RNNs and the IBVS method presented in the study by Zhang and Li (2018). In this regard, the RNN provided in the study by Zhang and Li (2018) is shown as

$\begin{array}{ll}\dot{\varphi }=P_{\gamma }(-{\kappa }_1{\mathcal{J}}^{\text{T}}(p-p_{\text{d}})-{\kappa }_2{\mathcal{J}}^{\text{T}}{\displaystyle {\int }_0^t(p-p_{\text{d}})}\text{d}t),& (18)\end{array}$

where parameters κ₁>0 and κ₂>0 determine the rate of error convergence. It is worth pointing out that the IBVS method in the study by Zhang and Li (2018) assisted with RNN (18) is constructed from the viewpoint of the velocity level, and that, RNN (18) requires exact structural information 𝒥 to maintain the normal operation.

In the first place, simulations are conducted on the Franka Emika Panda manipulator for IBVS task with Figures 5A–D synthesized by RNN (13) and Figures 5E–H synthesized by RNN (18). Notably, the results in Figure 5 are carried out on the premise of known structural information 𝒥 with parameters κ₁ = κ₂ = 2, α = 10, and β = 10. As shown in Figures 5A, E, the manipulator's end-effector is controlled to point toward the object. In Figure 5B, the joint acceleration generated by RNN (13) is safely confined within the joint limits, while the joint acceleration generated by RNN (18) exists a sudden change of ~38 rad/s² in Figure 5F, which may cause damage to the robot. Furthermore, in contrast to Figure 5G, the joint velocity shown in Figure 5C is smaller and exhibits smoother changes, making it more suitable for real-world scenarios. Figures 5D, H demonstrate that both RNN (13) and RNN (18) are able to quickly propel pixel errors to zero. Therefore, it is concluded from the above results that AIVS scheme (9) aided by RNN (13) is able to guarantee a better safety performance when controlling the manipulator.

Figure 5

Figure 5. Simulation results on a Franka Emika Panda manipulator with accurate structure information carrying out IBVS task. (A) Profiles of motion process assisted with RNN (13). (B) Profiles of joint acceleration assisted with RNN (13). (C) Profiles of joint velocity assisted with RNN (13). (D) Profiles of pixel error assisted with RNN (13). (E) Profiles of motion process assisted with RNN (18). (F) Profiles of joint acceleration assisted with RNN (18). (G) Profiles of joint velocity assisted with RNN (18). (H) Profiles of pixel error assisted with RNN (18).

Beyond that, in the case of the unknown visual system, corresponding comparison simulations are driven by the DAIVS scheme aided with the RNN (16) and the IBVS method in the study by Zhang and Li (2018) assisted with RNN (18). The results are shown in Figure 6 with parameters κ₁ = κ₂ = 2, α = 10, β = 40, and δ = 2 × 10⁴. To simulate the unknown visual system, $\bar{\mathcal{J}}$ in Equation (16) and 𝒥 in Equation (18) are random matrices of constants with the absolute value of each element < 100. Figures 6A, B well embody that, when encountering unknown structural information, the DAIVS scheme assisted with RNN (16) controls the Franka Emika Panda manipulator to preferably complete IBVS task with the pixel error converging to zero. Nevertheless, the generated joint velocity in Figure 6C changes dramatically within the joint limit in a mess. Even worse, the pixel error driven by RNN (18) does not converge and maintain a diffused state, which indicates the failure of the IBVS task. In conclusion, the proposed DAIVS scheme is able to deal with the unknown structural information in the robot system and fulfill the visual servo control with simultaneous learning and control performance.

Figure 6

Figure 6. Simulation results on a Franka Emika Panda manipulator carrying out IBVS task with unknown structure. (A) Profiles of joint velocity assisted with RNN (16). (B) Profiles of pixel error assisted with RNN (16). (C) Profiles of joint velocity assisted with RNN (18). (D) Profiles of pixel error assisted with RNN (18).

Furthermore, comparison results among different existing approaches (Hashimoto et al., 1991; Keshmiri et al., 2014; Zhang et al., 2017; Van et al., 2018; Zhang and Li, 2018; Anwar et al., 2019; Li et al., 2020; Zhu et al., 2022) for visual servoing of robot manipulators are presented in Table 1. It is worth emphasizing that, compared with the prior art, the proposed RNN (13) and RNN (16) are the first acceleration-level work, considering the multiple levels of joint constraints, and RNN (16) is the first study to dispose the unknown situations in the robot visual system with simultaneous learning and control ability. As a result, the above two points are the innovative contributions of this study.

Table 1

Table 1. Comparisons among different approaches for visual servoing of robot manipulators.

6 Experiments on real manipulators

To verify the effectiveness and practicability of the proposed DAIVS scheme , physical experiments on a real manipulator are conducted in this section, which are driven by the DAIVS scheme aided with RNN (16). Specifically, the experiments essentially rely on C++ and the visual servoing platform (ViSP) for embedding algorithms and control (Marchand et al., 2005), which are built on ubuntu 16.04 LTS operating system. In addition, the experiment platform consists of a Franka Emika Panda manipulator, an Intel RealSense Camera D435i, a personal computer, and an AprilTag (target). It is worth mentioning that the acceleration control commands generated by the proposed RNN (16) are transmitted in a discrete form with a frequency of 1,000 Hz, and parameter settings of RNN (16) are designed as follows. We choose α = 10, β = 10, δ = 10⁶, μ = ν = 20, ${\ddot{\varphi }}^{+}=-{\ddot{\varphi }}^{-}={\left[15,7.5,10,12.5,15,20,20\right]}^{\mathrm{\text{T}}}$ rad/s², ${\dot{\varphi }}^{+}=-{\dot{\varphi }}^{-}={\left[2.1,2.1,2.1,2.1,2.6,2.6,2.6\right]}^{\mathrm{\text{T}}}$ rad/s, ϕ⁺ = [2.8, 1.7, 2.8, −0.1, 2.9, 3.7, 2.8]^T rad, ϕ⁻ = [−2.8, −1.7, −2.8, −3.0, −2.8, −0, −2.8]^T rad, and $\bar{\mathcal{J}}(0)=\mathcal{J}(0)$. As for the parameter settings of the camera and pixel coordinates, they can be directly referenced to ViSP (Marchand et al., 2005). Different from the previous simulations, the physics experiments set the target as an AprilTag containing four features. As a result, the physical parameters associated with the features are expanded to 8 instead of 2.

Experiment results on the Franka Emika Panda manipulator tracking the fixed target are shown in Figures 7, 8 with ${\text{p}}_{\mathrm{\text{d}}}={\left[-0.06,-0.06,0.06,-0.06,0.06,0.06,-0.06,0.06\right]}^{\mathrm{\text{T}}}$ m for the given task in the camera system. It is worth mentioning that the robot manipulator adjusts the joint state to recognize and approach the target, and when the pixel error reaches the order of 10⁻⁵ pixel, the task automatically completes. It is important that the whole process of learning and control does not involve the real Jacobian matrix to simulate the situation of the unknown structure. In Figures 7A, B, the initial and final states of the manipulator and camera indicate that the visual servoing task is successfully realized by the DAIVS scheme with execution time of 1.25 s. Specifically, the joint acceleration in Figure 7C varies normally within the joint constraints. In the meantime, the tracking errors ξ of four features are presented in Figure 7D, which illustrate the precise control ability of the DAIVS scheme with global convergence to zero.

Figure 7

Figure 7. Physical experiments on a Franka Emika Panda manipulator assisted with RNN (16) for carrying out IBVS task with a fixed target. (A) Initial states of the manipulator and camera. (B) Final states of the manipulator and camera. (C) Profiles of joint acceleration. (D) Profiles of tracking error.

Figure 8

Figure 8. Physical experiments on a Franka Emika Panda manipulator assisted with RNN (16) for carrying out IBVS task with a moving target.

Beyond that, experiments on the Franka Emika Panda manipulator tracking the moving target are conducted to demonstrate the feasibility of the DAIVS scheme . In Figure 8, the AprilTag is moved artificially by the hand toward the left and right and simultaneously the manipulator constantly adjusts joint states to achieve the characteristics of real-time visual tracking. More vividly, the experiment videos corresponding to Figures 7, 8 are available at https://youtu.be/6uw35bidVcw.

7 Conclusion

This study has proposed an AIVS scheme for robot manipulators, taking into account joint limits at multiple levels. On this basis, incorporating data-driven techniques, a DAIVS scheme has been proposed to handle potential unknown situations in the robot visual system. Furthermore, RNNs have been exploited to generate the online solution corresponding to the proposed schemes with theoretical analyses, demonstrating the simultaneous learning and control ability of the proposed DAIVS scheme. Then, numerous simulations and experiments have been carried out on a Franka Emika Panda manipulator to track the desired feature. The results validate the theoretical analyses, demonstrate the feasibility of the AIVS scheme, and showcase the fast convergence and robustness of the DAIVS scheme. Compared with the method in the study by Zhang and Li (2018), the DAIVS scheme exhibits superior learning capability and achieves visual servoing control with the unknown Jacobian matrix.

In summary, this study provides a data-driven approach for the precise manipulation of robots in IBVS tasks, addressing unknown situations that could affect the robot's Jacobian matrix. In the future, we aim to expand our research to incorporate dynamic factors, utilizing joint torque as control signals and considering dynamic uncertainties.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

LW: Investigation, Methodology, Resources, Software, Writing – original draft. ZX: Data curation, Investigation, Writing – review & editing.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. This work was supported by the Key Laboratory of Tibetan Information Processing, Ministry of Education, under Grant 2024-Z-002, and supported by the Supercomputing Center of Lanzhou University.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Agarwal, S., and Akella, S. (2024). Line coverage with multiple robots: algorithms and experiments. IEEE Transact. Robot. 40, 1664–1683. doi: 10.1109/TRO.2024.3355802

Crossref Full Text | Google Scholar

Anwar, A., Lin, W., Deng, X., Qiu, J., and Gao, H. (2019). Quality inspection of remote radio units using depth-free image-based visual servo with acceleration command. IEEE Transact. Ind. Electron. 66, 8214–8223. doi: 10.1109/TIE.2018.2881948

Crossref Full Text | Google Scholar

Cigliano, P., Lippiello, V., Ruggiero, F., and Siciliano, B. (2015). Robotic ball catching with an eye-in-hand single-camera system. IEEE Transact. Cont. Syst. Technol. 23, 1657–1671. doi: 10.1109/TCST.2014.2380175

Crossref Full Text | Google Scholar

Gaz, C., Cognetti, M., Oliva, A., Robuffo Giordano, P., and De Luca, A. (2019). Dynamic identification of the Franka Emika Panda robot with retrieval of feasible parameters using penalty-based optimization. IEEE Robot. Automat. Lett. 4, 4147–4154. doi: 10.1109/LRA.2019.2931248

Crossref Full Text | Google Scholar

Hashimoto, K., Kimoto, T., Ebine, T., and Kimura, H. (1991). “Manipulator control with image-based visual servo,” in Proceedings IEEE International Conference on Robotics and Automation (New York, NY: Sacramento, CA: IEEE), 2267–2271.

Google Scholar

Jin, P., Lin, Y., Song, Y., Li, T., and Yang, W. (2023). Vision-force-fused curriculum learning for robotic contact-rich assembly tasks. Front. Neurorobot. 17:1280773. doi: 10.3389/fnbot.2023.1280773

PubMed Abstract | Crossref Full Text | Google Scholar

Jin, L., Liu, L., Wang, X., Shang, M., and Wang, F.-Y. (2024). Physical-informed neural network for MPC-based trajectory tracking of vehicles with noise considered. IEEE Transact. Intell. Vehicl. doi: 10.1109/TIV.2024.3358229

Crossref Full Text | Google Scholar

Keshmiri, M., Xie, W.-F., and Mohebbi, A. (2014). Augmented image-based visual servoing of a manipulator using acceleration command. IEEE Transact. Ind. Electron. 61, 5444–5452. doi: 10.1109/TIE.2014.2300048

Crossref Full Text | Google Scholar

Khalil, H. K. (2001). Nonlinear Systems. Englewood Cliffs, NJ: Prentice Hall.

Google Scholar

Li, S., Shao, Z., and Guan, Y. (2019). A dynamic neural network approach for efficient control of manipulators. IEEE Transact. Syst. Man Cybernet. Syst. 49, 932–941. doi: 10.1109/TSMC.2017.2690460

Crossref Full Text | Google Scholar

Li, W., Chiu, P. W. Y., and Li, Z. (2020). An accelerated finite-time convergent neural network for visual servoing of a flexible surgical endoscope with physical and RCM constraints. IEEE Transact. Neural Netw. Learn. Syst. 31, 5272–5284. doi: 10.1109/TNNLS.2020.2965553

Crossref Full Text | Google Scholar

Liang, X., Wang, H., Liu, Y.-H., Chen, W., and Jing, Z. (2018). Image-based position control of mobile robots with a completely unknown fixed camera. IEEE Trans. Automat. Contr. 63, 3016–3023. doi: 10.1109/TAC.2018.2793458

Crossref Full Text | Google Scholar

Liufu, Y., Jin, L., Shang, M., Wang, X., and Wang, F.-Y. (2024). ACP-incorporated perturbation-resistant neural dynamics controller for autonomous vehicles. IEEE Transact. Intelli. Vehicl. doi: 10.1109/TIV.2023.3348632

Crossref Full Text | Google Scholar

Malis, E., Mezouar, Y., and Rives, P. (2010). Robustness of image-based visual servoing with a calibrated camera in the presence of uncertainties in the three-dimensional structure. IEEE Transact. Robot. 26, 112–120. doi: 10.1109/TRO.2009.2033332

Crossref Full Text | Google Scholar

Marchand, E., Spindler, F., and Chaumette, F. (2005). ViSP for visual servoing: a generic software platform with a wide class of robot control skills. IEEE Robot. Automat. Mag. 12, 40–52. doi: 10.1109/MRA.2005.1577023

Crossref Full Text | Google Scholar

Na, J., Yang, J., Wang, S., Gao, G., and Yang, C. (2021). Unknown dynamics estimator-based output-feedback control for nonlinear pure-feedback systems IEEE Transact. Syst. Man Cybernet. Syst. 51, 3832–3843. doi: 10.1109/TSMC.2019.2931627

Crossref Full Text | Google Scholar

Park, D.-H., Kwon, J.-H., and Ha, I.-J. (2012). Novel position-based visual servoing approach to robust global stability under field-of-view constraint. IEEE Transact. Ind. Electron. 59, 4735–4752. doi: 10.1109/TIE.2011.2179270

Crossref Full Text | Google Scholar

Peng, G., Chen, C. L. P., and Yang, C. (2023). Robust admittance control of optimized robot-environment interaction using reference adaptation. IEEE Transact. Neural Netw. Learn. Syst. 34, 5804–5815. doi: 10.1109/TNNLS.2021.3131261

PubMed Abstract | Crossref Full Text | Google Scholar

Stanimirovic, P. S., Zivkovic, I. S., and Wei, Y. (2015). Recurrent neural network for computing the Drazin inverse. IEEE Transact. Neural Netw. Learn. Syst. 26, 2830–2843. doi: 10.1109/TNNLS.2015.2397551

PubMed Abstract | Crossref Full Text | Google Scholar

Tang, Z., and Zhang, Y. (2022). Refined self-motion scheme with zero initial velocities and time-varying physical limits via Zhang neurodynamics equivalency. Front. Neurorobot. 16:945346. doi: 10.3389/fnbot.2022.945346

PubMed Abstract | Crossref Full Text | Google Scholar

Van, M., Ge, S. S., and Ceglarek, D. (2018). Fault estimation and accommodation for virtual sensor bias fault in image-based visual servoing using particle filter. IEEE Transact. Ind. Inf. 14, 1312–1322. doi: 10.1109/TII.2017.2723930

Crossref Full Text | Google Scholar

Wang, X., Sun, Y., Xie, Y., Bin, J., and Xiao, J. (2023). Deep reinforcement learning-aided autonomous navigation with landmark generators. Front. Neurorobot. 17:1200214. doi: 10.3389/fnbot.2023.1200214

PubMed Abstract | Crossref Full Text | Google Scholar

Xie, Z., Jin, L., Luo, X., Hu, B., and Li, S. (2022). An acceleration-level data-driven repetitive motion planning scheme for kinematic control of robots with unknown structure. IEEE Transact. Syst. Man Cybernet. Syst. 5152, 5679–5691. doi: 10.1109/TSMC.2021.3129794

Crossref Full Text | Google Scholar

Xie, Z., and Jin, L. (2023). A fuzzy neural controller for model-free control of redundant manipulators with unknown kinematic parameters. IEEE Transact. Fuzzy Syst. 32, 1589–1601. doi: 10.1109/TFUZZ.2023.3328545

Crossref Full Text | Google Scholar

Xu, C., Sun, Z., Wang, C., Wu, X., Li, B., and Zhao, L. (2023). An advanced bionic knee joint mechanism with neural network controller. Front. Neurorobot. 17:1178006. doi: 10.3389/fnbot.2023.1178006

PubMed Abstract | Crossref Full Text | Google Scholar

Xu, F., Wang, H., Wang, J., Au, K. W. S., and Chen, W. (2019). Underwater dynamic visual servoing for a soft robot arm with online distortion correction. IEEE ASME Transact. Mechatron. 24, 979–989. doi: 10.1109/TMECH.2019.2908242

Crossref Full Text | Google Scholar

Yang, C., Chen, C., He, W., Cui, R., and Li, Z. (2019). Robot learning system based on adaptive neural control and dynamic movement primitives. IEEE Transact. Neural Netw. Learn. Syst. 30, 777–787. doi: 10.1109/TNNLS.2018.2852711

PubMed Abstract | Crossref Full Text | Google Scholar

Zeng, D., Liu, Y., Qu, C., Cong, J., Hou, Y., and Lu, W. (2024). Design and human-robot coupling performance analysis of flexible ankle rehabilitation robot. IEEE Robot. Automat. Lett. 9, 579–586. doi: 10.1109/LRA.2023.3330052

Crossref Full Text | Google Scholar

Zeng, Z., Wang, J., and Liao, X. (2003). Global exponential stability of a general class of recurrent neural networks with time-varying delays. IEEE Transact. Circ. Syst. I Fund. Theory Appl. 50, 1353–1358. doi: 10.1109/TCSI.2003.817760

Crossref Full Text | Google Scholar

Zhang, Y., and Li, S. (2018). A neural controller for image-based visual servoing of manipulators with physical constraints. IEEE Transact. Neural Netw. Learn. Syst. 29, 5419–5429. doi: 10.1109/TNNLS.2018.2802650

PubMed Abstract | Crossref Full Text | Google Scholar

Zhang, Y., and Li, S. (2023). Kinematic control of serial manipulators under false data injection attack. IEEE CAA J. Automat. Sin. 10, 1009–1019. doi: 10.1109/JAS.2023.123132

Crossref Full Text | Google Scholar

Zhang, Y., Li, S., Liao, B., Jin, L., and Zheng, L. (2017). “A recurrent neural network approach for visual servoing of manipulators,” in 2017 IEEE International Conference on Information and Automation (ICIA) (New York, NY: Macao: IEEE), 614–619.

Google Scholar

Zhang, Z., and Zhang, Y. (2012). Acceleration-level cyclic-motion generation of constrained redundant robots tracking different paths. IEEE Transact. Syst. Man Cybernet. Part B 42, 1257–1269. doi: 10.1109/TSMCB.2012.2189003

PubMed Abstract | Crossref Full Text | Google Scholar

Zheng, X., Liu, M., Jin, L., and Yang, C. (2024). Distributed collaborative control of redundant robots under weight-unbalanced directed graphs. IEEE Transact. Ind. Inf. 20, 681–690. doi: 10.1109/TII.2023.3268778

Crossref Full Text | Google Scholar

Zhu, M., Huang, C., Qiu, Z., Zheng, W., and Gong, D. (2022). Parallel image-based visual servoing/force control of a collaborative delta robot. Front. Neurorobot. 16:922704. doi: 10.3389/fnbot.2022.922704

PubMed Abstract | Crossref Full Text | Google Scholar