Visual-admittance-based model predictive control for nuclear collaborative robots

This paper presents a novel visual-admittance-based model predictive control scheme to cope with the problem of vision/force control and several constraints of a nuclear collaborative robotic visual servoing system. A visual-admittance model considering the desired image feature and force command in the image feature space is proposed. Moreover, a novel constraint scheme of the model predictive control (MPC) is proposed to cancel the overshoot in the interaction force control for most cases by taking the desired force command as the constraint of the proposed MPC. Via applying the robotic dynamic image-based visual servoing (IBVS) model, some other constraints, such as the actuator saturation, joint angle, and visual limits, can be satisfied simultaneously. The simulation results for the two-degrees-of-freedom (DOF) robot manipulator with an eye-to-hand camera are present to demonstrate the effectiveness of the proposed controller.

1 Introduction

In recent years, collaborative robots have become more and more popular in the nuclear industry since the increasing need for safe and secure nuclear power plants (NPPs). Interaction between the robot and the environment or human is a fundamental requirement for the construction of NPPs 1, such as moving objects, deburring, grinding, polishing, and high-precision assembling (Nabat et al., 2005; Yang et al., 2008; Xu et al., 2017; Wu et al., 2017; Song et al., 2022, Figure 1). How to realize the precision position control and compliant interaction force control simultaneously in the nuclear environment is crucial for the execution of an interaction task.

FIGURE 1

FIGURE 1. Application of collaborative robots in nuclear power plants (NPPs).

To cope with this problem, numerous force control schemes have been proposed. Two main categories of those control schemes are direct and indirect force control. Direct force control adds the feedback of the force sensor directly into the closed motion control loop. For example, in the hybrid position/force control approach, the force and position controllers work separately and are connected with a diagonal section matrix (Raibert and Craig, 1981). However, these kinds of schemes lead to a trade-off between the force controller and the position controller, which implies that neither force nor position can precisely converge to their desired target. The indirect force control includes impedance control and admittance control. They all create a mass–spring–damper (MSD) model to describe the interaction force between the robot and the environment (Mason, 1981; Hogan, 1985) and are inverse to each other. When the robot is controlled with the conventional model-based controller, precise interaction environment information and an accurate dynamic model of the robot are essential in compliant force control.

In the nuclear environment, where nuclear radiation can affect sensors and electronics, measuring accurate knowledge of the interaction environment and robot feedback is quite hard. Motivated by reducing or avoiding the need for precise preliminary knowledge of the environment, numerous researchers are trying to equip robot systems with vision and force sensors together. This method allows the robot system to get the environmental information and modify the vision/force controller online. In order to combine the visual and force information, several vision/force control schemes have been proposed to replace the motion control of the aforementioned force control with visual servoing (Bellakehal et al., 2011; Zhu et al., 2022b). For instance, in the hybrid vision/force control scheme (Zhu et al., 2022b), the force and vision are controlled separately, and a trade-off matrix is used to combine the output of two sensor controllers. Thus, these kinds of methods may result in local minima and reduce the robot control precision. In Lippiello et al. (2018), three vision-impedance controllers with the feedback of the camera and force sensor have been proposed to realize the physical interaction of a dual-arm unmanned aerial manipulator. However, the impedance control schemes can neither combine the vision and force sensing simultaneously nor avoid the trade-off between the two control loops.

To handle this problem, we propose a novel visual-admittance-based model to drive the robot with the trajectory of vision and the command of contact force. In this approach, the vision and force can be coupled in the image feature space successfully. Therefore, the convergence to a local minimum can be avoided, and the discrepancy between the modalities of vision and force sensors can be overcome.

However, due to the combination of force and vision in the image feature space, the controller is only employed to track the image feature commands generated from the aforementioned vision-admittance model without real-time perception of the contact force, which may lead to the unacceptable overshoot phenomenon and thus break the interaction environment or robot body. Therefore, we propose to consider the force command as a contact force constraint of the vision tracking controller. In practical applications, there are some other constraints on robot control systems, such as actuator saturation and visual limitation, that prevent the visual features from leaving the camera view.

Model predictive control (MPC) has been proven to be an efficient optimal control scheme aimed at addressing the disturbances and constraints of the system (Allibert et al., 2010; Deng et al., 2022). Recently, more and more researchers have concentrated on the advantages of MPC and applied this control scheme to the visual control of robots. In contrast to the aforementioned vision/force control scheme, MPC can explicitly handle the system constraints and resist disturbances. In Allibert et al. (2010), an MPC-based image-based visual servoing (IBVS) method is designed based on the conventional image Jacobian matrix and considers the constraints of image features. According to Hajiloo et al. (2016), a robust online MPC scheme based on the tensor product (TP) model of IBVS is proposed to realize the online control of MPC. By simplifying the image Jacobian matrix as a TP model, the computational speed of MPC is considerably increased. In Fusco et al. (2020b), the image feature acceleration is used to construct the model of MPC in order to get a shorter sensor trajectory and better motion in the feature space in Cartesian space. However, the aforementioned methods only consider the image feature constraints and kinematics constraints, such as the joint velocity or acceleration of the robot system, without taking into account the dynamic constraints of the robot system.

In general, in this paper, a novel visual-admittance-based MPC scheme with non-linear constraints based on the dynamic IBVS of the collaborative robot and desired force constraint is proposed to address the problem of the vision/force control with constraints. The visual-admittance model is formulated to converge the force and vision commands into image features to avoid the trade-off between two targets’ (vision and force) tracking control. Considering the overshoot of the visual-admittance model, force constraints are added to the predictive model of the MPC. Moreover, the non-linear input bounds are added to the proposed MPC based on the dynamic model of the robot.

This paper is organized as follows: in Section 2, some preliminaries of the robot dynamic model and IBVS are presented. The vision-admittance-based trajectory generator is introduced to combine the force and visual information in Section 3. Section 4 presents the MPC controller with the non-linear constraints based on the dynamic model of the robot and force constraints. In Section 5, simulations based on a robot manipulator model with an eye-to-hand camera are performed to verify the effects of the proposed control scheme. Finally, the conclusions are given in Section 6.

2 Preliminaries

2.1 Dynamic model of the robot manipulator

The dynamic model of the manipulator robot with n actuators in the task space is often described as an Euler–Lagrange second-order equation (Roy et al., 2018). When the end effector of the robot comes into contact with the environment, the environment exerts an interactive force on the robot system. Considering the external force developing from the contact between the end-effector of the robot and the environment, the dynamic model of a robot can be written as

$\mathbf{M}\left(q\right)\ddot{q}+\mathbf{C}\left(q,\dot{q}\right)\dot{q}+\mathbf{G}+{\mathbf{J}}^T{\mathbf{F}}_e=\mathbf{T}+{\mathbf{T}}_d,(1)$

where $q,\dot{q},\ddot{q}\in {\mathcal{R}}^n$ represent the position, velocity, and acceleration of the actuate joint angle, respectively. $\mathbf{M}(q(t))\in {\mathcal{R}}^{n\times n}$ is the inertia matrix of the robot. $\mathbf{C}(q,\dot{q})$ denotes the Coriolis–centripetal matrix of the robot torques. $\mathbf{G}\in {\mathcal{R}}^{6\times 1}$ is the gravitational vector. J is the Jacobian matrix of the robot system. ${\mathbf{F}}_e\in {\mathcal{R}}^{6\times 1}$ represents the interactive force vector between the robot and environment. $\mathbf{T},{\mathbf{T}}_d\in {\mathcal{R}}^n$ represent the input vector of the proposed controller and disturbance of the robotic system, respectively.

2.2 Dynamic model of image-based visual servoing

Image-based visual servoing (IBVS) is a sensor-based control scheme. It uses cameras as the main sensor to estimate the pose of the robot directly.

In this work, an eye-to-hand IBVS system is established to get the visual information, and we denote the visual feature estimated by the single fixed camera as $s\in {\mathcal{R}}^m$. For a point p = (X,Y,Z)^T in the Cartesian space, the projection in the 2D image space is given as

$\left\{\begin{array}{cc}\hfill x=& X/Z=\left(x_s-c_x\right)/P_x,\hfill \\ \hfill y=& Y/Z=\left(y_s-c_y\right)/P_y\hfill \end{array}\right.(2)$

where s = (x,y)^T is the projection of p in the image space and $s_i={(x_i,y_i)}^T$ represents the coordinates of the image point expressed in pixel units. c_x, c_y, p_x, and p_y are the intrinsic parameters of the camera. c_x and c_y denote the coordinates of the principal point. p_x and p_y are the ratios between the focal length and pixel size.

Differentiating Eq. 2, the relationship between the time variation of the visual feature vector $\dot{s}$ and the spatial relative camera–object velocity of the robot $\dot{p}$ can be written as described in Mariottini et al. (2007)as follows:

$\dot{s}={\mathbf{L}}_s\dot{p},(3)$

where L_s is the interaction matrix related to s (Chaumette and Hutchinson, 2006; Zhu et al., 2022a) given as

$\left[\begin{array}{cccccc}\hfill -1/Z\hfill & \hfill 0\hfill & \hfill x/Z\hfill & \hfill xy\hfill & \hfill -\left(1+x^2\right)\hfill & \hfill y\hfill \\ \hfill 0\hfill & \hfill -1/Z\hfill & \hfill y/Z\hfill & \hfill 1+y^2\hfill & \hfill -xy\hfill & \hfill -x\hfill \end{array}\right].(4)$

The robot spatial velocity $\dot{p}$ can be transformed to the actuator velocity $\dot{q}$ with the Jacobian matrix J. Then, Eq. 3 can be written as

$\dot{s}={\mathbf{L}}_s^c{\mathbf{T}}_e\mathbf{J}\dot{q},(5)$

where ^cT_e denotes the transform matrix from the kinematic screw to the camera frame. We define ${\mathbf{J}}_s={\mathbf{L}}_s^c{\mathbf{T}}_e\mathbf{J}$. Equation 5 can be rearranged as follows:

$\dot{s}={\mathbf{J}}_s\dot{q}.(6)$

In order to generate the actuator torque, the relationship between the image feature and acceleration of the actuator joint has been formulated in the IBVS dynamic model (Fusco et al., 2020a; Liang et al., 2022).

By differentiating Eq. 3, the second-order interaction model can be demonstrated as

$\ddot{s}={\mathbf{L}}_s\ddot{p}+{\dot{\mathbf{L}}}_s\dot{p}.(7)$

When differentiating Eq. 5, we derive

$\begin{array}{ll}\hfill \ddot{s}& ={\dot{\mathbf{L}}}_s^c{\mathbf{T}}_e\mathbf{J}\dot{q}+{\mathbf{L}}_s^c{\dot{\mathbf{T}}}_e\mathbf{J}\dot{q}+{\mathbf{L}}_s^c{\mathbf{T}}_e\dot{\mathbf{J}}\dot{q}+{\mathbf{L}}_s^c{\mathbf{T}}_e\mathbf{J}\ddot{q}\hfill \\ \hfill & ={\mathbf{J}}_s\ddot{q}+\mathbf{P}\dot{q},\hfill \end{array}(8)$

where $\mathbf{P}={\dot{\mathbf{L}}}_s^c{\mathbf{T}}_e\mathbf{J}+{\mathbf{L}}_s^c{\dot{\mathbf{T}}}_e\mathbf{J}+{\mathbf{L}}_s^c{\mathbf{T}}_e\dot{\mathbf{J}}$.

Substituting Eqs 5, 8 into Eq. 1, the dynamic model of IBVS can be written in the following form:

$\mathbf{T}=\mathbf{M}{\mathbf{J}}_s^{+}\ddot{s}+\widehat{\mathbf{C}}\dot{s}+\mathbf{G}+{\mathbf{J}}^T{\mathbf{F}}_e-{\mathbf{T}}_d,(9)$

where $\widehat{\mathbf{C}}=\mathbf{C}{\mathbf{J}}_s^{+}-\mathbf{M}{\mathbf{J}}_s^{+}\mathbf{P}{\mathbf{J}}_s^{+}$ and ⁺ represents the pseudo-inverse.

Assumption 1: During the control process, the robot and the IBVS controller do not encounter the controller singularity (Zhu et al., 2020). J and L_s are full-rank matrices.

3 Visual-admittance-based trajectory generator

In this section, a novel visual-admittance-based (VAB) trajectory generation is proposed and used to generate a reference trajectory for the robot end-effector online with feedback from the force sensor and camera in the image feature space. The structure of the robot manipulator system with an eye-to-hand visual servoing system is given in Figure 2.

FIGURE 2

FIGURE 2. Structure of the visual-admittance-based model predictive control.

The contact dynamic model between the robot end-effector and environment is often considered a second-order MSD model in the Cartesian space (Wu et al., 2022). We assume the contact model as follows:

${\mathbf{K}}_M{\ddot{e}}_p+{\mathbf{K}}_D{\dot{e}}_p+{\mathbf{K}}_S{e}_p={\mathbf{F}}_e,(10)$

where e_p = p_r − p_d represents the error between the reference trajectory and the desired trajectory of the robot end-effector in the Cartesian space. K_M, K_D, K_S denote the positive definite impedance parameters.

Due to the intrinsic technological constraints (such as the need for certain robot intrinsic parameters), the visual/force control cannot be realized directly (Oliva et al., 2021). Therefore, the vision-admittance-based trajectory generator is designed to avoid those limits.

First, after some simple manipulations with Eq. 9, the relationship between image features and external force can be rewritten as follows:

$\ddot{s}+\bar{\mathbf{C}}\dot{s}+d_s+f_{\mathit{pumi}}=f_u,(11)$

where

$\begin{array}{cc}\hfill \bar{\mathbf{C}}& ={\mathbf{J}}_s{\mathbf{M}}^{-1}\widehat{\mathbf{C}},\hfill \\ \hfill d_s& ={\mathbf{J}}_s{\mathbf{M}}^{-1}\left(\mathbf{G}-{\mathbf{T}}_d,\right)\hfill \\ \hfill f_{\mathit{pumi}}& ={\mathbf{J}}_s{\mathbf{M}}^{-1}\mathbf{J}{\mathbf{F}}_e={\mathbf{J}}_{sf}^T{\mathbf{F}}_e,\hfill \\ \hfill f_u& ={\mathbf{J}}_s{\mathbf{M}}^{-1}\mathbf{T},\hfill \end{array}(12)$

where f_pumi represents the vector of per unit mass/inertia (p.u.m.i) virtual forces applied on the image feature space corresponding to the robot in the Cartesian space. f_u denotes the actuator torque projection in the image feature space.

Multiplying both sides of Eq. 10 by J_s, the relationship between f_pumi and the position error e_p can be rearranged as follows:

$f_{\mathit{pumi}}={\mathbf{J}}_s{\mathbf{K}}_M{\ddot{e}}_p+{\mathbf{J}}_s{\mathbf{K}}_D{\dot{e}}_p+{\mathbf{J}}_s{\mathbf{K}}_S{e}_p.(13)$

Substituting Eqs 2, 3, 7 into Eq. 13 and denoting p = P_zs, the impedance model corresponding to the image feature can be rearranged as

$f_{\mathit{pumi}}={\widehat{\mathbf{K}}}_M{\ddot{e}}_s+{\widehat{\mathbf{K}}}_D{\dot{e}}_s+{\widehat{\mathbf{K}}}_S{e}_s,(14)$

where

$\begin{array}{cc}\hfill {\widehat{\mathbf{K}}}_M& ={\mathbf{J}}_s{\mathbf{K}}_M{\mathbf{L}}_s,\hfill \\ \hfill {\widehat{\mathbf{K}}}_D& ={\mathbf{J}}_s\left({\mathbf{K}}_D-{\mathbf{K}}_M{\mathbf{L}}_s^{+}{\dot{\mathbf{L}}}_s\right){\mathbf{L}}^{+},\hfill \\ \hfill {\widehat{\mathbf{K}}}_S& ={\mathbf{J}}_s{\mathbf{K}}_S{\mathbf{P}}_z,\hfill \end{array}(15)$

where e_s = s_r−s_d represents the error between the reference trajectory generated from the interaction force and the desired trajectory. When the contact force of the robot is zero, s_r = s_d. Otherwise, the vision-based admittance model can generate the reference trajectory to perform the force control of the system.

When the system is at equilibrium, the error of the image feature acceleration will converge to zero. To simplify the admittance model and reduce the calculation complexity, we set ${\ddot{e}}_s=0$ and the simplified model can be described as

${\dot{e}}_s=-\frac{{\widehat{\mathbf{K}}}_S}{{\widehat{\mathbf{K}}}_D}{e}_s+\frac{f_{\mathit{pumi}}}{{\widehat{\mathbf{K}}}_D}.(16)$

By solving Eq. 16, we obtain

$e_s\left(t\right)=s_0{e}^{\mathbf{A}\left(t\right)}+e^{\mathbf{A}\left(t\right)}{\int }_0^t\frac{f_{\mathit{pumi}}}{{\widehat{\mathbf{K}}}_D}{e}^{-\mathbf{A}\left(\tau \right)}d\tau ,(17)$

where $\mathbf{A}(t)=-{\int }_0^t\frac{{\widehat{\mathbf{K}}}_S(\tau )}{{\widehat{\mathbf{K}}}_D(\tau )}d\tau$ and s₀ is the initial state of the image feature. Then, the reference trajectory can be described as

$\begin{array}{ll}\hfill s_r& =e_s\left(t\right)+s_d\hfill \\ \hfill & =s_0{e}^{\mathbf{A}\left(t\right)}+e^{\mathbf{A}\left(t\right)}{\int }_0^t\frac{f_{\mathit{pumi}}}{{\widehat{\mathbf{K}}}_D}{e}^{-\mathbf{A}\left(\tau \right)}d\tau +s_d.\hfill \end{array}(18)$

Considering the desired force command ${\mathbf{F}}_d\in {\mathcal{R}}^6$, we propose a novel integral force control law (IFCL) to obtain the p.u.m.i virtual force f_pumi as follows:

$f_{\mathit{pumi}}\left(t\right)={\mathbf{J}}_{sf}\left({\mathbf{K}}_d{\dot{e}}_f+{\mathbf{K}}_p{e}_f+{\mathbf{K}}_i{\int }_0^t{e}_f\left(\tau \right)d\tau \right),(19)$

where e_f = F_d − F_e, F_d is the predefined desired contact force, and K_d = diag(k_d1, ⋯k_dn), K_p = diag(k_p1, ⋯k_pn), and K_i = diag(K_i1, ⋯k_in) are positive defined diagonal matrices.

4 Visual/force predictive controller

4.1 Model predictive control

MPC is an optimal control scheme proposed to solve the multi-variable constraint control problem. It determines the best input signal for a system by considering the finite future evolution of the system state. More precisely, it finds an optimal control sequence U* that minimizes a cost function J_m with a set of predefined constraints over a finite predictive horizon N_p. The obtained control sequence is described as $U_t^{*}=\{u_t^{*},\dots ,u_{t+N_c-1}^{*}\}$, where $u_{i-1}^{*}$ is the optimal input signal in step i and N_c ≤ N_p is called control horizon. The state predictions during the control horizon are calculated using the independent control inputs, while the remaining inputs are equal to the last elements of the control sequence. The cost function J_m is the sum of the quadratic error between the predictive state x_t+i and the desired state $x_{t+i}^{*}$. The optimal problem is shown as follows:

${\mathbf{U}}_t^{*}=\underset{{\mathbf{U}}_t}{\mathit{min}}{J}_m\left(t\right),(20)$

with

$J_m\left(t\right)=\sum _{i=1}^{N_p-1}{e}_{t+i}^T{\mathbf{Q}}_s{e}_{t+i}+\sum _{i=0}^{N_c-1}{u}_{t+i}^T{\mathbf{Q}}_u{u}_{t+i}+e_{N_p}^T{\mathbf{Q}}_p{e}_{N_p},(21)$

which is subjected to the constraints

$x_{t+i+1}=f\left(x_{t+i},u_i\right),(\mathrm{22a})$

$u_{t+i}\in \mathcal{U}\subseteq {\mathbf{R}}^{n_u},i=0,\dots ,N_c-1,(\mathrm{22b})$

$x_{t+i}\in \mathcal{X}\subseteq {\mathbf{R}}^{{\mathbf{n}}_{\mathbf{x}}},i=1,\dots ,N_p,(\mathrm{22c})$

where $e_{t+i}=x_{t+i}-x_{t+i}^{*}$ and i represents time i = iT_s, where T_s is the control sample. Q_s, Q_u, and Q_p are the positive definite matrices that denote the relative importance of different components in cost function (Eq. 21). The constraint in Eq. 22a is the predictive model of MPC and is often chosen as the discrete-time model of a system. $\mathcal{U}$ and $\mathcal{X}$ are, respectively, the user-defined bounds of the input signal and system state. Once the optimal problem is solved, only $u_t^{*}$ is fed to the system, and the obtained control sequence is used to generate the next initial control sequence of the optimal problem. Then, the optimal process is repeated.

4.2 The predictive model

In this work, we choose the joint configuration, interaction force, and image features as the state vectors of MPC. This allows the controller to explicitly take the image feature and force constraints into account and handle the redundancy during the optimization process. Then, the discrete predictive model of the IBVS robot system can be described as

$\left[\begin{array}{c}\hfill q_{t+i+1}\hfill \\ \hfill {\dot{q}}_{t+i+1}\hfill \\ \hfill s_{t+i+1}\hfill \\ \hfill F_{t+i+1}\hfill \end{array}\right]=\left[\begin{array}{c}\hfill q_{t+i}+{\dot{q}}_{t+}\mathrm{\Delta }t\hfill \\ \hfill {\dot{q}}_{t+i}\hfill \\ \hfill s_{t+i}+{\mathbf{J}}_s{\dot{q}}_{t+i}\mathrm{\Delta }t\hfill \\ \hfill F_{t+i}+K_{fm}\mathbf{J}{\dot{q}}_{t+i}\mathrm{\Delta }t\hfill \end{array}\right]+\left[\begin{array}{c}\hfill \frac{1}{2}\mathrm{\Delta }{t}^2\hfill \\ \hfill \mathrm{\Delta }t\hfill \\ \hfill {\mathbf{J}}_s\mathrm{\Delta }t\hfill \\ \hfill K_{fm}\mathbf{J}\mathrm{\Delta }t\hfill \end{array}\right]u_i,(23)$

where Δt is the sample time of MPC and u_i is the actuator acceleration at the step i. J_s and J are evaluated with the change in the state vector (Allibert et al., 2010). $q_t,{\dot{q}}_t,s_t,F_t$ are the initial states of the predictive model and can be obtained directly from the sensors of the robot system.

4.3 The joint acceleration constraints

The joint acceleration constraints aim at taking into account the physical boundaries of the actuators. Due to the interaction with the environment, it is possible that the input torque of the system exceeds the physical limit of the actuator torque, while the joint acceleration satisfies the rated acceleration constraints of the actuator. Thus, substituting the predictive state x_t+i into Eq. 1, the relationship between the joint acceleration and actuator torque can be rewritten as

$\begin{array}{ll}\hfill u_{t+i}& ={\mathbf{M}}^{-1}\left({\mathbf{T}}_{t+i}-\mathbf{C}{\dot{q}}_{t+i}-\mathbf{G}-{\mathbf{J}}^T{\mathbf{F}}_{t+i}\right)\hfill \\ \hfill & ={\mathbf{M}}^{-1}{\mathbf{T}}_{t+i}-{\mathrm{\Pi }}_{t+i}.\hfill \end{array}(24)$

Then, the joint acceleration constraints of the MPC can be proposed as follows:

${\widehat{u}}_{t+i}^{\mathit{min}}\le u\le {\widehat{u}}_{t+i}^{\mathit{max}},(\mathrm{25a})$

${\widehat{u}}_{t+i}^{\mathit{min}}={\mathbf{M}}^{-1}{\mathbf{T}}_{t+i}^{\mathit{min}}-{\mathrm{\Pi }}_{t+i},(\mathrm{25b})$

${\widehat{u}}_{t+i}^{\mathit{min}}={\mathbf{M}}^{-1}{\mathbf{T}}_{t+i}^{\mathit{max}}-{\mathrm{\Pi }}_{t+i},(\mathrm{25c})$

where i = 1, …, N_p and Π_t+i can be generated from the predictive state x_t+i. ${\mathbf{T}}_{t+i}^{\mathit{max}}$ and ${\mathbf{T}}_{t+i}^{\mathit{min}}$ represent the upper and lower bounds of the permissible actuator torque, respectively.

4.4 The terminal constraint (TC)

In this case, the stability of the proposed controller is ensured by a terminal constraint. TC imposes the last predicted visual features equal to the desired feature. However, it is difficult to satisfy a strict equality constraint when solving the optimizing problem. Thus, the constraints are converted into inequalities by defining a small enough threshold δ_tc.

$\Vert s_{t+N_p}-s_d\Vert -{\delta }_{tc}\le 0.(26)$

4.5 The image feature constraints

In the context of visual serving, the image feature must remain visible. The following constraint prevents the image feature from escaping the field of the camera pixel view:

$s_{t+i}^{\mathit{min}}\le s_{t+i}\le s_{t+i}^{\mathit{max}},(27)$

where i = 0, …, N_c − 1. $s_{t+i}^{\mathit{min}}$ and $s_{t+i}^{\mathit{max}}$, respectively, represent the maximum and minimum bounds of the image feature in the camera pixel in sample step i.

4.6 The force constraints

Considering that the possible overshoot of force control can break the interactive environment, we set the desired force as a state constraint given as

${\mathbf{F}}_{t+i}\le {\mathbf{F}}_{t+i}^{*}+{\delta }_f,i=1,\dots ,N_p,(28)$

where ${\mathbf{F}}_{t+i}^{*}$ is the desired force in step i, and δ_f is the permissible error of the interaction force control.

5 Simulation

In this section, to demonstrate the control effects of the proposed visual-admittance-based model predictive controller, several simulations are illustrated in the two-link manipulator with an eye-to-hand camera. The force sensor is inserted in the end-effector of the robot, and the robot is controlled to track the predefined force and image feature trajectory.

5.1 Simulation design

In this simulation, we choose a two-link robot manipulator as the control object of the proposed controller. The structure of the two-link robot manipulator with eye-to-hand camera configuration is given in Figure 3. The coordinate of the origin of the robot system is P_b = (0, 0, 0) m, and the dynamic model of this robot is given as

$\mathbf{M}\ddot{q}\left(t\right)+\mathbf{C}\dot{q}+\mathbf{G}+{\mathbf{J}}^T{\mathbf{F}}_e=\mathbf{T},(29)$

with

$\begin{array}{cc}\hfill \mathbf{M}\left(q\right)& =\left[\begin{array}{cc}\hfill {\mathbf{M}}_{11}\hfill & \hfill {\mathbf{M}}_{12}\hfill \\ \hfill {\mathbf{M}}_{21}\hfill & \hfill {\mathbf{M}}_{22},\hfill \end{array}\right]\hfill \\ \hfill \mathbf{C}\left(q,\dot{q}\right)& =\left[\begin{array}{cc}\hfill -p_1\dot{q_2}\hfill & \hfill -p_1\left({\dot{q}}_1+{\dot{q}}_2\right)\hfill \\ \hfill -p_1{\dot{q}}_1\hfill & \hfill 0,\hfill \end{array}\right]\hfill \\ \hfill \mathbf{G}\left(q\right)& =\left[\begin{array}{c}\hfill p_{2}g\mathit{cos}\left(q_1\right)+p_{3}g\mathit{cos}\left(q_1+q_2\right)\hfill \\ \hfill p_{3}g\mathit{cos}\left(q_1+q_2\right),\hfill \end{array}\right]\hfill \\ \hfill \mathbf{J}\left(q\right)& =\left[\begin{array}{cc}\hfill -l_{1}sin\left(q_1\right)-l_{2}sin\left(q_s\right)\hfill & \hfill -l_2\mathit{s}\mathit{i}\mathit{n}\left(q_s\right)\hfill \\ \hfill l_1\mathit{c}\mathit{o}\mathit{s}\left(q_1\right)+l_2\mathit{c}\mathit{o}\mathit{s}\left(q_s\right)\hfill & \hfill l_2\mathit{c}\mathit{o}\mathit{s}\left(q_s\right),\hfill \end{array}\right]\hfill \end{array}$

where,

$\begin{array}{cc}\hfill {\mathbf{M}}_{11}& =m_1{l}_{1c}^2+m_2\left(l_1^2+l_{2c}^2+2l_1{l}_{2c}\mathit{cos}\left(q_2\right)\right),\hfill \\ \hfill {\mathbf{M}}_{12}& =m_2\left(l_{2c}^2+l_1{l}_{2c}\mathit{cos}\left(q_2\right)\right)\hfill \\ \hfill {\mathbf{M}}_{21}& ={\mathbf{M}}_{12},{\mathbf{M}}_{22}=m_2{l}_{2c}^2,\hfill \\ \hfill p_1& =m_2{l}_1{l}_{2c}\mathit{sin}\left(q_2\right),p_2=m_1{l}_{1c}+m_2{l}_1,\hfill \\ \hfill p_3& =m_2{l}_{2c},q_s=q_1+q_2,\hfill \end{array}$

FIGURE 3

FIGURE 3. Structure of the two-link robot manipulator with eye-to-hand camera configuration.

where q₁ and q₂ are the joint angles of two actuators. m₁ and m₂ are the masses of links 1 and 2, respectively. l₁ and l₂ are the lengths of two links, respectively. l_1c and l_2c are the distances between the mass center of two links and the actuator joints, respectively. p₁(0) and p₂(0) are the initial joint angles. The aforementioned elements are listed in Table 1.

TABLE 1

TABLE 1. Parameters and initial joint angle of the robot.

The camera set is fixed at P_c(0, 0,−2) m, and the image plane of the camera (X’OZ’) is set to be parallel to the Cartesian plane (XOZ). The camera resolution is 1280 × 720 pixels, and the focal length of the camera is 10 mm. The ratio of a pixel and unit length is 100 pixels/mm along the two axes of the pixel plane. The frequency of the camera observation is set at 50 Hz.

The constraints of actuator torque, actuator acceleration, and the image feature are, respectively, given as

$\begin{array}{cc}\hfill -20Nm\le T& \le 20Nm,\hfill \\ \hfill -5rad/s^2\le \ddot{q}& \le 5rad/s^2,\hfill \\ \hfill \left[\begin{array}{c}\hfill 0px\hfill \\ \hfill 0px\hfill \end{array}\right]\le s& \le \left[\begin{array}{c}\hfill 1280px\hfill \\ \hfill 720px\hfill \end{array}\right].\hfill \end{array}(30)$

In this simulation, we use Eq. 10 as the dynamic model of the force sensor. In the real interactive environment, compared with k_S and K_D, the value of K_M is small enough to be ignored. Then, the parameters of this equation are given as K_S = 10000 and K_D = 0.57. The threshold of the TC is given as δ_tc = 100.

In the visual-admittance model, the parameters of Eq. 19 are given as K_p = 5 × 10⁻⁴, K_i = 6 × 10⁻⁴, and K_d = 2 × 10⁻⁵.

In order to validate the convergence capability and the control performance of the proposed controller, the position trajectory is predefined and consists of two phases: the approach phase and the interaction phase. The image feature trajectory we used in the MPC is generated by a virtual camera model from the position trajectory. During the approach phase, the robot needs to converge to P₁, (1.5, 0.4, 0) m in 1.5 s, and in the interaction phase, the robot moves from P₁ to P₂, (1.5, 0.365, 0) m in 3.5 s, and the position trajectory is given as follows:

$p_d=\left[\begin{array}{c}\hfill 1.5\hfill \\ \hfill 0.4+0.01*\left(t-1.5\right)\hfill \\ \hfill 0\hfill \end{array}\right].(31)$

During the interaction phase, a 10 N force command is applied along the x-axis from 2 to 4 s.

In order to verify the effects of the proposed controller, three sets of controllers are introduced in this simulation to drive the robot to track the desired trajectory:

•Controller 1: The MPC we proposed in this paper. The control and prediction horizon are set at n_c = 5 and n_p = 6, respectively, and the discretization time used in the controller is given as △t = 0.1 s. The parameters of the predictive model cost function are chosen as K_mf =, 10000, Q_x = diag{1000}, and Q_u = diag{0.001}.

•Controller 2: The classical MPC without the torque or interaction force constraints. The parameters of this controller are similar to Controller 1.

•Controller 3: The parallel visual and force control (PVFC) scheme proposed in Zhu et al. (2022b) with input constraints. The parameters of this controller are given as K_v = 7.5, K_p = 20, K_i = 0.01, and K_f = 0.01.

5.2 Results and analysis

In this section, to illustrate the performance of the proposed controller with the torque constraints, three controllers are investigated through the simulation on a two-DOF robot manipulator with an eye-to-hand camera. We use MATLAB to conduct the simulation and sequential quadratic programming algorithm (fmincon function in the MATLAB optimization toolbox) to solve the optimization problem of MPC. The frequency of the MPC is equal to that of the camera which is given as 50 Hz. Similarly, the frequency of the torque generator is equal to that of the joint and force sensor given as 1,000 Hz.

The image trajectories and the image errors of the three aforementioned controllers are given in Figure 4. From Figures 4A–C, we can find that the image trajectory of the IBVS system under the proposed MPC is smoother than that under the PVFC and is similar to that under the classical MPC. When the force command is applied, the overshoot of the IBVS system under the PVFC is much more serious than that of the classical MPC and proposed MPC. As is shown in Figures 4D–F, during the approach phase, the converge time of the proposed MPC (approximately 0.8 s) is shorter than that of the PVFC (approximately 1.25 s) and is very close to the classical MPC (approximately 0.75 s).

FIGURE 4

FIGURE 4. Image trajectories and errors of three controllers. (A) Image trajectory of the proposed MPC. (B) Image trajectory of the classical MPC. (C) Image trajectory of the PVFC. (D) Image errors of the proposed MPC. (E) Image errors of the classical MPC. (F) Image errors of the PVFC.

The position and force trajectories of the three aforementioned controllers are shown in Figure 5A and Figure 6, respectively. As is shown in Figure 5A, during the approach process, the length of the position trajectory in the Cartesian space under the proposed controller is shorter than that under the PVFC and, in the interaction phase, the positioning accuracy of the IBVS robot system under the proposed MPC is better than that under the PVFC. From Figures 6A, B, we can find that the overshoot and chatters of the force control in the IBVS robot system under the PVFC are the most serious within the three aforementioned controllers, and the overshoot of the force control in the IBVS system under the classical MPC still exists. Nevertheless, due to the effectiveness of the virtual interaction model and the force constraints, the overshoot and chatters of the force control in the IBVS robot system under the proposed controller are eliminated.

FIGURE 5

FIGURE 5. Position trajectories and input torques of three controllers. (A) Proposed and classical MPC. (B) PVFC.

FIGURE 6

FIGURE 6. Force trajectories of the three controllers. (A) Proposed and classical MPC. (B) PVFC.

The input torques of three controllers are given in Figure 5B. The red zone in Figure 5B is the predefined torque constraint. As is shown in Figure 5B, the classical MPC without the non-linear torque constraints cannot make the IBVS system obey the torque constraints strictly, and the proposed MPC can satisfy the torque constraints. Compared with the PVFC, in the interaction phase, the input torque of the IBVS system under the proposed MPC is smoother than that under the PVFC.

6 Conclusion

In this paper, a visual-admittance-based model predictive controller is developed to cope with the overshoot and trade-off of the classical position/force control scheme in the nuclear environment. A visual-admittance-based trajectory generator is presented to combine the desired image features and the force feedback into the reference image trajectory. Considering the integration element of the visual-admittance-based trajectory generator, we propose a novel predictive model constraints scheme. In this scheme, the desired force command is considered a constraint to avoid the overshoot of the force control, and the non-linear constraints based on the dynamic model of the robot are proposed to satisfy the actuator torque limits. A torque generator is used to generate the input signal of the robot system with the proposed MPC output and the real-time feedback of the robot joint sensors. An illustrative example of a two-DOF robot manipulator with an eye-to-hand camera is given to validate the effect of the proposed control scheme. Moreover, compared with the PVFC, the proposed MPC controller has better precision in force and position tracking. Compared with the classical MPC, the proposed controller can satisfy the image feature, torque, and force constraints and eliminate the overshoot of force control.

The proposed controller still needs information about the interaction environment, such as the elements of the virtual force model in the predictive model. Our ongoing research is finding a new observer to identify the elements of the interaction environment online. In addition, simplification of the optimization process in the MPC is in progress to construct a real-time MPC scheme of the IBVS robot system with non-linear constraints. In the future, experiments will be conducted to validate the proposed method.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary Material; further inquiries can be directed to the corresponding author.

Author contributions

JQ: methodology, software, validation, formal analysis, investigation, data curation, writing—original draft, and writing—review and editing. JC and ZX: validation, writing—review and editing, and data curation. MZ and YT: conceptualization, supervision, writing—review and editing, and funding acquisition. All authors contributed to the article and approved the submitted version.

Funding

This study was supported by the Natural Science Foundation of Sichuan Province (2023NSFSC0872) and the Shenzhen Science and Technology Program (CYJ20210324143004012).

Conflict of interest

Authors JQ, ZX, and JC were employed by the company China Nuclear Power Engineering Co., Ltd.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Allibert, G., Courtial, E., and Chaumette, F. (2010). Predictive control for constrained image-based visual servoing. IEEE Trans. Robot.26, 933–939. doi:10.1109/TRO.2010.2056590

CrossRef Full Text | Google Scholar

Bellakehal, S., Andreff, N., Mezouar, Y., and Tadjine, M. (2011). Vision/force control of parallel robots. Mech. Mach. Theory.46, 1376–1395. doi:10.1016/j.mechmachtheory.2011.05.010

CrossRef Full Text | Google Scholar

Chaumette, F., and Hutchinson, S. (2006). Visual servo control. i. basic approaches. IEEE Robot. Autom. Mag.13, 82–90. doi:10.1109/mra.2006.250573

CrossRef Full Text | Google Scholar

Deng, Y., Léchappé, V., Zhang, C., Moulay, E., Du, D., Plestan, F., et al. (2022). Designing discrete predictor-based controllers for networked control systems with time-varying delays: Application to a visual servo inverted pendulum system. IEEE/CAA J. Autom. Sin.9, 1763–1777. doi:10.1109/JAS.2021.1004249

CrossRef Full Text | Google Scholar

Fusco, F., Kermorgant, O., and Martinet, P. (2020a). “A comparison of visual servoing from features velocity and acceleration interaction models,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

Google Scholar

Fusco, F., Kermorgant, O., and Martinet, P. (2020b). Integrating features acceleration in visual predictive control. IEEE Robot. Autom. Lett.5, 5197–5204. doi:10.1109/lra.2020.3004793

CrossRef Full Text | Google Scholar

Hajiloo, A., Keshmiri, M., Xie, W.-F., and Wang, T.-T. (2016). Robust online model predictive control for a constrained image-based visual servoing. IEEE Trans. Indust. Electron.63, 1–2250. doi:10.1109/TIE.2015.2510505

CrossRef Full Text | Google Scholar

Hogan, N. (1985). Impedance control: An approach to manipulation: Part III—applications. J. Dyn. Syst. Meas. Control107, 17–24. doi:10.1115/1.3140701

CrossRef Full Text | Google Scholar

Liang, X., Wang, H., Liu, Y.-H., You, B., Liu, Z., Jing, Z., et al. (2022). Fully uncalibrated image-based visual servoing of 2dofs planar manipulators with a fixed camera. IEEE Trans. Cybern.52, 10895–10908. doi:10.1109/TCYB.2021.3070598

PubMed Abstract | CrossRef Full Text | Google Scholar

Lippiello, V., Fontanelli, G. A., and Ruggiero, F. (2018). Image-based visual-impedance control of a dual-arm aerial manipulator. IEEE Robot. Autom. Lett.3, 1856–1863. doi:10.1109/lra.2018.2806091

CrossRef Full Text | Google Scholar

Mariottini, G. L., Oriolo, G., and Prattichizzo, D. (2007). Image-based visual servoing for nonholonomic mobile robots using epipolar geometry. IEEE Trans. Robot.23, 87–100. doi:10.1109/TRO.2006.886842

CrossRef Full Text | Google Scholar

Mason, M. T. (1981). Compliance and force control for computer controlled manipulators. IEEE Trans. Syst. Man. Cybern.11, 418–432. doi:10.1109/tsmc.1981.4308708

CrossRef Full Text | Google Scholar

Nabat, V., de la O Rodriguez, M., Company, O., Krut, S., and Pierrot, F. (2005). “Par4: Very high speed parallel robot for pick-and-place,” in 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE), 553–558.

CrossRef Full Text | Google Scholar

Oliva, A. A., Giordano, P. R., and Chaumette, F. (2021). A general visual-impedance framework for effectively combining vision and force sensing in feature space. IEEE Robot. Autom. Lett.6, 4441–4448. doi:10.1109/LRA.2021.3068911

CrossRef Full Text | Google Scholar

Raibert, M. H., and Craig, J. J. (1981). Hybrid position/force control of manipulators. J. Dyn. Syst. Meas. Control.102, 126–133. doi:10.1115/1.3139652

CrossRef Full Text | Google Scholar

Roy, S., Roy, S. B., and Kar, I. N. (2018). Adaptive–robust control of euler–Lagrange systems with linearly parametrizable uncertainty bound. IEEE Trans. Contr. Syst. Technol.26, 1842–1850. doi:10.1109/TCST.2017.2739107

CrossRef Full Text | Google Scholar

Song, S., Zhu, M., Dai, X., and Gong, D. (2022). “Model-free optimal tracking control of nonlinear input-affine discrete-time systems via an iterative deterministic q-learning algorithm,” in IEEE Transactions on Neural Networks and Learning Systems, 1–14. doi:10.1109/TNNLS.2022.3178746

CrossRef Full Text | Google Scholar

Wu, J., Gao, Y., Zhang, B., and Wang, L. (2017). Workspace and dynamic performance evaluation of the parallel manipulators in a spray-painting equipment. Robot. Comput. Integr. Manuf.44, 199–207. doi:10.1016/j.rcim.2016.09.002

CrossRef Full Text | Google Scholar

Wu, R., Li, M., Yao, Z., Liu, W., Si, J., and Huang, H. (2022). Reinforcement learning impedance control of a robotic prosthesis to coordinate with human intact knee motion. IEEE Robot. Autom. Lett.7, 7014–7020. doi:10.1109/LRA.2022.3179420

CrossRef Full Text | Google Scholar

Xu, P., Li, B., and Chueng, C.-F. (2017). “Dynamic analysis of a linear delta robot in hybrid polishing machine based on the principle of virtual work,” in 2017 18th International Conference on Advanced Robotics (ICAR) (IEEE), 379–384.

CrossRef Full Text | Google Scholar

Yang, G., Chen, I.-M., Yeo, S. H., and Lin, W. (2008). “Design and analysis of a modular hybrid parallel-serial manipulator for robotised deburring applications,” in Smart devices and machines for advanced manufacturing (Springer), 167–188.

Google Scholar

Zhu, M., Chriette, A., and Briot, S. (2020). “Control-based design of a DELTA robot,” in ROMANSY 23 - Robot Design, Dynamics and Control, Proceedings of the 23rd CISM IFToMM Symposium.

CrossRef Full Text | Google Scholar

Zhu, M., Briot, S., and Chriette, A. (2022a). Sensor-based design of a delta parallel robot. Mechatronics87, 102893. doi:10.1016/j.mechatronics.2022.102893

CrossRef Full Text | Google Scholar

Zhu, M., Huang, C., Qiu, Z., Zheng, W., and Gong, D. (2022b). Parallel image-based visual servoing/force control of a collaborative delta robot. Front. Neurorobot.16, 922704. doi:10.3389/fnbot.2022.922704

PubMed Abstract | CrossRef Full Text | Google Scholar