Skip to main content

ORIGINAL RESEARCH article

Front. Robot. AI, 10 December 2021
Sec. Biomedical Robotics
This article is part of the Research Topic Rising Stars in Biomedical Robotics View all 7 articles

A Human Gesture Mapping Method to Control a Multi‐Functional Hand for Robot‐Assisted Laparoscopic Surgery: The MUSHA Case

  • 1Department of Electrical Engineering and Information Technology, University of Naples Federico II, Napoli, Italy
  • 2Department of Information Engineering and Mathematics, University of Siena, Siena, Italy
  • 3Department of Advanced Robotics, Istituto Italiano di Tecnologia, Genova, Italy

This work presents a novel technique to control multi-functional hand for robot-assisted laparoscopic surgery. We tested the technique using the MUSHA multi-functional hand, a robot-aided minimally invasive surgery tool with more degrees of freedom than the standard commercial end-effector of the da Vinci robot. Extra degrees of freedom require the development of a proper control strategy to guarantee high performance and avoid an increasing complexity of control consoles. However, developing reliable control algorithms while reducing the control side’s mechanical complexity is still an open challenge. In the proposed solution, we present a control strategy that projects the human hand motions into the robot actuation space. The human hand motions are tracked by a LeapMotion camera and mapped into the actuation space of the virtualized end-effector. The effectiveness of the proposed method was evaluated in a twofold manner. Firstly, we verified the Lyapunov stability of the algorithm, then an user study with 10 subjects assessed the intuitiveness and usability of the system.

1 Introduction

In minimally invasive surgery (MIS), surgical instruments are introduced into the patient’s body through small incisions, this practice allows to reduce trauma. As scientific research has amply demonstrated, MIS offers several benefits such as reduced pain, limited post-operative course, and short patient recovery times (Perigli et al., 2008; Sood et al., 2017; Hanley et al., 2019; Chadi et al., 2020). On the other hand, 1) the surgeon loses dexterity due to the limited range of movement of the instruments and their limited number of degrees of freedom (DoFs), 2) there is a reduction of tactile sensations with a consequent risk of tissue damage, and 3) the hand-eye coordination decrease. Nowadays, according to Tonutti et al. (2017), such of these issues are addressed by the introduction of technology in surgical rooms. Indeed, robotics has proven to be a viable solution to mitigate the limitations associated with the MIS. Rapid advances in image systems and computer technologies have led to significant advances in robotics applied to medicine. Robot-assisted surgery is now a common practice for several surgical operations and will probably become the most exploited surgical procedure of the future (Song et al., 2009; Parisi et al., 2017; Kulaylat et al., 2018; Sheetz et al., 2020). In this context, the da Vinci robot (Intuitive Surgical Inc.) demonstrated effective recovery of hand-eye coordination and motion dexterity. Research done in the last decade has been focused on overcoming existing deficiencies of robotic surgery systems, e.g., lack of haptic feedback (Okamura, 2009; Manganelli et al., 2010; Meli et al., 2014; Mohareri et al., 2014), enhancement of system integration (Handini et al., 2004; Leven et al., 2005; Takács et al., 2015), and combination of augmented reality navigation functionalities (Sugimoto et al., 2010; Yamamoto et al., 2012; Tabrizi and Mahvash, 2015). Other research and development activities aim to realize outstanding training systems, including the next generation of virtual reality simulators (Van der Meijden et al., 2010; Fontanelli et al., 2018).

More than one thousand publications are available in literature discussing medical robots control approaches and surgeon side console (Beasley, 2012). A bilateral control, with position-position architecture, is the standard approach to control short distance tele-robotic systems for surgery, according to Niemeyer and Slotine (1991). For instance, in setups equipped with the da Vinci robot the position of the surgeon’s hand is mapped into the position of the surgery tool and the foot-pedal enables a temporary disconnection between the two sides to reconfigure the control manipulator into a more comfortable and ergonomic pose. For some applications, impedance control and switched-impedance control are used as a valid alternative (Cheng and Tavakoli, 2018). For instance, a strategy exploiting force based model predictive control for beating heart surgery is presented by Bebek and Cavusoglu (2006). These approaches are usually reinforced by identification techniques of robotic arms as described in Fontanelli et al. (2017), and by techniques of shared autonomy as motion compensation, collision avoidance (Moccia et al., 2020), active constrains, and shared task space (Bowyer et al., 2013).

An innovative surgery control console is presented in Sukhoon et al. (2009), the developed robot controller fully replicate the reference movements. Similarly, in Kim et al. (2013) the authors propose a control device mechanically similar to the robot end effector, coupled with a system for regulating the joints stiffness in accordance with the exerted force. On this line, new consoles are designed to recover tactile perception integrating haptic technologies. For instance, (Yin et al., 2015) used magnetorheological fluids to return haptic sensations. Grounded commercial haptic devices as Omega (Force Dimension, CH) or Falcon (Novint Technologies Inc., United States) replace the control side in some experiments to project a more portable and sensorily complete experience of control as in the case reported in Tang et al. (2016). In Madhani et al. (1998) a solution built with two PHANToM haptic devices is presented, whereas (Cavusoglu et al., 1999) shows a six DoFs laparoscopic telesurgery workstation composed of an Immersion System Impulse Engine 3000 and two additional motors. the authors in Hu et al. (2016) developed a tendon driven exoskeleton to control a surgical robot and provide force stimuli, a camera system tracks the positions of fingers of the surgeon, while bilateral Bowden cables return kinesthetic information.

However, an open issue is the limited number of DoFs of robotic surgical end-effectors. In fact, the surgeon side manipulators of the da Vinci robot provides only eight DoFs to control the (Patient Side Manipulator) PSM position and the orientation and enclosure state of the common EndoWrist (Intuitive Surgical Inc., United States), bipolar and monopolar tools used to grasp, stamp or cut the organic tissues. Introducing extra degrees of freedom requires the development of suitable control (Gioioso et al., 2013; Meeker et al., 2018), or the development of alternative control consoles to move them independently. For instance, the MUSHA hand (depicted in Figure 1) is a three-soft-finger robotic gripper for surgical application (Liu et al., 2020). It is developed to be connected with the da Vinci robot, but it has more degrees of actuation (DoA) than DoFs of surgeon console interface. For this reason, the DoAs of the MUSHA hand are controlled synergistically, however, this reduces the potential of the instrument by limiting the number of configurations that can be reached by the fingers.

FIGURE 1
www.frontiersin.org

FIGURE 1. The MUSHA hand.

The contribution of this paper is a human to MUSHA hand mapping algorithm to perform full teleoperation of surgical hand DoAs, without increasing the mechanical complexity of the surgeon console. For this purpose, we use a camera system to track the human hand motion. The algorithm can easily be adapted to any robotic tool, the resulted algorithm is corroborated by a theoretical analysis of stability and it is followed by a preliminary study about forces exerted by fingertips.

2 Materials and Methods

The teleoperated MUSHA hand system is composed of surgical gripper and the da Vinci robot arm in patient side, while the surgeon side is comprehensive of a human hand tracker system. Communication between the surgeon and patient side is provided by Robot Operating System (ROS).

2.1 MUSHA Hand Modelling

The MUSHA hand is the main and innovative part of the teleoperation system, while the PSM of the da Vinci robot is already a known system. Then, in what follows we describe the multi-functionality of the MUSHA hand and its kinematic model which is crucial in the development of the entire control approach. According to Liu et al. (2019), and as anticipated in Section 1, the tool is a three-fingered underactuated miniature hand, each finger is composed of 12 segments and connected with a one DoF wrist, the kinematic model is composed of 37 joints connected by 36 links. An optoelectronic force sensor is designed, calibrated and presented in Liu et al. (2020), and located at the tip of each finger. In the range of the sensors, equal to 4 N, the maximal error on x-, y-, and z-direction is 0.28, 0.24, and 0.75 N, respectively. The MUSHA is actuated by the four motors of da Vinci PSM plus two additional motors. As a result, six joints are independent and directly connected to actuators by tendons, while the other ones are passive. For these reasons, it is possible to define a reduced kinematic model composed of actuated joints and equivalent links (Figure 2). The kinematic structure and the actuation system are designed to be multi-functional allowing the MUSHA hand performing three main methods of grasp, depicted in Figure 3: 1) power grasp (Po), 2) precision grasp (Pr) and 3) retractor (Rt). During the power grasp method, all fingertips and phalanges are involved to hold firmly tissue, in the precision grasp method only the fingertips are used to pinch tissue, and the retractor method allows the retraction of the tissues that come in contact with phalanges of index and middle fingertips. The relationship between joints and the tips position for a single robotic finger results as:

xr,ρ=cosq1,ρa2cosq2,ρ+a3cosq2,ρ+q3,ρyr,ρ=sinq1,ρa2cosq2,ρ+a3cosq2,ρ+q3,ρzr,ρ=a2sinq2,ρ+a3sinq2,ρ+q3,ρρT,I,M(1)

where qi,ρ is the value of the i-th joint of the robotic hand r ρ-th finger and ai is the length of the i-th link. We refer with T, I, M the thumb, index, and middle finger, respectively. Thanks to the mechanical properties of the robotic tool tendon-driven transmission, q3,ρ and q2,ρ are decoupled and thus they can be independently controlled, it follows that the model can be rewritten as

xr,ρ=cosq1,ρa2cosq2,ρ+a3cosq3,ρyr,ρ=sinq1,ρa2cosq2,ρ+a3cosq3,ρzr,ρ=a2sinq2,ρ+a3sinq3,ρρT,I,M.(2)

FIGURE 2
www.frontiersin.org

FIGURE 2. The full kinematic model of the MUSHA hand and the reduced model into joint actuated space.

FIGURE 3
www.frontiersin.org

FIGURE 3. Main grasping method of MUSHA, from left to right are respectively depicted the approach poses in case of precision grasp (Pr) in A, power grasp (Po) in B and retractor (Rt) in C.

The complete kinematic model of the MUSHA hand, pr = K(⋅), is obtained exploiting (2) for each finger, thus pr = K(q1,T, q2,T, q3,T, q1,I, q2,I, q3,I, q1,M, q2,M, q3,M). According to the sub-actuation constrains we define the wrist angle position, qW, and the index and medium proximal one, qPIP,:

qW=q1,IqW=q1,T+2π3qW=q1,M2π3 and qPIP=q2,MqPIP=q2,I(3)

we can simplify the kinematic relationship, obtaining:

pr=KqW,q2,T,q3,T,qPIP,q3,I,q3,M(4)

where pr is a vector containing the fingertips position expressed in Cartesian coordinate. It is worth noticing that in the exploited model, the length of the links a2 and a3 depends on the joint values. Indeed, the coupling motion of the phalanx segments (generated by the tendons) causes links shrinkage. In accordance with the chord theorem (see Figure 4), the length of the links can be computed as

ai,ρ=2ri,ρsinβi2.(5)

FIGURE 4
www.frontiersin.org

FIGURE 4. Details of a bended MUSHA finger.

Let qij,ρ be the joint j in the phalanx i of the finger ρ, ∀i ∈ {2, 3} and ∀ρ ∈ {T, I, M}. Consequently, it is possible to write the length of the chord in kinematic model and the length of the link in the actuated model as follows:

ai,ρ=2Li,ρj=26qij,ρ+j=15qij,ρsinj=26qij,ρ+j=15qij,ρ2Li,ρ if qi,ρ=0(6)

where Li, ρ is the length of the arc of the i-th phalanx of the finger ρ. Assuming that the action of the tendons is equal on all the segments, the angular displacements are equal to a fraction of the equivalent joint in the reduced model:

qij,ρ=qi,ρ6j1,6, i2,3 and ρT,I,M.(7)

As a consequence, ∀ρ ∈ {T, I, M} the single finger model becomes:

xr,ρ=cosq1,ρa2,ρcosq2,ρ+a3,ρcosq3,ρ(8)
yr,ρ=sinq1,ρa2,ρcosq2,ρ+a3,ρcosq3,ρ
zr,ρ=a2,ρsinq2,ρ+a3,ρsinq3,ρ
a2,ρ=6L2,ρ5q2,ρsin5q2,ρ6L2,ρ if q2,ρ=0
a3,ρ=6L3,ρ5q3,ρsin5q3,ρ6L3,ρ if q3,ρ=0

Therefore, the explicit full hand model (reported in the Supplementary Material) has 14 equations nine of which linearly independent, in six variables. Finally, we can express ṗr=Jrwq̇r, where Jrw is the Jacobian matrix of the robotic fingers of MUSHA expressed in reference frame of the wrist. The differential model is fully reported in the Supplementary (see Supplementary Section S5.2).

2.2 Human Hand Modelling

In this section, we report the simplified kinematic model of the hand, which will be instrumental in the design of the control technique. Even if a complete human hand model has about 30 DoF (Lee and Kunii, 1995), for the sake of simplicity and without loss of generality we use a simplified kinematic hand structure. We modeled each finger as a planar kinematic chain, with three 1-D hinges. As in Lisini et al. (2017), we assume that each finger has the metacarpal (MC) bone fixed with respect to the hand frame, and it is characterized by three DoFs. In a more complete model of the human hand, the thumb has at least five DoFs: two in the trapeziometacarpal (TM) joint, two in the metacarpophalangeal (MCP) joint, and one in the interphalangeal (IP) joint. However, the thumb abduction/adduction motion range of the MCP joint usually can be neglected so the thumb can be modeled with two DoFs. Index, middle, ring, and pinky fingers have one DoFs in the MCP revolute joint for flexion/extension, one in the proximal IP (PIP), and one in the distal IP (DIP). Figure 5 shows the model of the hand used in this work. Therefore, the exploited hand model has 20 DoFs. i.e., 14 DoFs describing the fingers and six DoFs (position and orientation) for the palm. Finally, we introduce the relationship between fingertips and joints velocities, which can be described by the differential model:

p̂̇h=Jhqhq̇h(9)

where the matrix JhR15×20 is the hand Jacobian, that maps the joint velocities (q̇hR20) to the Cartesian fingertip velocities (p̂̇hR15).

FIGURE 5
www.frontiersin.org

FIGURE 5. The simplified kinematic model of the human hand, structure with links, joints, base, and rotation axes are in evidence.

2.3 Mapping Algorithms to Teleoperate da Vinci Arm and MUSHA Hand

Starting from the definition of the human hand and the MUSHA hand models, the developed method to control the MUSHA hand and the da Vinci robot arm are explained. A standard point-to-point mapping technique is used to control the PSM, while an innovative approach based on the theory of mapping in object domain to teleoperate the MUSHA fingers is designed and implemented. We assume that the human and the robotic hands are constantly in touch with a similar virtual object, all possible contact points are called active points. The positions of MUSHA active points pr are defined as the positions of fingertips, while the human active points (php̂h) are given by the positions of palm (P), thumb, index, and middle fingertips:

ph=ph,P,ph,T,ph,I,ph,MT.(10)

We indicate with ph,ρ a vector of Cartesian positions of the ρ-th finger of the human hand h expressed in the world reference frame:

ph,ρ=xh,ρ,yh,ρ,zh,ρT,ρP,T,I,M.(11)

Depending on the grasping method, we define the vector of interaction points, ch, equal to a subset of active points of the human hand involved in the desired grasp, and evaluated as:

ch=tċhτdτ(12)

being

ċh=IΩṗh with IR12×12(13)

and

Ω=112×1if method=Pr01×3,11×9Tif method=Po11×3,01×3,11×6Tif method=Rt(14)

It is worth noticing that Ω is a selection vector that is evaluated on-line to allow switch control reference without discontinuity and according to the grasping method required. The palm-thumb distance is used to recognize the current grasping method.

The reference of PSM in cartesian space is calculated as integration over time τ of the velocity components of the user hand centroid c̃̇h until the time instant t. In this way, during a temporary disconnection between the patient side and control side, the relocation of the palm in more ergonomic poses or in a position far from the boundaries of the vision cone of the camera system is allowed. This results in:

pPSMt=pPSMtτ+tc̃̇hτdτ(15)

Where the value of centroid changes according to method of grasping, in fact:

c̃h=ρP,T,I,Mch,ρN with N=3 if methodPo4 if method=Po(16)

Similarly, the orientation of the human hand is exploited to orient the PSM. To this end Euler angles in roll (α), pitch (β), and yaw (γ) convention are computed as:

α=12tan12μ0,1,1μ0,2,0μ0,0,2,(17)
β=12tan12μ1,0,1μ0,0,2μ2,0,0,(18)
γ=12tan12μ1,1,0μ2,0,0μ0,2,0,(19)

being μi,j,k the second order central moments of cloud of contact points

μi,j,k=ρP,I,T,Mxh,ρx̃hjyh,ρỹhizh,ρz̃hk(21)

We exploited a sphere as virtual object to easily project the human hand motions into MUSHA fingers and wrist movements.

The roto-translations and deformations of a sphere can be completely defined by the following seven parameters: 1) ohR3 Cartesian coordinate of the center of sphere; 2) θhR3 the orientation of the frame attached to the sphere with respect to the world frame, expressed as roll, pitch, and yaw angles; 3) rhR the sphere radius. Combining (13) and the aforementioned seven parameters, it is possible to obtain a linear relationship between human hand motions and sphere transformation

ċh,ρ=ȯh+θ̇h×ch,ρoh+ṙhch,ρoh.(22)

Thus, there exists a Jacobian matrix Jhs such that its pseudo-inverse, Jhs, projects the human fingertip velocities into derivative parameters of the sphere:

oḣθḣrḣ=Jhsoh,chIΩpḣ(23)

The resulting Jacobian matrix depends on the current position of interaction points of human hand and on the sphere center that is a function of the active points of the human hand. Indeed, the sphere center is the circumcenter of a triangle with three of the current interaction points as vertices.

oh=ch,i+|Δh,ki|2Δh,ji×Δh,ki×Δh,ji2|Δh,ji×Δh,ki|2+|Δh,ji|2Δh,ki×Δh,ji×Δh,ki2|Δh,ji×Δh,ki|2(24)

where Δh,ki is the difference between the position vector of the k-th fingertip ch,k in contact with the sphere and the i-th fingertip ch,i, then, (23) can be rewritten as:

oḣθḣrḣ=JhsphIΩpḣ(25)

As a further step towards the mapping, the sphere manipulated by the human is properly transformed into the one manipulated by the MUSHA through a scale factor Ks, that is equal to the ratio between the radius of human and MUSHA manipulated spheres. The algorithm computes the Jacobian of the sphere manipulated by the robot (i.e., Jrs) and evaluates the velocities of the robotic fingertips,

ṗr=JrsprKsJhsphIΩpḣ.(26)

Velocities are estimated by a closed loop kinematic inversion

q̇r=JrwqrJrsprKsJhsphIΩpḣ+Kcp̃rpr(27)

where Jrw, Jrs and Jhs are the Jacobian matrices of the robot, of the sphere manipulated by the robot, and of the sphere manipulated by the user, respectively. Kc(p̃rpr) is the corrective action on the error between the current position of MUSHA interaction points pr and the desired ones p̃r, weighted by the gain Kc. It is worth pointing out that the pseudo-inversion of Jhs induces an indeterminacy in the estimated positions far from the current pose in case of multiple solutions, for this reason, the right position of the thumb in the retractor method may be not correctly computed. This situation is visually depicted in Figure 6. A projection into the null-space of mapping actions is used to overcome this limitation. To this end, (27) is extended as follows,

q̇r=JrwqrJrsprKsJhsphIΩpḣ+Kcp̃rpr+JrwI6×6JrsKsJhsIΩJrsKsJhsIΩu(28)

where

u=06×1if methodRtKRtqRtqrif method=Rt(29)

being KRt the gain that multiplies the error between current joint position qr and qRt, the expected joint position in retractor pose. Regarding the initial condition, at the beginning of any task, the MUSHA lies in a rest position with all the fingers closed with joint configuration qr0=06×1. This particular configuration results in a kinematic singularity, in fact when q(t) = qr0 the rank of Jacobian of robot decreases, Rank(Jrw)<6. For this reason, an initialization procedure is required. In accordance with

qr0=argmaxqdetJrwqJrwTqqi,ρqi,ρmin,qi,ρmaxi, ρ(30)

we select qr0=[0,π9,π9,π9,π9,π9] as initial position. It corresponds to the joint configuration with the maximum manipulability. In conclusion, combining (2830), we obtain the entire human-MUSHA hand joints mapping:

qr=qr0 if t=0tq̇rτdτ if t0(31)

with

q̇r=JrwqrJrsprKsJhsphIΩpḣ+Kcp̃rpr+JrwIJrsKsJhsIΩJrsKsJhsIΩu(32)

FIGURE 6
www.frontiersin.org

FIGURE 6. Graphic representation of thumb indeterminacy in retractor method with two reachable positions, in the correct one in (A) the thumb is between middle and index to allow the hand to reach the position scoop-like while in wrong one, depicted in (B), the thumb is in opposition respect the other two fingers and the reached posed of MUSHA hand is equal to approach pose in power grasp method. This is a consequence of the high similarity between the virtual spheres generated by the contact points in the Rt case and in the Po case.

2.4 User Interface, Software and Communication

The camera system LeapMotion (Ultraleap inc., United States) is used to acquire active points of human hand. This camera system is composed of two monochromatic IR cameras and three LEDs that observe a hemispherical area and capture snapshots of up to 200 frames per second, and it returns information about positions of some fixed notable points of acquired hand as tips of fingers, the center of palm and knuckles. A ROS node acquires these from LeapMotion and publishes the values of active points, selecting only those useful for the mapping algorithm. To test the control law a virtual model of MUSHA hand is imported in V-rep (Coppelia Robotics GmbH, CH), and here its kinematic and dynamic behavior is simulated.

3 Results

First of all, a theoretical analysis of applied forces is conducted to evaluate the effect of contact between MUSHA and a generic organic tissue for each method of grasp, then a validation of the control algorithm is pursued to demonstrate its stability. An experimental campaign follows, during which naive participants are asked to complete a pick and place operation using the virtual model of the MUSHA hand.

3.1 Theoretical Grasping Forces Evaluation

During surgery, the grasping and manipulation of tissue may generate both internal and external forces. The former impresses deformation and allows to avoid the sliding of the manipulated tissue, the latter generates twists. In accordance with the grasping theory (Prattichizzo and Trinkle, 2016), it is possible to evaluate from the applied force (i.e., λ) the components that generate the squeezing (i.e., ξ), and the ones that generate twisting (i.e., ω). This can be done by exploiting the grasping matrix GR6×18 and the null space projector NGR18×18, following the relationship

λ=Gω+NGξ(33)

with

GT=HG̃T.(34)

G̃T is the transposed grasping matrix that maps sources of twist from contact point reference to world reference. H is a selection matrix depending on the contact finger model. Considering the consistency of organic tissues, it is possible to approximate the grasp with the Soft Finger (SF) contact model. Interested reader is referred to Prattichizzo and Trinkle (2016) for further details. Therefore, admitting the applicability of the principle of kineto-static duality, it is possible to apply generalized forces exclusively in the directions of constrained motions. Directions of force application are shown in Figure 7 in case of contact with one finger. We assumed SF contact model for each point, thus the selection matrix H can be defined as

H=HT000HI000HMR18×18(35)

with

HT=HI=HM=100000010000001000000000000000000001.(36)

FIGURE 7
www.frontiersin.org

FIGURE 7. In (A), the contact model with one finger. The red area surrounding the contact point cj is the deformation zone. Red arrows display the directions in which motion is not allowed, thus it is possible to apply force or torques. In (B), the contact surface and the axes of the main reference systems are depicted.

Consequently, the complete grasp matrix GT can be modeled as GT=HG̃T, being

G̃T=RT̄T000RĪT000RM̄TPTTPITPMT,G̃TR18×6(37)

with

R̄jT=RjT00RjTjT,I,M,(38)

and

PjT=IScjpts0IjT,I,M(39)

where RTR3×3 are the rotation matrices describing the orientation of each contact point reference frame Σj, j = T, I, M with respect to the world reference frame ΣO, S(cjpts) is the cross-product matrix of the vector cjpts̄, and pts is center of mass of the grasped portion of tissue. Accounting the available grasping configurations, we can refine the general formula as follows:

λ=NGξif method=PoGω+NGξif method=PrGωif method=Rt(40)

At each time instant, the position of the centroid of the manipulated tissues and the positions of the contact points may be known, so the grasping matrix can be computed and a projector in its null space can be evaluated:

NG=IGG.(41)

Thus, given measure of contact force, λ̃, the value of ξ and ω can be evaluated as:

ξ=NG1λ̃if method=PoNGλ̃if method=Pr0if method=Rt(42)
ω=Gλ̃if methodPo0if method=Po(43)

3.2 Lyapunov Stability Proof

The stability of the proposed control law was assessed by Lyapunov conditions. To this aim, we choose as positive definite Lyapunov functions the function of error:

V=12ϵTKcTϵ.(44)

Where ϵ is the distance between the actual (pr) and desired (pr̃) MUSHA fingertip.

ϵ=p̃rpr(45)

It is possible to evaluate its time derivative value as:

ϵ̇=pr̃̇Jrŵqṙ(46)

Consequently, the time derivative of the Lyapunov function is

V̇=ϵTKcTpr̃̇Jrŵqṙ.(47)

Where Jrŵ is the Jacobian matrix of model of MUSHA hand evaluated in real current position. Substituting (28) in (47), the derivative Lyapunov function becomes

V̇=ϵTKcTpr̃̇ϵTKcTJrŵJrwMpḣ+JrwNMu+Kcp̃rpr.(48)

Where M is the mapping action of control law

M=JrsprKsJhsphIΩ,(49)

and NM is the projection action in null space of M used to overcome the duality of thumb pose in retractor method:

NM=IJrsprKsJhsphIΩJrsprKsJhsphIΩ.(50)

The desired velocity p̃̇r is chosen as the result of mapping M(qr,pr,ph) and projection in null space of mapping NM,

p̃̇r=Mṗh+JrwNMu.(51)

Introducing (48) in (51), it is obtained that:

V̇=ϵTKcTpr̃̇ϵTKcTJrŵJrwpr̃̇+Kcp̃rpr.(52)

Reintroducing (52) the error evaluated in accordance with (45), we obtain

V̇=ϵTKcTpr̃̇ϵTKcTJrŵJrwpr̃̇+Kcϵ.(53)

If MUSHA is accurately calibrated the Jacobian matrix evaluated on the real position of fingers and on the estimated one are approximately equal, JrŵJrw, and the time derivative Lyapunov function becomes

V̇ϵTKcTKcϵ.(54)

Since the Lyapunov function (47) is positive definite (V(t) > 0) and its time derivative is negative definite (V̇(t)<0), the system is Lyapunov stable.

It is possible assert the BIBO stability, too. In fact, substituting definition of desired position (51) in control law (28), we obtain that

q̇r=Jrwp̃̇rKcϵ(55)

It is possible to evaluate the tendency of error, using the definition of time derivative of error metric:

ϵ̇=Kcϵ(56)

The error evolves as first order linear system, so the Lyapunov stability implies the BIBO stability.

3.3 User Studies

Ten inexperienced participants (six males and four females, aged 24–59, all right-handed) took part in user studies. Each subject gave her/his consent to participate and was able to discontinue participation at any time during experiments. The experimental evaluation protocols followed the declaration of Helsinki, and there was no risk of harmful effects on subjects’ health. Participants performed 10 pick-and-place tasks of two cubes with 10 different initial scenarios pseudo-randomly sorted. The experimental set-up is shown in1 Figure 8. The dimension of each cube is 0.02 m × 0.02 m×0.02 m and the weight is equal to 20 g. To help the participant to complete the task, force feedback is provided by two vibromotors NFP-C1234L to inform them about internal and external forces applied on cubes. The two components of force are evaluated starting from the results of Section 3.1 using (42) and (43), respectively, and their norms in 2 space, ‖ξ‖ and ‖ω‖, are used to calculate the motor PWMs as:

PWMξ=1kξΔF255,(57)
PWMω=ωΔF255.(58)

where ΔF is the range of forces that can be acquired through MUSHA hand force sensors (ΔF = 4N). The dimension disparity between the internal force space (ξR18) and the external one (ωR6) introduces an imbalance of haptic stimulus intensity which has been compensated by scaling the 2 norm ‖ξ‖ with a k–factor equal to the ratio of the dimensions of the two spaces (k = 3).

FIGURE 8
www.frontiersin.org

FIGURE 8. The experimental set-up consisted in the LeapMotion controller, the haptic bracelet and a couple of screen to visualize the Virtual environment and the reconstructed model of tracked hand. In the virtual scene, cubes pseudo-randomly changed their position at the beginning of the single repetition of experiment.

At the end of each session, a NASA Task Load Index questionnaire (TLX) (Hart, 2006) was proposed to the participants, with the aim of assessing the perceived load in terms of Mental Demand (MD), Temporal Demand (TD), Physical Demand (PD), Performance (PE), Effort (EF), and Frustration (FR). This method assesses workload on a point-based scale. Each of the six questions has a scale of 21 levels, considering 1 as “very low” and 21 as “very high.” Questions are reported in what follows:

- Mental demand: “How mentally demanding was the task?”

- Physical demand: “How physically demanding was the task?”

- Temporal demand: “How hurried or rushed was the pace of the task?”

- Performance: “How successful were you in accomplishing what you were asked to do?”

- Effort: “How hard did you have to work to accomplish your level of performance?”

- Frustration: “How insecure, discouraged, irritated, stressed, and annoyed were you?”

All participants successfully completed the ten pick and place repetitions of the two cubes, so the completion times for each trial and the results of the questionnaires were considered as metrics for evaluation. Results are reported in Table 1 and Table 2. The mean time to complete the task is 16.49 s and no participant took more than 30s. The load of the task performed with the proposed control law is evaluated from answers of the NASA TLX questionnaire and its mean value is equal to 35%. According with the proposed task and experimental set-up we used the following weights: wMD = 3, wPD = 0, wTD = 5, wPE = 1, wFR = 3, wEF = 3. The physical demand was excluded from task load evaluation (PD = 0) because the physical load is due to the absence of arm support that is provided by the surgeon console in a real scenario.

TABLE 1
www.frontiersin.org

TABLE 1. Time to complete tasks (mean and ±std).

TABLE 2
www.frontiersin.org

TABLE 2. NASA TLX questionnaire results, tally of importance selections: wMD = 3, wPD = 0, wTD = 5, wPE = 1, wFR = 3, wEF = 3.

4 Discussion

In this work, we present a reliable and intuitive method for the bilateral asymmetric teleoperation of the MUSHA hand. A camera-based device is in charge of tracking the human hand, and a mapping algorithm projects the human motion in the end-effector actuation space through the virtual object domain. The developed technique consists in a remote control for the projection of the human gestures into the robot space of actuation by crossing the object domain and switching in the function of the current gripping method, which is recognized online by the posture of the user’s hand. A sphere is chosen as virtual objects due to the reduced number of parameters that describe its pose and deformation. A closed-loop inverse kinematic algorithm makes more robust the control action and guarantees the BIBO stability of the algorithm, while a projection in the null of the object space allows the correct estimation of indeterminate poses. Starting from the kinematics description of the MUSHA hand and its main grasping methods, we theoretically evaluated the grasping forces applied by the tool during each method and an experimental campaign assesses the functionality and the usability of the algorithm. All the participants completed the assigned task in less than 30 s (15 s per cube) with a limited cognitive load. The obtained results demonstrate the intuitiveness of the teleoperation system.

The proposed control algorithm has been designed, developed, and tested to control the MUSHA hand, nevertheless it can be generalized and extended to different grippers and robotic hands. Indeed the proposed methodology relies on a mapping algorithm that exploits contact points and grasping methods. Thus, a redefinition of the differential kinematic equations and of the related Jacobian Matrix of the gripper allow the reader adapting the algorithm to the hand of interest. A preliminary analysis of the instrument grasping capacity and the redefinition of the selection vector, Ω, allows to maintain the validity of the theoretical results regarding the evaluation of the applied forces. In future work, experiments will be conducted considering expert users asking them to complete elementary surgical procedures using the real MUSHA hand mounted on the da Vinci robot and we will test the compatibility of the algorithm with several standard end-effector during the same procedures. Finally, the entire system will be integrated with the surgeon console of the da Vinci robot.

Data Availability Statement

The raw data supporting the conclusion of this article will be made available by the authors, without undue reservation.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

FF and DP were project coordinators, they contributed to conception and design of the study. AV and TL worked on the experimental setup and runned the experiments. FF, AV, and DP proposed algorithm and supervised the experiments. FF, AV, and TL were principal investigators, and wrote the manuscript. All authors contributed to manuscript revision, read, and approved the submitted version.

Funding

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101017008.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frobt.2021.741807/full#supplementary-material

Footnotes

1A video demonstration of the experimental task is available at the following link: https://youtu.be/6DYI1MfqI7A.

References

Beasley, R. A. (2012). Medical Robots: Current Systems and Research Directions. J. Robotics 2012, 401613. doi:10.1155/2012/401613

CrossRef Full Text | Google Scholar

Bebek, O., and Cavusoglu, M. C. “Model Based Control Algorithms for Robotic Assisted Beating Heart Surgery,” in Proceedings of the 2006 International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE), New York, NY, USA, August 2006, 823–828. doi:10.1109/IEMBS.2006.260562

CrossRef Full Text | Google Scholar

Bowyer, S. A., Davies, B. L., and y Baena, F. R. (2013). Active Constraints/virtual Fixtures: A Survey. IEEE Trans. Robotics 30, 138–157. doi:10.1109/TRO.2013.2283410

CrossRef Full Text | Google Scholar

Cavusoglu, M. C., Tendick, F., Cohn, M., and Sastry, S. S. (1999). A Laparoscopic Telesurgical Workstation. IEEE Trans. Robot. Automat. 15, 728–739. doi:10.1109/70.782027

CrossRef Full Text | Google Scholar

Chadi, S. A., Guidolin, K., Caycedo-Marulanda, A., Sharkawy, A., Spinelli, A., Quereshy, F. A., et al. (2020). Current Evidence for Minimally Invasive Surgery during the COVID-19 Pandemic and Risk Mitigation Strategies. Ann. Surg. 272, e118–e124. doi:10.1097/sla.0000000000004010

PubMed Abstract | CrossRef Full Text | Google Scholar

Cheng, L., and Tavakoli, M. (2018). Switched-impedance Control of Surgical Robots in Teleoperated Beating-Heart Surgery. J. Med. Robot. Res. 03, 1841003. doi:10.1142/s2424905x18410039

CrossRef Full Text | Google Scholar

Fontanelli, G. A., Ficuciello, F., Villani, L., and Siciliano, B. “Modelling and identification of the da vinci research kit robotic arms,” in Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE), Vancouver, BC, Canada, September 2017, 1464–1469.

Google Scholar

Fontanelli, G. A., Selvaggio, M., Ferro, M., Ficuciello, F., Vendittelli, M., and Siciliano, B. “A v-rep simulator for the da vinci research kit robotic platform,” in Proceedings of the 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob) (IEEE), Enschede, Netherlands, August 2018, 1056–1061.

Google Scholar

Gioioso, G., Salvietti, G., Malvezzi, M., and Prattichizzo, D. (2013). Mapping Synergies from Human to Robotic Hands with Dissimilar Kinematics: an Approach in the Object Domain. IEEE Trans. Robot. 29, 825–837. doi:10.1109/tro.2013.2252251

CrossRef Full Text | Google Scholar

Handini, D., Yeong, T. M., and Hong, L. V. “System Integration of Neurobot: a Skull-Base Surgical Robotic System,” in Proceedings of the IEEE Conference on Robotics, Automation and Mechatronics, 2004. (IEEE), Singapore, December 2004, 43–48.

Google Scholar

Hanley, D. F., Thompson, R. E., Rosenblum, M., Yenokyan, G., Lane, K., McBee, N., et al. (2019). Efficacy and Safety of Minimally Invasive Surgery with Thrombolysis in Intracerebral Haemorrhage Evacuation (Mistie Iii): a Randomised, Controlled, Open-Label, Blinded Endpoint Phase 3 Trial. Lancet 393, 1021–1032. doi:10.1016/S0140-6736(19)30195-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Hart, S. G. (2006). Nasa-task Load index (Nasa-tlx); 20 Years Later. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 50, 904–908. doi:10.1177/154193120605000909

CrossRef Full Text | Google Scholar

Hu, Z., Yoon, C.-H., Park, S. B., and Jo, Y.-H. (2016). Design of a Haptic Device with Grasp and Push-Pull Force Feedback for a Master-Slave Surgical Robot. Int. J. CARS 11, 1361–1369. doi:10.1007/s11548-015-1324-9

CrossRef Full Text | Google Scholar

Jintae Lee, J., and Kunii, T. L. (1995). Model-based Analysis of Hand Posture. IEEE Comput. Grap. Appl. 15, 77–86. doi:10.1109/38.403831

CrossRef Full Text | Google Scholar

Kim, C. Y., Kang, B. H., Lee, M. C., and Yoon, S. M. (2013). Design of Master Console Haptic Handle for Robot Assisted Laparoscopy. Intell. Autonomous Syst. 12, 3–10. doi:10.1007/978-3-642-33932-5_1

CrossRef Full Text | Google Scholar

Kulaylat, A. S., Mirkin, K. A., Puleo, F. J., Hollenbeak, C. S., and Messaris, E. (2018). Robotic versus Standard Laparoscopic Elective Colectomy: where Are the Benefits? J. Surg. Res. 224, 72–78. doi:10.1016/j.jss.2017.11.059

CrossRef Full Text | Google Scholar

Leven, J., Burschka, D., Kumar, R., Zhang, G., Blumenkranz, S., Dai, X. D., et al. (2005). “Davinci Canvas: a Telerobotic Surgical System with Integrated, Robot-Assisted, Laparoscopic Ultrasound Capability,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, Palm Springs, CA, USA, October 2005 (Springer), 811–818. doi:10.1007/11566465_100

CrossRef Full Text | Google Scholar

Lisini, B. T., Scheggi, S., Meli, L., Mohammadi, M., and Prattichizzo, D. (2017). Gesto: A Glove for Enhanced Sensing and Touching Based on Inertial and Magnetic Sensors for Hand Tracking and Cutaneous Feedback. IEEE Trans. Human-mach. Syst. 47, 1066–1076. doi:10.1109/thms.2017.2720667

CrossRef Full Text | Google Scholar

Liu, H., Ferrentino, P., Selvaggio, M., Pirozzi, S., and Ficuciello, F. “Design of the Musha Hand Ii for Robotic-Assisted Laparoscopic Surgery,” in Proceedings of the 12th The Hamlyn Symposium on Medical Robotics, London, United Kingdom, June 2019.

Google Scholar

Liu, H., Selvaggio, M., Ferrentino, P., Moccia, R., Pirozzi, S., Bracale, U., et al. (2020). The Musha Hand Ii: A Multi-Functional Hand for Robot-Assisted Laparoscopic Surgery. Ieee/asme Trans. Mechatron. 26, 1. doi:10.1109/TMECH.2020.3022782

CrossRef Full Text | Google Scholar

Madhani, A. J., Niemeyer, G., and Salisbury, J. K. “The Black falcon: a Teleoperated Surgical Instrument for Minimally Invasive Surgery,” in Proceedings of the 1998 IEEE/RSJ International Conference on Intelligent Robots and Systems. Innovations in Theory, Practice and Applications (Cat. No. 98CH36190) (IEEE), Victoria, BC, Canada, October 1998, 936–944.

Google Scholar

Manganelli, R., Chinello, F., Formaglio, A., and Prattichizzo, D. (2010). A Teleoperation System for Micro-invasive Brain Surgery. Paladyn, J. Behav. Robotics 1, 198–203. doi:10.2478/s13230-011-0006-8

CrossRef Full Text | Google Scholar

Meeker, C., Rasmussen, T., and Ciocarlie, M. “Intuitive Hand Teleoperation by Novice Operators Using a Continuous Teleoperation Subspace,” in Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA) (IEEE), Brisbane, QLD, Australia, May 2018, 1–7.

Google Scholar

Meli, L., Pacchierotti, C., and Prattichizzo, D. (2014). Sensory Subtraction in Robot-Assisted Surgery: Fingertip Skin Deformation Feedback to Ensure Safety and Improve Transparency in Bimanual Haptic Interaction. IEEE Trans. Biomed. Eng. 61, 1318–1327. doi:10.1109/tbme.2014.2303052

PubMed Abstract | CrossRef Full Text | Google Scholar

Moccia, R., Iacono, C., Siciliano, B., and Ficuciello, F. (2020). Vision-based Dynamic Virtual Fixtures for Tools Collision Avoidance in Robotic Surgery. IEEE Robot. Autom. Lett. 5, 1650–1655. doi:10.1109/lra.2020.2969941

CrossRef Full Text | Google Scholar

Mohareri, O., Schneider, C., and Salcudean, S. “Bimanual Telerobotic Surgery with Asymmetric Force Feedback: a Davinci® Surgical System Implementation,” in In Proceedings of the 2014 Ieee/Rsj International Conference on Intelligent Robots and Systems (IEEE), Chicago, IL, USA, September 2014, 4272–4277.

Google Scholar

Niemeyer, G., and Slotine, J.-J. E. (1991). Stable Adaptive Teleoperation. IEEE J. Oceanic Eng. 16, 152–162. doi:10.1109/48.64895

CrossRef Full Text | Google Scholar

Okamura, A. M. (2009). Haptic Feedback in Robot-Assisted Minimally Invasive Surgery. Curr. Opin. Urol. 19, 102–107. doi:10.1097/mou.0b013e32831a478c

PubMed Abstract | CrossRef Full Text | Google Scholar

Parisi, A., Reim, D., Borghi, F., Nguyen, N. T., Qi, F., Coratti, A., et al. (2017). Minimally Invasive Surgery for Gastric Cancer: A Comparison between Robotic, Laparoscopic and Open Surgery. Wjg 23, 2376. doi:10.3748/wjg.v23.i13.2376

PubMed Abstract | CrossRef Full Text | Google Scholar

Perigli, G., Cortesini, C., Qirici, E., Boni, D., and Cianchi, F. (2008). Clinical Benefits of Minimally Invasive Techniques in Thyroid Surgery. World J. Surg. 32, 45–50. doi:10.1007/s00268-007-9259-0

CrossRef Full Text | Google Scholar

Prattichizzo, D., and Trinkle, J. C. (2016). “Grasping,” in Springer Handbook of Robotics. Editors B. Siciliano, and O. Khatib (Berlin, Germany: Springer), 955–988. doi:10.1007/978-3-319-32552-1_38

CrossRef Full Text | Google Scholar

Sheetz, K. H., Claflin, J., and Dimick, J. B. (2020). Trends in the Adoption of Robotic Surgery for Common Surgical Procedures. JAMA Netw. Open 3, e1918911. doi:10.1001/jamanetworkopen.2019.18911

PubMed Abstract | CrossRef Full Text | Google Scholar

Song, J., Kang, W. H., Oh, S. J., Hyung, W. J., Choi, S. H., and Noh, S. H. (2009). Role of robotic gastrectomy using da vinci system compared with laparoscopic gastrectomy: initial experience of 20 consecutive cases. Surg. Endosc. 23, 1204–1211. doi:10.1007/s00464-009-0351-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Sood, A., Meyer, C. P., Abdollah, F., Sammon, J. D., Sun, M., Lipsitz, S. R., et al. (2017). Minimally Invasive Surgery and its Impact on 30-day Postoperative Complications, Unplanned Readmissions and Mortality. J. Br. Surg. 104, 1372–1381. doi:10.1002/bjs.10561

CrossRef Full Text | Google Scholar

Sugimoto, M., Yasuda, H., Koda, K., Suzuki, M., Yamazaki, M., Tezuka, T., et al. (2010). Image Overlay Navigation by Markerless Surface Registration in Gastrointestinal, Hepatobiliary and Pancreatic Surgery. J. hepato-biliary-pancreatic Sci. 17, 629–636. doi:10.1007/s00534-009-0199-y

CrossRef Full Text | Google Scholar

Sukhoon, P., Bin, L. K., Muk, J. J., and Yong-San, Y. “Design of Master Console Robot for Natural Orifice Transluminal Endoscopic Surgery,” in Proceedings of the 2009 ICCAS-SICE (IEEE), Fukuoka, Japan, August 2009, 1152–1157.

Google Scholar

Tabrizi, L.B., and Mahvash, M. (2015). Augmented Reality-Guided Neurosurgery: Accuracy and Intraoperative Application of an Image Projection Technique. Jns 123, 206–211. doi:10.3171/2014.9.jns141001

CrossRef Full Text | Google Scholar

Takács, Á., Rudas, I., and Haidegger, T. (2015). Open-source Research Platforms and System Integration in Modern Surgical Robotics. Acta Universitatis Sapientiae; Electr. Mech. Eng. 14, 20–34.

Google Scholar

Tang, A., Cao, Q., Tan, H., Fujie, M. G., and Pan, T. (2016). “Motion Control of a Master-Slave Minimally Invasive Surgical Robot Based on the Hand-Eye-Coordination,” Computer Aided Surgery. Editor M. Fujie (Berlin, Germany: Springer), 57–71. doi:10.1007/978-4-431-55810-1_5

CrossRef Full Text | Google Scholar

Tonutti, M., Elson, D. S., Yang, G.-Z., Darzi, A. W., and Sodergren, M. H. (2017). The Role of Technology in Minimally Invasive Surgery: State of the Art, Recent Developments and Future Directions. Postgrad. Med. J. 93, 159–167. doi:10.1136/postgradmedj-2016-134311

PubMed Abstract | CrossRef Full Text | Google Scholar

Van der Meijden, O. A., Broeders, I. A., and Schijven, M. P. (2010). The SEP "robot": a Valid Virtual Reality Robotic Simulator for the Da Vinci Surgical System? Surg. Technol. Int. 19, 51–58.

PubMed Abstract | Google Scholar

Yamamoto, T., Abolhassani, N., Jung, S., Okamura, A. M., and Judkins, T. N. (2012). Augmented Reality and Haptic Interfaces for Robot-Assisted Surgery. Int. J. Med. Robotics Comput. Assist. Surg. 8, 45–56. doi:10.1002/rcs.421

CrossRef Full Text | Google Scholar

Yin, X., Guo, S., and Wang, Y. “Force Model-Based Haptic Master Console Design for Teleoperated Minimally Invasive Surgery Application,” in 2015 IEEE International Conference on Mechatronics and Automation (ICMA) (IEEE), Beijing, China, August 2015, 749–754.

Google Scholar

Keywords: medical robot, surgical gripper, robotics, surgical robot, teleoperation

Citation: Ficuciello F, Villani A, Lisini Baldi T and Prattichizzo D (2021) A Human Gesture Mapping Method to Control a Multi‐Functional Hand for Robot‐Assisted Laparoscopic Surgery: The MUSHA Case. Front. Robot. AI 8:741807. doi: 10.3389/frobt.2021.741807

Received: 15 July 2021; Accepted: 29 October 2021;
Published: 10 December 2021.

Edited by:

Ka-Wai Kwok, The University of Hong Kong, Hong Kong SAR, China

Reviewed by:

Antonia Tzemanaki, University of Bristol, United Kingdom
Luigi Manfredi, University of Dundee, United Kingdom

Copyright © 2021 Ficuciello, Villani, Lisini Baldi and Prattichizzo. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Fanny Ficuciello, fanny.ficuciello@unina.it

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.