Development of human collaborative robot to perform daily tasks based on multimodal vital information with cybernics space

Akira Uehara ^1,2,3* Hiroaki Kawamoto ^1,2 Yoshiyuki Sankai ^1,2,3,4

• ¹ Institute of Systems and Information Engineering, University of Tsukuba, Tsukuba, Japan
• ² Center for Cybernics Research, University of Tsukuba, Tsukuba, Ibaraki, Japan
• ³ Research and Development Center for Frontiers of MIRAI in Policy and Technology, Tsukuba, Ibaraki, Japan
• ⁴ Cyberdyne Inc., Tsukuba, Ibaraki, Japan

Due to the increasing employment of people with disabilities, support for the elderly, and childcare needs, enhancing the independence and freedom across generations and spaces is necessary. This study aimed to develop a human collaborative robot using multimodal vital information as input with cybernics space that is fused "Human" and "Cyber/Physical Space" and confirm its feasibility experimentally. The robot allows the user to operate it via gaze and bio-electrical signals (BES), reflecting the user's intentions, and to seamlessly transition among three modes (i.e., assistant, jockey, and ghost). In the assistant mode, the user collaborates with the robot in the physical space using a system that includes a head-mounted display (HMD) for gaze measurement, BES measurement unit, personal mobility system, and an arm-hand system. The HMD can be flipped up and down for hands-free control. The BES measurement unit captures leaked weak signals from the skin surface, indicating the user's voluntary movement intentions, which are processed by the main unit to generate control commands for the various actuators. The personal mobility system features omniwheels for tight turning, and the arm hand system can handle payloads up to 500 g. In the jockey mode, the user remotely operates a small mobile base with a display and camera, moving it through the physical space. In the ghost mode, the user navigates and inputs commands in a virtual space using a smart key and remote-control device integrated with IoT and wireless communication. The switching of each control mode is estimated using the BES from the user's upper arm, gaze direction, and position, thereby enabling movement, mobility, and manipulation without physical body movement. At basic experiments involving able-bodied participants, macro average of recall, precision, and F score were 1.00, 0.90, and 0.94 at assistant mode. The macro average of recall, precision, and F score were 0.85, 0.92, and 0.88 at ghost mode. Therefore, the human collaborative robot utilizing multimodal vital information has feasibility for supporting daily life tasks, contributing to a safer and more secure society by addressing various daily life challenges.