Haitao Yang1,2
Wenbo Wu1*- 1Department of Chemistry, Institute of Molecular Aggregation Science, Tianjin University, Tianjin, China
- 2Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Singapore
Compliant and soft sensors that detect machinal deformations become prevalent in emerging soft robots for closed-loop feedback control. In contrast to conventional sensing applications, the stretchy body of the soft robot enables programmable actuating behaviors and automated manipulations across a wide strain range, which poses high requirements for the integrated sensors of customized sensor characteristics, high-throughput data processing, and timely decision-making. As various soft robotic sensors (strain, pressure, shear, etc.) meet similar challenges, in this perspective, we choose strain sensor as a representative example and summarize the latest advancement of strain sensor-integrated soft robotic design driven by machine learning techniques, including sensor materials optimization, sensor signal analyses, and in-sensor computing. These machine learning implementations greatly accelerate robot automation, reduce resource consumption, and expand the working scenarios of soft robots. We also discuss the prospects of fusing machine learning and soft sensing technology for creating next-generation intelligent soft robots.
Introduction
Emerging soft robots have attracted more and more attention in robotic research due to their easy body/shape transformation (Shepherd et al., 2011; Rus and Tolley, 2015), high degree of freedom (DOF) deformation (Rus and Tolley, 2018; Walsh, 2018), energy-efficient actuation (Hu et al., 2018; Kim et al., 2018), and safe human-machine interfaces (Laschi et al., 2016; Kim et al., 2019). Efficient and accurate feedback control is an essential function for soft robots to safely interact in dynamic environments (Thuruthel et al., 2019; Wang et al., 2020; Zhou et al., 2020). Conventional robotic control strategy also involves simulation and coaching, which depends on advanced algorithms or mathematical models. But for soft robots, due to the very high degree of freedom of their soft body, it is challenging to model the robotic dynamics mathematically, making the simulation and coaching not suitable. As a result, feedback control, which relays on the integrated soft sensors to back actual electronic signals in response to external stimulus, becomes prevalent in soft robotic research. Among various soft sensing technologies, the compliant strain sensor, as a representative sensing technology with the facile device fabrication and testing, is widely attached to the robotic body, enabling the robot itself to capture mechanical deformations and achieve closed-loop control (Mirvakili and Hunter, 2018; Zhang et al., 2019). Until now, diverse strain sensing components are integrated into soft robots to enable bio-mimicking dexterous manipulations, such as fish-like robot to explore deep ocean (Li et al., 2021), ultra-gentle gripper to capture fragile lives (Sinatra et al., 2019; Sundaram et al., 2019), cheetah-inspired soft robot with fast-moving (Tang et al., 2020), and soft human exoskeletons for rehabilitation (Zhao et al., 2016; Cianchetti et al., 2018), etc.
With the prosperity of soft robots, the soft robotic body with high DOF manipulations proposes high requirements of the integrated strain sensors. Firstly, soft robots often exhibit multimodal locomotion, whereas different robotic units show very different actuating behaviors with strain deformations ranging from <5% (Araromi et al., 2020) to >200% (Wang et al., 2018; Yang et al., 2020). To track the coordinated locomotion of a soft robot, multiple strain sensors with on-demand sensor characteristics are necessary for tracking separate robotic units with distinct kinematics (Mengüç et al., 2014; Yang et al., 2020), yet it is hindered by the huge resource consumption to validate a large number of design principles and trial-and-error experiments. Secondly, with the increasing number of integrated strain sensors, the mass of sensing data increases rapidly, leading to a heavy load of data storage, transmission, and processing, as well as the concern of high energy consumption (Faisal et al., 2019; Wang et al., 2020; Yang et al., 2020). Besides, to achieve the ultimate goal of an intelligent soft robot, timely decision-making is significant, which relays on local sensor data processing (i.e., in-sensor computing) rather than remote external computing devices (Zhou and Chai, 2020; Moin et al., 2021), yet is rarely explored in the soft robotic system. To push the boundary of strain sensor-integrated soft robots, it is significant to develop new principles to address the above issues of customized sensor characteristics, high-throughput data processing, and timely decision-making.
To address these challenges, data-driven exploration, prediction, and recommendation by machine learning (ML) provide an alternative approach (Schmidt et al., 2019; Xia et al., 2020). ML is a powerful tool that could create a model to uncover the unknown relations or principles in data-rich systems (Butler et al., 2018). Recently, ML experiences a boom in the investigation of various scientific areas including material science (Kitchin, 2018; Zahrt et al., 2019; Toyao et al., 2020), statistics (Yu and Kumbier, 2018), medicine (Ekins et al., 2019; Vamathevan et al., 2019), neuroscience (Savage, 2019), and machine automation (Zhou et al., 2022), where a large number of data points are provided to train an accurate ML model. Considering the inherently strong capabilities and recent advancements in handling data, including 1) uncovering the material/device characteristics with absent first principles, 2) parallel computing for high-throughput data, and 3) edge chip to achieve localized data processing, it is of high value to fuse ML technology for addressing robotic sensing challenges (Shih et al., 2020). Herein, we choose strain sensor as a representative example of robotic sensing technologies and briefly summarize the latest advancement of using ML techniques to design strain sensor-integrated soft robots, including sensor materials optimization, sensor signal analyses, and in-sensor computing. These ML implementations address the above challenges of customized sensor characteristics, high-throughput data processing, and timely decision-making, respectively. In the end, several prospects are also discussed. We expect this mini perspective could provide a brief but timely review of the current status and future directions.
ML for sensor material optimization
There are generally two types of strain sensors: piezoresistive sensor and capacitive sensor. Under mechanical deformations, the piezoresistive sensor shows resistance changes while the capacitive sensor demonstrates changes in capacitance value. Compared with the capacitive sensor that requires a sandwich structure, the piezoresistive sensor with a simple single-layer structure shows the advantages of facile fabrication and tight integration with the robotic body. As a result, at this very initial research stage, most ML techniques in strain sensor-integrated soft robots are implemented on the piezoresistive sensor. Therefore, piezoresistive sensor is selected as the main topic to review recent ML advancements.
A variety of intrinsically conductive materials ranging from metallic materials (e.g., gold foil (Yang et al., 2016), liquid metal (Lin et al., 2022)) to nanomaterials (e.g., fullerenes (Shi et al., 2018a), carbon nanotubes (Yamada et al., 2011; Sun et al., 2020), graphene (Wang et al., 2014; Yang et al., 2018), MXene (Yang et al., 2020; Saeidi-Javash et al., 2021; Cao et al., 2022)) have been adopted for the fabrication of piezoresistive strain sensors. The sensing mechanism is based on varying conductive pathways of the sensor layer under strains (e.g., in-plane crack propagation), leading to increased resistance as the electrical indicator. Two characteristics of a strain sensor are generally evaluated, including strain sensitivity and linear working window. The sensitivity of a strain sensor is characterized by gauge factor (GF), as defined in the equation below,
where ε denotes the applied uniaxial strain, R0 and R represent the resistances before and after the applied strain, respectively. On the other hand, the linear working window of a strain sensor is determined by the strain range where the resistance of the sensor increased linearly with the applied strain.
The research experiences from past decades have revealed that the GF and linear working window of a strain sensor are highly correlated and balanced (Qiu et al., 2019; Afsarimanesh et al., 2020; Souri et al., 2020; Pei et al., 2021). A high GF generally exists within a narrow linear range, while a broad strain range is usually accompanied by a low GF (Amjadi et al., 2016; Shi et al., 2018b; Jayathilaka et al., 2019). This trade-off is caused by the different fracture behaviors of piezoresistive sensors: high GF comes from quick and drastic crack propagation under a narrow strain range, yet a broad linear working window requires slow and mild crack growth across a large strain. Based on these principles, many influential fabrication parameters including sensor material composition (Shi et al., 2018b; Cai et al., 2018), sensor shape/dimension (Jiang et al., 2018; Xin et al., 2020), and sensor topography (Yang et al., 2016; Pei et al., 2021) were proposed to engineer the sensor characteristics. For example, Shi et al. used fullerene as a lubricant agent to lower the friction between 2D graphene layers (Figure 1A). Under uniaxial strains, the graphene layer with lubricant experienced a mild fracture behavior instead of inducing long cracks to cut off the conductive pathways quickly, enabling the resulted strain sensor with a large linear working window (Shi et al., 2018a). Besides, X. Chen’s group designed an auxetic mechanical structure to improve the sensitivity (Jiang et al., 2018). As shown in Figure 1B, compared with conventional flat film, the auxetic structure showed a negative Poisson’s ratio and underwent 2D expansion in both parallel and perpendicular directions. Such auxetic metamaterials were incorporated into stretchable strain sensors to significantly enhance the crack density under strains and improved the sensitivity by up to 24-fold. In addition, P.-Y. Chen’s group demonstrated a hierarchical topography of MXene sensor with micro-crumples/wrinkles (Figure 1C) to regulate the in-plane crack propagation, which could control the crack growth in a programmable manner and realized highly tunable strain sensor characteristics (Yang et al., 2020).
FIGURE 1. ML for Strain Sensor-Integrated Soft Robots. (A) Schematic illustration of sensing mechanism for GO–AgNW–C60 sensing films under stretching, where the layers partially slide out inside the crack in the sensing film. Reproduced with permission from Shi et al. (2018a). Copyright: WILEY-VCH Verlag GmbH & Co. (B) Comparison of the sensor structures between conventional flat film structure and auxetic metamaterial structure, with corresponding deformation under 15% tensile strain. Reproduced with permission from Jiang et al. (2018). Copyright: WILEY-VCH Verlag GmbH & Co. (C) MXene strain sensor with hierarchical surface topography. Reproduced with permission from Yang et al. (2020). Copyright: American Chemical Society. (D) Three-stage framework for construction of an ML-enabled prediction model capable of automatic strain sensor design for soft robots. Reproduced with permission from Yang et al. (2022). Copyright: Springer Nature. (E) A robot hand is integrated with tactile sensors and the multilayer perceptron model. With the aid of ML tools, the robot hand could identify object material, shape, and size. Reproduced with permission from Li et al. (2020). Copyright: American Association for the Advancement of Science. (F) Digitally designed full-sized tactile sensing wearables, which collected the tactile frames during human–environment interactions and explored their applications using ML techniques. Reproduced with permission from Luo et al. (2021). Copyright: Springer Nature. (G) Wearable biosensing system with adaptive machine learning for real-time prediction of hand gestures. Reproduced with permission from Moin et al. (2021). Copyright: Springer Nature. (H) The physical picture of the Tianjicat robot, the development board, and the TianjicX chip array. By equipping with TianjicX and multimodal sensors, a robotic car is able to simultaneously process multisensory information locally for various tasks including object detection, obstacle avoidance, and decision-making. Reproduced with permission from Ma et al. (2022). Copyright: American Association for the Advancement of Science.
Through various design principles, as discussed by multiple important review articles (Amjadi et al., 2016; Jayathilaka et al., 2019; Qiu et al., 2019; Afsarimanesh et al., 2020), the state-of-the-art research studies focus on developing high-performance strain sensor that shows high sensitivity (GF > 100) and large linear working window (>100%), simultaneously. With the great progress in recent years, a lot of strain sensors with outstanding sensor characteristics were created (Tao et al., 2017; Shi et al., 2018a; Shi et al., 2018b; Cai et al., 2018; Jiang et al., 2018; Lee et al., 2019; Li and Du, 2019; Yang et al., 2020; Zhang et al., 2021), which gradually fulfilled the above challenging region. With this regard, instead of continuing to pursue high sensor performance, it is time to move to the next stage of developing strain sensors with customized sensor characteristics for desired applications such as soft robots.
Nevertheless, due to the complex relationship between sensing materials and sensor characteristics, the exploration efficiency is greatly limited through conventional trial-and-error experiments. Under this condition, ML tools show a unique opportunity to underlay the unclear relations, which greatly reduces time, manpower, and resource consumption (Butler et al., 2018; Wei et al., 2019). Recently, H. Yang et al. proposed an ML framework that enabled automatic strain sensor design across a wide strain range from <0.5% to 350% (Yang et al., 2022). As shown in Figure 1D, there was a three-stage machine learning process, including boundary definition, active learning, and data augmentation. Firstly, multiple influential variables of varying material components, varying sensor thicknesses, and varying sensor morphologies were included. Adequate delamination tests were conducted to examine the fabrication feasibility which was used to train a support vector machine (SVM) classifier that only allowed the fabrication recipes with high successful chances to be suggested by the model. Secondly, with the guidance of active learning, an increasing number of strain sensors were stagewise fabricated to serve as the representative dataset to enrich the multi-dimensional dataset of the ML model. Finally, data augmentation methods were implemented to create more than 10,000 virtual data points to cultivate the prediction model. After the three-stage process, the ultimate prediction model was capable of conducting two-way tasks of automatic strain sensor design for diverse soft robots, including 1) high-accuracy prediction of sensor characteristics based on proposed fabrication recipes and 2) suggestion of feasible fabrication recipes for interested strain ranges. In addition, by implementing statistical analyses based on the collected data points, the underlying sensing mechanisms were revealed that dictated data-driven design principles.
ML for sensor signal analyses
Besides the on-demand sensor characteristics, soft robots also face difficulties in processing large amounts of data generated from multiple sensors on separate robotic units (Shih et al., 2020). For example, when applying wearable robots to track full-body human motions, more than 70 sensors are needed to match various body joints from shoulders to toe. Also, to achieve high sensing resolution, S. Sundaram et al. showed a knitted soft glove with 548 sensors (Sundaram et al., 2019). Such a large number of integrated sensors lead to a heavy load in multi-channeled sensor data collection, transmission, and processing (Zhou and Chai, 2020). Emerging approaches adopt wireless technology such as Bluetooth or near-field communication modules to facilitate data collection and wireless transmission (Lin et al., 2020; Yang et al., 2020; Jeong et al., 2021; Lin et al., 2022), yet they failed in the aspect of signal analyses due to the absence of computing unit.
With the rapid development of mobile computing platforms like Raspberry Pi, an emerging strategy is to connect sensors, wireless modules, and ML-embedded computing hardware together to establish a comprehensive sensor platform (Ha et al., 2020; Wang et al., 2021). By introducing ML tools, without manual data calibration, one can adopt the well-known algorithms from the open-source library (e.g., pandas, scikit-learn, PyTorch, TensorFlow) to pre-process the sensor signals easily such as de-noising, filtering, and grouping (Manie et al., 2020; Zhou et al., 2020). On the other hand, powerful machine learning neural networks enable the ML model to solve more complex tasks like signal pattern recognition. For example, X. Chen group adopted deep convolutional neural networks to analyze the strain sensor signals from five fingers of a human hand to distinguish different hand gestures (Wang et al., 2020). In addition, by combining tactile sensing information and a multilayer perceptron model, G, Li et al. developed a smart robotic hand that could precisely recognize different shapes, sizes, and materials in a diverse set of garbage (Figure 1E) (Li et al., 2020). On the other hand, recurrent neural network is often used as a powerful architecture to process variables along a temporal sequence like speech recognition (Jung et al., 2020).
Besides executing a single task, multiple ML models and computing tools with different functions could be assembled and work together to satisfy the diverse needs of soft robots. For instance, J. Chen group recently reported a wearable sign-to-speech translation system based on a soft glove with stretchable strain sensor arrays (Zhou et al., 2020), which involved a three-step computing process including signal pre-processing (the generated sensor signals are merged into a matrix), feature extraction (by principal component analysis), and speech translation (by multi-class SVM classifier). In addition, as shown in Figure 1F, Y. Luo et al. reported an ML-powered sensing textile suit which had three models with different network architectures to cover self-supervised calibration, interaction identification, and full-body motion prediction, respectively (Luo et al., 2021).
ML for in-sensor computing
The integration of sensors, wireless modules, and ML-embedded computing devices ensured facile sensor data transmission and processing, but with the increasing number of sensors, continuous data steaming raises high concerns of bandwidth requirement, graphics processing units expense, and power consumption (Zhou and Chai, 2020; Iwendi et al., 2021). On the other hand, wireless communication usually faces interruption and disconnection challenges, especially for soft robotic navigation in distant fields or underwater scenarios (Shih et al., 2019; Pillai and Lohani, 2020). Additionally, the radio frequency communication between wireless hardware and external computing devices faces the inevitable risks of data security (Lin et al., 2020).
An innovative approach to address the above difficulties is to process the real-time sensor data locally to achieve in-sensor computing without steaming sensor data out (Zhou and Chai, 2020; Moin et al., 2021). The goal of in-sensor computing is to perform both data processing and decision-making tasks locally to achieve higher levels of robotic autonomy with onboard machine intelligence, without further analyses or external computing modules. For instance, J. M. Rabaey group recently proposed an in-sensor adaptive machine learning framework for hand gesture recognition (Figure 1G) (Moin et al., 2021). In particular, they implemented a neuro-inspired hyperdimensional computing algorithm that could analyze sensor data locally for real-time hand gesture classification, as well as automatic model training and updating under variable conditions such as different arm positions and sensor replacement. That report well filled the gap of self-supervised model training and updating during continuous usage, enabling optimal model performance under long-term operations. On the other hand, S. Ma et al. developed a neuromorphic computing chip of TianjicX, which enabled multiple neural network models to be executed locally (Ma et al., 2022). As shown in Figure 1H, by equipping with TianjicX and multimodal sensors, without steaming sensor data out and external computing devices, a robotic car is able to simultaneously process multisensory information locally for diverse tasks, including object recognition, obstacle avoidance, and decision-making. In comparison with the commercial NVIDIA Jetson TX2 (ML-embedded computing hardware developed by NVIDIA), the developed TianjicX chip showed less latency and 50% lower power consumption.
Prospects
Fusing ML and soft sensing technology is an emerging multidisciplinary field and shows great potential for soft robots (see a summary in Table 1). But there is still a long way to achieve the ultimate goal of the intelligent soft robot, and several viewpoints are discussed here. Firstly, the sensing capability of soft robot is not limited to body strain deformation to estimate the kinematics of the robot itself but also should include more functions such as contact modeling (by pressure sensor) and mapping of the surroundings (by chemical and biological sensors). Therefore, important progress is expected in integrating ML technology with multiple types of sensors (e.g., strain, pressure, chemical, biomolecule) to create a multifunctional soft robot with comprehensive perception capability. We believe these ML techniques, including sensor material optimization, sensor signal analyses, and in-sensor computing, are also workable for other types of soft sensors, as they meet similar challenges to realize the goal of intelligent soft robot. On the other hand, for ML-assisted sensor material optimization, although some promising examples have been identified, we envisioned that critical challenges still exist at this early stage, such as the lack of sufficient high-quality datasets and absent evaluation standards to fairly compare the developed models. Besides, considering different kind of sensor types, their unique sensing mechanisms and structures may lead to different ML implementations yet is rarely studied, which would be an important future research direction.
TABLE 1. Summary of ML implementations for strain sensor-integrated soft robots.
Secondly, many recent advances are conceptual studies to examine the feasibility of ML tools for soft robots yet lack the assessments of system robustness in practical conditions. For example, both compliant sensors and soft robotic bodies consist of soft matters, which will experience stress fatigue under long-term repeated mechanical deformations. This kind of phenomenon is commonly seen in robotics and affects the sensor signal and body kinematics a lot. It remains to explore suitable ML frameworks to ensure accurate feedback and control under such mechanical coupling conditions. Also, many other powerful ML algorithms like active learning, meta-learning, and reinforcement learning are waiting to be fully utilized for guiding soft robots to complete more challenging missions like automatic navigation or swarm under dynamic, unstructured, and even unknown environments.
At last, module integration and miniaturization are always pursued. For soft robotic applications, the separated sensors and computing modules inevitably increase the overall weight, cause signal deviation, and affect robotic locomotion. Therefore, the tight integration of multiple sensors, data collection/transmission modules, and computing units as well as the subsequent miniaturized circuit design are desired to achieve the goals of creating light, compliant, power-efficient, and eventually intelligent robots with multifunctionality. Moreover, it is expected to see potential innovations in constructing the data processing and computing modules with total soft materials, which would reduce the design restrictions of soft robots and expand their application scenarios.
Author contributions
All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.
Acknowledgments
We are grateful for the financial support of the National Natural Science Foundation of China (52103228, 22075199).
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Afsarimanesh, N., Nag, A., Sarkar, S., Sabet, G. S., Han, T., and Mukhopadhyay, S. C. (2020). A review on fabrication, characterization and implementation of wearable strain sensors. Sensors Actuators A Phys. 315, 112355. doi:10.1016/j.sna.2020.112355
Amjadi, M., Kyung, K. U., Park, I., and Sitti, M. (2016). Stretchable, skin‐mountable, and wearable strain sensors and their potential applications: A review. Adv. Funct. Mat. 26, 1678–1698. doi:10.1002/adfm.201504755
Araromi, O. A., Graule, M. A., Dorsey, K. L., Castellanos, S., Foster, J. R., Hsu, W.-H., et al. (2020). Ultra-sensitive and resilient compliant strain gauges for soft machines. Nature 587, 219–224. doi:10.1038/s41586-020-2892-6
Butler, K. T., Davies, D. W., Cartwright, H., Isayev, O., and Walsh, A. (2018). Machine learning for molecular and materials science. Nature 559, 547–555. doi:10.1038/s41586-018-0337-2
Cai, Y., Shen, J., Ge, G., Zhang, Y., Jin, W., Huang, W., et al. (2018). Stretchable Ti3C2TX MXene/carbon nanotube composite based strain sensor with ultrahigh sensitivity and tunable sensing range. ACS Nano 12, 56–62. doi:10.1021/acsnano.7b06251
Cao, Y., Guo, Y., Chen, Z., Yang, W., Li, K., He, X., et al. (2022). Highly sensitive self-powered pressure and strain sensor based on crumpled MXene film for wireless human motion detection. Nano Energy 92, 106689. doi:10.1016/j.nanoen.2021.106689
Cianchetti, M., Laschi, C., Menciassi, A., and Dario, P. (2018). Biomedical applications of soft robotics. Nat. Rev. Mat. 3, 143–153. doi:10.1038/s41578-018-0022-y
Ekins, S., Puhl, A. C., Zorn, K. M., Lane, T. R., Russo, D. P., Klein, J. J., et al. (2019). Exploiting machine learning for end-to-end drug discovery and development. Nat. Mat. 18, 435–441. doi:10.1038/s41563-019-0338-z
Faisal, A. I., Majumder, S., Mondal, T., Cowan, D., Naseh, S., and Deen, M. J. (2019). Monitoring methods of human body joints: State-of-the-Art and research challenges. Sensors 19, 2629. doi:10.3390/s19112629
Ha, N., Xu, K., Ren, G., Mitchell, A., and Ou, J. Z. (2020). Machine learning‐enabled smart sensor systems. Adv. Intell. Syst. 2, 2000063. doi:10.1002/aisy.202000063
Hu, W., Lum, G. Z., Mastrangeli, M., and Sitti, M. (2018). Small-scale soft-bodied robot with multimodal locomotion. Nature 554, 81–85. doi:10.1038/nature25443
Iwendi, C., Maddikunta, P. K. R., Gadekallu, T. R., Lakshmanna, K., Bashir, A. K., and Piran, M. J. (2021). A metaheuristic optimization approach for energy efficiency in the IoT networks. Softw. Pract. Exper. 51, 2558–2571. doi:10.1002/spe.2797
Jayathilaka, W. a. D. M., Qi, K., Qin, Y., Chinnappan, A., Serrano‐García, W., Baskar, C., et al. (2019). Significance of nanomaterials in wearables: A review on wearable actuators and sensors. Adv. Mat. 31, 1805921. doi:10.1002/adma.201805921
Jeong, H., Kwak, S. S., Sohn, S., Lee, J. Y., Lee, Y. J., O’brien, M. K., et al. (2021). Miniaturized wireless, skin-integrated sensor networks for quantifying full-body movement behaviors and vital signs in infants. Proc. Natl. Acad. Sci. U. S. A. 118, e2104925118. doi:10.1073/pnas.2104925118
Jiang, Y., Liu, Z., Matsuhisa, N., Qi, D., Leow, W. R., Yang, H., et al. (2018). Auxetic mechanical metamaterials to enhance sensitivity of stretchable strain sensors. Adv. Mat. 30, 1706589. doi:10.1002/adma.201706589
Jung, Y. H., Hong, S. K., Wang, H. S., Han, J. H., Pham, T. X., Park, H., et al. (2020). Flexible piezoelectric acoustic sensors and machine learning for speech processing. Adv. Mat. 32, 1904020. doi:10.1002/adma.201904020
Kim, Y., Parada, G. A., Liu, S., and Zhao, X. (2019). Ferromagnetic soft continuum robots. Sci. Robot. 4, eaax7329. doi:10.1126/scirobotics.aax7329
Kim, Y., Yuk, H., Zhao, R., Chester, S. A., and Zhao, X. (2018). Printing ferromagnetic domains for untethered fast-transforming soft materials. Nature 558, 274–279. doi:10.1038/s41586-018-0185-0
Kitchin, J. R. (2018). Machine learning in catalysis. Nat. Catal. 1, 230–232. doi:10.1038/s41929-018-0056-y
Laschi, C., Mazzolai, B., and Cianchetti, M. (2016). Soft robotics: technologies and systems pushing the boundaries of robot abilities. Sci. Robot. 1, eaah3690. doi:10.1126/scirobotics.aah3690
Lee, J., Pyo, S., Kwon, D. S., Jo, E., Kim, W., and Kim, J. (2019). Ultrasensitive strain sensor based on separation of overlapped carbon nanotubes. Small 15, 1805120. doi:10.1002/smll.201805120
Li, G., Chen, X., Zhou, F., Liang, Y., Xiao, Y., Cao, X., et al. (2021). Self-powered soft robot in the mariana trench. Nature 591, 66–71. doi:10.1038/s41586-020-03153-z
Li, G., Liu, S., Wang, L., and Zhu, R. (2020). Skin-inspired quadruple tactile sensors integrated on a robot hand enable object recognition. Sci. Robot. 5, eabc8134. doi:10.1126/scirobotics.abc8134
Li, H., and Du, Z. (2019). Preparation of a highly sensitive and stretchable strain sensor of MXene/silver nanocomposite-based yarn and wearable applications. ACS Appl. Mat. Interfaces 11, 45930–45938. doi:10.1021/acsami.9b19242
Lin, R., Kim, H.-J., Achavananthadith, S., Kurt, S. A., Tan, S. C., Yao, H., et al. (2020). Wireless battery-free body sensor networks using near-field-enabled clothing. Nat. Commun. 11, 444–510. doi:10.1038/s41467-020-14311-2
Lin, R., Kim, H.-J., Achavananthadith, S., Xiong, Z., Lee, J. K., Kong, Y. L., et al. (2022). Digitally-embroidered liquid metal electronic textiles for wearable wireless systems. Nat. Commun. 13, 2190–2210. doi:10.1038/s41467-022-29859-4
Luo, Y., Li, Y., Sharma, P., Shou, W., Wu, K., Foshey, M., et al. (2021). Learning human–environment interactions using conformal tactile textiles. Nat. Electron. 4, 193–201. doi:10.1038/s41928-021-00558-0
Ma, S., Pei, J., Zhang, W., Wang, G., Feng, D., Yu, F., et al. (2022). Neuromorphic computing chip with spatiotemporal elasticity for multi-intelligent-tasking robots. Sci. Robot. 7, eabk2948. doi:10.1126/scirobotics.abk2948
Manie, Y. C., Li, J.-W., Peng, P.-C., Shiu, R.-K., Chen, Y.-Y., and Hsu, Y.-T. (2020). Using a machine learning algorithm integrated with data de-noising techniques to optimize the multipoint sensor network. Sensors 20, 1070. doi:10.3390/s20041070
Mengüç, Y., Park, Y.-L., Pei, H., Vogt, D., Aubin, P. M., Winchell, E., et al. (2014). Wearable soft sensing suit for human gait measurement. Int. J. Rob. Res. 33, 1748–1764. doi:10.1177/0278364914543793
Mirvakili, S. M., and Hunter, I. W. (2018). Artificial muscles: Mechanisms, applications, and challenges. Adv. Mat. 30, 1704407. doi:10.1002/adma.201704407
Moin, A., Zhou, A., Rahimi, A., Menon, A., Benatti, S., Alexandrov, G., et al. (2021). A wearable biosensing system with in-sensor adaptive machine learning for hand gesture recognition. Nat. Electron. 4, 54–63. doi:10.1038/s41928-020-00510-8
Pei, Y., Zhang, X., Hui, Z., Zhou, J., Huang, X., Sun, G., et al. (2021). Ti3C2TX MXene for sensing applications: Recent progress, design principles, and future perspectives. ACS Nano 15, 3996–4017. doi:10.1021/acsnano.1c00248
Pillai, R. R., and Lohani, R. B. (2020). “Emergency data detection using hidden markov model during temporary disconnection of wireless body area networks,” in 2020 International Conference for Emerging Technology (INCET), Belgaum, India, Jun 5-7, 1–5.
Qiu, A., Li, P., Yang, Z., Yao, Y., Lee, I., and Ma, J. (2019). A path beyond metal and silicon: Polymer/nanomaterial composites for stretchable strain sensors. Adv. Funct. Mat. 29, 1806306. doi:10.1002/adfm.201806306
Rus, D., and Tolley, M. T. (2018). Design, fabrication and control of origami robots. Nat. Rev. Mat. 3, 101–112. doi:10.1038/s41578-018-0009-8
Rus, D., and Tolley, M. T. (2015). Design, fabrication and control of soft robots. Nature 521, 467–475. doi:10.1038/nature14543
Saeidi-Javash, M., Du, Y., Zeng, M., Wyatt, B. C., Zhang, B., Kempf, N., et al. (2021). All-printed MXene–graphene nanosheet-based bimodal sensors for simultaneous strain and temperature sensing. ACS Appl. Electron. Mat. 3, 2341–2348. doi:10.1021/acsaelm.1c00218
Savage, N. (2019). How AI and neuroscience drive each other forwards. Nature 571, S15–S17. doi:10.1038/d41586-019-02212-4
Schmidt, J., Marques, M. R. G., Botti, S., and Marques, M. a. L. (2019). Recent advances and applications of machine learning in solid-state materials science. npj Comput. Mat. 5, 83. doi:10.1038/s41524-019-0221-0
Shepherd, R. F., Ilievski, F., Choi, W., Morin, S. A., Stokes, A. A., Mazzeo, A. D., et al. (2011). Multigait soft robot. Proc. Natl. Acad. Sci. U. S. A. 108, 20400–20403. doi:10.1073/pnas.1116564108
Shi, X., Liu, S., Sun, Y., Liang, J., and Chen, Y. (2018a). Lowering internal friction of 0D–1D–2D ternary nanocomposite‐based strain sensor by fullerene to boost the sensing performance. Adv. Funct. Mat. 28, 1800850. doi:10.1002/adfm.201800850
Shi, X., Wang, H., Xie, X., Xue, Q., Zhang, J., Kang, S., et al. (2018b). Bioinspired ultrasensitive and stretchable MXene-based strain sensor via nacre-mimetic microscale “brick-and-mortar” architecture. ACS Nano 13, 649–659. doi:10.1021/acsnano.8b07805
Shih, B., Shah, D., Li, J., Thuruthel, T. G., Park, Y.-L., Iida, F., et al. (2020). Electronic skins and machine learning for intelligent soft robots. Sci. Robot. 5, eaaz9239. doi:10.1126/scirobotics.aaz9239
Shih, Y., Hsiu, P., and Pang, A. (2019). A data parasitizing scheme for effective health monitoring in wireless body area networks. IEEE Trans. Mob. Comput. 18, 13–27. doi:10.1109/TMC.2018.2830779
Sinatra, N. R., Teeple, C. B., Vogt, D. M., Parker, K. K., Gruber, D. F., and Wood, R. J. (2019). Ultragentle manipulation of delicate structures using a soft robotic gripper. Sci. Robot. 4, eaax5425. doi:10.1126/scirobotics.aax5425
Souri, H., Banerjee, H., Jusufi, A., Radacsi, N., Stokes, A. A., Park, I., et al. (2020). Wearable and stretchable strain sensors: Materials, sensing mechanisms, and applications. Adv. Intell. Syst. 2, 2000039. doi:10.1002/aisy.202000039
Sun, X., Qin, Z., Ye, L., Zhang, H., Yu, Q., Wu, X., et al. (2020). Carbon nanotubes reinforced hydrogel as flexible strain sensor with high stretchability and mechanically toughness. Chem. Eng. J. 382, 122832. doi:10.1016/j.cej.2019.122832
Sundaram, S., Kellnhofer, P., Li, Y., Zhu, J.-Y., Torralba, A., and Matusik, W. (2019). Learning the signatures of the human grasp using a scalable tactile glove. Nature 569, 698–702. doi:10.1038/s41586-019-1234-z
Tang, Y., Chi, Y., Sun, J., Huang, T.-H., Maghsoudi, O. H., Spence, A., et al. (2020). Leveraging elastic instabilities for amplified performance: Spine-inspired high-speed and high-force soft robots. Sci. Adv. 6, eaaz6912. doi:10.1126/sciadv.aaz6912
Tao, L.-Q., Wang, D.-Y., Tian, H., Ju, Z.-Y., Liu, Y., Pang, Y., et al. (2017). Self-adapted and tunable graphene strain sensors for detecting both subtle and large human motions. Nanoscale 9, 8266–8273. doi:10.1039/C7NR01862B
Thuruthel, T. G., Shih, B., Laschi, C., and Tolley, M. T. (2019). Soft robot perception using embedded soft sensors and recurrent neural networks. Sci. Robot. 4, eaav1488. doi:10.1126/scirobotics.aav1488
Toyao, T., Maeno, Z., Takakusagi, S., Kamachi, T., Takigawa, I., and Shimizu, K.-I. (2020). Machine learning for catalysis informatics: Recent applications and prospects. ACS Catal. 10, 2260–2297. doi:10.1021/acscatal.9b04186
Vamathevan, J., Clark, D., Czodrowski, P., Dunham, I., Ferran, E., Lee, G., et al. (2019). Applications of machine learning in drug discovery and development. Nat. Rev. Drug Discov. 18, 463–477. doi:10.1038/s41573-019-0024-5
Walsh, C. (2018). Human-in-the-Loop development of soft wearable robots. Nat. Rev. Mat. 3, 78–80. doi:10.1038/s41578-018-0011-1
Wang, H., Totaro, M., and Beccai, L. (2018). Toward perceptive soft robots: Progress and challenges. Adv. Sci. 5, 1800541. doi:10.1002/advs.201800541
Wang, M., Wang, T., Luo, Y., He, K., Pan, L., Li, Z., et al. (2021). Fusing stretchable sensing technology with machine learning for human–machine interfaces. Adv. Funct. Mat. 31, 2008807. doi:10.1002/adfm.202008807
Wang, M., Yan, Z., Wang, T., Cai, P., Gao, S., Zeng, Y., et al. (2020). Gesture recognition using a bioinspired learning architecture that integrates visual data with somatosensory data from stretchable sensors. Nat. Electron. 3, 563–570. doi:10.1038/s41928-020-0422-z
Wang, Y., Wang, L., Yang, T., Li, X., Zang, X., Zhu, M., et al. (2014). Wearable and highly sensitive graphene strain sensors for human motion monitoring. Adv. Funct. Mat. 24, 4666–4670. doi:10.1002/adfm.201400379
Wei, J., Chu, X., Sun, X. Y., Xu, K., Deng, H. X., Chen, J., et al. (2019). Machine learning in materials science. InfoMat 1, 338–358. doi:10.1002/inf2.12028
Xia, B., Miriyev, A., Trujillo, C., Chen, N., Cartolano, M., Vartak, S., et al. (2020). Improving the actuation speed and multi-cyclic actuation characteristics of silicone/ethanol soft actuators. Actuators 9, 62. doi:10.3390/act9030062
Xin, M., Li, J., Ma, Z., Pan, L., and Shi, Y. (2020). MXenes and their applications in wearable sensors. Front. Chem. 8, 297. doi:10.3389/fchem.2020.00297
Yamada, T., Hayamizu, Y., Yamamoto, Y., Yomogida, Y., Izadi-Najafabadi, A., Futaba, D. N., et al. (2011). A stretchable carbon nanotube strain sensor for human-motion detection. Nat. Nanotechnol. 6, 296–301. doi:10.1038/nnano.2011.36
Yang, H., Li, J., Lim, K. Z., Pan, C., Van Truong, T., Wang, Q., et al. (2022). Automatic strain sensor design via active learning and data augmentation for soft machines. Nat. Mach. Intell. 4, 84–94. doi:10.1038/s42256-021-00434-8
Yang, H., Xiao, X., Li, Z., Li, K., Cheng, N., Li, S., et al. (2020). Wireless Ti3C2TX MXene strain sensor with ultrahigh sensitivity and designated working windows for soft exoskeletons. ACS Nano 14, 11860–11875. doi:10.1021/acsnano.0c04730
Yang, T., Li, X., Jiang, X., Lin, S., Lao, J., Shi, J., et al. (2016). Structural engineering of gold thin films with channel cracks for ultrasensitive strain sensing. Mat. Horiz. 3, 248–255. doi:10.1039/C6MH00027D
Yang, Z., Pang, Y., Han, X.-L., Yang, Y., Ling, J., Jian, M., et al. (2018). Graphene textile strain sensor with negative resistance variation for human motion detection. ACS Nano 12, 9134–9141. doi:10.1021/acsnano.8b03391
Yu, B., and Kumbier, K. (2018). Artificial intelligence and statistics. Front. Inf. Technol. Electron. Eng. 19, 6–9. doi:10.1631/FITEE.1700813
Zahrt, A. F., Henle, J. J., Rose, B. T., Wang, Y., Darrow, W. T., and Denmark, S. E. (2019). Prediction of higher-selectivity catalysts by computer-driven workflow and machine learning. Science 363, eaau5631. doi:10.1126/science.aau5631
Zhang, H., Liu, D., Lee, J.-H., Chen, H., Kim, E., Shen, X., et al. (2021). Anisotropic, wrinkled, and crack-bridging structure for ultrasensitive, highly selective multidirectional strain sensors. Nano-Micro Lett. 13, 122–215. doi:10.1007/s40820-021-00615-5
Zhang, J., Sheng, J., O’neill, C. T., Walsh, C. J., Wood, R. J., Ryu, J., et al. (2019). Robotic artificial muscles: Current progress and future perspectives. IEEE Trans. Robot. 35, 761–781. doi:10.1109/TRO.2019.2894371
Zhao, H., O’brien, K., Li, S., and Shepherd, R. F. (2016). Optoelectronically innervated soft prosthetic hand via stretchable optical waveguides. Sci. Robot. 1, eaai7529. doi:10.1126/scirobotics.aai7529
Zhou, F., and Chai, Y. (2020). Near-sensor and in-sensor computing. Nat. Electron. 3, 664–671. doi:10.1038/s41928-020-00501-9
Zhou, X., Wen, X., Wang, Z., Gao, Y., Li, H., Wang, Q., et al. (2022). Swarm of micro flying robots in the wild. Sci. Robot. 7, eabm5954. doi:10.1126/scirobotics.abm5954
Keywords: artificial intelligence, strain sensors, soft robots, machine intelligence, closed-loop control
Citation: Yang H and Wu W (2022) A review: Machine learning for strain sensor-integrated soft robots. Front. Electron. Mater. 2:1000781. doi: 10.3389/femat.2022.1000781
Received: 22 July 2022; Accepted: 31 October 2022;
Published: 14 November 2022.
Edited by:
Ctirad Uher, Department of Physics, University of Michigan, United StatesReviewed by:
Kean C. Aw, The University of Auckland, New ZealandCopyright © 2022 Yang and Wu. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Wenbo Wu, wuwb@tju.edu.cn