Guanqiu Qi
Zhiqin Zhu
Yu Liu
Huafeng Li
Bo Xiao
Jinxing Li6- 1Computer Information Systems Department, State University of New York at Buffalo, Buffalo, NY, United States
- 2College of Automation, Chongqing University of Posts and Telecommunications, Chongqing, China
- 3Department of Biomedical Engineering, Hefei University of Technology, Hefei, China
- 4Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China
- 5Department of Computing, Imperial College London, London, United Kingdom
- 6School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China
Editorial on the Research Topic
Multi-sensor imaging and fusion: methods, evaluations, and applications, volume II
Introduction
Multi-sensor imaging and fusion technology plays an increasingly important role in medical imaging [1, 2], medical image segmentation [3, 4], engineering construction [5, 6], complex task object detection [7, 8] and other fields [9, 10]. Multi-sensor image fusion mainly processes images of the same object or scene captured by multiple sensors [11], which complement each other by combining multi-level and multi-spatial information, ultimately providing a consistent interpretation of the observed environment. In recent years, multi-sensor image fusion has become a highly active Research Topic, and various fusion methods have been proposed. In addition, the performance evaluation and downstream applications of multi-sensor imaging and fusion technology [12] are also receiving increasing attention. This Research Topic focuses on cutting-edge research related to multi-sensor imaging and fusion technology, including image detection and fusion methods [13], objective evaluation methods [14], and specific applications in engineering problems [15]. After a thorough peer review process, all fifteen articles submitted to this Research Topic were accepted for publication. The main research results of these works are summarized in the following three aspects.
Imaging detection, feature extraction, and fusion methods in multi-sensors
Chen et al. proposed a structure similarity virtual map generation network (SVGNet) for optical and SAR image matching. This method uses an Attention U-Net and a conditional GAN to reduce modal differences, significantly improving the matching accuracy by more than two times compared to direct image matching.
Feng et al. proposed a cross-modal fusion framework based on YOLOv5 to improve night-time pedestrian detection under low-light conditions. This dual-stream architecture processes visible and infrared images separately, using a cross-modal feature rectification module (CMFRM) to fine-tune features and reduce noise. The two-stage feature fusion module (FFM) enhances feature output through cross-attention and mixed-channel embedding, significantly improving the accuracy and robustness of night-time pedestrian detection.
Xiong et al. proposed a sparse hair cluster detection model based on improved object detection neural networks and dermoscopic images. This model utilizes a multi-level feature fusion module to extract and fuse features at different levels, and a channel-space dual attention module to enhance representation capabilities and detection accuracy. The model, tested on self-annotated data, can accurately identify and count sparse hair clusters, which outperforms existing methods in accuracy and efficiency, making it a valuable tool for early detection and treatment of hair loss.
Chen et al. proposed the spatial-channel synergistic optimization net (SCSONet), a lightweight network for skin lesion segmentation designed to run efficiently with limited computing resources. This model introduces a ConvStem module with full-dimensional attention to enhance the recognition of irregularly shaped lesion regions while reducing parameters and computational burden. The SCF block further optimizes the model by fusing spatial and channel features to reduce feature redundancy. SCSONet was validated on two public skin lesion segmentation datasets, showing high effectiveness and robustness with low computational resource requirements.
Wang et al. developed a long-depth-of-field (DOF) full-field optical angiography (FFOA) imaging system to address the limitations of capturing complete blood flow information. A novel multi-focus image fusion scheme based on gradient feature detection was proposed. This method uses non-subsampled contourlet transform (NSCT) to decompose FFOA images and applies fusion rules based on gradient feature detection. Experimental results on phantoms and animal cases showed that this method effectively expands the DOF and solves the defocus issues, providing a more comprehensive description of blood information compared with a single FFOA image.
Song et al. investigated the application of deep learning in medical ultrasound imaging with a focus on reducing computational complexity and assisting novices. They explored deep learning solutions for improving image reconstruction and clinical diagnosis.
Objective evaluation methods in multi-sensor imaging
Peng et al. developed a novel method for detecting intracerebral hemorrhage (ICH) based on the frequency-dependent variations in permittivity, eliminating the need for non-hemorrhagic baseline data. By identifying the frequency points with the maximal permittivity differences between blood and other brain tissues, the method enables absolute detection of ICH. Experimental results show that specific frequency ranges can effectively detect blood in different tissue environments, bringing promise for rapid and accurate ICH detection.
Song et al. used functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI) to detect brain alterations in intensive care unit (ICU) patients developing delirium and assess their predictive value. The study compared fifteen ICU patients with delirium to fifteen healthy controls and found significant differences in brain activity and structure. In the delirium group, the regional homogeneity (ReHo) values of the left caudate nucleus and frontal lobe were lower, the amplitude of low-frequency fluctuations (ALFF) in the hippocampus and frontal lobe was altered, and the mean diffusivity (MD), radial diffusivity (RD), fractional anisotropy (FA), and axial diffusivity (AD) in several brain regions were reduced. Early fMRI and DTI examinations are recommended to predict delirium and facilitate early intervention, potentially improving patient outcomes.
Huang et al. used electrical capacitance tomography (ECT) with a symmetrical cancellation method to detect intracerebral hemorrhage (ICH). This method places electrodes symmetrically around the head and subtracts the measured capacitances to isolate hemorrhagic events. Testing on various models shows this method can achieve absolute imaging of ICH, although mirroring artifacts and the need for precise electrode alignment pose challenges. Nonetheless, this method shows promise for pre-hospital emergency detection of ICH.
Specific applications of multi-sensor technology in engineering problems
Li et al. proposed a dense metal corrosion depth estimation method based on image segmentation and inpainting to accurately measure corrosion depth using X-ray images. This method also includes virtual data generation techniques to create training images with ground-truth annotations, thereby improving the accuracy and reliability of the corrosion depth estimates. Experimental results confirm the effectiveness of the method on both virtual and real datasets.
Yang et al. applied a digital twin to highway tunnels using a multi-modal information fusion method based on convolutional neural network (CNN)–long short-term memory (LSTM)–attention. This system solves the challenges of sensor breakdown and insufficient data support in tunnel management. By realizing closed-loop management of “accurate perception–risk assessment–decision warning–emergency management,” the digital twin enhances traffic safety, reduces management costs, and improves driving comfort in highway tunnels.
Xian et al. developed an auto-verbalizer filtering method for prompt-based aspect category detection (ACD) in sentiment analysis. This approach automatically builds the verbalizer in prompt learning, enhancing the reliability of aspect categories in predictions. By leveraging the semantic extension of category labels and an indicator mechanism, their model significantly improves performance, improving by 7.5% on the zero-shot tasks and 2% on the few-shot tasks compared with the second-best models, especially excelling in handling general or miscellaneous aspect categories.
Luo et al. introduced an enhanced YOLOv5s + Deep SORT method for highway vehicle speed detection and multi-sensor verification. This approach optimizes data augmentation and incorporates the Swin Transformer module to improve local feature recognition. The model enhances vehicle detection using complete IoU (CIoU) loss for higher accuracy and Mish activation function for better convergence. Modified Deep SORT mitigates identity switching, and an image-to-coordinate transformation is used to calculate vehicle speed and average it over multiple frames. Multi-sensor verification shows the mean average precision (mAP) exceeds 90% and the speed measurement error is within 1–8 km/h, proving the model’s reliability and applicability for highway scenarios.
Wang et al. developed a novel algorithm for road surface detection that combines LiDAR point clouds with 2D images to predict drivable areas for autonomous vehicle navigation. The method constructs an altitude discrepancy map from LiDAR data to exploit the height uniformity of the road surface. An innovative attention mechanism with adaptive weighting coefficients is introduced to integrate altitude disparity images with image features for semantic segmentation. Empirical evaluation using the KITTI dataset demonstrates the superior accuracy of this method in road surface detection, advancing 3Dperception technology in autonomous driving.
Tang et al. applied mixed reality navigation technology (MRNT) to brainstem hematoma puncture and drainage surgery in seven patients with primary brainstem hemorrhage (PBH). This study aims to verify the feasibility and safety of MRNT. The technology demonstrates high precision, low cost, and an immersive operating experience. The results show that the average hematoma evacuation rate was 50.39%, and the postoperative GCS scores of patients improved significantly. No intraoperative deaths or postoperative complications were reported, indicating the potential of MRNT to improve surgical outcomes in patients with PBH.
Conclusion
Overall, the Research Topic collects a wide range of relevant topics. In particular, there are research hotspots in the fields of object detection, medical image analysis and evaluation, signal monitoring and fault detection.
Special thanks to Frontier in Physics for its support and efforts in this Research Topic. We would also like to thank all authors who contributed original work to this Research Topic and all reviewers for sharing their thoughts on the submissions. We hope that this Research Topic will inspire researchers in this field and push the research on multi-sensor imaging and fusion to new frontiers.
Author contributions
GQ: Writing–original draft, Writing–review and editing. ZZ: Writing–original draft, Writing–review and editing. YL: Writing–original draft, Writing–review and editing. HL: Writing–original draft, Writing–review and editing. BX: Writing–original draft, Writing–review and editing. JL: Writing–original draft, Writing–review and editing.
Funding
The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1. Zhu Z, He X, Qi G, Li Y, Cong B, Liu Y. Brain tumor segmentation based on the fusion of deep semantics and edge information in multimodal MRI. Inf Fusion (2023) 91, 376–387.
2. Xu Y, He X, Xu G, Qi G, Yu K, Yin L, et al. A medical image segmentation method based on multi-dimensional statistical features. Front Neurosci, Sec Brain Imaging Methods (2022) 16. doi:10.3389/fnins.2022.1009581
3. Liu Y, Shi Y, Mu F, Cheng J, Chen X. Glioma segmentation-oriented multi-modal MR image fusion with adversarial learning. IEEE/CAA J Autom Sin (2022) 9(10): 1528–1531.
4. Liu Y, Mu F, Shi Y, Chen X. SF-net: a multi-task model for brain tumor segmentation in multimodal MRI via image fusion. IEEE Signal Process Lett (2022) 29, 1799–1803.
5. Qi G, Zhang Y, Wang K, Mazur N, Liu Y, Malaviya D. Small object detection method based on adaptive spatial parallel convolution and fast multi-scale fusion. Remote Sens (2022) 14 (2), 420.
6. Zhu Z, Wang S, Gu S, Li Y, Li J, Shuai L, et al. Driver distraction detection based on lightweight networks and tiny object detection. Math Biosci Eng (2023) 20 (10), 18248–18266.
7. Zhu Z, Zheng R, Qi G, Li S, Li Y, Gao X. Small object detection method based on global multi-level perception and dynamic region aggregation. IEEE Trans Circuits Syst Video Technol (2024). doi:10.1109/TCSVT.2024.3402097
8. Li Y, Zhou Z, Qi G, Hu G, Zhu Z, Huang X. Remote sensing micro-object detection under global and local attention mechanism. Remote Sens (2024) 16 (4), 644.
9. Zhu Z, Wei H, Hu G, Li Y, Qi G, Mazur N. A novel fast single image dehazing algorithm based on artificial multiexposure image fusion. IEEE Trans Instrum Meas (2021) 70, 1–23.
10. Zheng M, Qi G, Zhu Z, Li Y, Wei H, Liu Y. Image dehazing by an artificial image fusion method based on adaptive structure decomposition. IEEE Sens J (2020) 20 (14), 8062–8072.
11. Zhu Z, Sun M, Qi G, Li Y, Gao X, Liu Y. Sparse dynamic volume TransUNet with multi-level edge fusion for brain tumor segmentation. Comput Biol Med (2024) 172, 108284.
12. Li Y, Wang Z, Yin L, Zhu Z, Qi G, Liu Y. X-net: a dual encoding–decoding method in medical image segmentation. Vis Comput (2023) 39, 2223–2233.
13. Qi G, Wang H, Haner M, Weng C, Chen S, Zhu Z. Convolutional neural network based detection and judgement of environmental obstacle in vehicle operation. CAAI trans intell technol (2019) 4 (2), 80–91.
14. Liu L, Qi Z, Cheng J, Chen X. Rethinking the effectiveness of objective evaluation metrics in multi-focus image fusion: a statistic-based approach. IEEE Trans Pattern Anal Mach Intell (2024). doi:10.1109/TPAMI.2024.3367905
Keywords: multi-sensor fusion, imaging, multi-sensor image segmentation, machine learning, performance evaluation, deep learning
Citation: Qi G, Zhu Z, Liu Y, Li H, Xiao B and Li J (2024) Editorial: Multi-sensor imaging and fusion: methods, evaluations, and applications, volume II. Front. Phys. 12:1450685. doi: 10.3389/fphy.2024.1450685
Received: 18 June 2024; Accepted: 24 June 2024;
Published: 10 July 2024.
Edited and reviewed by:
Cinzia Da Via, The University of Manchester, United KingdomCopyright © 2024 Qi, Zhu, Liu, Li, Xiao and Li. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Guanqiu Qi, qig@buffalostate.edu