Pan Ruiying ^*

• The College of Henan Procuratorial Profession, Zhengzhou Henan province, 450003, Zhengzhou, China

Speech recognition and multimodal learning are two critical areas in machine learning.Current multimodal speech recognition systems often encounter challenges such as high computational demands and model complexity. To overcome these issues, we propose a novel framework-EnglishAL-Net, a Multimodal Fusion-powered English Speaking Robot. This framework leverages the ALBEF model, optimizing it for real-time speech and multimodal interaction, and incorporates a newly designed text and image editor to fuse visual and textual information. The robot processes dynamic spoken input through the integration of Neural Machine Translation (NMT), enhancing its ability to understand and respond to spoken language. In our experimental evaluation, we developed a dataset of diverse scenarios and oral commands for testing. Results demonstrate that compared to traditional unimodal approaches, our model significantly improves both language comprehension accuracy and response speed. This research advances the capabilities of multimodal interaction in robotics, offering promising applications in education, disaster relief, customer service, and beyond, with substantial theoretical and practical implications.