Development of A Multi-modal Deep-Learning-Based Web Application for Image Classification of Marine Decapods in Conservation Practice

Peng Zhou ^1* Zhinuo Zhou ² Yu-Xin Bu ¹ Chunsheng Wang ¹ Hong-Bin Shen ² Xiaoyong Pan ²

• ¹ Second Institute of Oceanography, Ministry of Natural Resources, Hangzhou, China
• ² Shanghai Jiao Tong University, Shanghai, Shanghai Municipality, China

Marine invasive Decapoda species have caused huge losses to biodiversity and world fisheries. Early awareness of non-indigenous species (NIS) is critical to prompt response and mitigate impacts. Citizen support has emerged as a valuable tool for the early detection of NIS worldwide. However, the great biodiversity of Decapoda species in global oceans poses challenges for the public to the recognize marine Decapoda species, especially for the uncommon or unfamiliar specimens, which sometimes might be NIS. However, despite remarkable performance of Deep learning (DL) techniques in automated image analysis, there remains a scarcity of professional tools tailored specifically for the image classification of diverse decapods. To tackle this challenge, a web application for automated image classification of marine Decapoda species, termed DecapodAI, was developed by training the Fine-tuning Contrastive Language–Image Pretraining model with the images from World Register of Marine Species. For the test dataset, DecapodAI achieved an average accuracy at 0.717 (family), 0.719 (genus), 0.773 (species), respectively. Online service is provided at http://www.csbio.sjtu.edu.cn/bioinf/DecapodAI/. DecapodAI can help alleviate the burden of manually analyzing images. It is expected to promote public participation and has promising application prospects in exploring and monitoring the biodiversity of decapods in global oceans, including early awareness of NIS.