Zero-Shot Instance Segmentation for Plant Phenotyping in Vertical Farming with Foundation Models and VC-NMS

Qinzhou Bao ¹ Yixin Yang ² Qing Li ¹ Haichao Yang ^1*

• ¹ Dali University, Dali, China
• ² Xijing University, Xi'an, Shaanxi Province, China

Image instance segmentation is an efficient technique for plant phenotyping. However, the diverse plant types and limited availability of annotated image data in vertical farms limits the effectiveness of traditional supervised segmentation techniques. To overcome these challenges, we propose a zero-shot instance segmentation framework that integrates Grounding DINO with the Segment Anything Model (SAM). We use Vegetation Cover Aware Non-Maximum Suppression (VC-NMS), which incorporates the Normalized Crop Greenness Index (NCGI) to enhance box prompts. Additionally, similarity maps with the max distance criterion are combined to improve point prompts. Experiments show that these enhanced box and point prompts significantly outperform SAM's anything mode and Grounded SAM in zero-shot segmentation.Compared to supervised methods like YOLOv11, our approach exhibits exceptional zero-shot generalization. It achieves the best segmentation performance on two test sets, providing an effective solution to scarce annotation data in vertical farming.