基于RGB-IR多模态融合的番茄小目标检测

doi:10.11896/jsjkx.250700173

计算机科学 ›› 2026, Vol. 53 ›› Issue (6A): 250700173-8.doi: 10.11896/jsjkx.250700173

• 图像处理&多媒体技术 • 上一篇下一篇

基于RGB-IR多模态融合的番茄小目标检测

董烨¹, 连心悦¹, 王雨阳¹, 欧新宇^1,2

1 云南财经大学信息学院昆明 650600
2 云南省服务计算重点实验室昆明 650221

出版日期:2026-06-16 发布日期:2026-06-12
通讯作者: 欧新宇(ouxinyu@alumni.hust.edu.cn)
作者简介:(dongyeamy@163.com)
基金资助:
国家自然科学基金(62462065);教育部人文社会科学研究项目(25YJCZH220);云南财经大学云彩学者项目(2024D45)

RGB-IR Multi-modal Fusion-based Tomato Small Object Detection

DONG Ye¹, LIAN Xinyue¹, WANG Yuyang¹, OU Xinyu^1,2

1 School of Information,Yunnan University of Finance and Economics,Kunming 650600,China
2 Yunnan Key Laboratory of Service Computing,Kunming 650221,China

Published:2026-06-16 Online:2026-06-12
About author:DONG Ye,born in 2004,undergraduate.Her main research interests include computer vision, image recognition and deep learning algorithms for practical applications.
OU Xinyu,born in 1982,Ph.D,professor.His main research interests include computer vision,multi-modality,and federated learning.
Supported by:
National Natural Science Foundation of China(62462065),Ministry of Education Humanities and Social Sciences Research Project(25YJCZH220) and Yunnan University of Finance and Economics Yuncai Scholar Project(2024D45).

摘要/Abstract

摘要： 番茄自动化采摘在现代农业生产中至关重要,能够显著提升效率,降低成本,并保障果实品质。然而,现有基于单一模态的自动采摘技术在果园环境中仍面临诸多挑战:复杂光照条件与枝叶遮挡容易导致番茄目标出现误检和漏检;同时,番茄作为小目标,在传统RGB图像中的特征易被背景噪声淹没;此外,单一传感器数据难以全面捕捉目标特征,限制了模型的鲁棒性与适应性。针对上述问题,提出一种基于多模态特征融合的RTDMF模型(Robust Tomato Detection with Multi-modal Fusion),通过整合RGB与IR图像的互补特性,增强番茄检测的精准性与稳定性。该模型在YOLOv5基础上进行优化,针对小目标检测引入轻量化深度可分离卷积与自适应锚框策略,以提升对细微特征的捕捉能力;采用双分支架构分别处理RGB和IR图像,并借助自注意力机制实现多模态特征的高效融合,充分结合色彩、纹理与温度信息;此外,引入Mosaic数据增强与动态学习率策略,进一步提升模型的泛化性能与收敛效率。实验结果表明,在包含多种光照与遮挡条件的“Multimodal image dataset of tomato fruits with different maturity”数据集上,RTDMF模型将使平均精度(mAP)提升9.7%,召回率提高0.6%,漏检率降低2.3%,误检率降低3.1%。可视化结果进一步验证了该模型在低对比度、遮挡等复杂场景下的强适应性。

关键词: 番茄小目标检测, 多模态融合, 自适应锚框优化, 双分支架构, 目标特征细化

Abstract: Automated tomato harvesting plays a pivotal role in enhancing agricultural efficiency and ensuring produce quality,but faces significant challenges in complex orchard environments where single-modal systems often fall short.Variabilities in lighting,occlusions by foliage,and the subtle characteristics of small targets in RGB images,coupled with inadequate feature extraction from single-sensor data,significantly impede precise detection.This study introduces the robust tomato detection with multi-modal fusion(RTDMF) model,designed to address these limitations by integrating RGB and infrared(IR) imaging technologies to bolster detection robustness.Constructed on the YOLOv5 framework,RTDMF incorporates lightweight depthwise separable convolutions and adaptive anchor boxes to enhance sensitivity towards small targets.The dual-branch architecture of RTDMF processes RGB and IR data independently,fusing color,texture,and thermal features effectively through self-attention mechanisms and specialized fusion modules.Furthermore,Mosaic data augmentation and dynamic learning rate strategies are employed to further enhance the model's generalization and convergence capabilities.Evaluated on a multimodal tomato dataset encompassing varying levels of maturity,diverse lighting conditions,and occlusion scenarios,RTDMF demonstrates a notable improvement,it achieves a 9.7% increase in mean average precision(mAP) and a0.6% higher recall rate compared to single-modal models.It also significantly reduces the miss and false detection rates by 2.3% and 3.1%,respectively.Visual analysis confirms the model's effectiveness in low-contrast and heavily occluded scenarios,showcasing its superior adaptability to real-world agricultural challenges.This multi-modal approach delivers a robust solution for automated harvesting systems in dynamic environments,marking a significant advancement in the field of agricultural automation.

Key words: Tomato small object detection, Multi-modal fusion, Adaptive anchor box optimization, Dual-branch architecture, Target feature refinement

中图分类号:

TP391

董烨, 连心悦, 王雨阳, 欧新宇. 基于RGB-IR多模态融合的番茄小目标检测[J]. 计算机科学, 2026, 53(6A): 250700173-8. https://doi.org/10.11896/jsjkx.250700173

DONG Ye, LIAN Xinyue, WANG Yuyang, OU Xinyu. RGB-IR Multi-modal Fusion-based Tomato Small Object Detection[J]. Computer Science, 2026, 53(6A): 250700173-8. https://doi.org/10.11896/jsjkx.250700173

参考文献

[1] RAM P P V S,YASWANTH K V S,KAMEPALLI S S B S,et al.Deep learning model YOLOv5 for red chilies detection from chilly crop images[C]//Proc of the 8th IEEE Int Conf on Technology Fusion(I2CT).IEEE,Piscataway,NJ,2023.
[2] ZHANG Y,RAO Y,CHEN W J,et al.Multimodal image dataset of tomato fruits with different maturity[DB/OL].Science Data Bank.Elsevier,2023.
[3] LI S Q,TANG L,LIU K Y,et al.A fast and adaptive objecttracking method[J].Journal of Computer Research and Deve-lopment,2012,49(2):383-391.
[4] XU S F,CHEN X,LI H W,et al.Airborne small target detection method based on multimodal and adaptive feature fusion[J].IEEE Transactions on Aerospace and Electronic Systems,2024,62(2):1-15.
[5] ZHAO J M,SHI Z L,YU C,et al.Infrared small target detection based on adjustable sensitivity strategy and multi-scale fusion[J].Infrared Physics & Technology,2024,2407(20090):1-14.
[6] BADRINARAYANAN V,KENDALL A,CIPOLLA R.Seg-Net:A deep convolutional encoder-decoder architecture for image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,39(12):2481-2495.
[7] NIU S Q,XU X L,LIANG A,et al.Research on a lightweight method for maize seed quality detection based on improved YOLOv8[J].IEEE Access,2024,12:32927-32937.
[8] WANG C Y,BOCHKOVSKIY A,LIAO H Y.YOLOv7:Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[C]//Proc of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR 2021).IEEE,Vancouver,BC,Canada,2023.
[9] SHANG Y Y,ZHANG Q R,SONG H B.Application of deep learning using YOLOv5s to apple flower detection in natural scenes[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2022,38(9):222-229.
[10] OMER S M,GHAFOOR K,ASKAR S.Lightweight improvedthe YOLOv5 model for cucumber leaf disease and pest detection based on deep learning.Computers and Electronics in Agriculture[J].Signal,Image and Video Processing,2024,18:1329-1342.
[11] WANG F Z,WANG P,ZHANG X,et al.An overview of parametric modeling and methods for radar target detection with limited data[J].IEEE Access,2021,9:60459-60469.
[12] LIU L B,CHEN J Q,WU H F,et al.Cross-modal collaborative representation learning and a large-scale RGBT benchmark for crowd counting[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).IEEE,2021:479-488.
[13] XIE J,NIE J,DING B A,et al.Cross-Modal Local Calibration and Global Context Modeling Network for RGB-Infrared Remote-Sensing Object Detection[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing,2023,16:8933-8942.
[14] MEES O,EITEL A,BURGARD W.Choosing smartly:Adaptive multimodal fusion for object detection in changing environments[C]//Proc of the IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS).Daejeon,Korea(South):IEEE,2016:775-9048.
[15] TANG H,LI Z C,ZHANG D,et al.Divide-and-Conquer:Confluent Triple-Flow Network for RGB-T Salient Object Detection[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2025,47(3):1958-1974.
[16] TIAN H K,SONG K C,TONG L,et al.Robot Unknown Objects Instance Segmentation Based on Collaborative Weight Assignment RGB-Depth Fusion[J].IEEE/ASME Transactions on Mechatronics,2024,29(3):2032-2043.
[17] WANG M C,WANG H,LI Y C,et al.MSAFusion:Object Detection Based on Multisensor Adaptive Fusion Under BEV[J]. IEEE Transactions on Instrumentation and Measurement,2025,74:9509212.
[18] WANG L C,LI C X.Research on multimodal fusion object detection algorithm of image design and computational vision[C]// International Proc of 2024 5th International Conference on Computer Vision,Image and Deep Learning(CVIDL).2024:19-21.
[19] CHEN J,REN H Z,YANG H T,et al.M2FNet:Multi-modal fusion network for object detection from visible and thermal infrared images[J].International Journal of Applied Earth Observation and Geoinformation,2024,130(103918):1-16.
[20] YUN S D,HAN D Y,OH S J,et al.CutMix:Regularizationstrategy to train strong classifiers with localizable features[C]//Proc of IEEE/CVF CVPR.2019:10.27-11.02:612.
[21] ZHONG Z,LIANG Z,KANG G L,et al,Random Erasing data augmentation[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2020:13001-13008.
[22] DAI H C,WEI X Y,XU Y X,et al.Multimodal fusion detection method for tomato fruits based on phase and hyperspectral imaging[J].Acta Photonica Sinica,2024,53(1):1-10.
[23] LI R J,SONG T,GAO J,et al.Tomato diseased leaf detection model in natural environment based on improved YOLOv5[J].Jiangsu Journal of Agricultural Sciences,2024,40(6):1028-1037.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

基于RGB-IR多模态融合的番茄小目标检测

RGB-IR Multi-modal Fusion-based Tomato Small Object Detection

PDF (PC)

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 0

Metrics

本文评价

推荐阅读 0