基于可见光与红外特征融合的轻量化目标检测方法

doi:10.12305/j.issn.1001-506X.2026.05.04

Abstract

Abstract:

To address the problems of low efficiency and high computational complexity of dual-stream object detection models, a lightweight object detection method based on visible and infrared feature fusion is proposed. Firstly, you only look once （YOLO） v8 is expanded into a dual-stream object detection model, the dual-stream backbone network is optimized using group convolution, and the two independent backbone networks are merged into one backbone network, which realizes the synchronous extraction of two modal features, greatly improving the model operation efficiency. Secondly, the faster cross-stage partial bottleneck with two convolution with cross-modal feature interaction （C2f-CMFI） module and the spatial pyramid pooling fast with cross-modal feature fusion （SPPF-CMFF） module are designed, while reducing the complexity of the model, fusion and interaction of the two modal features during the feature extraction process are realized. Finally, the experimental results on the public visible-infrared dataset show that compared with the traditional dual-stream object detection models, the parameter amount and computational complexity of the proposed method are reduced by 19.5% and 17.7% respectively, and the mean average precision 50:95 is improved by 1.9%. On a NVIDIA RTX 2080Ti graphics processing unit, the inference speed is 140 frames per second, which proved the effectiveness of the proposed method.

Key words: visible-infrared image, you only look once （YOLO）, lightweighting, object detection, dual-stream structure

CLC Number:

TP 391.4

Jie ZHANG, Tianqing CHANG, Xiaowei WANG, Wenlong HAO, Xin TANG. Lightweight object detection method based on visible and infrared feature fusion[J]. Systems Engineering and Electronics, 2026, 48(5): 1481-1491.

Figures/Tables 18

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Fig.13

Table 1

Table 2

Table 3

Table 4

Fig.14

References 35

1	CHENG G, YUAN X, YAO X W, et al. Towards large-scale small object detection: survey and benchmark[J]. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2023, 45 (11): 13467- 13488.
2	GHAHREMANNEZHAD H, SHI H, LIU C J. Object detection in traffic videos: a survey[J]. IEEE Trans. on Intelligent Transportation Systems, 2023, 24 (7): 6780- 6799. doi: 10.1109/TITS.2023.3258683
3	WANG C Y, BOCHKOVSKIY A, LIAO H Y M. YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[C]//Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 7464−7475.
4	LI R M, XIANG J J, SUN F X, et al. Multiscale cross-modal homogeneity enhancement and confidence-aware fusion for multispectral pedestrian detection[J]. IEEE Trans. on Multimedia, 2024, 26, 852- 863. doi: 10.1109/TMM.2023.3272471
5	SHAO Y H, HUANG Q M, MEI Y Y, et al. MOD-YOLO: multispectral object detection based on transformer dual-stream YOLO[J]. Pattern Recognition Letters, 2024, 183, 26- 34. doi: 10.1016/j.patrec.2024.05.001
6	CHEN Y Y, JHONG S Y, LO Y J. Reinforcement-and-alignment multispectral object detection using visible-thermal vision sensors in intelligent vehicles[J]. IEEE Sensors Journal, 2023, 23 (21): 26873- 26886. doi: 10.1109/JSEN.2023.3319230
7	张睿, 李允臣, 王家宝, 等. 多尺度特征融合的双模态目标检测方法[J]. 计算机工程与应用, 2024, 60 (17): 233- 242.
	ZHANG R, LI Y C, WANG J B, et al. Multiscale feature fusion approach for dual-modal object detection[J]. Computer Engineering and Applications, 2024, 60 (17): 233- 242.
8	LI Q, ZHANG C Q, HU Q H, et al. Stabilizing multispectral pedestrian detection with evidential hybrid fusion[J]. IEEE Trans. on Circuits and Systems for Video Technology, 2024, 34 (4): 3017- 3029. doi: 10.1109/TCSVT.2023.3306870
9	YOU S A, XIE X D, FENG Y J, et al. Multi-scale aggregation transformers for multispectral object detection[J]. IEEE Signal Processing Letters, 2023, 30, 1172- 1176. doi: 10.1109/LSP.2023.3309578
10	LI Q, ZHANG C Q, HU Q H, et al. Confidence-aware fusion using Dempster-Shafer theory for multispectral pedestrian detection[J]. IEEE Trans. on Multimedia, 2023, 25, 3420- 3431. doi: 10.1109/TMM.2022.3160589
11	ZHU J H, CHEN X, ZHANG H, et al. Transformer based remote sensing object detection with enhanced multispectral feature extraction[J]. IEEE Geoscience and Remote Sensing Letters, 2023, 20, 5001405.
12	SONG K H, ZHAO Y, HUANG L M, et al. RGB-T image analysis technology and application: a survey[J]. Engineering Applications of Artificial Intelligence, 2023, 120, 105919. doi: 10.1016/j.engappai.2023.105919
13	LIU J Y, FAN X, HUANG Z B, et al. Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection[C]//Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 5802−5811.
14	HUANG L, PENG Z J, CHEN F, et al. Cross-modality interaction for few-shot multispectral object detection with semantic knowledge[J]. Neural Networks, 2024, 173, 106156. doi: 10.1016/j.neunet.2024.106156
15	CHEN K Y, LIU J Q, ZHANG H. IGT: Illumination-guided RGB-T object detection with transformers[J]. Knowledge-Based Systems, 2023, 268, 110423. doi: 10.1016/j.knosys.2023.110423
16	韩子硕, 范喜全, 付强, 等. 面向无人机视角的多源信息融合目标检测[J]. 系统工程与电子技术, 2025, 47 (1): 52- 61.
	HAN Z S, FAN X Q, FU Q. Target detection based on multi-source information fusion from the perspective of drones[J]. Systems Engineering and Electronics, 2025, 47 (1): 52- 61.
17	HU W J, FU C L, CAO R L, et al. Joint dual-stream interaction and multi-scale feature extraction network for multi-spectral pedestrian detection[J]. Applied Soft Computing, 2023, 147, 110768. doi: 10.1016/j.asoc.2023.110768
18	AN Z J, LIU C L, HAN Y Q. Effectiveness guided cross-modal information sharing for aligned RGB-T object detection[J]. IEEE Signal Processing Letters, 2022, 29, 2562- 2566. doi: 10.1109/LSP.2022.3229571
19	FU H L, WANG S X, DUAN P H, et al. LRAF-Net: long-range attention fusion network for visible-infrared object detection[J]. IEEE Trans. on Neural Networks and Learning Systems, 2023, 35 (10): 13232- 13245.
20	LIU Z, LIN Y T, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 10012−10022.
21	GONG Y, WANG L, XU L S. A feature aggregation network for multispectral pedestrian detection[J]. Applied Intelligence, 2023, 53 (19): 22117- 22131. doi: 10.1007/s10489-023-04628-y
22	XIE Y M, ZHANG L W, YU X Y, et al. YOLO-MS: multispectral object detection via feature interaction and self-attention guided fusion[J]. IEEE Trans. on Cognitive and Developmental Systems, 2023, 15 (4): 2132- 2143. doi: 10.1109/TCDS.2023.3238181
23	SHEN J F, CHEN Y F, LIU Y, et al. ICAFusion: iterative cross-attention guided feature fusion for multispectral object detection[J]. Pattern Recognition, 2024, 145, 109913. doi: 10.1016/j.patcog.2023.109913
24	ZHOU K L, CHEN L S, CAO X. Improving multispectral pedestrian detection by addressing modality imbalance problems[C]//Proc. of the 16th European Conference on Computer Vision, 2020: 787−803.
25	SUN Y M, CAO B, ZHU P F, et al. Drone-based RGB-infrared cross-modality vehicle detection via uncertainty-aware learning[J]. IEEE Trans. on Circuits and Systems for Video Technology, 2022, 32 (10): 6700- 6713. doi: 10.1109/TCSVT.2022.3168279
26	ZHAO T Y, YUAN M X, JIANG F, et al. Removal and selection: improving RGB-infrared object detection via coarse-to-fine fusion [EB/OL]. [2025-02-12]. https: //arxiv.org/abs/2401.10731.
27	WANG H Y, WANG C P, FU Q, et al. Cross-modal oriented object detection of UAV aerial images based on image feature[J]. IEEE Trans. on Geoscience and Remote Sensing, 2024, 62, 5403021. doi: 10.1109/tgrs.2024.3367934
28	WANG H Y, WANG C P, FU Q, et al. YOLOFIV: object detection algorithm for around-the-clock aerial remote sensing images by fusing infrared and visible features[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2024, 17, 15269- 15287. doi: 10.1109/JSTARS.2024.3447649
29	王晓军, 陈高宇, 李晓航. 应用动态激活函数的轻量化YOLOv8行人检测算法[J]. 计算机工程与应用, 2024, 60 (15): 221- 233.
	WANG X J, CHEN G Y, LI X H. Lightweight YOLOv8 pedestrian detection algorithm using dynamic activation function[J]. Computer Engineering and Applications, 2024, 60 (15): 221- 233.
30	吴磊, 储钰昆, 杨洪刚, 等. 面向铝合金焊缝DR图像缺陷的Sim-YOLOv8目标检测模型[J]. 中国激光, 2024, 51 (16): 29- 38. doi: 10.3788/CJL231485
	WU L, CHU Y K, YANG H G, et al. Sim-YOLOv8 object detection model for DR image defects in aluminum alloy welds[J]. Chinese Journal of Lasers, 2024, 51 (16): 29- 38. doi: 10.3788/CJL231485
31	QU J L, LI Q, PAN J, et al. SS-YOLOv8: small-size object detection algorithm based on improved YOLOv8 for UAV imagery[J]. Multimedia Systems, 2025, 31 (1): 42. doi: 10.1007/s00530-024-01622-3
32	HAN K, WANG Y H, TIAN Q, et al. Ghostnet: more features from cheap operations[C]//Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 1580−1589.
33	ZHANG H, FROMONT E, LEFEVRE S, et al. Multispectral fusion for object detection with cyclic fuse-and-refine blocks[C]///Proc. of the IEEE International Conference on Image Processing, 2020: 276−280.
34	FANG Q Y, HANG D P, WANG Z K. Cross-modality fusion transformer for multispectral object detection[EB/OL]. [2025-02-04]. https://arxiv.org/abs/2111.00273.
35	解宇敏, 张浪文, 余孝源, 等. 可见光–红外特征交互与融合的YOLOv5目标检测算法[J]. 控制理论与应用, 2024, 41 (5): 914- 922.
	XIE Y M, ZHANG L W, YU X Y, et al. YOLOv5 object detection algorithm with visible-infrared feature interaction and fusion[J]. Control Theory and Applications, 2024, 41 (5): 914- 922.

检测模型	骨干网络		$w$	Params（×10⁶）	FLOPs（×10⁹）	FPS/（帧/秒）	mAP50:95/%
检测模型	DSB	DSBBGCO	$w$	Params（×10⁶）	FLOPs（×10⁹）	FPS/（帧/秒）	mAP50:95/%
DSODM-n	√	－	0.25	4.4	9.2	145	40.0
DSODM-n	－	√	0.25	4.4	9.2	166（+21）	40.0（+0.0）
DSODM-s	√	－	0.5	16.9	33.8	124	41.1
DSODM-s	－	√	0.5	16.9	33.8	135（+11）	41.2（+0.1）

检测模型	$ w $	骨干网络构成		Params（×10⁶）	FLOPs（×10⁹）	FPS/（帧/秒）	mAP50:95/%
检测模型	$ w $	GC+DSSPPF +DSC2f	GC+DSSPPF +C2f-CMFI	Params（×10⁶）	FLOPs（×10⁹）	FPS/（帧/秒）	mAP50:95/%
DSODM-n	0.25	√	－	4.4	9.2	166（+0）	40.0
DSODM-n	0.25	－	√	3.9	8.0	168（+2）	40.8（+0.8）
DSODM-s	0.5	√	－	16.9	33.8	135（+0）	41.2
DSODM-s	0.5	－	√	14.6	28.5	138（+3）	42.8（+1.6）

检测算法	$w$	骨干网络构成		Params（×10⁶）	FLOPs（×10⁹）	FPS/（帧/秒）	mAP50:95/%
检测算法	$w$	GC+C2f-CMFI +DSSPPF	GC+C2f-CMFI +SPPF-CMFF	Params（×10⁶）	FLOPs（×10⁹）	FPS/（帧/秒）	mAP50:95/%
DSODM-n	0.25	√	－	3.9	8.0	167	40.8
DSODM-n	0.25	－	√	3.6（-0.3）	7.8（-0.2）	170（+3）	40.9（+0.1）
DSODM-s	0.5	√	－	14.6	28.5	138	42.8
DSODM-s	0.5	－	√	13.6（-1.0）	27.8（-0.7）	140（+2）	43.0（+0.2）

检测算法	模态	参数（×10⁶）	FLOPs（×10⁹）	mAP50:95/%
YOLOv8-n	可见光	3.0	6.5	28.7
YOLOv8-n	红外	3.0	6.5	39.3
YOLOv8-s	可见光	11.1	22.8	30.3
YOLOv8-s	红外	11.1	22.8	40.9
CFT[34]	可见光+红外	206.0	224.4	40.2
YOLO-MS[22]	可见光+红外	15.3	36.9	38.3
ICAFusion[23]	可见光+红外	120.21	−	41.4
文献[35]	可见光+红外	13.01	33	37.9
LRAF-Net[19]	可见光+红外	18.8	40.5	42.8
DSODM-n	可见光+红外	4.4	9.2	40.0
本文-n	可见光+红外	3.6	7.8	40.9
DSODM-s	可见光+红外	16.9	33.8	41.1
本文-s	可见光+红外	13.6	27.8	43.0

[1]	Shuo LIU, Yu ZHOU, Zhehao ZHANG, Song SHANG, Song CHENG. Anti-same-frequency interference technology based on time-domain multi-pulse correlation and clustering analysis [J]. Systems Engineering and Electronics, 2026, 48(4): 1186-1194.
[2]	Kun CHENG, Hongtao LEI, Zhixuan LYU. Lightweight unmanned aerial vehicle tracking method of improved YOLOv8 [J]. Systems Engineering and Electronics, 2026, 48(3): 737-750.
[3]	Ying YU, Chunping WANG, Jinhui XU, Shuhang LYU, Qiang FU, Ming CHEN. PE-Net: a real-time landslide detection network with optimized pruning [J]. Systems Engineering and Electronics, 2025, 47(8): 2475-2485.
[4]	Kang NI, Wenjie JIA, Minrui ZOU, Zhizhong ZHENG. SAR object detection based on dynamic aggregation network [J]. Systems Engineering and Electronics, 2025, 47(8): 2527-2539.
[5]	Jie JIANG, Limin ZHANG, Kai LIU, Wenjun YAN, Meng WANG. Infrared ship target detection method based on task alignment learning [J]. Systems Engineering and Electronics, 2025, 47(1): 34-40.
[6]	Zishuo HAN, Xiquan FAN, Qiang FU, Chuanyan MA, Dongdong ZHANG. Target detection based on multi-source information fusion from the perspective of drones [J]. Systems Engineering and Electronics, 2025, 47(1): 52-61.
[7]	Duanyang SHI, Qiang LIN, Bing HU, Xiaoshuai DU. Target detection method of primary surveillance radar based on YOLO [J]. Systems Engineering and Electronics, 2024, 46(1): 143-151.
[8]	Meng WANG, Bing ZHU. Application of uncertainty modeling in 2D and 3D object detection [J]. Systems Engineering and Electronics, 2023, 45(8): 2370-2376.
[9]	Daozhi WEI, Zhaoyu ZHANG, Jiahao XIE, Ning LI. Multi-sensor cross cueing technique based on improved Actor-Critic algorithm [J]. Systems Engineering and Electronics, 2023, 45(6): 1624-1632.
[10]	Renfei CHEN, Yong PENG, Zhongwen LI. A novel detector for floating objects based on continual unsupervised domain adaptation strategy [J]. Systems Engineering and Electronics, 2023, 45(11): 3391-3401.
[11]	Shuang SONG, Yue ZHANG, Linna ZHANG, Yigang CEN, Yidong LI. Lightweight target detection algorithm based on deep learning [J]. Systems Engineering and Electronics, 2022, 44(9): 2716-2725.
[12]	Hongyao LI, Xiaoqiang LI, Xinzhong HAN, Xueli XIE, Jianxiang XI. Cooperative object detection and recognition algorithm for multiple UAVs based on decision fusion [J]. Systems Engineering and Electronics, 2022, 44(3): 746-754.
[13]	Jiali FAN, Shaobing TIAN, Kui HUANG, Xingdong ZHU. Multi-scale object detection algorithm for aircraft carrier surface based on Faster R-CNN [J]. Systems Engineering and Electronics, 2022, 44(1): 40-46.
[14]	Yiming ZHANG, Jianliang AI. Positioning of aerial refueling drogue and docking control based on binocular vision [J]. Systems Engineering and Electronics, 2021, 43(10): 2940-2953.
[15]	Zhenzhen YANG, Jun LE, Yongpeng YANG, Lu FAN. Object detection algorithm of nonconvex motion-assisted low rank and sparse decomposition [J]. Systems Engineering and Electronics, 2020, 42(6): 1218-1225.

Lightweight object detection method based on visible and infrared feature fusion

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 18

References 35

Related Articles 15

Recommended Articles

Metrics

Comments