面向无人机视角的多源信息融合目标检测

doi:10.12305/j.issn.1001-506X.2025.01.06

Abstract

Abstract:

To enhance the target detection performance of drones in complex environments, a target detection method based on multi-source information fusion is proposed. This method takes visible light and infrared images as inputs, applies the dual-branch Swin-Transformer structure to extract multi-level features of the two, and autonomously fuses the features at different levels in a self-learning manner to enhance information complementarity. On this basis, a bidirectional feature pyramid network is constructed to facilitate the fusion of shallow and deep features, that allows to capture multi-scale information of the target. Finally, multi-scale detection heads are utilized to independently predict targets on feature maps at different levels to improve the performance of the detector. Simulation and comparative experiments on multiple public datasets show that the proposed algorithm not only exhibits reasonable settings and superior performance, but also showcases robustness and generalization capabilities.

Key words: multi-source information fusion, drone, feature fusion, object detection

CLC Number:

TP391

Zishuo HAN, Xiquan FAN, Qiang FU, Chuanyan MA, Dongdong ZHANG. Target detection based on multi-source information fusion from the perspective of drones[J]. Systems Engineering and Electronics, 2025, 47(1): 52-61.

Figures/Tables 18

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Table 1

Fig.10

Table 2

Table 3

Table 4

Fig.11

Fig.12

Table 5

Fig.13

References 30

1	王波, 巫晨云, 杨晓露. 无人机应用难点分析及解决方案[J]. 通信与信息技术, 2024, (1):110-112.
	WANG B , WU C Y , YANG X L . Analysis of difficulties and solutions for drone applications[J]. Communication and Information Technology, 2024, (1): 110- 112.
2	LUO D H , LIU G , PRASAD D , et al. Infrared and visible ima-ge fusion based on VPDE model and VGG network[J]. Applied Intelligence, 2023, 53 (21): 24739- 24764. doi: 10.1007/s10489-023-04692-4
3	JEE S , KANG M G . Sensitivity improvement of extremely low light scenes with RGB-NIR multispectral filter array sensor[J]. Sensors, 2019, 19 (5): 1255- 1256. doi: 10.3390/s19051255
4	YANG L Z, MA R H, ZAKHOR A. Drone object detection using RGB/IR fusion[EB/OL]. [2023-08-08]. https://arxiv.org/abs/2201.03786.
5	BAO C , CAO J , HAO Q , et al. Dual-YOLO architecture from infrared and visible images for object detection[J]. Sensors, 2023, 23 (6): 2306- 2934.
6	宁大海, 郑晟. 可见光和红外图像决策级融合目标检测算法[J]. 红外技术, 2023, 45 (3): 282- 291.
	NING D H , ZHENG S . An object detection algorithm based on decision-level fusion of visible and infrared images[J]. Infrared Technology, 2023, 45 (3): 282- 291.
7	TANG C , LING Y S , YANG H , et al. Decision-level fusion detection for infrared and visible spectra based on deep learning[J]. Infrared and Laser Engineering, 2019, 48 (6): 626001. doi: 10.3788/IRLA201948.0626001
8	JOSEPH R, ALI F. YOLOv3: an incremental improvement[EB/OL]. [2023-08-06]. https://arxiv.org/pdf/1804.02767.
9	LIU Z, LIN Y T, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proc. of the IEEE/CVF International Conference on Computer Vision, 2021: 9992-10002.
10	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2023-08-08]. https://arXivpreprintarXiv:1409.1556.
11	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
12	HU J , SHEN L , ALBANIE S , et al. Squeeze-and-excitation networks[J]. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2020, 42 (8): 2011- 2023. doi: 10.1109/TPAMI.2019.2913372
13	NICOLAS C, FRANCISCO M, GABRIEL S, et al. End-to-end object detection with Transformers[EB/OL]. [2023-08-08]. https://arXivpreprintarXiv:2005.12872.
14	LEE J, LEE Y, KIM J, et al. Set Transformer: a framework for attention-based permutation-invariant neural networks[C]// Proc. of the 36th International Conference on Machine Learning, 2021.
15	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// Proc. of the 31st International Conference on Neural Information Processing Systems, 2017: 6000-6010.
16	DING X H, ZHANG X Y, MA N N, et al. RepVGG: making VGG-style ConvNets great again[C]//Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 13728-13737.
17	WANG Q , ZHOU L K , YAO Y C , et al. An interconnected feature pyramid networks for object detection[J]. Journal of Visual Communication and Image Representation, 2021, 79 (3): 103260.
18	HUANG G, LIU Z, LAURENS V D, et al. Densely connected convolutional networks[C]//Proc. of the Conference on Computer Vision and Pattern Recognition, 2017: 2261-2269.
19	LIN T, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 936-994.
20	LIU S, QI L, QIN H F, et al. Path aggregation network for instance segmentation[C]//Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 8759-8768.
21	SUN Y M , CAO B , ZHU P F , et al. Drone-based RGB-Infrared cross-modality vehicle detection via uncertainty-aware learning[J]. IEEE Trans. on Circuits and Systems for Video Technology, 2022, 32 (10): 6700- 6713. doi: 10.1109/TCSVT.2022.3168279
22	张冬冬, 王春平, 付强. 基于特征增强网络的SAR图像舰船目标检测[J]. 系统工程与电子技术, 2023, 45 (4): 1032- 1039.
	ZHANG D D , WANG C P , FU Q . Ship target detection in SAR image based on feature-enhanced network[J]. Systems Engineering and Electronics, 2023, 45 (4): 1032- 1039.
23	YANG T L , LI J H . Remote sensing image object detection based on improved YOLOv3 in deep learning environment[J]. Journal of Circuits, Systems and Computers, 2023, 32 (15): 2350265. doi: 10.1142/S0218126623502651
24	LIN T Y , GOYAL P , GIRSHICK R , et al. Focal loss for dense object detection[J]. IEEE Trans. on Pattern Analysis & Machine Intelligence, 2020, 42 (2): 318- 327.
25	REN S Q , HE K M , GIRSHICK R , et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137- 1149. doi: 10.1109/TPAMI.2016.2577031
26	HE K M , GKIOXARI G , DOLLAR P , et al. Mask R-CNN[J]. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2017, 42 (2): 386- 397.
27	白玉, 侯志强, 刘晓义, 等. 基于可见光图像与红外图像决策级融合的目标检测算法[J]. 空军工程大学学报, 2020, 21 (6): 53- 59.
	BAI Y , HOU Z Q , LIU X Y , et al. Target detection algorithm based on decision level fusion of visible image and infrared image[J]. Journal of Air Force Engineering University, 2020, 21 (6): 53- 59.
28	XIAO X W , WANG B , MIAO L J , et al. Infrared and visible image object detection via focused feature enhancement and cascaded semantic extension[J]. Remote Sensing, 2021, 13 (13): 2538. doi: 10.3390/rs13132538
29	JIA X Y, ZHU C, LI M Z, et al. LLVIP: a visible-infrared paired dataset for low-light vision[C]//Proc. of the IEEE/CVF International Conference on Computer Vision, 2021: 3489-3497.
30	RAZAKARIVONY S , JURIE F . Vehicle detection in aerial imagery: a small target detection benchmark[J]. Journal of Visual Communication and Image Representation, 2016, 34, 187- 203. doi: 10.1016/j.jvcir.2015.11.002

分类	图像对数	目标数
分类	图像对数	小汽车	卡车	巴士	面包车	货车
训练集	17 990	276 220	16 025	11 410	7 950	11 231
验证集	1 469	20 929	1 478	799	753	931
测试集	8 980	140 429	8 750	4 524	4 404	5 098

对应特征层	锚框
N₂特征层	(34, 18)	(20, 42)	(47, 26)
N₃特征层	(28, 57)	(44, 45)	(56, 67)
N₄特征层	(108, 46)	(38, 134)	(110, 118)

网络	FPS/(frame·s^-1)	AP/%					mAP/%
网络	FPS/(frame·s^-1)	小汽车	卡车	巴士	面包车	货车	mAP/%
可见光支路	69	86.80	71.84	88.21	72.78	62.55	76.44
红外支路	69	96.95	76.86	92.62	75.40	69.71	82.31
特征相加融合	48	97.47	81.38	95.57	84.30	72.83	86.31
特征拼接融合	47	97.60	81.65	96.22	85.47	73.29	86.85
特征自适应融合	43	98.02	85.56	97.59	86.23	75.72	88.62

算法	模式	FPS/(frame·s^-1)	AP/%					mAP/%
算法	模式	FPS/(frame·s^-1)	小汽车	卡车	巴士	面包车	货车	mAP/%
RetinaNet	可见光	48	67.50	28.24	62.05	19.26	13.72	38.16
Faster R-CNN	可见光	28	67.88	38.59	66.98	23.20	26.31	44.59
Mask R-CNN	可见光	23	68.52	39.84	66.75	25.35	26.83	45.46
所提算法(单支路)	可见光	69	86.80	71.84	88.21	72.78	62.55	76.44
RetinaNet	红外	48	79.86	32.84	67.32	16.44	28.05	44.90
Faster R-CNN	红外	28	88.63	42.51	77.92	28.52	35.16	54.55
Mask R-CNN	红外	22	88.77	48.86	78.38	32.16	36.63	56.96
所提算法(单支路)	红外	69	96.95	76.86	92.62	75.40	69.71	82.31
文献[27]	可见光+红外	52	77.94	55.41	85.59	36.80	39.47	59.04
文献[28]	可见光+红外	30	87.79	58.89	86.80	37.26	46.64	63.48
UA-CMDet	可见光+红外	36	87.51	60.70	87.08	37.95	46.80	64.01
所提算法	可见光+红外	43	98.02	85.56	97.59	86.23	75.72	88.62

算法	LLVIP	VEDAI
	AP	AP			mAP
	行人	汽车	飞机	船	mAP
所提算法	92.95	76.57	89.84	84.34	83.58

Target detection based on multi-source information fusion from the perspective of drones

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 18

References 30

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	Jie JIANG, Limin ZHANG, Kai LIU, Wenjun YAN, Meng WANG. Infrared ship target detection method based on task alignment learning [J]. Systems Engineering and Electronics, 2025, 47(1): 34-40.
[2]	Lei YU, Qiuyue DENG, Liying ZHENG, Haoyu WU. Second-order progressive feature fusion network for image super-resolution reconstruction [J]. Systems Engineering and Electronics, 2024, 46(2): 391-400.
[3]	Fengtao XUE, Tianyu SUN, Yimin YANG, Jian YANG. Rotated ship target detection algorithm in SAR images based on global feature fusion [J]. Systems Engineering and Electronics, 2024, 46(12): 4044-4053.
[4]	Jiayi LUO, Xuanpeng LI, Jianghao LI, Qifan XUE, Feng YANG, Weigong ZHANG. Radiation signal sorting method based on PDW multi-feature fusion [J]. Systems Engineering and Electronics, 2024, 46(1): 80-87.
[5]	Meng WANG, Bing ZHU. Application of uncertainty modeling in 2D and 3D object detection [J]. Systems Engineering and Electronics, 2023, 45(8): 2370-2376.
[6]	Haijun LI, Fancheng KONG, Yun LIN. Infrared ship detection algorithm based on improved YOLOv5s [J]. Systems Engineering and Electronics, 2023, 45(8): 2415-2422.
[7]	Daozhi WEI, Zhaoyu ZHANG, Jiahao XIE, Ning LI. Multi-sensor cross cueing technique based on improved Actor-Critic algorithm [J]. Systems Engineering and Electronics, 2023, 45(6): 1624-1632.
[8]	Yao LIU, Yangsheng XIA, Jianmai SHI, Chao CHEN, Jincai HUANG. Path planning method for multi-area coverage by cooperated ground vehicle multi-drone [J]. Systems Engineering and Electronics, 2023, 45(5): 1380-1390.
[9]	Zhengyuan LIU, Qinghua WANG. Review and prospect under the mapping mode of coordinated delivery of drones and vehicles [J]. Systems Engineering and Electronics, 2023, 45(3): 785-796.
[10]	Renfei CHEN, Yong PENG, Zhongwen LI. A novel detector for floating objects based on continual unsupervised domain adaptation strategy [J]. Systems Engineering and Electronics, 2023, 45(11): 3391-3401.
[11]	Jia LIU, Qunyu XU, Weishi CHEN. Motion feature extraction and ensembled classification method based on radar tracks for drones [J]. Systems Engineering and Electronics, 2023, 45(10): 3122-3131.
[12]	Wenqing SHI, Haifeng WANG, Haixin CHEN. Fighter-drone teaming system requirements elicitation and verification [J]. Systems Engineering and Electronics, 2023, 45(1): 108-118.
[13]	Shuang SONG, Yue ZHANG, Linna ZHANG, Yigang CEN, Yidong LI. Lightweight target detection algorithm based on deep learning [J]. Systems Engineering and Electronics, 2022, 44(9): 2716-2725.
[14]	Xiaofeng ZHAO, Fei WU, Yebin XU, Jiahui NIU, Wei CAI, Zhili ZHANG. Evaluation method of infrared camouflage effect based on background restoration [J]. Systems Engineering and Electronics, 2022, 44(8): 2554-2561.
[15]	Lin HUANG, Li GONG, Wei JIANG, Kangbo WANG. Remaining useful life prediction based on multi-source information fusion and HMM [J]. Systems Engineering and Electronics, 2022, 44(5): 1747-1756.