基于多级别融合的可见光和SAR图像联合识别

doi:10.12305/j.issn.1001-506X.2026.03.09

Abstract

Abstract:

To address the issues of information loss and detail blurring in the fusion process of synthetic aperture radar （SAR） and visible light images, which lead to insufficient accuracy in target detection and recognition, a multi-level image fusion method spanning pixel, feature, and decision levels is proposed. Firstly, perform preliminary fusion at the pixel level to enhance the sensitivity of the detection model to texture and morphological details. Then, conduct further fusion at the feature level to make the model focus more on global semantic information and capture target features in complex backgrounds. Finally, integrate and fuse the information at the decision level, selecting detection results with high reliability and accuracy. The model is compared and ablated experiments on a self-constructed target recognition dataset, with experimental results confirming its effectiveness.

Key words: multi-level fusion, joint recognition, synthetic aperture radar (SAR), visible light

CLC Number:

TP 391.4

Jia ZHAI, Qijia HUA, Ziquan WANG, Zikai ZHANG, Ruiqi WANG, Jinling LIU, Meiqi HU, Chenhui WU. Joint recognition of visible light and SAR images based on multi-level fusion[J]. Systems Engineering and Electronics, 2026, 48(3): 817-825.

Figures/Tables 11

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Table 1

Table 2

Fig.6

Table 3

Fig.7

Table 4

References 31

1	GHAMISI P, RASTI B, YOKOYA N, et al. Multisource and multitemporal data fusion in remote sensing: a comprehensive review of the state of the art[J]. IEEE Geoscience and Remote Sensing Magazine, 2019, 7 (1): 6- 39. doi: 10.1109/MGRS.2018.2890023
2	HAME T, SIRRO L, KILPI J, et al. A hierarchical clustering method for land cover change detection and identification[J]. Remote Sensing, 2020, 12 (11): 1751. doi: 10.3390/rs12111751
3	TSOKAS A, RYSZ M, PARDALOS P M, et al. SAR data applications in earth observation: an overview[J]. Expert Systems with Applications, 2022, 205, 117342. doi: 10.1016/j.eswa.2022.117342
4	ZHU X X, MONTAZERI S, ALI M, et al. Deep learning meets SAR: concepts, models, pitfalls, and perspectives[J]. IEEE Geoscience and Remote Sensing Magazine, 2021, 9 (4): 143- 172.
5	FREEMAN A. SAR calibration: an overview[J]. IEEE Trans. on Geoscience and Remote Sensing, 1992, 30 (6): 1107- 1121.
6	MIAO Z H, ZHU L J, ZHAO C C, et al. Image enhancement of low-light parking space based on Retinex[J]. Automotive Engineering, 2023, 45 (6): 989- 996.
7	AMARSAIKHAN D, BLOTEVOGEL H H, GENDEREN J L, et al. Fusing high-resolution SAR and optical imagery for improved urban land cover study and classification[J]. International Journal of Image and Data Fusion, 2010, 1 (1): 83- 97. doi: 10.1080/19479830903562041
8	GUO Q, LIU J N, KALIUZHNYI M. YOLOX-SAR: high-precision object detection system based on visible and infrared sensors for SAR remote sensing[J]. IEEE Sensors Journal, 2022, 22 (17): 17243- 17253. doi: 10.1109/JSEN.2022.3186889
9	BHAT S, KOUNDADL D. Multi-focus image fusion using neutrosophic based wavelet transform[J]. Applied Soft Computing, 2021, 106, 107307. doi: 10.1016/j.asoc.2021.107307
10	萧明伟, 李素敏, 李雁. 基于区域性IHS_NSST的国产高分合成孔径雷达和光学图像融合[J]. 激光与光电子学进展, 2020, 57(18): 181013.
	XIAO M W, LI S M, LI Y. Fusion of domestic high resolution synthetic aperture radar and optical images based on regional IHS_NSST[J]. Laser & Optoelectronics Progress, 2020, 57(18): 181013.
11	SHAH E, JAYAPRASAD P, JAMES M E. Image fusion of SAR and optical images for identifying antarctic ice features[J]. Journal of the Indian Society of Remote Sensing, 2019, 47, 2113- 2127. doi: 10.1007/s12524-019-01040-3
12	KARAKAYA D, ULUCAN O, TURKAN M. PAS-MEF: multi-exposure image fusion based on principal component analysis, adaptive well-exposedness and saliency map[C]//Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2022: 2345−2349.
13	CHEN Y X, BRUZZONE L. Self-supervised SAR-optical data fusion of sentinel-1/-2 images[J]. IEEE Trans. on Geoscience and Remote Sensing, 2022, 60, 5406011. doi: 10.1109/tgrs.2021.3128072
14	LI J, LI C C, XU W M, et al. Fusion of optical and SAR images based on deep learning to reconstruct vegetation NDVI time series in cloud-prone regions[J]. International Journal of Applied Earth Observation and Geoinformation, 2022, 112, 102818. doi: 10.1016/j.jag.2022.102818
15	ZHOU K X, ZHANG M, WANG H, et al. Ship detection in SAR images based on multi-scale feature extraction and adaptive feature fusion[J]. Remote Sensing, 2022, 14 (3): 755. doi: 10.3390/rs14030755
16	ZHANG H, XU H, TIAN X, et al. Image fusion meets deep learning: a survey and perspective[J]. Information Fusion, 2021, 76, 323- 336. doi: 10.1016/j.inffus.2021.06.008
17	WANG Z B, MA Y K, ZHANG Y N. Review of pixel-level remote sensing image fusion based on deep learning[J]. Information Fusion, 2023, 90, 36- 58. doi: 10.1016/j.inffus.2022.09.008
18	JIANG C C, REN H Z, YANG H, et al. M2FNet: multi-modal fusion network for object detection from visible and thermal infrared images[J]. International Journal of Applied Earth Observation and Geoinformation, 2024, 130, 103918. doi: 10.1016/j.jag.2024.103918
19	DENG B, LV H. Research on image fusion method of SAR and visible image based on CNN[C]//Proc. of the IEEE 4th International Conference on Civil Aviation Safety and Information Technology, 2022: 1400−1403.
20	WANG J P, SU N, ZHAO C H, et al. Multi-modal object detection method based on dual-branch asymmetric attention backbone and feature fusion pyramid network[J]. Remote Sensing, 2024, 16 (20): 3904. doi: 10.3390/rs16203904
21	SONG B Z, LIU P, LI J. MLFF-GAN: a multilevel feature fusion with GAN for spatiotemporal remote sensing images[J]. IEEE Trans. on Geoscience and Remote Sensing, 2022, 60, 4410816. doi: 10.1109/tgrs.2022.3169916
22	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770−778.
23	LIN T Y, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proc. of the IEEE Conference on Computer Vision And Pattern Recognition, 2017: 2117−2125.
24	REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal network[J]. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137- 1149. doi: 10.1109/TPAMI.2016.2577031
25	ZHANG J. RPN 2: on interdependence function learning towards unifying and advancing CNN, RNN, GNN, and transformer[EB/OL]. [2025-01-10]. https://arxiv.org/pdf/2411.11162.
26	GIRSHICK R. Fast R-CNN[C]//Proc. of the IEEE International Conference on Computer Vision, 2015.
27	HUANG M Y, XU Y, QIAN L X, et al. The QXS-SAROPT dataset for deep learning in SAR-optical data fusion[EB/OL]. [2025-01-10]. https://arxiv.org/abs/2103.08259.
28	郝永平, 曹昭睿, 白帆, 等. 基于兴趣区域掩码卷积神经网络的红外-可见光图像融合与目标识别算法研究[J]. 光子学报, 2021, 50 (2): 76- 90. doi: 10.3788/gzxb20215002.0210002
	HAO Y P, CAO Z R, BAI F, et al. Research on infrared visible image fusion and target recognition algorithm based on region of interest mask convolution neural network[J]. Acta Photonica Sinica, 2021, 50 (2): 76- 90. doi: 10.3788/gzxb20215002.0210002
29	邝楚文, 何望. 基于红外与可见光图像的目标检测算法[J]. 红外技术, 2022, 44 (9): 912- 919.
	KUANG C W, HE W. Object detection algorithm based on infrared and visible light images[J]. Infrared Technology, 2022, 44 (9): 912- 919.
30	赵松璞, 杨利萍, 赵昕, 等. 基于自适应注意力机制的红外与可见光图像目标检测[J]. 红外技术, 2024, 46 (4): 443- 451.
	ZHAO S P, YANG L P, ZHAO X, et al. Object detection in visible light and infrared images based on adaptive attention mechanism[J]. Infrared Technology, 2024, 46 (4): 443- 451.
31	何南, 石书桦. 融合红外特征的可见光图像目标检测算法研究[J]. 微电子学与计算机, 2023, 40 (3): 29- 36. doi: 10.19304/J.ISSN1000-7180.2022.0388
	HE N, SHI S H. Research on visible image object detection algorithm based on infrared features fusion[J]. Microelectronics & Computer, 2023, 40 (3): 29- 36. doi: 10.19304/J.ISSN1000-7180.2022.0388

类别	物体个数
船	672
车	988
飞机	468

类别	物体个数
岸边	328
陆地	556
海面	244

方法	车		船		飞机
方法	漏警率	精确率	漏警率	精确率	漏警率	精确率
文献[28]	12.4	95.7	12.2	96.3	14.3	96.6
文献[29]	13.1	96.1	12.7	97.0	13.4	97.7
文献[30]	12.1	95.4	14.4	96.2	12.8	97.1
文献[31]	12.6	96.4	13.1	96.9	12.5	97.4
本文方法	11.2	97.1	11.3	97.8	11.5	98.9

实验配置	最大值	最小值	平均值
去除像素级	0.910	0.845	0.887
去除特征级	0.915	0.856	0.892
去除决策级	0.928	0.831	0.905
完整模型	0.935	0.869	0.915

[1]	Xueying YANG, Gaopeng LI, Yun ZHANG. Semantic information-assisted 3D reconstruction method for satellite-borne multi-view SAR [J]. Systems Engineering and Electronics, 2026, 48(2): 430-446.
[2]	Mengli LIU, Gaofeng SHU, Ning LI. Deceptive jamming method against multi-channel SAR-GMTI via cooperative modulation of multiple jammers [J]. Systems Engineering and Electronics, 2026, 48(2): 490-502.
[3]	Jijun HU, Wei HAN, Long JIN, Xiwa ZHOU, Guoyu ZHANG, Qinyue ZHOU. Multi-station network one-bit jamming methods against radar seeker [J]. Systems Engineering and Electronics, 2026, 48(2): 503-514.
[4]	Mingyu JIANG, Shunsheng ZHANG, Siyao XIAO. SAR target recognition based on lightweight cross-attention convolutional neural network [J]. Systems Engineering and Electronics, 2025, 47(9): 2853-2861.
[5]	Xiang LI, Ding ZENG, Junjun YIN, Xianyu GUO, Jian YANG. Guided filtering for polarimetric SAR image based on gradient fusion [J]. Systems Engineering and Electronics, 2025, 47(9): 2890-2904.
[6]	Guohui ZHU, Lin YANG, Yang WANG, Taoye ZHENG. A modified PRF design method for high-speed maneuvering-platform SAR imaging [J]. Systems Engineering and Electronics, 2025, 47(9): 2905-2912.
[7]	Weihong FU, Wenhong PENG, Naian LIU. SAR image small target detection method with hybrid attention optimization [J]. Systems Engineering and Electronics, 2025, 47(8): 2519-2526.
[8]	Kang NI, Wenjie JIA, Minrui ZOU, Zhizhong ZHENG. SAR object detection based on dynamic aggregation network [J]. Systems Engineering and Electronics, 2025, 47(8): 2527-2539.
[9]	Xinzheng ZHANG, Mengke YAN, Xiaolin ZHU. Noise pseudo-label tolerant semi-supervised SAR target recognition [J]. Systems Engineering and Electronics, 2025, 47(6): 1796-1805.
[10]	Ning LIU, Fangfang LI, Xinwu LI, Wen HONG. 3D reconstruction of high-rise buildings from SAR combined with footprint and phase information [J]. Systems Engineering and Electronics, 2025, 47(5): 1469-1486.
[11]	Yujia JIA, Siqian ZHANG, Tao TANG, Gangyao KUANG. Blind super-resolution reconstruction of airborne SAR real-time transmission images with enhanced scattering features [J]. Systems Engineering and Electronics, 2025, 47(3): 753-767.
[12]	Zhiwei YANG, Andong YANG, Gengchen LIANG, Xianghai LI, Xiaorui LI, Jie LIU. Multi-subband fusion for radial velocity ambiguity resolution [J]. Systems Engineering and Electronics, 2025, 47(3): 788-796.
[13]	Shilong CHEN, Lin LIU, Xiaobei WANG, Hansen ZENG, Yabo LIU, Xiang LIU. Fast compensation frequency-domain imaging processing algorithm for inter-pules frequency agility SAR [J]. Systems Engineering and Electronics, 2025, 47(3): 797-806.
[14]	Leilei JIA, Limin LIU, Jian DONG. Fast registration of optical and SAR images based on image structural information [J]. Systems Engineering and Electronics, 2025, 47(2): 428-441.
[15]	Xianwu ZHENG, Hongyan ZHAO, Tian TIAN, Jinwen TIAN. Research on simulation and evaluation method of SAR typical active jamming [J]. Systems Engineering and Electronics, 2025, 47(12): 4068-4077.

Joint recognition of visible light and SAR images based on multi-level fusion

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 31

Related Articles 15

Recommended Articles

Metrics

Comments