基于深度特征融合的SAR图像与AIS信息关联方法

doi:10.12305/j.issn.1001-506X.2023.11.15

摘要/Abstract

摘要：

星载合成孔径雷达(synthetic aperture radar, SAR)和自动识别系统(automatic identification system, AIS)都可以获取到探测目标的相关信息, 将两者获取的信息进行关联融合, 有益于实现高效的海上侦察监视。由于数据之间存在的异构性, 传统方法多依赖人工特征建立SAR图像与AIS信息的关联关系, 但这些方法存在精度差、效率低等缺点。本文提出了一种基于深度特征融合的SAR图像与AIS信息关联方法, 针对两种模态数据的特点分别设计了对应的特征学习网络获取单模态特征表示, 进一步融合不同模态的特征信息以增强跨模态信息间的语义相关性, 然后通过设计的关联学习目标函数进行跨模态特征之间关联学习。在构建的数据集上验证表明, 所提方法关联精度高、适应性强, 验证了所提数据集和方法的有效性。

关键词: 多源数据关联, 深度学习, 遥感图像, 跨模态检索

Abstract:

Spaceborne synthetic aperture radar (SAR) and automatic identification system (AIS) can obtain the information of the detected target. And the association and fusion of the information obtained from the two sensors is beneficial to realize efficient maritime reconnaissance and surveillance. Due to the heterogeneity gas between different data, traditional methods mostly rely on artificial features to establish correlation between SAR images and AIS information, but these methods have disadvantages of poor accuracy and low efficiency. In this paper, the association method between SAR images and AIS information based on depth feature fusion is proposed. According to the characteristics of the two modal data, the corresponding feature learning network is designed respectively to obtain the unimodal feature representation, and the feature information of different modals is further fused to enhance the semantic correlation of cross-modal information. Then, the association learning between cross-modal features is carried out by the designed association learning objective function. Verification on the constructed dataset shows that the proposed method can achieve high correlation accuracy and strong adaptability, and the effectiveness of the proposed dataset and method is verified.

Key words: multi-source data association, deep learning, remote sensing image, cross-modal retrieval

中图分类号:

TP751.1

李浩然, 熊伟, 崔亚奇. 基于深度特征融合的SAR图像与AIS信息关联方法[J]. 系统工程与电子技术, 2023, 45(11): 3491-3497.

Haoran LI, Wei XIONG, Yaqi CUI. An association method between SAR images and AIS information based on depth feature fusion[J]. Systems Engineering and Electronics, 2023, 45(11): 3491-3497.

图/表 11

图1

图2

图3

表1

图4

表2

图5

表3

图6

图7

表4

参考文献 31

1	SOLDI G , GAGLIONE D , FORTI N , et al. Space-based global maritime surveillance. Part Ⅱ: artificial intelligence and data fusion techniques[J]. IEEE Aerospace and Electronic Systems Magazine, 2021, 36 (9): 30- 42. doi: 10.1109/MAES.2021.3070884
2	LEHNER S, BRUSCH S, FRITZ T. Ship surveillance by joint use of SAR and AIS[C]//Proc. of the IEEE Oceans Europe, 2009: 1-5.
3	BRUSCH S , LEHNER S , FRITZ T , et al. Ship surveillance with TerraSAR-X[J]. IEEE Trans.on Geoscience and Remote Sensing, 2010, 49 (3): 1092- 1103.
4	SANTAMARIA C , ALVAREZ M , GREIDANUS H , et al. Mass processing of Sentinel-1 images for maritime surveillance[J]. Remote Sensing, 2017, 9 (7): 678. doi: 10.3390/rs9070678
5	PELICH R , CHINI M , HOSTACHE R , et al. Large-scale automatic vessel monitoring based on dual-polarization Sentinel-1 and AIS data[J]. Remote Sensing, 2019, 11 (9): 1078. doi: 10.3390/rs11091078
6	UIBOUPIN R, RAUDSEPP U, SIPELGAS L. Detection of oil spills on SAR images, identification of polluters and forecast of the slicks trajectory[C]//Proc. of the IEEE/OES US/EU-Baltic International Symposium, 2008: 1-5.
7	LONGÉPÉ N , MOUCHE A A , GOACOLOU M , et al. Polluter identification with spaceborne radar imagery, AIS and forward drift modeling[J]. Marine Pollution Bulletin, 2015, 101 (2): 826- 833. doi: 10.1016/j.marpolbul.2015.08.006
8	GARELLO R, KERBAOL V. Oil pollution monitoring: An integrated approach[C]//Proc. of the IEEE Workshop on Environmental, Energy, and Structural Monitoring Systems, 2017: 1-6.
9	SONG J, KIM D. Identification of unclassified ships implementing AIS information and SAR image-based ship detection results[C]//Proc. of the IEEE International Geoscience and Remote Sensing Symposium, 2021: 3557-3560.
10	TOUZI R , VACHON P W . RCM polarimetric SAR for enhanced ship detection and classification[J]. Canadian Journal of Remote Sensing, 2015, 41 (5): 473- 484. doi: 10.1080/07038992.2015.1110010
11	VELOTTO D , BENTES C , TINGS B , et al. First comparison of Sentinel-1 and TerraSAR-X data in the framework of maritime targets detection: South Italy case[J]. IEEE Journal of Oceanic Engineering, 2016, 41 (4): 993- 1006. doi: 10.1109/JOE.2016.2520216
12	KUREKIN A A , LOVEDAY B R , CLEMENTS O , et al. Operational monitoring of illegal fishing in Ghana through exploitation of satellite earth observation and AIS data[J]. Remote Sensing, 2019, 11 (3): 293. doi: 10.3390/rs11030293
13	LONGÉPÉ N , HAJDUCH G , ARDIANTO R , et al. Completing fishing monitoring with spaceborne vessel detection system (VDS) and automatic identification system (AIS) to assess illegal fishing in Indonesia[J]. Marine Pollution Bulletin, 2018, 131, 33- 39. doi: 10.1016/j.marpolbul.2017.10.016
14	ROWLANDS G , BROWN J , SOULE B , et al. Satellite surveillance of fishing vessel activity in the ascension island exclusive economic zone and marine protected area[J]. Marine Policy, 2019, 101, 39- 50. doi: 10.1016/j.marpol.2018.11.006
15	International Telecommunication Union. Technical characteristics for an automatic identification system using time division multiple access in the VHF maritime mobile frequency band[EB/OL]. [2022-03-18]. https://www.itu.int/dms_pubrec/itu-r/rec/m/R-REC-M.1371-5-201402-I!!PDF-E.pdf
16	BAR-SHALOM Y , FORTMANN T E , CABLE P G . Tracking and data association[J]. Acoustical Society of America Journal, 1990, 87 (2): 918- 919. doi: 10.1121/1.398863
17	ACHIRI L, GUIDA R, IERVOLINO P. SAR and AIS fusion for maritime surveillance[C]//Proc. of the IEEE 4th International Forum on Research and Technology for Society and Industry (RTSI), 2018: 1-4.
18	LANG H T , WU S W , XU Y J . Ship classification in SAR images improved by AIS knowledge transfer[J]. IEEE Geoscience and Remote Sensing Letters, 2018, 15 (3): 439- 443. doi: 10.1109/LGRS.2018.2792683
19	GRAZIANO M D , RENGA A , MOCCIA A . Integration of automatic identification system (AIS) data and single-channel synthetic aperture radar (SAR) images by SAR-based ship velocity estimation for maritime situational awareness[J]. Remote Sensing, 2019, 11 (19): 2196. doi: 10.3390/rs11192196
20	RODGER M , GUIDA R . Classification-aided SAR and AIS data fusion for space-based maritime surveillance[J]. Remote Sensing, 2020, 13 (1): 104. doi: 10.3390/rs13010104
21	WANG L W , LI Y , HUANG J , et al. Learning two-branch neural networks for image-text matching tasks[J]. IEEE Trans.on Pattern Analysis and Machine Intelligence, 2018, 41 (2): 394- 407.
22	YAO S S, NIU B N, LIU J Q. Enhancing sampling and counting method for audio retrieval with time-stretch resistance[C]// Proc. of the IEEE Fourth International Conference on Multimedia Big Data (BigMM), 2018: 1-5.
23	TIAN S , YIN X C , SU Y , et al. A unified framework for tracking based text detection and recognition from web videos[J]. IEEE Trans.on Pattern Analysis and Machine Intelligence, 2017, 40 (3): 542- 554.
24	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
25	DENG Z, HU X, ZHU L, et al. R3net: recurrent residual refinement network for saliency detection[C]//Proc. of the 27th International Joint Conference on Artificial Intelligence, 2018: 684-690.
26	XU K, BA J, KIROS R, et al. Show, attend and tell: neural image caption generation with visual attention[C]//Proc. of the International Conference on Machine Learning, 2015: 2048-2057.
27	FAGHRI F, FLEET D J, KIROS J R, et al. Vse++: improving visual-semantic embeddings with hard negatives[J]. arXiv preprint arXiv: 1707. 05612, 2017.
28	ZHANG T W , ZHANG X L , LI J W , et al. SAR ship detection dataset (SSDD): official release and comprehensive data analysis[J]. Remote Sensing, 2021, 13 (18): 3690. doi: 10.3390/rs13183690
29	WANG K Y, YIN Q Y, WANG W, et al. A comprehensive survey on cross-modal retrieval[J]. arXiv preprint arXiv: 1607. 06215, 2016.
30	LEE K H, CHEN X, HUA G, et al. Stacked cross attention for image-text matching[C]//Proc. of the European Conference on Computer Vision (ECCV), 2018: 201-216.
31	YUAN Z Q , ZHANG W K , FU K , et al. Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval[J]. IEEE Trans.on Geoscience and Remote Sensing, 2021, 60, 1- 19.

网络层	输入尺寸	输出尺寸	网络参数
Layer1	256×256	128×128	7×7, 64, 2
Layer2	128×128	64×64	3×3 maxpool, 2 [3×3, 64 3×3, 64]×2
Layer3	64×64	32×32	[3×3, 128 3×3, 128]×2
Layer4	32×32	16×16	[3×3, 256 3×3, 256]×2
Layer5	16×16	8×8	[3×3, 512 3×3, 512]×2

网络模型	输入维度	输出维度	网络参数
GRU	300	512	6层
1D-CNN	300	512	2层

任务	模型	召回率(Recall)
任务	模型	R@1	R@5	R@10	R_m
(SAR-＞AIS)	A-MFMN	50.77	56.92	77.23	68.82 (AMFMN)
	SCAN	55.38	61.54	70.77	68.82 (AMFMN)
	本文	68.31	74.15	84.92	70.00 (SCAN)
(AIS-＞SAR)	AMFMN	53.54	85.54	89.54	70.00 (SCAN)
	SCAN	58.46	84.62	89.23	79.44 (本文)
	本文	67.69	89.85	91.69	79.44 (本文)

模型	(SAR-＞AIS)				(AIS-＞SAR)		R_m
模型	R@1	R@5	R@10	R@1	R@5	R@10	R_m
M1	32.92	36.92	53.54	32.00	77.85	86.46	53.28
M2	51.38	58.75	73.23	54.46	85.23	89.54	68.67
M3	57.85	65.23	79.08	60.62	89.85	92.62	74.21
M4	62.46	70.46	81.85	67.08	87.69	91.08	76.77
M5	56.62	63.69	79.38	60.92	85.54	89.23	72.56
M6	60.13	65.54	81.54	63.08	88.92	90.77	75.03
M7	64.62	71.38	83.38	66.46	87.69	90.15	77.28
M8	67.08	72.92	83.08	66.77	89.23	90.77	78.31
M	68.31	74.15	84.92	67.69	89.85	91.69	79.44

[1]	汪萌, 诸兵. 不确定性建模在2D和3D目标检测中的应用[J]. 系统工程与电子技术, 2023, 45(8): 2370-2376.
[2]	邵凯, 杜自群, 王光宇. 基于模型剪枝动态调整压缩率的CSI反馈方法[J]. 系统工程与电子技术, 2023, 45(8): 2615-2622.
[3]	崔天舒, 王栋, 黄振. 面向空间认知通信的轻量化网络自动调制分类方法[J]. 系统工程与电子技术, 2023, 45(7): 2220-2226.
[4]	姜雨, 袁琪, 胡志韬, 吴薇薇, 顾欣. 基于气象因素的机场进离港延误预测[J]. 系统工程与电子技术, 2023, 45(6): 1722-1731.
[5]	成倩, 李佳, 杜娟. 基于YOLOv5的光学遥感图像舰船目标检测算法[J]. 系统工程与电子技术, 2023, 45(5): 1270-1276.
[6]	陈洋, 廖灿辉, 张锟, 刘建, 王鹏举. 基于自监督对比学习的信号调制识别算法[J]. 系统工程与电子技术, 2023, 45(4): 1200-1206.
[7]	张晔, 侯毅, 欧阳克威, 周石琳. 单变量序列数据分类方法综述[J]. 系统工程与电子技术, 2023, 45(2): 313-335.
[8]	邵正途, 许登荣, 徐文利, 王晗中. 基于LSTM和残差网络的雷达有源干扰识别[J]. 系统工程与电子技术, 2023, 45(2): 416-423.
[9]	陈任飞, 彭勇, 李忠文. 基于持续无监督域适应策略的水面漂浮物目标检测方法[J]. 系统工程与电子技术, 2023, 45(11): 3391-3401.
[10]	陶臣嵩, 陈思伟, 肖顺平. 基于深度学习模型的SAR图像间歇采样转发干扰检测[J]. 系统工程与电子技术, 2023, 45(11): 3465-3473.
[11]	何诗扬, 汪玲, 朱岱寅, 钱君. 基于时空记忆解耦RNN的雷暴预测方法[J]. 系统工程与电子技术, 2023, 45(11): 3474-3480.
[12]	刘子超, 王江, 何绍溟. 基于深度学习的时间角度控制制导律[J]. 系统工程与电子技术, 2023, 45(11): 3579-3587.
[13]	季策, 宋博翰, 耿蓉, 梁敏骏. 快时变信道下基于深度学习的OFDM系统信道估计[J]. 系统工程与电子技术, 2023, 45(11): 3649-3655.
[14]	丁建立, 张琪琪, 王静, 霍纬纲. 基于Transformer-VAE的ADS-B异常检测方法[J]. 系统工程与电子技术, 2023, 45(11): 3680-3689.
[15]	宋婷, 兀泽朝, 高艾, 袁建平. 基于CycleGAN的月表图像数据增强方法[J]. 系统工程与电子技术, 2023, 45(10): 3041-3048.