基于低秩双线性池化注意力网络的舰船目标识别

doi:10.12305/j.issn.1001-506X.2023.05.06

摘要/Abstract

摘要：

针对多模态舰船图像融合识别质量不高等问题，构建了一种端到端的低秩双线性池化注意力网络。首先对各模态原始特征向量基于跨模类别中心进行注意力加权重构, 使不同模态特征更好地关注公共类别信息; 然后采用双线性池化捕获不同模态图像的交互信息, 并引入权重低秩分解降低网络参数规模; 最后依靠特征级联实现模态信息的交互与互补, 并设计联合损失提升网络跨模态融合识别效果。实验结果表明, 相比现有融合方法, 所提方法可有效提升多模态遥感舰船图像的融合识别效果, 在公开的遥感舰船数据集上取得了较高的识别准确率。

关键词: 舰船识别, 双线性池化, 跨模类别中心, 注意力加权, 跨模联合损失

Abstract:

In order to solve the problem of low quality of multimodal ship image fusion recognition, an end-to-end low rank bilinear pooling attention network is constructed. Firstly, the original feature vector of each modal are reconstructed based on the cross-modal category center, so that different modal features can pay more attention to the common category information. Then, bilinear pooling is used to capture the interactive information of different modal images, and the weight low rank decomposition is introduced to reduce the scale of network parameters. Finally, the interaction and complementarity of modal information are realized by feature cascaded, and the joint loss is designed to improve the effect of network cross-modal fusion recognition. The experimental results show that compared with the existing fusion methods, the proposed method can effectively improve the fusion recognition effect of multimodal remote sensing ship images, and achieve high recognition accuracy on the public remote sensing ship datasets.

Key words: ship recognition, bilinear pooling, cross-modal category center, attention weighting, cross-modal joint loss

中图分类号:

TP391.4

关欣, 国佳恩, 衣晓. 基于低秩双线性池化注意力网络的舰船目标识别[J]. 系统工程与电子技术, 2023, 45(5): 1305-1314.

Xin GUAN, Jiaen GUO, Xiao YI. Ship target recognition based on low rank bilinear pooling attention network[J]. Systems Engineering and Electronics, 2023, 45(5): 1305-1314.

图/表 14

图1

图2

图3

图4

表1

表2

表3

图5

图6

图7

图8

表4

表5

表6

参考文献 35

1	LAN J H , WAN L L . Automatic ship target classification based on aerial images[J]. Proceedings of SPIE-the International Society for Optical Engineering, 2009, 7156 (12): 715612.
2	URŠKA K , GREIDANUS H , KRIŠTOF O . Vessel detection and classification from spaceborne optical images: a literature survey[J]. Remote Sensing of Environment, 2018, 207 (15): 1- 26.
3	张岩, 李建增, 李德良, 等. 快速自适应鲁棒性尺度不变的特征检测子[J]. 系统工程与电子技术, 2017, 39 (6): 1406- 1413.
	ZHANG Y , LI J Z , LI D L . Fast adaptive robust invariant scalable feature detector[J]. Systems Engineering and Electronics, 2017, 39 (6): 1406- 1413.
4	苏亚, 高新波, 王博, 等. 基于Gabor相位和局部二值模式的AAM纹理表示[J]. 系统工程与电子技术, 2010, 32 (5): 1051- 1054. 1051-1054, 1104 doi: 10.3969/j.issn.1001-506X.2010.05.037
	SU Y , GAO X B , WANG B , et al. Gabor phase and LBP based texture representation in AAM[J]. Systems Engineering and Electronics, 2010, 32 (5): 1051- 1054. 1051-1054, 1104 doi: 10.3969/j.issn.1001-506X.2010.05.037
5	谌德荣, 王文斌, 刘丙太, 等. 旋转不变梯度直方图目标描述方法[J]. 电子与信息学报, 2016, 38 (1): 23- 28.
	CHEN D R , WANG W B , LIU B T , et al. Rotation-invariant histogram of oriented gradients for target description[J]. Jounal of Electronics & Information Technology, 2016, 38 (1): 23- 28.
6	苏娟, 杨龙, 黄华, 等. 用于SAR图像小目标舰船检测的改进SSD算法[J]. 系统工程与电子技术, 2020, 42 (5): 1026- 1034.
	SU J , YANG L , HUANG H , et al. Improved SSD algorithm for small-sized SAR ship dection[J]. Systems Engineering and Electronics, 2020, 42 (5): 1026- 1034.
7	罗畅, 王洁, 王世强, 等. 基于泛化深度迁移特征的高分遥感场景分类[J]. 系统工程与电子技术, 2018, 40 (3): 682- 691.
	LUO C , WANG J , WANG S Q , et al. General deep transfer features based high resolution remote scene classification[J]. Systems Engineering and Electronics, 2018, 40 (3): 682- 691.
8	UCAR F , KORKMAZ D . A novel ship classification network with cascade deep features for line-of-sight sea data[J]. Machine Vision and Applications, 2021, 32 (3): 1- 15.
9	LIU K , YU S T , LIU S D . An improved inceptionV3 network for obscured ship classification in remote sensing images[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13, 4738- 4747. doi: 10.1109/JSTARS.2020.3017676
10	卢旺, 张雅声, 徐灿, 等. 基于双谱-谱图特征和深度卷积神经网络的HRRP目标识别方法[J]. 系统工程与电子技术, 2020, 42 (8): 1703- 1709.
	LU W , ZHANG Y S , XU C , et al. HRRP target recognition method based on bispectrum-spectrogram feature and deep convolutional neural network[J]. Systems Engineering and Electronics, 2020, 42 (8): 1703- 1709.
11	XU Y J , LANG H T , NIU L H , et al. Discriminative adaptation regularization framework-based transfer learning for ship classification in SAR images[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 16 (11): 1786- 1790. doi: 10.1109/LGRS.2019.2907139
12	SHI J W , JIANG Z G , ZHANG H P . Few-shot ship classification in optical remote sensing images using nearest neighbor prototype representation[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14, 3581- 3590. doi: 10.1109/JSTARS.2021.3066539
13	ZHU J W , QIU X L , PAN Z X , et al. An improved shape contexts based ship classification in SAR images[J]. Remote Sensing, 2017, 9 (2): 145. doi: 10.3390/rs9020145
14	HUANG L H , LI W , CHEN C , et al. Multiple features learning for ship classification in optical imagery[J]. Multimedia Tools & Applications, 2018, 77, 13363- 13389.
15	HUANG L H , CHEN C , LI W , et al. Remote sensing image scene classification using multi-scale completed local binary patterns and Fisher vectors[J]. Remote Sensing, 2016, 8 (6): 483.
16	ZHANG X H , LV Y F , YAO L B , et al. A new benchmark and an attribute-guided multi-level feature representation network for fine-grained ship classification in optical remote sensing images[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13, 1258- 1271.
17	JEON H K , YANG C S . Enhancement of ship type classification from a combination of CNN and KNN[J]. Electronics, 2021, 10 (10): 1169.
18	HUANG S Z , XU H S , XIA X Z , et al. Multi-feature fusion of convolutional neural networks for fine-grained ship classification[J]. Journal of Intelligent & Fuzzy Systems, 2019, 37 (1): 125- 135.
19	CHEN X Q , YANG Y S , WANG S Z , et al. Ship type recognition via a coarse-to-fine cascaded convolution neural network[J]. Journal of Navigation, 2020, 73 (4): 813- 832.
20	ZHANG T W , ZHANG X L . Injection of traditional hand-crafted features into modern CNN-based models for SAR ship classification: what, why, where, and how[J]. Remote Sensing, 2021, 13 (11): 2091.
21	SHI Q Q , LI W , ZHANG F , et al. Deep CNN with multi-scale rotation invariance features for ship classification[J]. IEEE Access, 2018, 6, 38656- 38668.
22	REN Y M , YANG J , ZHANG Q N , et al. Multi-feature fusion with convolutional neural network for ship classification in optical images[J]. Applied Sciences, 2019, 9 (20): 4209.
23	SHI Q Q , LI W , TAO R , et al. Ship classification based on multifeature ensemble with convolutional neural network[J]. Remote Sensing, 2019, 11 (4): 419.
24	REN Y M , YANG J , ZHANG Q N , et al. Ship recognition based on Hu invariant moments and convolutional neural network for video surveillance[J]. Multimedia Tools and Applications, 2021, 80, 1343- 1373.
25	邱晓华, 李敏, 邓光芒, 等. 多层卷积特征融合的双波段决策级船舶识别[J]. 光学精密工程, 2021, 29 (1): 183- 190.
	QIU X H , LI M , DENG G M , et al. Multi-layer convolutional features fusion for dual-band decision-level ship recognition[J]. Optics and Precision Engineering, 2021, 29 (1): 183- 190.
26	SANTOS C E, BHANU B. Dyfusion: dynamic IR/RGB fusion for maritime vessel recognition[C]//Proc. of the International Conference on Image Processing, 2018: 1328-1332.
27	GAETANO R, IENCO D, OSE K, et al. MRFusion: a deep learning architecture to fuse PAN and MS imagery for land cover mapping[EB/OL]. [2022-01-05]. https://arxiv.org/abs.1806.11452v1.
28	LIN T Y, ROYCHOWDHURY A, MAJI S. Bilinear CNN models for fine-grained visual recognization[C]//Proc. of the IEEE International Conference on Computer Vision, 2015.
29	ZADEH A, CHEN M, PORIA S, et al. Tensor fusion network for multimodal sentiment analysis[EB/OL]. [2022-01-05]. https://arxiv.org/abs/1707.07250.
30	LIU Z, SHEN Y, LAKSHMINARASIMHAN V B, et al. Efficient low-rank multimodal fusion with modality-specific factors[EB/OL]. [2022-01-05]. https://arxiv.org/abs/1806.00064.
31	JING L L, VAHDANI E, TAN J X, et al. Cross-modal center loss for 3D cross-modal retrieval[EB/OL]. [2022-01-05]. https://arxiv.org/abs/2008.03561.
32	LI M Y , SUN W W , DU X , et al. Ship classification by the fusion of panchromatic image and multi-spectral image based on pseudo siamese light weight network[J]. Journal of Physics Conference Series, 2021, 1757 (1): 12- 22.
33	ZHANG M M, CHOI J, DANⅡLIDIS K, et al. VAIS: a dataset for recognizing maritime imagery in the visible and infrared spectrums[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2015: 10-16.
34	HUANG G , LIU Z , PLEISS G , et al. Convolutional networks with dense connectivity[J]. IEEE Trans.on Pattern Analysis and Machine Intelligence, 2022, 44 (12): 8704- 8716.
35	ZHANG E H , WANG K K , LIN G F . Classification of marine vessels with multi-feature structure fusion[J]. Applied Sciences, 2019, 9 (10): 2153.

数据集	类别	数量
MPSC	驱逐舰	99(86/13)
	护卫舰	179(143/36)
	战斗船	261(213/48)
	散货船	919(745/174)
	集装箱船	387(314/73)
	邮轮	787(631/156)
VAIS	商船	146(83/63)
	其他中型船	138(62/76)
	中型客船	117(58/59)
	帆船	284(148/136)
	小型船	353(158/195)
	拖船	50(30/20)

模态	算法
模态	MRF^[27]	MPF^[32]	Siamese VGG16	Two-branch CNN^[34]	TFN^[29]	LMF^[30]	LRBPAN
融合分支	68.32	75.59	69.15	69.9	83.2±0.3	84.1±0.7	86.7±0.4
PAN分支	-	71.77	65.52	67.3	79.8±0.2	78.9±0.8	80.4±0.2
MS分支	-	66.88	62.94	61.2	71.2±0.3	71.1±0.3	70.7±0.7

模态	算法
模态	MCFF^[25]	Dyfusion^[26]	CNN+Gnostic Field^[33]	SRDA^[35]	TFN^[29]	LMF^[30]	LRBPAN
融合分支	89.7±0.3	88.2±0.2	87.4	88.0	90.7±0.4	90.4±0.9	92.6±0.4
VIS分支	88.8±0.4	-	81.0	87.6	89.1±0.4	88.8±0.4	89.4±0.4
IR分支	70.3±0.5	-	56.8	74.7	73.5±0.6	73.2±0.2	72.9±0.7

β	γ
β	1.0	0.5	0.2	0.1	0.01
0.1	0.706	0.686	0.684	0.696	0.700
0.01	0.798	0.774	0.796	0.788	0.786
0.001	0.802	0.812	0.822	0.812	0.838
0.000 1	0.834	0.826	0.828	0.867	0.828
0.000 01	0.824	0.820	0.824	0.830	0.838

参数	方法
参数	〇	①	②	③	①②	①③	②③	①②③
识别准确率(融合分支)	80.7±0.1	84.7±0.2	83.6±0.6	81.9±0.8	86.1±0.4	85.5±0.7	85.0±0.7	86.7±0.4