基于低秩双线性池化注意力网络的舰船目标识别

doi:10.12305/j.issn.1001-506X.2023.05.06

Abstract

Abstract:

In order to solve the problem of low quality of multimodal ship image fusion recognition, an end-to-end low rank bilinear pooling attention network is constructed. Firstly, the original feature vector of each modal are reconstructed based on the cross-modal category center, so that different modal features can pay more attention to the common category information. Then, bilinear pooling is used to capture the interactive information of different modal images, and the weight low rank decomposition is introduced to reduce the scale of network parameters. Finally, the interaction and complementarity of modal information are realized by feature cascaded, and the joint loss is designed to improve the effect of network cross-modal fusion recognition. The experimental results show that compared with the existing fusion methods, the proposed method can effectively improve the fusion recognition effect of multimodal remote sensing ship images, and achieve high recognition accuracy on the public remote sensing ship datasets.

Key words: ship recognition, bilinear pooling, cross-modal category center, attention weighting, cross-modal joint loss

CLC Number:

TP391.4

Xin GUAN, Jiaen GUO, Xiao YI. Ship target recognition based on low rank bilinear pooling attention network[J]. Systems Engineering and Electronics, 2023, 45(5): 1305-1314.

Figures/Tables 14

Fig.1

Fig.2

Fig.3

Fig.4

Table 1

Table 2

Table 3

Fig.5

Fig.6

Fig.7

Fig.8

Table 4

Table 5

Table 6

References 35

1	LAN J H , WAN L L . Automatic ship target classification based on aerial images[J]. Proceedings of SPIE-the International Society for Optical Engineering, 2009, 7156 (12): 715612.
2	URŠKA K , GREIDANUS H , KRIŠTOF O . Vessel detection and classification from spaceborne optical images: a literature survey[J]. Remote Sensing of Environment, 2018, 207 (15): 1- 26.
3	张岩, 李建增, 李德良, 等. 快速自适应鲁棒性尺度不变的特征检测子[J]. 系统工程与电子技术, 2017, 39 (6): 1406- 1413.
	ZHANG Y , LI J Z , LI D L . Fast adaptive robust invariant scalable feature detector[J]. Systems Engineering and Electronics, 2017, 39 (6): 1406- 1413.
4	苏亚, 高新波, 王博, 等. 基于Gabor相位和局部二值模式的AAM纹理表示[J]. 系统工程与电子技术, 2010, 32 (5): 1051- 1054. 1051-1054, 1104 doi: 10.3969/j.issn.1001-506X.2010.05.037
	SU Y , GAO X B , WANG B , et al. Gabor phase and LBP based texture representation in AAM[J]. Systems Engineering and Electronics, 2010, 32 (5): 1051- 1054. 1051-1054, 1104 doi: 10.3969/j.issn.1001-506X.2010.05.037
5	谌德荣, 王文斌, 刘丙太, 等. 旋转不变梯度直方图目标描述方法[J]. 电子与信息学报, 2016, 38 (1): 23- 28.
	CHEN D R , WANG W B , LIU B T , et al. Rotation-invariant histogram of oriented gradients for target description[J]. Jounal of Electronics & Information Technology, 2016, 38 (1): 23- 28.
6	苏娟, 杨龙, 黄华, 等. 用于SAR图像小目标舰船检测的改进SSD算法[J]. 系统工程与电子技术, 2020, 42 (5): 1026- 1034.
	SU J , YANG L , HUANG H , et al. Improved SSD algorithm for small-sized SAR ship dection[J]. Systems Engineering and Electronics, 2020, 42 (5): 1026- 1034.
7	罗畅, 王洁, 王世强, 等. 基于泛化深度迁移特征的高分遥感场景分类[J]. 系统工程与电子技术, 2018, 40 (3): 682- 691.
	LUO C , WANG J , WANG S Q , et al. General deep transfer features based high resolution remote scene classification[J]. Systems Engineering and Electronics, 2018, 40 (3): 682- 691.
8	UCAR F , KORKMAZ D . A novel ship classification network with cascade deep features for line-of-sight sea data[J]. Machine Vision and Applications, 2021, 32 (3): 1- 15.
9	LIU K , YU S T , LIU S D . An improved inceptionV3 network for obscured ship classification in remote sensing images[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13, 4738- 4747. doi: 10.1109/JSTARS.2020.3017676
10	卢旺, 张雅声, 徐灿, 等. 基于双谱-谱图特征和深度卷积神经网络的HRRP目标识别方法[J]. 系统工程与电子技术, 2020, 42 (8): 1703- 1709.
	LU W , ZHANG Y S , XU C , et al. HRRP target recognition method based on bispectrum-spectrogram feature and deep convolutional neural network[J]. Systems Engineering and Electronics, 2020, 42 (8): 1703- 1709.
11	XU Y J , LANG H T , NIU L H , et al. Discriminative adaptation regularization framework-based transfer learning for ship classification in SAR images[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 16 (11): 1786- 1790. doi: 10.1109/LGRS.2019.2907139
12	SHI J W , JIANG Z G , ZHANG H P . Few-shot ship classification in optical remote sensing images using nearest neighbor prototype representation[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14, 3581- 3590. doi: 10.1109/JSTARS.2021.3066539
13	ZHU J W , QIU X L , PAN Z X , et al. An improved shape contexts based ship classification in SAR images[J]. Remote Sensing, 2017, 9 (2): 145. doi: 10.3390/rs9020145
14	HUANG L H , LI W , CHEN C , et al. Multiple features learning for ship classification in optical imagery[J]. Multimedia Tools & Applications, 2018, 77, 13363- 13389.
15	HUANG L H , CHEN C , LI W , et al. Remote sensing image scene classification using multi-scale completed local binary patterns and Fisher vectors[J]. Remote Sensing, 2016, 8 (6): 483.
16	ZHANG X H , LV Y F , YAO L B , et al. A new benchmark and an attribute-guided multi-level feature representation network for fine-grained ship classification in optical remote sensing images[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13, 1258- 1271.
17	JEON H K , YANG C S . Enhancement of ship type classification from a combination of CNN and KNN[J]. Electronics, 2021, 10 (10): 1169.
18	HUANG S Z , XU H S , XIA X Z , et al. Multi-feature fusion of convolutional neural networks for fine-grained ship classification[J]. Journal of Intelligent & Fuzzy Systems, 2019, 37 (1): 125- 135.
19	CHEN X Q , YANG Y S , WANG S Z , et al. Ship type recognition via a coarse-to-fine cascaded convolution neural network[J]. Journal of Navigation, 2020, 73 (4): 813- 832.
20	ZHANG T W , ZHANG X L . Injection of traditional hand-crafted features into modern CNN-based models for SAR ship classification: what, why, where, and how[J]. Remote Sensing, 2021, 13 (11): 2091.
21	SHI Q Q , LI W , ZHANG F , et al. Deep CNN with multi-scale rotation invariance features for ship classification[J]. IEEE Access, 2018, 6, 38656- 38668.
22	REN Y M , YANG J , ZHANG Q N , et al. Multi-feature fusion with convolutional neural network for ship classification in optical images[J]. Applied Sciences, 2019, 9 (20): 4209.
23	SHI Q Q , LI W , TAO R , et al. Ship classification based on multifeature ensemble with convolutional neural network[J]. Remote Sensing, 2019, 11 (4): 419.
24	REN Y M , YANG J , ZHANG Q N , et al. Ship recognition based on Hu invariant moments and convolutional neural network for video surveillance[J]. Multimedia Tools and Applications, 2021, 80, 1343- 1373.
25	邱晓华, 李敏, 邓光芒, 等. 多层卷积特征融合的双波段决策级船舶识别[J]. 光学精密工程, 2021, 29 (1): 183- 190.
	QIU X H , LI M , DENG G M , et al. Multi-layer convolutional features fusion for dual-band decision-level ship recognition[J]. Optics and Precision Engineering, 2021, 29 (1): 183- 190.
26	SANTOS C E, BHANU B. Dyfusion: dynamic IR/RGB fusion for maritime vessel recognition[C]//Proc. of the International Conference on Image Processing, 2018: 1328-1332.
27	GAETANO R, IENCO D, OSE K, et al. MRFusion: a deep learning architecture to fuse PAN and MS imagery for land cover mapping[EB/OL]. [2022-01-05]. https://arxiv.org/abs.1806.11452v1.
28	LIN T Y, ROYCHOWDHURY A, MAJI S. Bilinear CNN models for fine-grained visual recognization[C]//Proc. of the IEEE International Conference on Computer Vision, 2015.
29	ZADEH A, CHEN M, PORIA S, et al. Tensor fusion network for multimodal sentiment analysis[EB/OL]. [2022-01-05]. https://arxiv.org/abs/1707.07250.
30	LIU Z, SHEN Y, LAKSHMINARASIMHAN V B, et al. Efficient low-rank multimodal fusion with modality-specific factors[EB/OL]. [2022-01-05]. https://arxiv.org/abs/1806.00064.
31	JING L L, VAHDANI E, TAN J X, et al. Cross-modal center loss for 3D cross-modal retrieval[EB/OL]. [2022-01-05]. https://arxiv.org/abs/2008.03561.
32	LI M Y , SUN W W , DU X , et al. Ship classification by the fusion of panchromatic image and multi-spectral image based on pseudo siamese light weight network[J]. Journal of Physics Conference Series, 2021, 1757 (1): 12- 22.
33	ZHANG M M, CHOI J, DANⅡLIDIS K, et al. VAIS: a dataset for recognizing maritime imagery in the visible and infrared spectrums[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2015: 10-16.
34	HUANG G , LIU Z , PLEISS G , et al. Convolutional networks with dense connectivity[J]. IEEE Trans.on Pattern Analysis and Machine Intelligence, 2022, 44 (12): 8704- 8716.
35	ZHANG E H , WANG K K , LIN G F . Classification of marine vessels with multi-feature structure fusion[J]. Applied Sciences, 2019, 9 (10): 2153.

数据集	类别	数量
MPSC	驱逐舰	99(86/13)
	护卫舰	179(143/36)
	战斗船	261(213/48)
	散货船	919(745/174)
	集装箱船	387(314/73)
	邮轮	787(631/156)
VAIS	商船	146(83/63)
	其他中型船	138(62/76)
	中型客船	117(58/59)
	帆船	284(148/136)
	小型船	353(158/195)
	拖船	50(30/20)

模态	算法
模态	MRF^[27]	MPF^[32]	Siamese VGG16	Two-branch CNN^[34]	TFN^[29]	LMF^[30]	LRBPAN
融合分支	68.32	75.59	69.15	69.9	83.2±0.3	84.1±0.7	86.7±0.4
PAN分支	-	71.77	65.52	67.3	79.8±0.2	78.9±0.8	80.4±0.2
MS分支	-	66.88	62.94	61.2	71.2±0.3	71.1±0.3	70.7±0.7

模态	算法
模态	MCFF^[25]	Dyfusion^[26]	CNN+Gnostic Field^[33]	SRDA^[35]	TFN^[29]	LMF^[30]	LRBPAN
融合分支	89.7±0.3	88.2±0.2	87.4	88.0	90.7±0.4	90.4±0.9	92.6±0.4
VIS分支	88.8±0.4	-	81.0	87.6	89.1±0.4	88.8±0.4	89.4±0.4
IR分支	70.3±0.5	-	56.8	74.7	73.5±0.6	73.2±0.2	72.9±0.7

β	γ
β	1.0	0.5	0.2	0.1	0.01
0.1	0.706	0.686	0.684	0.696	0.700
0.01	0.798	0.774	0.796	0.788	0.786
0.001	0.802	0.812	0.822	0.812	0.838
0.000 1	0.834	0.826	0.828	0.867	0.828
0.000 01	0.824	0.820	0.824	0.830	0.838

参数	方法
参数	〇	①	②	③	①②	①③	②③	①②③
识别准确率(融合分支)	80.7±0.1	84.7±0.2	83.6±0.6	81.9±0.8	86.1±0.4	85.5±0.7	85.0±0.7	86.7±0.4

Ship target recognition based on low rank bilinear pooling attention network

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 14

References 35

Related Articles 15

Recommended Articles

Metrics

Comments

方法	单轮训练时间/s	单图训练时间/ms	单图测试时间/ms	FLOPs(×10²¹)	网络层数	参数量(×10⁶)
MCFF^[25]	-	-	-	7.92	80	276.9
MPF^[32]	-	-	-	0.24	25	6.49
TFN^[29]	22.72	10.66	10.64	2.17	258	56.5
LMF^[30]	6.77	3.48	1.97	2.16	278	48.1
LRBPAN	7.53	3.59	2.04	2.16	292	48.2

[1]	Qian CHENG, Jia LI, Juan DU. Ship target detection algorithm of optical remote sensing image based on YOLOv5 [J]. Systems Engineering and Electronics, 2023, 45(5): 1270-1276.
[2]	Yifei XU, Xiaodong LI, Xinde LI. A method of camouflaged object segmentation with locating and asymmetric compensation [J]. Systems Engineering and Electronics, 2022, 44(9): 2707-2715.
[3]	Tao ZHANG, Xiaogang YANG, Ruitao LU, Xueli XIE, Chuang LIU. Key-point based method for ship detection in remote sensing images [J]. Systems Engineering and Electronics, 2022, 44(8): 2437-2447.
[4]	Cheng FAN, Buhong WANG, Jiwei TIAN. Node classification of airline network based on the graph convolution network model with multi-task learning [J]. Systems Engineering and Electronics, 2022, 44(7): 2341-2349.
[5]	Xiaofeng ZHAO, Yebin XU, Fei WU, Jiahui NIU, Wei CAI, Zhili ZHANG. Ground infrared target detection method based on global sensing mechanism [J]. Systems Engineering and Electronics, 2022, 44(5): 1461-1467.
[6]	Huachao WANG, Jing LIU, Haowen CHENG, Xiyan PENG. Fast star map recognition algorithm based on fuzzy decision [J]. Systems Engineering and Electronics, 2022, 44(5): 1447-1453.
[7]	Tongdian WANG, Jieyu LIU, Zongshou WU, Qiang SHEN, Erliang YAO. Visual-inertial SLAM method based on multi-scale optical flow fusion feature point [J]. Systems Engineering and Electronics, 2022, 44(3): 977-985.
[8]	Hongyao LI, Xiaoqiang LI, Xinzhong HAN, Xueli XIE, Jianxiang XI. Cooperative object detection and recognition algorithm for multiple UAVs based on decision fusion [J]. Systems Engineering and Electronics, 2022, 44(3): 746-754.
[9]	Zizhuang SONG, Jiawei YANG, Dongfang ZHANG, Shiqiang WANG, Shuo ZHANG. Real-time infrared multi-class multi-target anchor-free tracking network [J]. Systems Engineering and Electronics, 2022, 44(2): 401-409.
[10]	Jiali FAN, Shaobing TIAN, Kui HUANG, Xingdong ZHU. Multi-scale object detection algorithm for aircraft carrier surface based on Faster R-CNN [J]. Systems Engineering and Electronics, 2022, 44(1): 40-46.
[11]	Xin LYU, Xiaodong MU, Jun ZHANG. Multi-threshold image segmentation based on improved sparrow search algorithm [J]. Systems Engineering and Electronics, 2021, 43(2): 318-327.
[12]	Guoming XU, Hongwu YUAN, Mogen XUE, Feng WANG. Full polarization parameters low-rank and sparse factorization for pseudo color image fusion [J]. Systems Engineering and Electronics, 2020, 42(11): 2450-2460.
[13]	Qian XIANG, Xiaodan WANG, Rui LI, Jie LAI, Guoling ZHANG. HRRP image recognition of midcourse ballistic targets based on DCNN [J]. Systems Engineering and Electronics, 2020, 42(11): 2426-2433.
[14]	Jun LUO, Jianqiang LIU, Yanan PANG. Multi-threshold image segmentation of 2D Otsu based on neighborhood search JADE [J]. Systems Engineering and Electronics, 2020, 42(10): 2164-2171.
[15]	Siqi HUA, Wei ZHAO, Jianye LIU. Background suppression algorithms based on improved filter and image multi-scale transformation [J]. Systems Engineering and Electronics, 2020, 42(8): 1679-1684.