基于几何约束孪生卷积网络的相机6DOF定位研究

doi:10.12305/j.issn.1001-506X.2023.06.27

摘要/Abstract

摘要：

视觉定位技术是视觉导航和自动驾驶领域的重要组成部分。提出一种基于几何约束孪生卷积网络的相机六自由度(6 degree of freedom, 6DOF)定位方法, 采用卷积网络、以学习查询图像与参考图像之间相对位姿关系的几何约束方式, 获得查询图像的绝对位姿; 并使用性能优异的主干特征提取网络, 以及多任务联合损失函数同时训练的策略, 进一步提高方法的定位精度、稳定性以及泛化能力。同时, 设计了特征距离度量损失函数, 增强了对于相似图像的区分性。在室内及室外公开数据集上的验证数据表明, 与同类型方法相比，所提方法更具竞争力。

关键词: 相机定位, 孪生卷积网络, 相对位姿, 泛化能力, 定位精度

Abstract:

Visual localization technique is an important component in the field of visual navigation and autonomous driving. A camera 6 degree of freedom(6DOF) localization method based on geometrically constrained Siamese convolutional networks is proposed, which uses a geometrically constrained way for the convolutional network to learn the relative positional relationship between the query image and the reference image to obtain the global position of the query image. Meanwhile, this method uses a high-performing backbone feature extraction network and a multi-task joint loss function training strategy simultaneously to further improve the localization accuracy, stability, and generalization of the proposed method. The feature distance metric loss function is also designed to enhance the differentiation of similar images. The validation data on indoor and outdoor public datasets show that the proposed method is more competitive when it is compared with similar methods.

Key words: camera location, Siamese convolutional network, relative pose, generalization capability, pose accuracy

中图分类号:

TP301

董思强, 邓年茂. 基于几何约束孪生卷积网络的相机6DOF定位研究[J]. 系统工程与电子技术, 2023, 45(6): 1828-1835.

Siqiang DONG, Nianmao DENG. 6DOF camera location research based on geometric constraint Siamese convolution network[J]. Systems Engineering and Electronics, 2023, 45(6): 1828-1835.

图/表 9

图1

图2

图3

图4

图5

表1

表2

图6

图7

参考文献 32

1	MURILLOA C, KOSECKA J. Experiments in place recognition using gist panoramas[C]//Proc. of the IEEE 12th International Conference on Computer Vision Workshops, 2010.
2	SATTLER T, WEYAND T, LEIBE B, et al. Image retrieval for image-based localization revisited[C]//Proc. of the British Machine Vision Conference, 2012.
3	ULRICH I, NOURBAKHSH I. Appearance-based place recognition for topological localization[C]// Proc. of the IEEE International Conference on Robotics and Automation, 2000: 1023-1029.
4	WOLF J , BURGARD W , BURKHARDT H . Robust vision-based localization by combining an image-retrieval system with Monte Carlo localization[J]. IEEE Trans.on Robotics and Automation, 2005, 21 (2): 208- 216. doi: 10.1109/TRO.2004.835453
5	KUBELOVA Z, BUJNAK M, PAJDLA T. Real-time solution to the absolute pose problem with unknown radial distortion and focal length[C]//Proc. of the IEEE International Conference on Computer Vision, 2013: 2816-2823.
6	BRAHMBHAAT S, GU J, KIM K, et al. Geometry-aware learning of maps for camera localization[C]//Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 2616-2625.
7	NILWONG S , HOSSAIN D , KANEKO S , et al. Deep learning-based landmark detection for mobile robot outdoor localization[J]. Machines, 2019, 7 (2): 25. doi: 10.3390/machines7020025
8	KENDALL A, GRIMES M, CIPOLLA R. PoseNet: convolutional network for real-time 6-DOF camera relocalization[C]//Proc. of the IEEE International Conference on Computer Vision, 2015: 2938-2946.
9	SZEGEDY C, LIU W, JIA Y, et al. Going deeper with convolutions[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.
10	PIASCO N, SIDIBE D, DEMONCEAUX C, et al. Geometric camera pose refinement with learned depth maps[C]//Proc. of the IEEE International Conference on Image Processing, 2019.
11	LI Q , ZHU J S , CAO R , et al. Relative geometry-aware Siamese neural network for 6DOF camera relocalization[J]. Neurocomputing, 2021, 426, 134- 146. doi: 10.1016/j.neucom.2020.09.071
12	BOCHKOVSKIY A, WANG C Y, LIAO H. YOLOv4: optimal speed and accuracy of object detection[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2020: 2930-2937.
13	VALADA A, RADWAN N, BURGARD W. Deep auxiliary learning for visual localization and odometry[C]// Proc. of the IEEE International Conference on Robotics and Automation, 2008.
14	RADWAN N , VALADA A , BURGARD W . VLocNet++: deep multitask learningfor semantic visual localization and odometry[J]. IEEE Robotics and Automation Letters, 2018, 3 (4): 4407- 4414. doi: 10.1109/LRA.2018.2869640
15	DONOSER M, SCHMALSTIEG D. Discriminative feature-to-point matching in image-based localization[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2014: 516-523.
16	SATTLER T, HAVLENA M, RADENOVIC F, et al. Hyper points and fine vocabularies for large-scale location recognition[C]// Proc. of the IEEE International Conference on Computer Vision, 2015: 2102-2110.
17	SATTLER T , LEIBE B , KOBBELT L . Efficient & effective prioritized matching for large-scale image-based localization[J]. IEEE Trans.on Pattern Analysis and Machine Intelligence, 2017, 39 (9): 1744- 1756. doi: 10.1109/TPAMI.2016.2611662
18	LIM H, SINHA S, COHEN M, et al. Real-time imagebased 6-DOF localization in large-scale environments[C]//Proc. of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2012: 1043-1050.
19	RUBLEE E, RABAUD V, KONOLIGE K, et al. Orb: an efficient alternative to sift or surf[C]//Proc. of the IEEE International Conference on Computer Vision, 2011: 2564-2571.
20	KENDALL A, CIPOLLA R. Modelling uncertainty in deep learning for camera relocalization[C]//Proc. of the IEEE International conference on Robotics and Automation, 2016: 4762-4769.
21	MELEKHOV I, YLIOINAS J, KANNALA J, et al. Image-based localization using hourglass networks[C]// Proc. of the IEEE International Conference on Computer Vision, 2017: 879-886.
22	CLARK R, WANG S, MARKHAM A, et al. Vidloc: a deep spatio-temporal model for 6-DOF video-clip relocalization[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
23	BROMLEY J , BENTZ JW , BOTTOU L , et al. Signature verification using a "Siamese" time delay neural network[J]. International Journal of Pattern Recognition and Artificial Intelligence, 1993, 7 (4): 669- 688. doi: 10.1142/S0218001493000339
24	WANG C Y, LIAO H, WU Y H, et al. CSPNet: a new backbone that can enhance learning capability of CNN[C]//Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020.
25	SRIVASTAVA N , HINTON G E , KRIZHEVSKY A , et al. Dropout: a simple way to prevent neural networks from over-fitting[J]. Journal of Robotics & Machine Learning, 2014, 15 (1): 1929- 1958.
26	KENDALL A, CIPOLLA R. Geometric loss functions for camera pose regression with deep learning[C]// Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
27	SCHROFF F, KALENICHENKO D, PHILBIN J. Facenet: A unified embedding for face recognition and clustering[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 815-823.
28	BELLET A, HABRARD A, SEBBAN M. A survey on metric learning for feature vectors and structured data[EB/OL]. [2022-01-10]. https://arxiv.org/abs/1306.6709.
29	RUSSAKOVSKY O , DENG J , SU H , et al. ImageNet large scale visual recognition challenge[J]. International Journal of Computer Vision, 2015, 115 (3): 211- 252.
30	SHOTTON B, GLOCKER C, ZACH S, et al. Scene coordinate regression forests for camera relocalization in RGB-D images[C]// Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2013: 2930-2937.
31	NEWCOMBE R A, IZADI S, HILLIGES O, et al. Kinectfusion: real-time dense surface mapping and tracking[C]//Proc. of the IEEE International Symposium on Mixed and Augmented Reality, 2011: 127-136.
32	WEYAND T, KOSTRIKOV I, PHILBIN J. PlaNet-photo geo-location with convolutional neural networks[C]//Proc. of the European Conference on Computer Vision, 2016: 37-55.

场景	方法
场景	BayesianPoseNet^[32]	PoseNet^[8]	PoseNet2^[26]	LSTM-Net^[20]	GPRM	RPRM
Chess	0.37 m, 7.24°	0.32 m, 8.12°	0.13 m, 4.48°	0.24 m, 5.77°	0.11 m, 5.19°	0.09 m, 4.82°
Fire	0.43 m, 13.70°	0.47 m, 14.40°	0.27 m, 11.30°	0.34 m, 11.90°	0.25 m, 11.64°	0.21 m, 9.55°
Heads	0.31 m, 12.00°	0.29 m, 12.00°	0.17 m, 13.00°	0.21 m, 13.70°	0.13 m, 13.20°	0.11 m, 11.23°
Office	0.48 m, 8.04°	0.48 m, 7.68°	0.19 m, 5.55°	0.30 m, 8.08°	0.16 m, 7.61°	0.12 m, 5.36°
Pumpkin	0.61 m, 7.08°	0.47 m, 8.42°	0.26 m, 4.75°	0.33 m, 7.00°	0.16 m, 6.62°	0.13 m, 5.24°
Red Kitchen	0.58 m, 7.54°	0.59 m, 8.64°	0.23 m, 5.35°	0.37 m, 8.83°	0.17 m, 8.24°	0.12 m, 6.58°
Stairs	0.48 m, 13.10°	0.47 m, 13.80°	0.35 m, 12.40°	0.40 m, 13.70°	0.25 m, 13.11°	0.21 m, 10.18°

场景	方法
场景	BayesianPoseNet^[32]	PoseNet^[8]	PoseNet2^[26]	LSTM-Net^[20]	GPRM	RPRM
King’s College	1.74 m, 4.06°	1.92 m, 5.40°	0.88 m, 1.04°	0.99 m, 3.68°	0.87 m, 1.95°	0.83 m, 1.85°
Old Hospital	2.57 m, 5.14°	2.31 m, 5.38°	3.20 m, 3.29°	1.51 m, 4.29°	1.62 m, 2.67°	1.52 m, 2.37°
Shop Facade	1.25 m, 7.54°	1.46 m, 8.08°	0.88 m, 3.78°	1.18 m, 7.44°	0.83 m, 5.15°	0.81 m, 4.65°
StMary’s Church	2.11 m, 8.38°	2.65 m, 8.46°	1.57 m, 3.32°	1.52 m, 6.68°	1.65 m, 3.10°	1.21 m, 2.85°

[1]	沈子涵, 赵修斌, 张闯, 张良, 刘鑫贤. 基于长短期记忆神经网络的自适应容错方法[J]. 系统工程与电子技术, 2023, 45(3): 831-838.
[2]	王晓伟, 詹亚锋, 谢浩然, 陈曦. 通导一体化环月星座设计初探[J]. 系统工程与电子技术, 2023, 45(1): 241-249.
[3]	罗畅, 王洁, 王世强, 史通, 任卫华. 基于泛化深度迁移特征的高分遥感场景分类[J]. 系统工程与电子技术, 2018, 40(3): 682-691.
[4]	许兆新, 李航洲, 李亮, 赵琳, 刘亚. 基于EKF/SF的BDS短基线多径抑制方法[J]. 系统工程与电子技术, 2017, 39(6): 1334-1340.
[5]	赵琳, 张胜宗, 李亮, 王雪. #br# 基于自适应EKF的BDS/GPS精密单点定位方法[J]. 系统工程与电子技术, 2016, 38(9): 2142-2148.
[6]	马旭, 程咏梅, 郝帅. 基于矢量约束的随机特征点选取算法[J]. 系统工程与电子技术, 2016, 38(10): 2367-2374.
[7]	任文娟, 周志鑫, 吕守业, 师栋锋. 基于AIS信息的舰船位置标校方法[J]. 系统工程与电子技术, 2016, 38(10): 2381-2388.
[8]	陈君, 彭小奇, 唐秀明, 宋彦坡, 刘征. 支持向量回归机预测误差校正方法[J]. 系统工程与电子技术, 2015, 37(8): 1832-1836.
[9]	徐景峰1, 韩树平1, 舒象兰1, 马鑫1, 董永峰1, 柳洪刚2. T-R型双基地声纳定位算法及系统精度研究[J]. 系统工程与电子技术, 2015, 37(10): 2234-2238.
[10]	张养瑞，李云杰，高梅国. 协同干扰资源优化分配模型及算法[J]. 系统工程与电子技术, 2014, 36(9): 1744-1749.
[11]	马可锌，王惠南. 基于视觉的近距离编队对偶四元数相对位姿估计算法[J]. Journal of Systems Engineering and Electronics, 2013, 35(2): 391-396.
[12]	赵孔瑞, 周共健, 于长军, 权太范. 双基地高频地波雷达三维曲面定位精度分析[J]. Journal of Systems Engineering and Electronics, 2012, 34(7): 1344-1349.
[13]	彭芳, 左继章, 吴军. 基于高斯—牛顿法改进的复合双基地雷达目标空间定位算法[J]. Journal of Systems Engineering and Electronics, 2009, 31(3): 557-559.