VDAS中基于单目红外图像的深度估计方法

doi:10.12305/j.issn.1001-506X.2021.05.07

系统工程与电子技术 ›› 2021, Vol. 43 ›› Issue (5): 1210-1217.doi: 10.12305/j.issn.1001-506X.2021.05.07

VDAS中基于单目红外图像的深度估计方法

李旭¹(), 丁萌¹(), 魏东辉^2,*(), 吴晓舟¹(), 曹云峰³()

1. 南京航空航天大学民航学院, 江苏南京 211106
2. 北京机电工程研究所复杂系统控制与智能协同技术重点实验室, 北京 100074
3. 南京航空航天大学航天学院, 江苏南京 211106

收稿日期:2020-08-03 出版日期:2021-05-01 发布日期:2021-04-27
通讯作者: 魏东辉 E-mail:lixunuaa@nuaa.edu.cn;nuaa_dm@nuaa.edu.cn;weidonghui2652@sina.com;wuxiaozhou@nuaa.edu.cn;cyfac@nuaa.edu.cn
作者简介:李旭(1995—), 男, 硕士研究生, 主要研究方向为机器视觉。E-mail: lixunuaa@nuaa.edu.cn|丁萌(1981—), 男, 副教授, 博士, 主要研究方向为无人系统自主环境感知。E-mail: nuaa_dm@nuaa.edu.cn|魏东辉(1987—), 男, 高级工程师, 博士, 主要研究方向为飞行器制导导航控制技术。E-mail: weidonghui2652@sina.com|吴晓舟(1995—), 男, 硕士研究生, 主要研究方向为机器视觉。E-mail: wuxiaozhou@nuaa.edu.cn|曹云峰(1964—), 男, 教授, 博士, 主要研究方向为导航、制导与控制。E-mail: cyfac@nuaa.edu.cn
基金资助:
装备预研航天科工联合基金(6141B07090114);南京航空航天大学研究生创新基地(实验室)开放基金(kfjj20190730);南京航空航天大学研究生创新基地(实验室)开放基金(kfjj20190723)

Depth estimation method based on monocular infrared image in VDAS

Xu LI¹(), Meng DING¹(), Donghui WEI^2,*(), Xiaozhou WU¹(), Yunfeng CAO³()

1. College of Civil Aviation, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China
2. Science and Technology on Complex System Control and Intelligent Agent Cooperation Laboratory, Beijing Electro-Mechanical Engineering Institute, Beijing 100074, China
3. College of Astronautics, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China

Received:2020-08-03 Online:2021-05-01 Published:2021-04-27
Contact: Donghui WEI E-mail:lixunuaa@nuaa.edu.cn;nuaa_dm@nuaa.edu.cn;weidonghui2652@sina.com;wuxiaozhou@nuaa.edu.cn;cyfac@nuaa.edu.cn

摘要/Abstract

摘要：

针对视觉辅助驾驶系统(vision-based driving assistance system, VDAS)对夜间等低能见度场景下的车辆前视深度感知需求, 提出一种基于深度学习的单目红外图像深度估计方法。该方法采用端对端的多任务自监督学习框架, 利用单目红外视频帧之间的立体几何约束构建损失函数, 无须场景的真实深度信息。取前后两帧重投影误差的最小值, 解决了单目视频帧之间的遮挡问题, 同时削弱了红外图像噪点多的影响。网络解码器将多尺度深度图统一上采样到较高分辨率并计算重投影误差, 改善了深度图中的空洞现象。在FLIR车载红外数据集上的定性实验表明, 所提方法能利用单目红外图像得到对应的像素级稠密深度; 真实道路上的分析试验表明, 所提方法能够在夜间情况下利用红外图像有效感知目标物的深度信息, 15 m以内的相对误差为13.2%, 可以满足多数突发情况下情况下的避撞要求。

关键词: 红外图像, 深度估计, 卷积神经网络, 自监督学习

Abstract:

In view of the demand of vision-based driving assistance system (VDAS) for vehicle forward-looking depth perception in low visibility scenes, a depth learning based monocular infrared image depth estimation method is proposed. In this method, an end-to-end multi-task self-monitoring learning framework is adopted, and the loss function is constructed by using the stereo geometric constraints between monocular infrared video frames, so the real depth information of the scene is not need. The minimum value of the reprojection error between the two frames is taken to solve the occlusion problem, and the influence of infrared image noise is weaken; The network decoder samples the multi-scale depth map to higher resolutions and calculates the reprojection error, which avoids the hole phenomenon in the depth map. Qualitative experiments on FLIR infrared datasets show that the proposed method can obtain pixel-level dense depth from monocular infrared images. Experiments on real roads show that the proposed method can effectively perceive the depth information of the target in the night, and the absolute error is 13.2% within 15 m. It can meet the requirements of collision avoidance in most emergency situations.

Key words: infrared image, depth estimation, convolution neural network, self-supervised learning

中图分类号:

TP39

李旭, 丁萌, 魏东辉, 吴晓舟, 曹云峰. VDAS中基于单目红外图像的深度估计方法[J]. 系统工程与电子技术, 2021, 43(5): 1210-1217.

Xu LI, Meng DING, Donghui WEI, Xiaozhou WU, Yunfeng CAO. Depth estimation method based on monocular infrared image in VDAS[J]. Systems Engineering and Electronics, 2021, 43(5): 1210-1217.

图/表 11

图1

图2

图3

图4

图5

图6

图7

表1

表2

图8

图9

参考文献 19

1	RANFT B , STILLER C . The role of machine vision for intelligent vehicles[J]. IEEE Trans.on Intelligent Vehicles, 2016, 1 (1): 8- 19. doi: 10.1109/TIV.2016.2551553
2	HORGAN J, HUGHES C, MCDONALD J, et al. Vision-based driver assistance systems: survey, taxonomy and advances[C]//Proc. of the IEEE 18th International Conference on Intelligent Transportation Systems, 2015: 2032-2039.
3	ULLMAN S . The interpretation of structure from motion[J]. Proceedings of the Royal Society of London, 1979, 203 (1153): 405- 426.
4	ZHAO C Q, SUN Q Y, ZHANG C Z, et al. Monocular depth estimation based on deep learning: an overview[EB/OL]. [2020-03-14]. https://arxiv.org/pdf/2003.06620.
5	HUANG J , WANG C , LIU Y , et al. The progress of monocular depth estimation technology[J]. Journal of Image and Gra-phics, 2019, 24 (12): 2081- 2097.
6	EIGEN D, PUHRSCH C, FERGUS R. Depth map prediction from a single image using a multi-scale deep network[EB/OL]. [2014-06-09]. https://arxiv.org/abs/1406.2283.
7	KENDALL A, MARTIROSYAN H, DASGUPTA S, et al. End-to-end learning of geometry and context for deep stereo regression[C]//Proc. of the IEEE International Conference on Computer Vision, 2017: 66-75.
8	XU D, WANG W, TANG H, et al. Structured attention guided convolutional neural fields for monocular depth estimation[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 3917-3925.
9	GARG R, VIJAY K B G, GUSTAVO C, et al. Unsupervised CNN for single view depth estimation: geometry to the rescue[C]//Proc. of the European Conference on Computer Vision, 2016: 740-756.
10	ZHOU T, BROWN M, SNAVELY N, et al. Lowe, unsupervised learning of depth and ego-motion from video[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 1851-1858.
11	GEIGER A, LENZ P, URTASUN R. Are we ready for autonomous driving? the kitti vision benchmark suite[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2012: 3354-3361.
12	SAXENA A , SUN M , NG A Y . Make3D: learning 3D scene structure from a single still image[J]. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2008, 31 (5): 824- 840.
13	RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proc. of the Medical Image Computing and Computer-Assisted Intervention, 2015: 234-241.
14	WANG Z , BOVIK A C , SHEIKH H R , et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Trans.on Image Processing, 2004, 13 (4): 600- 612. doi: 10.1109/TIP.2003.819861
15	GODARD C, AODHA O M, BROSTOW G J. Unsupervised monocular depth estimation with left-right consistency[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 270-279.
16	VIJAYANARASIMHAN S, RICCO S, SCHMID C, et al. SfM-net: learning of structure and motion from video[EB/OL]. [2017-02-25]. https://arxiv.org/abs/1704.07804.
17	CASSER V, PIRK S, MAHJOURIAN R, et al. Depth prediction without the sensors: leveraging structure for unsupervised learning from monocular videos[EB/OL]. [2019-06-12] https://arxiv.org/abs/1906.05717.
18	ODARD C, AODHA O M, FIRMAN M, et al. Digging into self-supervised monocular depth estimation[C]//Proc. of the IEEE International Conference on Computer Vision, 2019: 3828-3838.
19	KAIMING H , GEORGIA G , PIOTR D , et al. Mask R-CNN[J]. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2018, 42 (2): 386- 397.

距离/m	分布空间
距离/m	0~0.1	0.1~0.2	0.2~0.3	＞0.3
0~5	60.0	30.0	10.0	0
5~10	16.7	66.7	16.7	0
10~15	0	50.0	50.0	0
15~20	0	33.3	50.0	16.7

距离/m	分布空间
距离/m	＜0.1	＜0.2	＜0.3	＞0.3
0~5	60.0	90.0	100.0	0
5~10	16.7	83.4	100.0	0
10~15	0	50.0	100.0	0
15~20	0	33.3	83.3	16.7

[1]	宋爽, 张悦, 张琳娜, 岑翼刚, 李浥东. 基于深度学习的轻量化目标检测算法[J]. 系统工程与电子技术, 2022, 44(9): 2716-2725.
[2]	王彩云, 吴钇达, 王佳宁, 马璐, 赵焕玥. 基于改进的CNN和数据增强的SAR目标识别[J]. 系统工程与电子技术, 2022, 44(8): 2483-2487.
[3]	韦娟, 杨皇卫, 宁方立. 基于NMF与CNN联合优化的声学场景分类[J]. 系统工程与电子技术, 2022, 44(5): 1433-1438.
[4]	陈冬, 句彦伟. 基于语义分割实现的SAR图像舰船目标检测[J]. 系统工程与电子技术, 2022, 44(4): 1195-1201.
[5]	方伟, 王玉, 闫文君, 林冲. 基于神经网络的符号化飞行动作识别[J]. 系统工程与电子技术, 2022, 44(3): 737-745.
[6]	孙晶明, 虞盛康, 孙俊. 基于深度学习的HRRP识别姿态敏感性分析[J]. 系统工程与电子技术, 2022, 44(3): 802-807.
[7]	刘恒燕, 张立民, 闫文君, 钟兆根, 凌青, 梁晓军. 基于WBP-CNN算法的LDPC译码[J]. 系统工程与电子技术, 2022, 44(3): 1030-1035.
[8]	邵凯, 朱苗苗, 王光宇. 基于生成对抗与卷积神经网络的调制识别方法[J]. 系统工程与电子技术, 2022, 44(3): 1036-1043.
[9]	张玺, 金正猛, 姜亚琴. 融合深度图像先验的全变差图像着色算法[J]. 系统工程与电子技术, 2022, 44(2): 385-393.
[10]	金涛, 王晓峰, 田润澜, 张歆东. 基于改进1DCNN+TCN的雷达辐射源快速识别方法[J]. 系统工程与电子技术, 2022, 44(2): 463-469.
[11]	吕勤哲, 全英汇, 沙明辉, 董淑仙, 邢孟道. 基于集成深度学习的有源干扰智能分类[J]. 系统工程与电子技术, 2022, 44(12): 3595-3602.
[12]	高涌荇, 王旭东, 汪玲, 朱岱寅, 郭军, 孟凡旺. 基于RCNN的双极化气象雷达天气信号检测[J]. 系统工程与电子技术, 2022, 44(11): 3380-3387.
[13]	李永刚, 朱卫纲, 黄琼男, 李云涛, 何永华. 复杂背景下SAR图像近岸舰船目标检测[J]. 系统工程与电子技术, 2022, 44(10): 3096-3103.
[14]	但波, 付哲泉, 高山, 简涛. 基于卷积神经网络的海面目标全极化高分辨距离像识别技术[J]. 系统工程与电子技术, 2022, 44(1): 108-116.
[15]	刘紫燕, 马珊珊, 梁静, 朱明成, 袁磊. 注意力机制CNN的毫米波大规模MIMO系统信道估计算法[J]. 系统工程与电子技术, 2022, 44(1): 307-312.

VDAS中基于单目红外图像的深度估计方法

Depth estimation method based on monocular infrared image in VDAS

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 19

相关文章 15

编辑推荐

Metrics

本文评价