基于深度强化学习的海战场目标搜寻路径规划

doi:10.12305/j.issn.1001-506X.2022.11.24

系统工程与电子技术 ›› 2022, Vol. 44 ›› Issue (11): 3486-3495.doi: 10.12305/j.issn.1001-506X.2022.11.24

基于深度强化学习的海战场目标搜寻路径规划

杨清清, 高盈盈*, 郭玙, 夏博远, 杨克巍

国防科技大学系统工程学院, 湖南长沙 410073

收稿日期:2021-09-01 出版日期:2022-10-26 发布日期:2022-10-29
通讯作者: 高盈盈
作者简介:杨清清(1982—), 女, 副教授, 硕士研究生导师, 博士，主要研究方向为应急管理智能决策、资源优化、任务规划方法|高盈盈(1996—), 女, 博士研究生, 主要研究方向为海上搜救任务规划、智能决策|郭玙(1993—), 女, 博士研究生, 主要研究方向为海上搜救任务规划、智能决策|夏博远(1994—), 男, 博士研究生, 主要研究方向为作战体系建模、评估与优化、复杂系统机理性研究|杨克巍(1977—), 男, 教授, 博士研究生导师, 博士, 主要研究方向为大数据与体系工程、复杂系统机理性
基金资助:
国家自然科学基金(72071206);国家自然科学基金(71690233);湖南省科技创新计划(2020RC4046);中国博士后基金(2019M653923)

Target search path planning for naval battle field based on deep reinforcement learning

Qingqing YANG, Yingying GAO*, Yu GUO, Boyuan XIA, Kewei YANG

College of Systems Engineering, National University of Defense Technology, Changsha 410073, China

Received:2021-09-01 Online:2022-10-26 Published:2022-10-29
Contact: Yingying GAO

摘要/Abstract

摘要：

海战场是未来大国冲突的主阵地之一, 强大的海战场目标搜寻能力是执行海上训练和作战的最后一道屏障, 同时也因其复杂多变的环境和重要战略地位成为战场联合搜救中最艰难最核心的部分。面向海战场目标搜寻的存活时间短、实时性要求高等特点, 提出一种基于深度强化学习的海战场目标搜寻规划方法。首先, 构建了海战场目标搜寻场景数学规划模型, 并将其映射为一种强化学习模型; 然后, 基于Rainbow深度强化学习算法, 设计了海战场目标搜寻规划的状态向量、神经网络结构以及算法框架与流程。最后, 用一个案例, 验证了所提方法的可行性与有效性, 与常规应用的平行搜寻模式相比大大提高了搜寻成功率。

关键词: 海战场, 目标搜寻, 路径规划, 动态规划, 深度强化学习

Abstract:

The naval battle field is one of the main situations of the future great power conflicts. The powerful target search capability of the naval battle field is the last protection for the implementation of maritime training and combat, and becomes the most difficult and core part of the battlefield joint search and rescue because of its complex and changeable environment and important strategic position. A path planning method based on deep reinforcement learning is proposed to solve the problem of short time cycle and high real-time requirement of target search in naval battle field. Firstly, the mathematical programming model of naval battle field target search is constructed and mapped into a reinforcement learning model. Then, based on Rainbow deep reinforcement learning algorithm, the state vector, neural network structure and algorithm framework and flow of target search planning in naval battle field are designed. Finally, a case is used to verify the feasibility and effectiveness of the proposed method, which greatly improves the search success rate compared with the conventional parallel search mode.

Key words: naval battle field, target search, path planning, dynamic planning, deep reinforcement learning

中图分类号:

N945.15

杨清清, 高盈盈, 郭玙, 夏博远, 杨克巍. 基于深度强化学习的海战场目标搜寻路径规划[J]. 系统工程与电子技术, 2022, 44(11): 3486-3495.

Qingqing YANG, Yingying GAO, Yu GUO, Boyuan XIA, Kewei YANG. Target search path planning for naval battle field based on deep reinforcement learning[J]. Systems Engineering and Electronics, 2022, 44(11): 3486-3495.

图/表 13

图1

图2

表1

图3

图4

图5

图6

图7

图8

表2

图9

图10

图11

参考文献 28

1	OTOTE D A , LI B , AI B , et al. A decision-making algorithm for maritime search and rescue plan[J]. Sustainability, 2019, 11 (7): 2084- 2099. doi: 10.3390/su11072084
2	JIN Y Q , WANG N , SONG Y T , et al. Optimization model and algorithm to locate rescue bases and allocate rescue vessels in remote oceans[J]. Soft Computing, 2021, 25 (4): 3317- 3334. doi: 10.1007/s00500-020-05378-6
3	GUO Y , YE Y Q , YANG Q Q , et al. A multi-objective INLP model of sustainable resource allocation for long-range maritime search and rescue[J]. Sustainability, 2019, 11 (3): 929- 953. doi: 10.3390/su11030929
4	RAHMES M, CHESTER D, HUNT J, et al. Optimizing cooperative cognitive search and rescue UAVs[C]//Proc. of the Autonomous Systems: Sensors, Vehicles, Security and the Internet of Everything, 2018.
5	LIANG X Y , DU X S , WANG G L , et al. A deep reinforcement learning network for traffic light cycle control[J]. IEEE Trans.on Vehicular Technology, 2019, 68 (2): 1243- 1253. doi: 10.1109/TVT.2018.2890726
6	WANG Y D , LIU H , ZHENG W B , et al. Multi-objective workflow scheduling with deep-Q-network-based multi-agent reinforcement learning[J]. IEEE Access, 2019, 7, 39974- 39982. doi: 10.1109/ACCESS.2019.2902846
7	LUONG N C , HOANG D T , GONG S , et al. Applications of deep reinforcement learning in communications and networking: a survey[J]. IEEE Communications Surveys and Tutorials, 2019, 21 (4): 3133- 3174. doi: 10.1109/COMST.2019.2916583
8	MNIH V , KAVUKCUOGLU K , SILVER D , et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518 (7540): 529- 533. doi: 10.1038/nature14236
9	史腾飞, 王莉, 黄子蓉. 序列多智能体强化学习算法[J]. 模式识别与人工智能, 2021, 34 (3): 206- 213.
	SHI T F , WANG L , HUANG Z R . Sequence to sequence multi-agent reinforcement learning algorithm[J]. Pattern Recognition and Artificial Intelligence, 2021, 34 (3): 206- 213.
10	MNIH V, KAVUKCUOGLU K, SILVER D, et al. Playing atari with deep reinforcement learning[EB/OL]. [2021-10-12]. https://arxiv.org/abs/1312.5602.
11	SCHAUL T, QUAN J, ANTONOGLOU I, et al. Prioritized experience replay[EB/OL]. [2021-10-12]. https://arxiv.org/abs/1511.05952.
12	VAN H H, GUEZ A, SILVER D. Deep reinforcement learning with double q-learning[C]//Proc. of the AAAI conference on Artificial Intelligence, 2016.
13	WANG Z Y, SCHAUL T, HESSEL M, et al. Dueling network architectures for deep reinforcement learning[C]//Proc. of the International Conference on Machine Learning, 2016: 1995-2003.
14	BELLEMARE M G, DABNEY W, MUNOS R. A distributional perspective on reinforcement learning[C]//Proc. of the International Conference on Machine Learning, 2017: 449-458.
15	FORTUNATO M, AZAR M G, PIOT B, et al. Noisy networks for exploration[EB/OL]. [2021-10-12]. https://arxiv.org/abs/1706.10295.
16	HESSEL M, MODAYIL J, VAN H H, et al. Rainbow: combining improvements in deep reinforcement learning[C]//Proc. of the National Conference on Artificial Intelligence, 2018.
17	SUTTON R S , BARTO A G . Reinforcement learning: an introduction[M]. Cambridge: Massachusetts Institute of Technology press, 1998.
18	SUTTON R S . Learning to predict by the methods of temporal differences[J]. Machine learning, 1988, 3 (1): 9- 44.
19	HAUSKNECHT M, STONE P. Deep recurrent Q-learning for partially observable MDPs[EB/OL]. [2021-10-12]. https://arxiv.org/abs/1507.06527v4.
20	轩永波, 黄长强, 吴文超, 等. 运动目标的多无人机编队覆盖搜索决策[J]. 系统工程与电子技术, 2013, 35 (3): 539- 544. doi: 10.3969/j.issn.1001-506X.2013.03.15
	XUN Y B , HUANG C Q , WU W C , et al. Coverage search strategies for moving targets using multiple unmanned aerial vehicle teams[J]. Systems Engineering and Electronics, 2013, 35 (3): 539- 544. doi: 10.3969/j.issn.1001-506X.2013.03.15
21	高盈盈. 海上搜救中无人机搜寻规划方法及应用研究[D]. 长沙: 国防科技大学, 2020.
	GAO Y Y. Research on UAV search planning method and application in maritime search and rescue[D]. Changsha: National University of Defense Technology, 2020.
22	XIONG W T , GELDER P V , YANG K W . A decision support method for design and operationalization of search and rescue in maritime emergency[J]. Ocean Engineering, 2020, 207, 107399- 107415. doi: 10.1016/j.oceaneng.2020.107399
23	GALLEGO A J , PERTUSA A , GIL P , et al. Detection of bodies in maritime rescue operations using unmanned aerial vehicles with multispectral cameras[J]. Journal of Field Robotics, 2019, 36 (4): 782- 796. doi: 10.1002/rob.21849
24	高春庆, 寇英信, 李战武, 等. 小型无人机协同覆盖侦察路径规划[J]. 系统工程与电子技术, 2019, 41 (6): 1294- 1299.
	GAO C Q , KOU Y X , LI Z W . Cooperative coverage path planning for small UAVs[J]. Systems Engineering and Electronics, 2019, 41 (6): 1294- 1299.
25	YUE W , GUAN X H , WANG L Y . A novel searching method using reinforcement learning scheme for multi-UAVs in unknown environments[J]. Applied Sciences, 2019, 9 (22): 4964- 4978. doi: 10.3390/app9224964
26	CHENG Y , ZHANG W D . Concise deep reinforcement learning obstacle avoidance for underactuated unmanned marine vessels[J]. Neurocomputing, 2018, 272, 63- 73. doi: 10.1016/j.neucom.2017.06.066
27	LI R P , ZHAO Z F , SUN Q , et al. Deep reinforcement learning for resource management in network slicing[J]. IEEE Access, 2018, 6, 74429- 74441. doi: 10.1109/ACCESS.2018.2881964
28	TAMPUU A , MATⅡSEN T , KODELJA D , et al. Multiagent cooperation and competition with deep reinforcement learning[J]. Plos One, 2017, 12 (4): e0172395.

要素	数据类似	转化方法	转化后长度
区域大小	数值	独热编码	2
区域当前POC矩阵	数值	独热编码	L_x×L_y
区域各栅格是否被搜索过	布尔型	独热编码	L_x×L_y
当前动作	枚举	定量化映射	1
初始位置	数值	独热编码	2

参数名称	参数大小	参数名称	参数大小
隐含层个数	1	累计回报折扣因子	0.5
隐含层神经元个数	1 500	ε-greedy探索率	0.9
输出层神经元个数	8	目标网络更新间隔步数	300
神经元激活函数	ReLU	经验池容量	10⁴
神经网络学习率	0.001	经验块最小容量	200
动作划分个数	8	模型训练次数	10⁵

[1]	冯豪博, 胡桥, 赵振轶. 基于精英族系遗传算法的AUV集群路径规划[J]. 系统工程与电子技术, 2022, 44(7): 2251-2262.
[2]	王冠, 茹海忠, 张大力, 马广程, 夏红伟. 弹性高超声速飞行器智能控制系统设计[J]. 系统工程与电子技术, 2022, 44(7): 2276-2285.
[3]	孟泠宇, 郭秉礼, 杨雯, 张欣伟, 赵柞青, 黄善国. 基于深度强化学习的网络路由优化方法[J]. 系统工程与电子技术, 2022, 44(7): 2311-2318.
[4]	尹洋, 杨全顺, 王征, 刘洋. 通信距离约束下的无人船集群覆盖搜索方法[J]. 系统工程与电子技术, 2022, 44(12): 3821-3828.
[5]	马威强, 高永琪, 赵苗. 基于全局最优和差分变异的头脑风暴优化算法[J]. 系统工程与电子技术, 2022, 44(1): 270-278.
[6]	焦志强, 张杰勇, 姚佩阳, 王勋, 何宜超. 基于分层结构的C4ISR服务部署分布式演化方法[J]. 系统工程与电子技术, 2021, 43(6): 1572-1585.
[7]	郭建国, 苏亚鲁. 高超飞行器自适应动态规划的控制系统设计[J]. 系统工程与电子技术, 2021, 43(6): 1628-1635.
[8]	高昂, 董志明, 李亮, 宋敬华, 段莉. MADDPG算法并行优先经验回放机制[J]. 系统工程与电子技术, 2021, 43(2): 420-433.
[9]	马文, 李辉, 王壮, 黄志勇, 吴昭欣, 陈希亮. 基于深度随机博弈的近距空战机动决策[J]. 系统工程与电子技术, 2021, 43(2): 443-451.
[10]	王文明, 杜佳璐. 基于正六边形栅格JPS算法的智能体路径规划[J]. 系统工程与电子技术, 2021, 43(12): 3635-3642.
[11]	高昂, 郭齐胜, 董志明, 杨绍卿. 基于EAS+MADRL的多无人车体系效能评估方法研究[J]. 系统工程与电子技术, 2021, 43(12): 3643-3651.
[12]	李亚南, 黄海滨, 陈亮名, 庄宇飞, 王晓丽. 变化海流环境下AUV能量最优三维路径规划[J]. 系统工程与电子技术, 2021, 43(12): 3667-3674.
[13]	李文刚, 汪流江, 方德翔, 李玉玮, 黄郡. 联合A^*与动态窗口法的路径规划算法[J]. 系统工程与电子技术, 2021, 43(12): 3694-3702.
[14]	周鑫, 王维平, 朱一凡, 王涛, 井田. 基于顺次分配机制的无人装备体系架构方案空间搜索方法[J]. 系统工程与电子技术, 2021, 43(11): 3211-3219.
[15]	陈岱岱, 李玩幽. 带拖线阵的水面无人艇局部路径规划算法[J]. 系统工程与电子技术, 2020, 42(9): 1988-1994.

基于深度强化学习的海战场目标搜寻路径规划

Target search path planning for naval battle field based on deep reinforcement learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 13

参考文献 28

相关文章 15

编辑推荐

Metrics

本文评价