基于深度强化学习的UAV航路自主引导机动控制决策算法

doi:10.3969/j.issn.1001-506X.2020.07.19

Abstract

Abstract:

To solve a specific problem involved in autonomous guidance maneuver control of the unmanned aerial vehicle (UAV) route under terminal position constraints, the autonomous flight model of the UAV is described based on Markov decision processes and the simulation environment for the training algorithm is constructed. Meanwhile, an autonomous guidance maneuver control algorithm of UAV is proposed based on deep deterministic policy gradient (DDPG) and the guidance maneuvering control function and the state-action value function are fitted by the neural network. Finally, the simulation results show that the UAV using the proposed algorithm can fly to a fixed position in horizontal plane from any position and attitude. It is proved that the proposed algorithm can effectively improve the autonomy of the UAV.

Key words: autonomous guidance, maneuver control and decision-making, Markov decision process, deep deterministic policy gradient (DDPG) method, deep reinforcement learning

CLC Number:

V249.4

Kun ZHANG, Ke LI, Haotian SHI, Zhenchong ZHANG, Zekun LIU. Autonomous guidance maneuver control and decision-making algorithm[J]. Systems Engineering and Electronics, 2020, 42(7): 1567-1574.

Figures/Tables 11

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Table 1

Fig.9

Fig.10

References 26

1	尹欣繁, 章贵川, 彭先敏, 等. 军用无人机技术智能化发展及应用[J]. 国防科技, 2018, 39 (5): 30- 34.
	YIN X F , ZHANG G C , PENG X M , et al. Intelligent development and application of military UAV technology[J]. National Defense Science and Technology, 2018, 39 (5): 30- 34.
2	黄长强. 未来空战过程智能化关键技术研究[J]. 航空兵器, 2019, 26 (1): 11- 19.
	HUANG C Q . Research on key technology of future air combat process intelligentization[J]. Aero Weaponry, 2019, 26 (1): 11- 19.
3	周思羽, 吴文海, 张楠, 等. 自主空战机动决策方法综述[J]. 航空计算技术, 2012, 24 (1): 27- 31.
	ZHOU S Y , WU W H , ZHANG N , et al. Overview of autonomous air combat maneuver decision[J]. Aeronautical Computing Technique, 2012, 24 (1): 27- 31.
4	李世豪, 丁勇, 高振龙. 基于直觉模糊博弈的无人机空战机动决策[J]. 系统工程与电子技术, 2019, 41 (5): 1063- 1070.
	LI S H , DING Y , GAO Z L . UAV air combat maneuvering decision based on intuitionistic fuzzy game theory[J]. Systems Engineering and Electronics, 2019, 41 (5): 1063- 1070.
5	李世豪.复杂空战环境下基于博弈模型的无人机机动决策方法研究[D].南京:南京航空航天大学, 2019.
	LI S H. Research on UAV maneuvering decision method based on game theory in complex air combat[D]. Nanjing: Nanjing University of Aeronautics and Astronautics, 2019.
6	AUSTIN F , CARBONE G , HINZ H , et al. Game theory for automated maneuvering during air-to-air combat[J]. Journal of Guidance, Control and Dynamics, 1990, 13 (6): 1143- 1149.
7	邓可, 彭宣淇, 周德云. 基于矩阵对策与遗传算法的无人机空战决策[J]. 火力与指挥控制, 2019, 44 (12): 61- 66, 71.
	DENG K , PENG X Q , ZHOU D Y . Study on air combat decision method of UAV based on matrix game and genetic algorithm[J]. Fire Control & Command Control, 2019, 44 (12): 61- 66, 71.
8	孟光磊, 罗元强, 梁宵, 等. 基于动态贝叶斯网络的空战决策方法[J]. 指挥控制与仿真, 2017, 39 (3): 49- 54.
	MENG G L , LUO Y Q , LIANG X , et al. Air combat decision-making method based on dynamic Bayesian network[J]. Command Control & Simulation, 2017, 39 (3): 49- 54.
9	董彦非, 申洋, 张恒喜. 空战机动决策中的影响图方法[J]. 电光与控制, 2001, 30 (1): 49- 53.
	DONG Y F , SHEN Y , ZHANG H X . Influence diagram used in air combat maneuvering decision[J]. Electronics Optics & Control, 2001, 30 (1): 49- 53.
10	傅莉, 谢福怀, 孟光磊, 等. 基于滚动时域的无人机空战决策专家系统[J]. 北京航空航天大学学报, 2015, 41 (11): 1994- 1999.
	FU L , XIE F H , MENG G L , et al. An UAV air-combat decision expert system based on receding horizon control[J]. Journal of Beijing University of Aeronautics and Astronautics, 2015, 41 (11): 1994- 1999.
11	付昭旺, 李战武, 强晓明, 等. 基于滚动时域控制的战斗机空战机动决策[J]. 电光与控制, 2013, 20 (3): 20- 24.
	FU Z W , LI Z W , QIANG X M , et al. Tactical decision-making method based on receding horizon control for air combat[J]. Electronics Optics & Control, 2013, 20 (3): 20- 24.
12	黄长强, 赵克新, 韩邦杰, 等. 一种近似动态规划的无人机机动决策方法[J]. 电子与信息学报, 2018, 40 (10): 166- 171.
	HUANG C Q , ZHAO K X , HAN B J , et al. Maneuvering decision-making method of UAV based on approximate dynamic programming[J]. Journal of Electronics & Information Technology, 2018, 40 (10): 166- 171.
13	MCGREW J S , HOW J P , WILLIAMS B , et al. Air-combat strategy using approximate dynamic programming[J]. Journal of Guidance, Control and Dynamics, 2010, 33 (5): 1641- 1654.
14	董慧芬, 王晓丽, 高庆吉. 基于离散化分布密度的无人机粗糙决策方法[J]. 系统工程与电子技术, 2019, 41 (1): 105- 111.
	DONG H F , WANG X L , GAO Q J . Method on rough decision-making for UAV based on discrete distribution density[J]. Systems Engineering and Electronics, 2019, 41 (1): 105- 111.
15	SUTTON R S , BARTO A G . Reinforcement learning: an introduction[M].
16	YANG Q M , ZHANG J D , SHI G Q . Modeling of UAV path planning based on IMM under POMDP framework[J]. Journal of Systems Engineering and Electronics, 2019, 30 (3): 545- 554.
17	SONG H , LIU C C , LAWARREE J , et al. Optimal electricity supply bidding by Markov decision process[J]. IEEE Trans.on Power Systems, 2000, 15 (2): 618- 624.
18	LILLICRAP T P , HUNT J J , PRITZEL A , et al. Continuous control with deep reinforcement learning[J]. Computer Science, 2015, 8 (6): A187.
19	IOFFE S, SZEGEDY C. Batch normalization: accelerating deep network training by reducing internal covariate shift[C]//Proc.of the International Conference on Machine Learning, 2015: 448-456.
20	KINGMA D P, BA J. Adam: a method for stochastic optimization[C]//Proc.of the 3rd International Conference for Learning Representations, 2014.
21	TESAURO G . Temporal difference learning and TD-Gammon[J]. Communications of the ACM, 1995, 38 (3): 58- 68.
22	LECUN Y , BENGIO Y , HINTON G . Deep learning[J]. Nature, 2015, 521 (7553): 436- 444.
23	WATKINS C J C H , DAYAN P . Technical note: Q-learning[J]. Machine Learning, 1992, 8 (3/4): 279- 292.
24	MNIH V , KAVUKCUOGLU K , SILVER D , et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518 (7540): 529.
25	PETERS J , SCHAAL S . Reinforcement learning of motor skills with policy gradients[J]. Neural Networks, 2008, 21 (4): 682- 697.
26	BARNDORFF-NIELSEN O E , SHEPHARD N . Non-Gaussian Ornstein-Uhlenbeck based models and some of their uses in financial economics[J]. Journal of the Royal Statistical Society, 2001, 63 (2): 167- 241.

测试次数	成功完成次数	成功率/%	平均决策时间/ms
5 000	4 390	87.8	0.438

[1]	Bakun ZHU, Weigang ZHU, Wei LI, Ying YANG, Tianhao GAO. Research on decision-making modeling of cognitive jamming for multi-functional radar based on Markov [J]. Systems Engineering and Electronics, 2022, 44(8): 2488-2497.
[2]	Guan WANG, Haizhong RU, Dali ZHANG, Guangcheng MA, Hongwei XIA. Design of intelligent control system for flexible hypersonic vehicle [J]. Systems Engineering and Electronics, 2022, 44(7): 2276-2285.
[3]	Lingyu MENG, Bingli GUO, Wen YANG, Xinwei ZHANG, Zuoqing ZHAO, Shanguo HUANG. Network routing optimization approach based on deep reinforcement learning [J]. Systems Engineering and Electronics, 2022, 44(7): 2311-2318.
[4]	Shihan TAN, Fenglin JIN, Congying DUN. Task assignment strategy for space-air-ground integrated vehicular networks oriented to user demand [J]. Systems Engineering and Electronics, 2022, 44(5): 1717-1727.
[5]	Bakun ZHU, Weigang ZHU, Wei LI, Ying YANG, Tianhao GAO. Multi-function radar intelligent jamming decision method based on prior knowledge [J]. Systems Engineering and Electronics, 2022, 44(12): 3685-3695.
[6]	Qingqing YANG, Yingying GAO, Yu GUO, Boyuan XIA, Kewei YANG. Target search path planning for naval battle field based on deep reinforcement learning [J]. Systems Engineering and Electronics, 2022, 44(11): 3486-3495.
[7]	Ang GAO, Zhiming DONG, Liang LI, Jinghua SONG, Li DUAN. Parallel priority experience replay mechanism of MADDPG algorithm [J]. Systems Engineering and Electronics, 2021, 43(2): 420-433.
[8]	Wen MA, Hui LI, Zhuang WANG, Zhiyong HUANG, Zhaoxin WU, Xiliang CHEN. Close air combat maneuver decision based on deep stochastic game [J]. Systems Engineering and Electronics, 2021, 43(2): 443-451.
[9]	Ang GAO, Qisheng GUO, Zhiming DONG, Shaoqing YANG. Research on efficiency evaluation method of multi unmanned ground vehicle system based on EAS+MADRL [J]. Systems Engineering and Electronics, 2021, 43(12): 3643-3651.
[10]	LAI Zuomei, QIAO Wensheng, GU Bo, WANG Shiyi. Research on sensor cooperative radiation control strategy under task performance constraints [J]. Systems Engineering and Electronics, 2019, 41(8): 1749-1754.
[11]	XIE Hao, GUO Aihuang, SONG Chunlin, JIAO Runze. eNB selection for LTE-V using deep reinforcement learning [J]. Systems Engineering and Electronics, 2019, 41(7): 1652-1657.
[12]	QIAO Chenglin, SHAN Ganlin, DUAN Xiusheng, LIU Xinyi. Scheduling algorithm of active sensors for tracking task requirement [J]. Systems Engineering and Electronics, 2017, 39(11): 2515-2521.
[13]	LI Chenxi, CAO Lei, ZHANG Yongliang, CHEN Xiliang, ZHOU Yuhuan, DUAN Liwen. Knowledge-based deep reinforcement learning: a review [J]. Systems Engineering and Electronics, 2017, 39(11): 2603-2613.
[14]	LIN Xiaohui, TAN Yu， ZHANG Junling, YANG Chao, LIU Jing. MDPbased energy efficient policy for wireless transmission [J]. Systems Engineering and Electronics, 2014, 36(7): 1433-1438.
[15]	SHAN Ganlin， ZHANG Zining. Non-myopic sensor scheduling in a single platform for target tracking [J]. Systems Engineering and Electronics, 2014, 36(3): 458-463.

Autonomous guidance maneuver control and decision-making algorithm

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 26

Related Articles 15

Recommended Articles

Metrics

Comments