基于DQN的旋翼无人机着陆控制算法

doi:10.12305/j.issn.1001-506X.2023.05.21

Abstract

Abstract:

Aiming at the problem of landing control for unmanned aerial vehicle (UAV), a landing control algorithm of rotor UAV based on deep reinforcement learning (DRL) theory is studied. The UAV agent is generated by DRL training, and the action command is given according to the observation results to achieve autonomous landing control. Firstly, based on the random process theory, the landing control problem of rotor UAV is transformed into a Markov decision process (MDP). Secondly, a reward function is designed to consider the horizontal and vertical control processes of UAV respectively, and the landing control problem is transferred to the reinforcement learning framework. Then, the deep Q network (DQN) algorithm is used to solve the reinforcement learning problem, and the landing control agent is obtained through a large number of training. Finally, the effectiveness of the algorithm is verified by a large number of numerical simulations and simulation analysis of landing platforms in various operating conditions.

Key words: deep reinforcement learning (DRL), Markov decision process (MDP), deep Q network (DQN) algorithm, rotor unmanned aerial vehicle, landing control

CLC Number:

V448.2

Jin TANG, Yangang LIANG, Zhihui BAI, Kebo LI. Landing control algorithm of rotor UAV based on DQN[J]. Systems Engineering and Electronics, 2023, 45(5): 1451-1460.

Figures/Tables 18

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Table 1

Fig.7

Table 2

Table 3

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Fig.13

Fig.14

Fig.15

References 32

1	龚有敏. 四旋翼无人机轨迹跟踪与自主着陆控制研究[D]. 哈尔滨: 哈尔滨工业大学, 2017.
	GONG Y M. Research on trajectory tracking and autonomous landing control of four rotor UAV[D]. Harbin: Harbin Institute of Technology, 2017.
2	BELTRAN-CARBAJAL F , YANEZ-BADILLO H , TAPIA-DLVERA R , et al. On active vibration absorption in motion control of a quadrotor UAV[J]. Mathe-matics, 2022, 10 (2): 235.
3	LI X H , FANG C , FAN T . Analysis of reflected signal of quad rotor UAV based on model fitting in mobile communication system[J]. Journal of Systems Engineering and Electronics, 2022, 33 (1): 97- 104. doi: 10.23919/JSEE.2022.000011
4	宁东方. 无人机自动着陆控制系统的设计与实现研究[D]. 西安: 西北工业大学, 2006.
	NING D F. Design and implementation of automatic landing control system for UAV[D]. Xi'an: Northwestern Polytechnical University, 2006.
5	JING X D , WANG X F . PSO algorithm tuning PI_PID controller parameters of quad-rotor UAV[J]. Journal of Physics: Conference Series, 2022, 2228, 012017. doi: 10.1088/1742-6596/2228/1/012017
6	YU Z Q , ZHANG Y M , JINANG B . PID-type fault-tolerant prescribed performance control of fixed-wing UAV[J]. Journal of Systems Engineering and Electronics, 2021, 32 (5): 1053- 1061. doi: 10.23919/JSEE.2021.000090
7	HUANG S B , HUANG J F , CAI Z Q , et al. Adaptive backstepping sliding mode control for quadrotor UAV[J]. Scientific Programming, 2021, 2021, 13.
8	CHEN J , ZHAO H C . Sliding mode disturbance observer and sliding mode controller for quadrotor UAV[J]. Journal of Phy-sics: Conference Series, 2022, 2296, 012030. doi: 10.1088/1742-6596/2296/1/012030
9	KANG B , MIAO Y , LIU F , et al. A second-order sliding mode controller of quad-rotor UAV based on PID sliding mode surface with unbalanced load[J]. Journal of Systems Science & Complexity, 2021, 34, 520- 536.
10	LI Z X , ZHANG R . Time-varying sliding mode control of missile based on suboptimal method[J]. Journal of Systems Engineering and Electronics, 2021, 32 (3): 700- 710. doi: 10.23919/JSEE.2021.000060
11	MNIH V , KAVUKCUOGLU K , SILVER D , et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518 (7540): 529- 533. doi: 10.1038/nature14236
12	LI Y F , YANG B , YAN L , et al. RETRACTED: energy-aware resource management for uplink non-orthogonal multiple access: multi-agent deep reinforcement learning[J]. Future Generation Computer Systems, 2020, 105, 684- 694. doi: 10.1016/j.future.2019.12.047
13	KHAN A , JIANG F , LIU S H , et al. Playing a FPS doom video game with deep visual reinforcement learning[J]. Automatic Control and Computer Sciences, 2019, 53 (3): 214- 222. doi: 10.3103/S0146411619030052
14	PI C H , HU K C , CHENG S , et al. Low-level autonomous control and tracking of quadrotor using reinforcement learning[J]. Control Engineering Practice, 2020, 95, 104222. doi: 10.1016/j.conengprac.2019.104222
15	LI Y , QIU X H , LIU X D , et al. Deep reinforcement learning and its application in autonomous fitting optimization for attack areas of UCAVs[J]. Journal of Systems Engineering and Electronics, 2020, 31 (4): 734- 742. doi: 10.23919/JSEE.2020.000048
16	LIN J , WANG Y N , MIAO Z Q , et al. Low-complexity control for vision-based landing of quadrotor UAV on unknown moving platform[J]. IEEE Trans.on Industrial Informatics, 2022, 18 (8): 5348- 5358. doi: 10.1109/TII.2021.3129486
17	CHANG C W , LO L Y , CHENG H C , et al. Proactive gui-dance for accurate UAV landing on a dynamic platform: a visua -linertial approach[J]. Sensors, 2022, 22 (1): 404. doi: 10.3390/s22010404
18	邱鹏瑞, 刘筠, 刘聪. 四旋翼无人机单目视觉自主着陆系统研究[J]. 云南民族大学学报(自然科学版), 2019, 28 (3): 289- 292.
	QIU P R , LIU J , LIU C . Research on monocular vision auto-nomous landing system of four rotor UAV[J]. Journal of Yunnan Nationalities University, 2019, 28 (3): 289- 292.
19	朱飞翔, 高永, 孟浩. 基于参考轨迹的无人机自主着陆控制系统设计与仿真[J]. 海军航空工程学院学报, 2017, 32 (5): 463- 468.
	ZHU F X , GAO Y , MENG H . Design and simulation of UAV autonomous landing control system based on reference trajectory[J]. Journal of Naval Aeronautical and Astronautical University, 2017, 32 (5): 463- 468.
20	许陈元, 李春涛. 无人机快速着陆控制律设计及仿真验证[J]. 计算机仿真, 2016, 33 (7): 141- 146.
	XU C Y , LI C T . Design and simulation verification of UAV rapid landing control law[J]. Computer Simulation, 2016, 33 (7): 141- 146.
21	JOHNSON J D , LI J , CHEN Z . Reinforcement learning: an introduction[J]. Neurocomputing, 2000, 35 (1-4): 205- 206. doi: 10.1016/S0925-2312(00)00324-6
22	LOU W J , GUO X . Adaptive trajectory tracking control using reinforcement learning for quadrotor[J]. International Journal of Advanced Robotic Systems, 2016, 13, 679- 709.
23	宋欣屿, 王英勋, 蔡志浩, 等. 基于深度强化学习的无人机着陆轨迹跟踪控制[J]. 航空科学技术, 2020, 31 (1): 68- 75.
	SONG X Y , WANG Y X , CAI Z H , et al. UAV landing trajectory tracking control based on deep reinforcement learning[J]. Aeronautical Science & Technology, 2020, 31 (1): 68- 75.
24	裴培, 何绍溟, 王江, 等. 一种深度强化学习制导控制一体化算法[J]. 宇航学报, 2021, 42 (10): 1293- 1304.
	PEI P , HE S M , WANG J , et al. An integrated algorithm of deep reinforcement learning guidance and control[J]. Journal of Astronautics, 2021, 42 (10): 1293- 1304.
25	ZHANG Y J , HUANG Y J , LIANG K , et al. High-precision modeling and collision simulation of small rotor UAV[J]. Aero-space Science and Technology, 2021, 118, 106977.
26	KIENINGER S , DONATI L , KELLER B G . Dynamical reweighting methods for Markov models[J]. Current Opinion in Structural Biology, 2020, 61, 124- 131.
27	LIU Y , CHONG E , PEZESHKI A , et al. Submodular optimization problems and greedy strategies: a survey[J]. Discrete Event Dynamic Systems, 2020, 30, 381- 412.
28	DOGRU O , CHIPLUNKAR R , HUANG B . Reinforcement learning with constrained uncertain reward function through particle filtering[J]. IEEE Trans.on Industrial Electronics, 2022, 69 (7): 7491- 7499.
29	GAO X , FANG Y W , WU Y L . Fuzzy Q learning algorithm for dual-aircraft path planning to cooperatively detect targets by passive radars[J]. Journal of Systems Engineering and Electronics, 2013, 24 (5): 800- 810.
30	GUO J A , WANG Y L , AN H , et al. ⅡDQN: an incentive improved DQN algorithm in EBSN recommender system[J]. Security & Communication Networks, 2022, 2022, 7502248.
31	SU J J , MA C H , LI S , et al. An AGC dynamic control method based on DQN algorithm[J]. IOP Conference Series: Materials Science and Engineering, 2020, 729 (1): 012009.
32	MANIATOPOULOS A , MITIANOUDIS N . Learnable leaky ReLU (LeLeLU): an alternative accuracy-optimized activation function[J]. Information, 2021, 12 (12): 513.

参数	水平方向	垂直方向
经验回放池大小	10⁶	10⁴
目标网络更新步长	10	10
仿真时长/s	120	120
控制周期/s	0.2	0.2
折扣因子	0.99	0.99
学习率	0.000 1	0.000 1
贪婪度	0.9	0.9
训练回合数	1 000	500
批量大小	128	48

参数	无人机	目标小车
初始位置/m	[0, 0, 0]	[0, 20, 0]
初始速度/(m/s)	[0, 0, 0]	[5, 5, 0]

[1]	Jiaqi BAI, Yankai WANG, Hao XING. Fixed-time heterogeneous formation control of unmanned boats and quadrotor unmanned aerial vehicle [J]. Systems Engineering and Electronics, 2023, 45(4): 1152-1163.
[2]	Bakun ZHU, Weigang ZHU, Wei LI, Ying YANG, Tianhao GAO. Multi-function radar intelligent jamming decision method based on prior knowledge [J]. Systems Engineering and Electronics, 2022, 44(12): 3685-3695.
[3]	Wenqi YANG, Jianhua LU, Xu JIANG, Yuanxin WANG. Design of quadrotor attitude active disturbance rejection controller based on improved ESO [J]. Systems Engineering and Electronics, 2022, 44(12): 3792-3799.
[4]	XIE Hao, GUO Aihuang, SONG Chunlin, JIAO Runze. eNB selection for LTE-V using deep reinforcement learning [J]. Systems Engineering and Electronics, 2019, 41(7): 1652-1657.
[5]	WU Wenhai, ZHANG Yang, HU Yun’an, ZHANG Yuanyuan. Research development in nonlinear backstepping control method of carrier-based aircraft landing [J]. Systems Engineering and Electronics, 2018, 40(7): 1578-1587.
[6]	SHEN Zhipeng, CAO Xiaoming. Extended state observer based dynamic surface output feedback control for quadrotor UAV trajectory tracking with input constraints#br# [J]. Systems Engineering and Electronics, 2018, 40(12): 2766-2774.
[7]	ZHU Wei, MA Weiming, YANG Xidang, XIAO Huan. Tracking control of ship borne quad-roto UAV based on robust fault diagnosis LPV model [J]. Systems Engineering and Electronics, 2018, 40(11): 2540-.
[8]	WU Zhenglong1, LI Jie1, GUAN Zhenyu2, NIU Sanku1, YANG Chengwei1, LI Yang3. Optical flowbased autonomous landing control for fixedwing small UAV [J]. Systems Engineering and Electronics, 2016, 38(12): 2827-2834.

Landing control algorithm of rotor UAV based on DQN

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 18

References 32

Related Articles 8

Recommended Articles

Metrics

Comments

参数	神经元数量	激活函数
fc1	32	ReLU
fc2	32	ReLU
fc3	32	ReLU
fc4	1	-
fc5	32	ReLU