基于强化学习的变形飞行器抗扰补偿控制方法

doi:10.12305/j.issn.1001-506X.2025.12.25

Abstract

Abstract:

Aiming at the attitude control problem of morphing aircraft with internal model uncertainty and external multi-source disturbance in the morphing process, reinforcement learning algorithm is combined with active disturbance rejection controller to adapt to the aircraft shape change and combat the internal and external disturbance of the system through training, which is an optimal compensation control strategy of disturbance rejection compensation based on reinforcement learning （RL-DRCC）. In the experiment, RL-DRCC is used for simulation, and the control effect before compensation is compared. The results verify the superiority of the proposed control strategy, and the disturbace rejection compensation controller based on the dual delay depth deterministic policy gradient algorithm has the best effect. The output jitter of the control quantity under disturbance is effectively suppressed, the overall tracking accuracy of the attitude angle is improved, and the robustness of the attitude control in complex environments is promoted. Finally, random morphing commands and disturbance conditions are verified. The results show that the proposed method can effectively deal with various morphing commands and complex unknown conditions, and has good generalization and adaptive ability.

Key words: deep reinforcement learning, morphing aircraft, multi-source disturbances, compensation control

CLC Number:

V 249.1

Yang LIU, Fanyi MENG, Gang CHEN. Reinforcement learning based disturbance rejection compensation control method for morphing aircraft[J]. Systems Engineering and Electronics, 2025, 47(12): 4130-4142.

Figures/Tables 23

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Table 1

Table 2

Table 3

Fig.7

Table 4

Fig.8

Fig.9

Table 5

Fig.10

Fig.11

Fig.12

Table 6

Table 7

Fig.13

Fig.14

Fig.15

Table 8

References 31

1	WEISSHAAR T A. Morphing aircraft systems: historical perspectives and future challenges[J]. Journal of Aircraft, 2013, 50 (2): 337- 353. doi: 10.2514/1.C031456
2	AFONSO F, VALE J, LAU F, et al. Performance based multidisciplinary design optimization of morphing aircraft[J]. Aerospace Science and Technology, 2017, 67, 1- 12. doi: 10.1016/j.ast.2017.03.029
3	AJAJ R M, FRISWELL M I, BOURCHAK M, et al. Span morphing using the GNATSpar wing[J]. Aerospace Science and Technology, 2016, 53, 38- 46. doi: 10.1016/j.ast.2016.03.009
4	周雨欣, 王鹏, 汤国建, 等. 基于干扰观测器的变形飞行器预设性能控制[J]. 战术导弹技术, 2024 (4): 72- 82.
	ZHOU Y X, WANG P, TANG G J, et al. Disturbance observer-based prescribed performance control for morphing aircraft[J]. Tactical Missile Technology, 2024 (4): 72- 82.
5	WU Q, LIU Z H, LIU F N, et al. LPV-based self-adaption integral sliding mode controller with L₂ gain performance for a morphing aircraft[J]. IEEE Access, 2019, 7, 81515- 81531. doi: 10.1109/ACCESS.2019.2923313
6	YUE T, ZHANG X Y, WANG L X, et al. Flight dynamic modeling and control for a telescopic wing morphing aircraft via asymmetric wing morphing[J]. Aerospace Science and Technology, 2017, 70, 328- 338. doi: 10.1016/j.ast.2017.08.013
7	李珂澄, 刘小雄, 李煜, 等. 基于自抗扰控制的变体飞机机动控制研究[J]. 西北工业大学学报, 2024, 42 (4): 662- 672. doi: 10.1051/jnwpu/20244240662
	LI K C, LIU X X, LI Y, et al. Research on morphing aircraft maneuver control based on active disturbance rejection control[J]. Journal of Northwestern Polytechnical University, 2024, 42 (4): 662- 672. doi: 10.1051/jnwpu/20244240662
8	孟志鹏, 杨柳庆, 王波, 等. 基于改进平衡优化算法的折叠翼飞行器自抗扰控制器设计[J]. 北京航空航天大学学报, 2024, 50 (8): 2449- 2460.
	MENG Z P, YANG L Q, WANG B, et al. ADRC design for folding wing vehicles based on improved equilibrium optimization algorithm[J]. Journal of Beijing University of Aeronautics and Astronautics, 2024, 50 (8): 2449- 2460.
9	宋慧心, 金磊. 折叠翼飞行器的动力学建模与稳定控制[J]. 力学学报, 2020, 52 (6): 1548- 1559.
	SONG H X, JIN L. Dynamic modeling and stability control of folding wing aircraft[J]. Chinese Journal of Theoretical and Applied Mechanics, 2020, 52 (6): 1548- 1559.
10	MENG F Y, WANG T J, CHEN G. Prescribed performance-based active anti-disturbance backstepping control for morphing aircraft[J]. Aerospace Science and Technology, 2024, 152, 109386. doi: 10.1016/j.ast.2024.109386
11	PU J L, ZHANG Y H, GUAN Y Z, et al. Recurrent neural network-based predefined time control for morphing aircraft with asymmetric time-varying constraints[J]. Applied Mathematical Modelling, 2024, 135, 578- 600. doi: 10.1016/j.apm.2024.06.024
12	浦甲伦, 詹韬, 李博皓, 等. 助推-滑翔飞行器再入过程强化学习自抗扰控制[J]. 战术导弹技术, 2024 (2): 117- 125.
	PU J L, ZHAN T, LI B H, et al. Reinforcement learning-based active disturbance rejection control for boost-glide vehicle in re-entry phase[J]. Tactical Missile Technology, 2024 (2): 117- 125.
13	何海洋, 赵振根, 孔飞. 基于深度强化学习的固定翼无人机纵向控制[EB/OL]. [2024-11-09]. https://doi.org/10.13700/j.bh.1001-5965.2024.0075
	HE H Y, ZHAO Z G, KONG F, Longitudinal control of fixed-wing UAV based on deep reinforcement learning[EB/OL]. [2024-11-09]. https://doi.org/10.13700/j.bh.1001-5965.2024.0075.
14	黄旭, 柳嘉润, 贾晨辉, 等. 强化学习控制方法及在类火箭飞行器上的应用[J]. 宇航学报, 2023, 44 (5): 708- 718.
	HUANG X, LIU J R, JIA C H, et al. Reinforcement learning control and its application on rocket-like vehicle[J]. Journal of Astronautics, 2023, 44 (5): 708- 718.
15	LIU Y C, HUANG C Y. DDPG-based adaptive robust tracking control for aerial manipulators with decoupling approach[J]. IEEE Trans. on Cybernetics, 2021, 52 (8): 8258- 8271.
16	WU Z H, LU J C, ZHOU Q, et al. Modified adaptive neural dynamic surface control for morphing aircraft with input and output constraints[J]. Nonlinear Dynamics, 2017, 87 (4): 2367- 2383. doi: 10.1007/s11071-016-3196-0
17	WU Z H, LU J C, SHI J P, et al. Robust adaptive neural control of morphing aircraft with prescribed performance[EB/OL]. [2024-11-09]. https://doi.org/10.1155/2017/1401427.
18	路坤锋, 贾晨辉, 黄旭, 等. 面向变构型飞行器的强化学习位置姿态一体化控制方法[J]. 宇航学报, 2024, 45 (7): 1100- 1110.
	LU K F, JIA C H, HUANG X, et al. Reinforcement learning-based integrated position and attitude control method towards morphing flight vehicles[J]. Journal of Astronautics, 2024, 45 (7): 1100- 1110.
19	丁溶, 曹承钰, 李繁飙, 等. 基于深度强化学习的变外形飞行器姿态控制[J]. 航天控制, 2024, 42 (2): 55- 61.
	DING R, CAO C Y, LI F B, et al. Attitude control of morphing vehicle based on reinforcement learning[J]. Aerospace Control, 2024, 42 (2): 55- 61.
20	PI C H, YE W Y, CHENG S. Robust quadrotor control through reinforcement learning with disturbance compensation[J]. Applied Sciences, 2021, 11 (7): 3257. doi: 10.3390/app11073257
21	马少捷, 惠俊鹏, 王宇航, 等. 变形飞行器深度强化学习姿态控制方法研究[J]. 航天控制, 2022, 40 (6): 3- 10.
	MA S J, HUI J P, WANG Y H, et al. Research on attitude method of morphing aircraft based on deep reinforcement learning[J]. Aerospace Control, 2022, 40 (6): 3- 10.
22	ZHENG Y M, TAO J, SUN Q L, et al. Soft actor-critic based active disturbance rejection path following control for unmanned surface vessel under wind and wave disturbances[J]. Ocean Engineering, 2022, 247, 110631. doi: 10.1016/j.oceaneng.2022.110631
23	王思鹏, 杜昌平, 郑耀. 基于强化学习的扑翼飞行器路径规划算法[J]. 控制与决策, 2022, 37 (4): 851- 860.
	WANG S P, DU C P, ZHENG Y. Path planning algorithm for flapping-wing aircraft based on reinforcement learning[J]. Control and Decision, 2022, 37 (4): 851- 860.
24	JIN H Y, GAO Z Q. On the notions of normality, locality, and operational stability in ADRC[J]. Control Theory and Technology, 2023, 21, 97- 109. doi: 10.1007/s11768-023-00131-4
25	韩京清. 自抗扰控制技术——估计补偿不确定因素的控制技术[M]. 北京: 国防工业出版社, 2008.
	HAN J Q. Active disturbance rejection control technique—the technique for estimating and compensating the uncertainties[M]. Beijing: National Defense Industry Press, 2008.
26	韩京清. 自抗扰控制器及其应用[J]. 控制与决策, 1998 (1): 19- 23.
	HAN J Q. Active disturbance rejection controller and its application[J]. Control and Decision, 1998 (1): 19- 23.
27	LILLICRAP T P. Continuous control with deep reinforcement learning[EB/OL]. [2024-11-09]. https://arxiv.org/abs/1509.02971.
28	FUJIMOTO S, HOOF H, MEGER D. Addressing function approximation error in actor-critic methods[C]//Proc. of the 35th International Conference on Machine Learning, 2018: 1587−1596.
29	LIU J H, SHAN J Y, WANG J N, et al. Incremental sliding-mode control and allocation for morphing-wing aircraft fast manoeuvring[J]. Aerospace Science and Technology, 2022, 131, 107959. doi: 10.1016/j.ast.2022.107959
30	HAARNOJA T, ZHOU A, HARTIKAINEN K, et al. Soft actor-critic algorithms and applications[EB/OL]. [2024-11-14]. http://arxiv.org/abs/1812.05905.
31	HAARNOJA T, ZHOU A, ABBEEL P, et al. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]//Proc. of the 35th International Conference on Machine Learning, 2018: 1861−1870.

扰动参数	拉偏范围
大气密度	±10%
风向角	45°, 135°, 225°, 315°
质心位置${X}_{c},{Y}_{c} $	±3%
绕机体系各轴的转动惯量	±10%
机体系各平面的惯性积	±20%
滚转力矩系数	±20%
偏航力矩系数	±20%
俯仰力矩系数	±20%
气动舵伺服稳态增益	±5%
气动舵伺服时间常数	±12.5%
气动舵伺服自然频率1	±8 rad/s
气动舵伺服阻尼比1	±2%
气动舵伺服自然频率2	±10 rad/s
气动舵伺服阻尼比2	±15%

训练超参数	数值	训练超参数	数值
训练回合数	500	策略网络延迟更新步数	4
单回合最大步长	1500	软更新速率	0.001
评价网络学习率	0.003	探索噪声标准差	0.5
策略网络学习率	0.0005	噪声衰减率	0.000001
折扣因子$ \gamma $	0.99	平滑噪声标准差	0.2
经验池大小	1000000	批次样本大小	128

权重参数	数值
$ {k}_{1}, {k}_{2}, {k}_{3} $	−1, −2, −3
$ {w}_{1}, {w}_{2}, {w}_{3}, {w}_{4}, {w}_{5} $	−10, −10, 5, 2, 10
$ {e}_{1}, {e}_{2}, {e}_{3}, {e}_{4}, {e}_{5} $	1.5°, 5°, 0.05°, 0.05°, 0.05°
$ {\sigma }_{1}, {\sigma }_{2}, {\sigma }_{3} $	−0.1, −0.1, −0.2
$ {\lambda }_{1}, {\lambda }_{2}, {\lambda }_{3} $	0, −0.5, 0

参数项	取值
$ {\beta }_{0}\left(\theta \right), {\beta }_{1}\left(\theta \right), {\beta }_{2}\left(\theta \right) $	100, 300, 500
$ {\beta }_{0}\left(\varphi \right), {\beta }_{1}\left(\varphi \right), {\beta }_{2}\left(\varphi \right) $	100, 300, 500
$ {\beta }_{0}\left(\psi \right), {\beta }_{1}\left(\psi \right), {\beta }_{2}\left(\psi \right) $	100, 300, 500
$ {c}_{1},{c}_{2},{c}_{3} $	1, 1, 1
$ {h}_{1},{h}_{2},{h}_{3} $	0.1, 0.1, 0.1

参数项	取值
$ {\beta }_{0}\left(\theta \right),{\beta }_{1}\left(\theta \right),{\beta }_{2}\left(\theta \right) $	100,500,1300
$ {\beta }_{0}\left(\phi \right),{\beta }_{1}\left(\phi \right),{\beta }_{2}\left(\phi \right) $	100,800,1300
$ {\beta }_{0}\left(\psi \right),{\beta }_{1}\left(\psi \right),{\beta }_{2}\left(\psi \right) $	100,1300,5900
$ {c}_{1},{c}_{2},{c}_{3} $	8,1,1
$ {h}_{1},{h}_{2},{h}_{3} $	0.1,0.2,0.3

Reinforcement learning based disturbance rejection compensation control method for morphing aircraft

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 23

References 31

Related Articles 15

Recommended Articles

Metrics

Comments

参数		TD3-DRCC	SAC-DRCC	抗扰基础控制器
$ \overline{{\mathrm{IAE}}}/\left(\right(^{\circ})\cdot {\mathrm{s}}) $	$ \theta $	0.4186	0.5489	0.5908
	$ \varphi $	0.2060	0.2858	0.4596
	$ \psi $	0.4799	0.4963	0.5116
$ \overline{{\mathrm{ISE}}}/\left(\right(^{\circ}{)}^{2}\cdot {\mathrm{s}}) $	$ \theta $	1.5334	1.5429	2.0372
	$ \varphi $	0.1635	0.3324	0.7113
	$ \psi $	0.5643	0.5841	0.6034

[1]	Xiaolong WEI, Yarong WU, Dengkai YAO, Guhao ZHAO. Hierarchical decision-making algorithm for UAV air combat maneuvering based on deep reinforcement learning [J]. Systems Engineering and Electronics, 2025, 47(9): 2993-3003.
[2]	Yundou ZHU, Haiquan SUN, Xiaoxuan HU. Multi-satellite cooperative imaging task planning method based on pointer network architecture [J]. Systems Engineering and Electronics, 2025, 47(7): 2246-2255.
[3]	Linzhi MENG, Xiaojuan SUN, Yuxin HU, Bin GAO, Guoqing SUN, Wenhao MU. Reinforcement learning task scheduling algorithm for satellite on-orbit processing [J]. Systems Engineering and Electronics, 2025, 47(6): 1917-1929.
[4]	Kangjie ZHENG, Xinyu ZHANG, Weisong WANG, Zhensheng LIU. Intelligent ship dynamic autonomous obstacle avoidance decision based on DQN and rule [J]. Systems Engineering and Electronics, 2025, 47(6): 1994-2001.
[5]	Shuhan LIU, Tong LI, Fuqiang LI, Chungang YANG. Intent and situation-dual driven anti-jamming communication mechanism for data link [J]. Systems Engineering and Electronics, 2025, 47(6): 2055-2064.
[6]	Wei XIONG, Dong ZHANG, Zhi REN, Shuheng YANG. Research on intelligent decision-making methods for coordinated attack by manned aerial vehicles and unmanned aerial vehicles [J]. Systems Engineering and Electronics, 2025, 47(4): 1285-1299.
[7]	Peng MA, Rui JIANG, Bin WANG, Mengfei XU, Changbo HOU. Strategy reconstruction for resilience against intelligence jamming based on implicit opponent modeling [J]. Systems Engineering and Electronics, 2025, 47(4): 1355-1363.
[8]	Kaiqiang TANG, Huiqiao FU, Jiasheng LIU, Guizhou DENG, Chunlin CHEN. Hierarchical optimization research of constrained vehicle routing based on deep reinforcement learning [J]. Systems Engineering and Electronics, 2025, 47(3): 827-841.
[9]	Xiarong CHEN, Jichao LI, Gang CHEN, Peng LIU, Jiang JIANG. Portfolio of weapon system-of-systems based on heterogeneous information networks [J]. Systems Engineering and Electronics, 2025, 47(3): 855-861.
[10]	Yaozhong ZHANG, Zhuoran WU, Jiandong ZHANG, Qiming YANG, Guoqing SHI, Zixiang XU. UAV many-to-one pursuit-evasion game based on ME-DDPG algorithm [J]. Systems Engineering and Electronics, 2025, 47(10): 3288-3299.
[11]	Dong SUI, Xiangrong CAI. Continual learning mechanism for intelligent flight conflict resolution algorithm [J]. Systems Engineering and Electronics, 2025, 47(10): 3300-3312.
[12]	Tingyu ZHANG, Ying ZENG, Nan LI, Hongzhong HUANG. Spacecraft power-signal composite network optimization algorithm based on DRL [J]. Systems Engineering and Electronics, 2024, 46(9): 3060-3069.
[13]	Yuqi XIA, Yanyan HUANG, Qia CHEN. Path planning for unmanned vehicle reconnaissance based on deep Q-network [J]. Systems Engineering and Electronics, 2024, 46(9): 3070-3081.
[14]	Zhipeng YANG, Zihao CHEN, Chang ZENG, Song LIN, Jindi MAO, Kai ZHANG. Online route planning decision-making method of aircraft in complex environment [J]. Systems Engineering and Electronics, 2024, 46(9): 3166-3175.
[15]	Hongda GUO, Jingtao LOU, Youchun XU, Peng YE, Yongle LI, Jinsheng CHEN. Event-triggered communication of multiple unmanned ground vehicles collaborative based on MADDPG [J]. Systems Engineering and Electronics, 2024, 46(7): 2525-2533.

参数		TD3-DRCC	SAC-DRCC	抗扰基础控制器
$ \overline{{\mathrm{IAE}}}/\left(\right(^{\circ})\cdot {\mathrm{s}}) $	$ \theta $	0.3642	0.4594	0.7171
	$ \varphi $	0.0552	0.0734	0.0724
	$ \psi $	0.3529	0.3606	0.4024
$ \overline{{\mathrm{ISE}}}/\left(\right(^{\circ}{)}^{2}\cdot {\mathrm{s}}) $	$ \theta $	1.1952	1.2355	1.7970
	$ \varphi $	0.0171	0.0106	0.0127
	$ \psi $	0.3176	0.3154	0.3624