基于强化学习的变形飞行器抗扰补偿控制方法

doi:10.12305/j.issn.1001-506X.2025.12.25

摘要/Abstract

摘要：

针对变形飞行器变形过程存在内部模型不确定性和外部多源扰动的姿态控制问题，将强化学习算法结合主动抗扰控制器，通过训练以适应飞行器外形变化及对抗系统内外扰动的最优补偿控制策略，即一种基于强化学习的抗扰补偿控制（disturbance rejection compensation control based on reinforcement learning, RL-DRCC）策略。实验中用RL-DRCC进行仿真，结合补偿前的控制效果进行对比实验，验证了所提控制策略的优越性，其中基于双延迟深度确定性策略梯度算法的抗扰补偿控制器效果最好，干扰下的控制量输出抖动被有效抑制，姿态角跟踪精度整体提高，复杂环境下姿态控制的鲁棒性有所提升。最后，对随机变形指令及扰动工况进行验证，结果表明所提方法能有效应对各种变形指令和复杂未知工况，具备很好的泛化性和自适应能力。

关键词: 深度强化学习, 变形飞行器, 多源扰动, 补偿控制

Abstract:

Aiming at the attitude control problem of morphing aircraft with internal model uncertainty and external multi-source disturbance in the morphing process, reinforcement learning algorithm is combined with active disturbance rejection controller to adapt to the aircraft shape change and combat the internal and external disturbance of the system through training, which is an optimal compensation control strategy of disturbance rejection compensation based on reinforcement learning （RL-DRCC）. In the experiment, RL-DRCC is used for simulation, and the control effect before compensation is compared. The results verify the superiority of the proposed control strategy, and the disturbace rejection compensation controller based on the dual delay depth deterministic policy gradient algorithm has the best effect. The output jitter of the control quantity under disturbance is effectively suppressed, the overall tracking accuracy of the attitude angle is improved, and the robustness of the attitude control in complex environments is promoted. Finally, random morphing commands and disturbance conditions are verified. The results show that the proposed method can effectively deal with various morphing commands and complex unknown conditions, and has good generalization and adaptive ability.

Key words: deep reinforcement learning, morphing aircraft, multi-source disturbances, compensation control

中图分类号:

V 249.1

刘洋, 孟凡一, 陈刚. 基于强化学习的变形飞行器抗扰补偿控制方法[J]. 系统工程与电子技术, 2025, 47(12): 4130-4142.

Yang LIU, Fanyi MENG, Gang CHEN. Reinforcement learning based disturbance rejection compensation control method for morphing aircraft[J]. Systems Engineering and Electronics, 2025, 47(12): 4130-4142.

图/表 23

图1

图2

图3

图4

图5

图6

表1

表2

表3

图7

表4

图8

图9

表5

图10

图11

图12

表6

表7

图13

图14

图15

表8

参考文献 31

1	WEISSHAAR T A. Morphing aircraft systems: historical perspectives and future challenges[J]. Journal of Aircraft, 2013, 50 (2): 337- 353. doi: 10.2514/1.C031456
2	AFONSO F, VALE J, LAU F, et al. Performance based multidisciplinary design optimization of morphing aircraft[J]. Aerospace Science and Technology, 2017, 67, 1- 12. doi: 10.1016/j.ast.2017.03.029
3	AJAJ R M, FRISWELL M I, BOURCHAK M, et al. Span morphing using the GNATSpar wing[J]. Aerospace Science and Technology, 2016, 53, 38- 46. doi: 10.1016/j.ast.2016.03.009
4	周雨欣, 王鹏, 汤国建, 等. 基于干扰观测器的变形飞行器预设性能控制[J]. 战术导弹技术, 2024 (4): 72- 82.
	ZHOU Y X, WANG P, TANG G J, et al. Disturbance observer-based prescribed performance control for morphing aircraft[J]. Tactical Missile Technology, 2024 (4): 72- 82.
5	WU Q, LIU Z H, LIU F N, et al. LPV-based self-adaption integral sliding mode controller with L₂ gain performance for a morphing aircraft[J]. IEEE Access, 2019, 7, 81515- 81531. doi: 10.1109/ACCESS.2019.2923313
6	YUE T, ZHANG X Y, WANG L X, et al. Flight dynamic modeling and control for a telescopic wing morphing aircraft via asymmetric wing morphing[J]. Aerospace Science and Technology, 2017, 70, 328- 338. doi: 10.1016/j.ast.2017.08.013
7	李珂澄, 刘小雄, 李煜, 等. 基于自抗扰控制的变体飞机机动控制研究[J]. 西北工业大学学报, 2024, 42 (4): 662- 672. doi: 10.1051/jnwpu/20244240662
	LI K C, LIU X X, LI Y, et al. Research on morphing aircraft maneuver control based on active disturbance rejection control[J]. Journal of Northwestern Polytechnical University, 2024, 42 (4): 662- 672. doi: 10.1051/jnwpu/20244240662
8	孟志鹏, 杨柳庆, 王波, 等. 基于改进平衡优化算法的折叠翼飞行器自抗扰控制器设计[J]. 北京航空航天大学学报, 2024, 50 (8): 2449- 2460.
	MENG Z P, YANG L Q, WANG B, et al. ADRC design for folding wing vehicles based on improved equilibrium optimization algorithm[J]. Journal of Beijing University of Aeronautics and Astronautics, 2024, 50 (8): 2449- 2460.
9	宋慧心, 金磊. 折叠翼飞行器的动力学建模与稳定控制[J]. 力学学报, 2020, 52 (6): 1548- 1559.
	SONG H X, JIN L. Dynamic modeling and stability control of folding wing aircraft[J]. Chinese Journal of Theoretical and Applied Mechanics, 2020, 52 (6): 1548- 1559.
10	MENG F Y, WANG T J, CHEN G. Prescribed performance-based active anti-disturbance backstepping control for morphing aircraft[J]. Aerospace Science and Technology, 2024, 152, 109386. doi: 10.1016/j.ast.2024.109386
11	PU J L, ZHANG Y H, GUAN Y Z, et al. Recurrent neural network-based predefined time control for morphing aircraft with asymmetric time-varying constraints[J]. Applied Mathematical Modelling, 2024, 135, 578- 600. doi: 10.1016/j.apm.2024.06.024
12	浦甲伦, 詹韬, 李博皓, 等. 助推-滑翔飞行器再入过程强化学习自抗扰控制[J]. 战术导弹技术, 2024 (2): 117- 125.
	PU J L, ZHAN T, LI B H, et al. Reinforcement learning-based active disturbance rejection control for boost-glide vehicle in re-entry phase[J]. Tactical Missile Technology, 2024 (2): 117- 125.
13	何海洋, 赵振根, 孔飞. 基于深度强化学习的固定翼无人机纵向控制[EB/OL]. [2024-11-09]. https://doi.org/10.13700/j.bh.1001-5965.2024.0075
	HE H Y, ZHAO Z G, KONG F, Longitudinal control of fixed-wing UAV based on deep reinforcement learning[EB/OL]. [2024-11-09]. https://doi.org/10.13700/j.bh.1001-5965.2024.0075.
14	黄旭, 柳嘉润, 贾晨辉, 等. 强化学习控制方法及在类火箭飞行器上的应用[J]. 宇航学报, 2023, 44 (5): 708- 718.
	HUANG X, LIU J R, JIA C H, et al. Reinforcement learning control and its application on rocket-like vehicle[J]. Journal of Astronautics, 2023, 44 (5): 708- 718.
15	LIU Y C, HUANG C Y. DDPG-based adaptive robust tracking control for aerial manipulators with decoupling approach[J]. IEEE Trans. on Cybernetics, 2021, 52 (8): 8258- 8271.
16	WU Z H, LU J C, ZHOU Q, et al. Modified adaptive neural dynamic surface control for morphing aircraft with input and output constraints[J]. Nonlinear Dynamics, 2017, 87 (4): 2367- 2383. doi: 10.1007/s11071-016-3196-0
17	WU Z H, LU J C, SHI J P, et al. Robust adaptive neural control of morphing aircraft with prescribed performance[EB/OL]. [2024-11-09]. https://doi.org/10.1155/2017/1401427.
18	路坤锋, 贾晨辉, 黄旭, 等. 面向变构型飞行器的强化学习位置姿态一体化控制方法[J]. 宇航学报, 2024, 45 (7): 1100- 1110.
	LU K F, JIA C H, HUANG X, et al. Reinforcement learning-based integrated position and attitude control method towards morphing flight vehicles[J]. Journal of Astronautics, 2024, 45 (7): 1100- 1110.
19	丁溶, 曹承钰, 李繁飙, 等. 基于深度强化学习的变外形飞行器姿态控制[J]. 航天控制, 2024, 42 (2): 55- 61.
	DING R, CAO C Y, LI F B, et al. Attitude control of morphing vehicle based on reinforcement learning[J]. Aerospace Control, 2024, 42 (2): 55- 61.
20	PI C H, YE W Y, CHENG S. Robust quadrotor control through reinforcement learning with disturbance compensation[J]. Applied Sciences, 2021, 11 (7): 3257. doi: 10.3390/app11073257
21	马少捷, 惠俊鹏, 王宇航, 等. 变形飞行器深度强化学习姿态控制方法研究[J]. 航天控制, 2022, 40 (6): 3- 10.
	MA S J, HUI J P, WANG Y H, et al. Research on attitude method of morphing aircraft based on deep reinforcement learning[J]. Aerospace Control, 2022, 40 (6): 3- 10.
22	ZHENG Y M, TAO J, SUN Q L, et al. Soft actor-critic based active disturbance rejection path following control for unmanned surface vessel under wind and wave disturbances[J]. Ocean Engineering, 2022, 247, 110631. doi: 10.1016/j.oceaneng.2022.110631
23	王思鹏, 杜昌平, 郑耀. 基于强化学习的扑翼飞行器路径规划算法[J]. 控制与决策, 2022, 37 (4): 851- 860.
	WANG S P, DU C P, ZHENG Y. Path planning algorithm for flapping-wing aircraft based on reinforcement learning[J]. Control and Decision, 2022, 37 (4): 851- 860.
24	JIN H Y, GAO Z Q. On the notions of normality, locality, and operational stability in ADRC[J]. Control Theory and Technology, 2023, 21, 97- 109. doi: 10.1007/s11768-023-00131-4
25	韩京清. 自抗扰控制技术——估计补偿不确定因素的控制技术[M]. 北京: 国防工业出版社, 2008.
	HAN J Q. Active disturbance rejection control technique—the technique for estimating and compensating the uncertainties[M]. Beijing: National Defense Industry Press, 2008.
26	韩京清. 自抗扰控制器及其应用[J]. 控制与决策, 1998 (1): 19- 23.
	HAN J Q. Active disturbance rejection controller and its application[J]. Control and Decision, 1998 (1): 19- 23.
27	LILLICRAP T P. Continuous control with deep reinforcement learning[EB/OL]. [2024-11-09]. https://arxiv.org/abs/1509.02971.
28	FUJIMOTO S, HOOF H, MEGER D. Addressing function approximation error in actor-critic methods[C]//Proc. of the 35th International Conference on Machine Learning, 2018: 1587−1596.
29	LIU J H, SHAN J Y, WANG J N, et al. Incremental sliding-mode control and allocation for morphing-wing aircraft fast manoeuvring[J]. Aerospace Science and Technology, 2022, 131, 107959. doi: 10.1016/j.ast.2022.107959
30	HAARNOJA T, ZHOU A, HARTIKAINEN K, et al. Soft actor-critic algorithms and applications[EB/OL]. [2024-11-14]. http://arxiv.org/abs/1812.05905.
31	HAARNOJA T, ZHOU A, ABBEEL P, et al. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]//Proc. of the 35th International Conference on Machine Learning, 2018: 1861−1870.

扰动参数	拉偏范围
大气密度	±10%
风向角	45°, 135°, 225°, 315°
质心位置${X}_{c},{Y}_{c} $	±3%
绕机体系各轴的转动惯量	±10%
机体系各平面的惯性积	±20%
滚转力矩系数	±20%
偏航力矩系数	±20%
俯仰力矩系数	±20%
气动舵伺服稳态增益	±5%
气动舵伺服时间常数	±12.5%
气动舵伺服自然频率1	±8 rad/s
气动舵伺服阻尼比1	±2%
气动舵伺服自然频率2	±10 rad/s
气动舵伺服阻尼比2	±15%

训练超参数	数值	训练超参数	数值
训练回合数	500	策略网络延迟更新步数	4
单回合最大步长	1500	软更新速率	0.001
评价网络学习率	0.003	探索噪声标准差	0.5
策略网络学习率	0.0005	噪声衰减率	0.000001
折扣因子$ \gamma $	0.99	平滑噪声标准差	0.2
经验池大小	1000000	批次样本大小	128

权重参数	数值
$ {k}_{1}, {k}_{2}, {k}_{3} $	−1, −2, −3
$ {w}_{1}, {w}_{2}, {w}_{3}, {w}_{4}, {w}_{5} $	−10, −10, 5, 2, 10
$ {e}_{1}, {e}_{2}, {e}_{3}, {e}_{4}, {e}_{5} $	1.5°, 5°, 0.05°, 0.05°, 0.05°
$ {\sigma }_{1}, {\sigma }_{2}, {\sigma }_{3} $	−0.1, −0.1, −0.2
$ {\lambda }_{1}, {\lambda }_{2}, {\lambda }_{3} $	0, −0.5, 0

参数项	取值
$ {\beta }_{0}\left(\theta \right), {\beta }_{1}\left(\theta \right), {\beta }_{2}\left(\theta \right) $	100, 300, 500
$ {\beta }_{0}\left(\varphi \right), {\beta }_{1}\left(\varphi \right), {\beta }_{2}\left(\varphi \right) $	100, 300, 500
$ {\beta }_{0}\left(\psi \right), {\beta }_{1}\left(\psi \right), {\beta }_{2}\left(\psi \right) $	100, 300, 500
$ {c}_{1},{c}_{2},{c}_{3} $	1, 1, 1
$ {h}_{1},{h}_{2},{h}_{3} $	0.1, 0.1, 0.1

参数项	取值
$ {\beta }_{0}\left(\theta \right),{\beta }_{1}\left(\theta \right),{\beta }_{2}\left(\theta \right) $	100,500,1300
$ {\beta }_{0}\left(\phi \right),{\beta }_{1}\left(\phi \right),{\beta }_{2}\left(\phi \right) $	100,800,1300
$ {\beta }_{0}\left(\psi \right),{\beta }_{1}\left(\psi \right),{\beta }_{2}\left(\psi \right) $	100,1300,5900
$ {c}_{1},{c}_{2},{c}_{3} $	8,1,1
$ {h}_{1},{h}_{2},{h}_{3} $	0.1,0.2,0.3