基于双动态PPO算法的高超声速飞行器姿态控制

doi:10.12305/j.issn.1001-506X.2026.02.29

Abstract

Abstract:

To address the strong nonlinearities and significant uncertainties in hypersonic vehicle attitude control, as well as the limitations of traditional reinforcement learning algorithms in training convergence and control accuracy under multiple control requirements, a dual-dynamic adaptive proximal policy optimization （PPO） algorithm is proposed. The algorithm balances control precision and actuator protection through a soft dynamic clipping mechanism and a policy-driven entropy adjustment strategy. An integrated simulation environment incorporating aerodynamic characteristics and actuator dynamic characteristics is subsequently established. By integrating proportional-integral-derivative control principles, the state observation space is optimally redesigned. Simulation results demonstrate that compared with the baseline PPO algorithm, the proposed method improves convergence speed by 22% while significantly enhancing both control accuracy and action smoothness. Under different flight conditions, this method exhibits excellent strategic adaptability and robustness, and effectively improves the attitude control performance of the hypersonic vehicle.

Key words: hypersonic vehicle, intelligent control, deep reinforcement learning, proximal policy optimization (PPO), dynamic adaptive mechanism

CLC Number:

TP 183

Xu WANG, Guangbin CAI, Xiaoya YU, Ziqi YE, Bin SHAN. Attitude control of hypersonic vehicle based on dual-dynamic PPO algorithm[J]. Systems Engineering and Electronics, 2026, 48(2): 694-704.

Figures/Tables 16

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Table 1

Table 2

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Fig.13

Fig.14

References 35

1	XU H, CAI G B, MU C X, et al. Analytical reentry guidance framework based on swarm intelligence optimization and altitude-energy profile[J]. Chinese Journal of Aeronautics, 2023, 36 (12): 336- 348. doi: 10.1016/j.cja.2023.07.029
2	LIU S X, YAN B B, HUANG W, et al. Current status and prospects of terminal guidance laws for intercepting hypersonic vehicles in near space: a review[J]. Journal of Zhejiang University-SCIENCE A, 2023, 24 (5): 387- 403. doi: 10.1016/j.cja.2021.10.037
3	张远, 黄旭, 路坤锋, 等. 高超声速飞行器控制技术研究进展与展望[J]. 宇航学报, 2022, 43 (7): 866- 879.
	ZHANG Y, HUANG X, LU K F, et al. Research progress and prospects of control technology for hypersonic vehicles[J]. Journal of Astronautics, 2022, 43 (7): 866- 879.
4	DING Y B, YUE X K, CHEN G S, et al. Review of control and guidance technology on hypersonic vehicle[J]. Chinese Journal of Aeronautics, 2022, 35 (7): 1- 18. doi: 10.1016/j.cja.2021.10.037
5	HAN X, ZHENG Z Z, LIU L, et al. Online policy iteration ADP-based attitude-tracking control for hypersonic vehicles[J]. Aerospace Science and Technology, 2020, 106, 106233.
6	GAMBHIRE S J, KISHORE D R, LON-DHE P S, et al. Review of sliding mode based control techniques for control system applications[J]. International Journal of Dynamics and Control, 2021, 9 (1): 363- 378. doi: 10.1007/s40435-020-00638-7
7	DONE Z, BIN J, HAO Y. Backstepping-based decentralized fault-tolerant control of hypersonic vehicles in PDE-ODE form[J]. IEEE Trans. on Automatic Control, 2021, 67 (3): 1210- 1225.
8	赵昱宇, 索超, 王雨潇. 基于微分平坦的高超声速飞行器跟踪控制方法[J]. 系统工程与电子技术, 2024, 46 (3): 1084- 1092.
	ZHAO Y Y, SUO C, WANG Y X. Differential flatness-based tracking control method for hypersonic vehicle[J]. Systems Engineering and Electronics, 2024, 46 (3): 1084- 1092.
9	CAI G B, WU T, HAO M R, et al. Dynamic event-triggered gain-scheduled H∞ control for a polytopic LPV model of morphing aircraft[J]. IEEE Trans. on Aerospace and Electronic Systems, 2024, 61(1): 93−106.
10	ZHANG X, HU W, WEI C, et al. Nonlinear disturbance observer based adaptive super-twisting sliding mode control for generic hypersonic vehicles with coupled multisource disturbances[J]. European Journal of Control, 2021, 5 (7): 253- 262.
11	GUO R Y, DING Y B, YUE X K. Active adaptive continuous nonsingular terminal sliding mode controller for hypersonic vehicle[J]. Aerospace Science and Technology, 2023, 137, 108279.
12	ZHAO H W, YANG L. Global adaptive neural backstepping control of a flexible hypersonic vehicle with disturbance estimation[J]. Aircraft Engineering and Aerospace Technology, 2022, 94 (4): 492- 504. doi: 10.1108/AEAT-08-2020-0178
13	唐伟强, 甲成超, 石文科, 等. 高超声速飞行器积分滑模自抗扰控制研究[J]. 现代防御技术, 2024, 52 (5): 40- 50.
	TANG W Q, JIA C C, SHI W K, et al. Research on integral sliding mode active disturbance rejection control for hypersonic vehicle[J]. Modern Defence Technology, 2024, 52 (5): 40- 50.
14	LIU L, LIU Y X, ZHOU L L, et al. Cascade ADRC with neural network-based ESO for hypersonic vehicle[J]. Journal of the Franklin Institute, 2023, 360 (12): 9115- 9138. doi: 10.1016/j.jfranklin.2022.09.019
15	AZAR A T, KOUBAA A, ALI M N, et al. Drone deep reinforcement learning: a review[J]. Electronics, 2021, 10 (9): 999. doi: 10.3390/electronics10090999
16	ZHANG M H, WU Y H, LI C Y. Reinforcement learning strategy for spacecraft attitude hyperagile tracking control with uncertainties[J]. Aerospace Science and Technology, 2021, 119, 107126.
17	LIU Y C, HUANG C Y. DDPG-based a-daptive robust tracking control for aerial manipulators with decoupling approach[J]. IEEE Trans. on Cybernetics, 2022, 52 (8): 8258- 8271. doi: 10.1109/TCYB.2021.3049555
18	黄旭, 柳嘉润, 贾晨辉, 等. 强化学习控制方法及在类火箭飞行器上的应用[J]. 宇航学报, 2023, 44 (5): 708- 718.
	HUANG X, LIU J R, JIA C H, et al. Reinforcement learning control methods and their applications to rocket-like vehicles[J]. Journal of Astronautics, 2023, 44 (5): 708- 718.
19	XU L, YUE H J, YU S, et al. Modified deep deterministic policy gradient based on active disturbance rejection control for hypersonic vehicles[J]. Neural Computing and Applications, 2024, 36 (8): 4071- 4081. doi: 10.1007/s00521-023-09302-5
20	马少捷, 惠俊鹏, 王宇航, 等. 变形飞行器深度强化学习姿态控制方法研究[J]. 航天控制, 2022, 40 (6): 3- 10.
	MA S J, HUI J P, WANG Y H, et al. Research on deep reinforcement learning attitude control method for deformed vehicle[J]. Aerospace Control, 2022, 40 (6): 3- 10.
21	SHI L, WANG X S, CHENG Y H. Safe reinf-orcement learning-based robust approximate optimal control for hypersonic flight vehicles[J]. IEEE Trans. on Vehicular Technology, 2023, 72 (9): 11401- 11414. doi: 10.1109/TVT.2023.3264243
22	ZHU Y, PAN M, ZHOU W, et al. Intelligent direct thrust control for multivariable turbofan engine based on reinforcement and deep learning methods[J]. Aerospace Science and Technology, 2022, 13 (1): 107972.
23	SONG J, LUO Y, ZHAO M, et al. Fault-tolerant integrated guidance and control design for hypersonic vehicle based on PPO[J]. Mathematics, 2022, 10 (18): 3401. doi: 10.3390/math10183401
24	PAPINI M, PIROTTA M, RESTELLI M. Smoothing policies and safe policy gradients[J]. Machine Learning, 2022, 111 (11): 4081- 4137. doi: 10.1007/s10994-022-06232-6
25	王冠, 茹海忠, 张大力, 等. 弹性高超声速飞行器智能控制系统设计[J]. 系统工程与电子技术, 2022, 44 (7): 2276- 2285.
	WANG G, RU H Z, ZHANG D L, et al. Design of intelligent control system for flexible hypersonic vehicle[J]. Systems Engineering and Electronics, 2022, 44 (7): 2276- 2285.
26	TANG W Q, LONG W K, GAO H Y. Model predictive control of hypersonic vehicles accommodating constraints[J]. IET Control Theory & Applications, 2017, 11 (15): 2599- 2606.
27	ZHANG J W, ZHANG Z H, HAN S, et al. Proximal policy optimization via enhanced exploration efficiency[J]. Information Sciences, 2022, 6 (9): 750- 765.
28	DAI J T, JI J M, YANG L, et al. Augmented proximal policy optimization for safe rein-forcement learning[C]//Proc. of the AAAI Conference on Artificial Intelligence, 2023: 7288−7295.
29	CHEN X, DIAO D C, CHEN H C, et al. The sufficiency of off-policyness and soft clipping: PPO is still insufficient according to an off-policy measure[C]// Proc. of the AAAI Conference on Artificial Intelligence. 2023: 7078-7086.
30	SUMIEA E H, ABDULKADIR S J, ALHUSSIAN H S, et al. Deterministic policy gradient algorithms[C]// Proc. of the International Conference on Machine Learning, 2014: 387−395.
31	GU Y, CHENG Y, CHEN C L P, et al. Proximal policy optimization with policy feedback[J]. IEEE Trans. on Systems, Man, and Cybernetics: Systems, 2021, 52 (7): 4600- 4610.
32	CHENG Y H, HUANG L Y, WANG X S. Authentic boundary proximal policy optimization[J]. IEEE Trans. on Cybernetics, 2021, 52 (9): 9428- 9438.
33	LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015, 521 (7553): 436- 444. doi: 10.1038/nature14539
34	SRIVASTAVA A, SALAOAKA S M. Parameterized MDPs and reinforcement learning problems—a maximum entropy principle-based framework[J]. IEEE Trans. on Cybernetics, 2021, 52 (9): 9339- 9351.
35	胡军. 高超声速飞行器非线性自适应姿态控制[J]. 宇航学报, 2017, 38 (12): 1281- 1288.
	HU J. Nonlinear adaptive attitude controlfor hypersonic vehicles[J]. Journal of Astronautics, 2017, 38 (12): 1281- 1288.

参数	数值	参数	数值
最大训练回合	2×10⁴	Actor 学习率${\alpha _1}$	10⁻⁵
每回合最大步数	1000	Critic学习率${\alpha _2}$	10⁻⁵
系统采样时间	0.02	折扣因子$\gamma $	0.99
GAE系数	0.95	Mini-batch	32
Clip	0.2	熵正则系数${\beta _0}$	10⁻²
$\kappa $取值范围	[1,8]	熵系数衰减率$\eta $	0.99
更新周期$K$	10	正则系数$\omega $	10⁻³

网络层	策略网络		价值网络
网络层	神经元数	激活函数	神经元数	激活函数
输入层	3	None	6	None
隐藏层1	150	Tanh	150	Tanh
隐藏层2	150	Tanh	150	Tanh
输出层	2	Softplus	1	Linear

[1]	Jinyan XUE, Yasheng ZHANG, Xuefeng TAO, Mingqi YANG, Shuailong ZHAO. Advances in orbital maneuver control for GEO spacecraft [J]. Systems Engineering and Electronics, 2026, 48(1): 290-300.
[2]	Chuanlong SONG, Qianwu ZHANG, Jian HE, Wenjun ZHOU, Hui WANG, Weiwei KONG, Wenbo TIAN. Edge computing task offloading method of satellite-ground collaborative based on MADDPG algorithm [J]. Systems Engineering and Electronics, 2026, 48(1): 350-360.
[3]	Xiaolong WEI, Yarong WU, Dengkai YAO, Guhao ZHAO. Hierarchical decision-making algorithm for UAV air combat maneuvering based on deep reinforcement learning [J]. Systems Engineering and Electronics, 2025, 47(9): 2993-3003.
[4]	Yundou ZHU, Haiquan SUN, Xiaoxuan HU. Multi-satellite cooperative imaging task planning method based on pointer network architecture [J]. Systems Engineering and Electronics, 2025, 47(7): 2246-2255.
[5]	Linzhi MENG, Xiaojuan SUN, Yuxin HU, Bin GAO, Guoqing SUN, Wenhao MU. Reinforcement learning task scheduling algorithm for satellite on-orbit processing [J]. Systems Engineering and Electronics, 2025, 47(6): 1917-1929.
[6]	Kangjie ZHENG, Xinyu ZHANG, Weisong WANG, Zhensheng LIU. Intelligent ship dynamic autonomous obstacle avoidance decision based on DQN and rule [J]. Systems Engineering and Electronics, 2025, 47(6): 1994-2001.
[7]	Shuhan LIU, Tong LI, Fuqiang LI, Chungang YANG. Intent and situation-dual driven anti-jamming communication mechanism for data link [J]. Systems Engineering and Electronics, 2025, 47(6): 2055-2064.
[8]	Xun HUANG, Boyi CHEN, Shouyong PENG, Yanbin LIU, Ben YANG, Haoran PANG. Trajectory optimization strategy for hypersonic vehicle under control constraints [J]. Systems Engineering and Electronics, 2025, 47(5): 1646-1654.
[9]	Wei XIONG, Dong ZHANG, Zhi REN, Shuheng YANG. Research on intelligent decision-making methods for coordinated attack by manned aerial vehicles and unmanned aerial vehicles [J]. Systems Engineering and Electronics, 2025, 47(4): 1285-1299.
[10]	Peng MA, Rui JIANG, Bin WANG, Mengfei XU, Changbo HOU. Strategy reconstruction for resilience against intelligence jamming based on implicit opponent modeling [J]. Systems Engineering and Electronics, 2025, 47(4): 1355-1363.
[11]	Lan ZHANG, Biao ZHANG, Tianyi LIANG, Huijie ZHU. Research progress on generative adversarial network for electromagnetic information intelligent control [J]. Systems Engineering and Electronics, 2025, 47(3): 730-744.
[12]	Kaiqiang TANG, Huiqiao FU, Jiasheng LIU, Guizhou DENG, Chunlin CHEN. Hierarchical optimization research of constrained vehicle routing based on deep reinforcement learning [J]. Systems Engineering and Electronics, 2025, 47(3): 827-841.
[13]	Xiarong CHEN, Jichao LI, Gang CHEN, Peng LIU, Jiang JIANG. Portfolio of weapon system-of-systems based on heterogeneous information networks [J]. Systems Engineering and Electronics, 2025, 47(3): 855-861.
[14]	Shaowei HUANG, Yanli DU, Yanbin LIU, Yueping WANG, Wu LIU. Adaptive sliding cooperative terminal guidance with finite-time convergence [J]. Systems Engineering and Electronics, 2025, 47(3): 961-969.
[15]	Yang LIU, Fanyi MENG, Gang CHEN. Reinforcement learning based disturbance rejection compensation control method for morphing aircraft [J]. Systems Engineering and Electronics, 2025, 47(12): 4130-4142.

Attitude control of hypersonic vehicle based on dual-dynamic PPO algorithm

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 16

References 35

Related Articles 15

Recommended Articles

Metrics

Comments