基于改进PPO算法的无人机/无人船协同导航方法

doi:10.12305/j.issn.1001-506X.2026.07.29

Abstract

Abstract:

Aiming at the problems of the perception limitations of light detection and ranging (LiDAR) sensing system in “U-shaped” obstacle and occlusion scenarios, as well as the insufficient autonomous navigation efficiency of unmanned surface vessel （USV） in complex water environments, a cooperative unmanned aerial vehicle （UAV）/USV navigation method based on an improved proximal policy optimization algorithm is proposed. By integrating the high-altitude visual perception capability of UAVs with the surface execution function of USVs, the proposed method compensates for the perceptual deficiencies of LiDAR in specific scenarios, thereby enhancing the navigation performance of USVs in complex environments. In terms of algorithm design, a long short-term memory network is adopted to model the temporal dependency features of the USV’s motion state, while incorporating Ornstein-Uhlenbeck noise and generalized advantage estimation to enhance policy exploration efficiency and improve the stability of value estimation. Furthermore, by designing a set of auxiliary reward functions, the method optimizes path length while constraining rudder angle fluctuations, resulting in smoother navigation trajectories. This study provides a theoretical foundation for UAV/USV navigation in complex water environments.

Key words: unmanned aerial vehicle （UAV）, unmanned surface vessel （USV）, cooperative navigation, proximal policy optimization （PPO）

CLC Number:

U 644.82

Wei GUAN, Tongbo HU, Xianku ZHANG, Chunqi LUO, Sheng QU, Zhewen CUI. Cooperative navigation method for UAV/ USV based on the improved PPO algorithm[J]. Systems Engineering and Electronics, 2026, 48(7): 2457-2468.

Figures/Tables 20

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Table 1

Table 2

Table 3

Table 4

Fig.8

Fig.9

Fig.10

Table 5

Fig.11

Table 6

Fig.12

Fig.13

Fig.14

References 30

1	李远芳, 张莉, 陈钜龙. 水下无人船构建海洋环境监测系统研究[J]. 舰船科学技术, 2021, 43 (10): 166- 168. doi: 10.3404/j.issn.1672-7649.2021.5A.056
	LI Y F, ZHANG L, CHEN J L. Research on the construction of marine environmental monitoring system for unmanned underwater ships[J]. Ship Science and Technology, 2021, 43 (10): 166- 168. doi: 10.3404/j.issn.1672-7649.2021.5A.056
2	XIE Y C, NANLAL C, LIU Y C. Reliable LiDAR-based ship detection and tracking for autonomous surface vehicles in busy maritime environments[J]. Ocean Engineering, 2024, 312 (3): 119288.
3	SUN S, LYU H G, GAO Z J, et al. Grid map assisted radar target tracking in a detection occluded maritime environment[J]. IEEE Trans. on Instrumentation and Measurement, 2024, 73
4	WEI W, WANG J J, FANG Z R, et al. 3-U: joint design of UAV-USV-UUV networks for cooperative target hunting[J]. IEEE Trans. on Vehicular Technology, 2022, 72 (3): 4085- 4090.
5	REN Z G, ZHENG H, CHEN J, et al. Integrating UAV, UGV and UAV-UGV collaboration in future industrialized agriculture: analysis, opportunities and challenges[J]. Computers and Electronics in Agriculture, 2024, 227 (2): 109631.
6	YAN J, LIN J S, XIAN Y, et al. Cooperation detection and tracking of underwater target via aerial-surface-underwater vehicles[J]. IEEE Trans. on Automatic Control, 2025, 70 (2): 1068- 1083. doi: 10.1109/TAC.2024.3447976
7	WANG Y D, LIU W Z, LIU J, et al. Cooperative USV/UAV marine search and rescue with visual navigation and reinforcement learning based control[J]. ISA Transactions, 2023, 137, 222- 235. doi: 10.1016/j.isatra.2023.01.007
8	NIU H L, JI Z, PIETRO L, et al. Design, integration and sea trials of 3D printed unmanned aerial vehicle and unmanned surface vehicle for cooperative missions[C]// Proc. of the IEEE/SICE International Symposium on System Integration, 2021: 2747−2325.
9	SHAO G M, MA Y, REZA M, et al. A novel cooperative platform design for coupled USV-UAV systems[J]. IEEE Trans. on Industrial Informatics, 2019, 15 (9): 4913- 4922. doi: 10.1109/TII.2019.2912024
10	GUAN W, WANG K. Autonomous collision avoidance of unmanned surface vehicles based on improved a-star and dynamic window approach algorithms[J]. IEEE Intelligent Transportation Systems Magazine, 2023, 15 (3): 36- 50. doi: 10.1109/MITS.2022.3229109
11	ZHANG W L, SHAN L, CHANG L, et al. SVF-RRT: a stream-based VF-RRT for USVs path planning considering ocean currents[J]. IEEE Robotics and Automation Letters, 2023, 8 (4): 2413- 2420. doi: 10.1109/LRA.2023.3245409
12	李文刚, 汪流江, 方德翔, 等. 联合A*与动态窗口法的路径规划算法[J]. 系统工程与电子技术, 2021, 43 (12): 3694- 3702. doi: 10.12305/j.issn.1001-506X.2021.12.33
	LI W G, WANG L J, FANG D X, et al. A path planning algorithm combining A* and dynamic window method[J]. Systems Engineering and Electronics, 2021, 43 (12): 3694- 3702. doi: 10.12305/j.issn.1001-506X.2021.12.33
13	YANG C P, PAN J C, WEI K, et al. A novel unmanned surface vehicle path planning algorithm based on A* and artificial potential field in ocean currents[J]. Journal of Marine Science and Engineering, 2024, 12 (2): 285. doi: 10.3390/jmse12020285
14	CHEN Z H, YU J B, ZHAO Z Y, et al. A path planning method considering environmental disturbance based on VPF-RRT[J]. Drones, 2023, 7 (2): 145. doi: 10.3390/drones7020145
15	夏雨奇, 黄炎焱, 陈恰. 基于深度Q网络的无人车侦察路径规划[J]. 系统工程与电子技术, 2024, 46 (9): 3070- 3081.
	XIA Y Q, HUANG Y Y, CHEN Q. Unmanned vehicle reconnaissance path planning based on deep Q network[J]. Systems Engineering and Electronics, 2024, 46 (9): 3070- 3081.
16	关巍, 曲胜, 张显库, 等. 基于改进DQN算法的船舶全局路径规划研究[J]. 中国舰船研究, 2025, 20 (1): 107- 114.
	GUAN W, QU S, ZHANG X K, et al. Global path planning for ships based on improved DQN algorithm[J]. Chinese Journal of Ship Research, 2025, 20 (1): 107- 114.
17	ZHAO J, WANG P R, LI B Y, et al. A DDPG based USV path planning algorithm[J]. Applied Sciences, 2023, 13 (19): 10567. doi: 10.3390/app131910567
18	CUI Z, GUAN W, ZHANG X K, et al. Autonomous collision avoidance decision-making method for USV based on ATL-TD3 algorithm[J]. Ocean Engineering, 2024, 312 (3): 119297.
19	SUN P Q, YANG C X, ZHOU X J, et al. Path planning for unmanned surface vehicles with strong generalization ability based on improved proximal policy optimization[J]. Sensors, 2023, 23 (21): 8864. doi: 10.3390/s23218864
20	LI W Z, GE Y, GUAN Z H, et al. Synchronized motion based UAV–USV cooperative autonomous landing[J]. Journal of Marine Science and Engineering, 2022, 10 (9): 1214. doi: 10.3390/jmse10091214
21	LI W Z, GE Y, GUAN Z H, et al. NMPC-based UAV-USV cooperative tracking and landing[J]. Journal of the Franklin Institute, 2023, 360 (11): 7481- 7500. doi: 10.1016/j.jfranklin.2023.06.023
22	LUO W B, WANG X, HAN F, et al. Research on LSTM-PPO obstacle avoidance algorithm and training environment for unmanned surface vehicles[J]. Journal of Marine Science and Engineering, 2025, 13 (3): 479. doi: 10.3390/jmse13030479
23	LIU L S, CHEN J H, ZHANG Y Y, et al. Unmanned ground vehicle path planning based on improved DRL algorithm[J]. Electronics, 2024, 13 (13): 2479. doi: 10.3390/electronics13132479
24	王涛, 张卫华, 蒲亦非. 适用于强化学习惯性环境的分数阶改进OU噪声[J]. 四川大学学报（自然科学版）, 2023, 60 (2): 63- 69.
	WANG T, ZHANG W H, PU Y F, et al. Fractional order improved OU noise for reinforcement learning inertial environments[J]. Journal of Sichuan University （Natural Science Edition）, 2023, 60 (2): 63- 69.
25	CHEN Y A, YE R S, TAO Z Y, et al. Reinforcement learning for robot navigation with adaptive forward simulation time (AFST) in a semi-Markov model[C]//Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 2023: 3597−3604.
26	乔志洋, 余娅荣. 压缩感知理论在图像处理中的应用[J]. 云南大学学报（自然科学版）, 2017, 39 (S1): 63- 69.
	QIAO Z Y, YU Y R. Application of compressive sensing theory in image processing[J]. Journal of Yunnan University （Natural Science Edition）, 2017, 39 (S1): 63- 69.
27	LIANG C Q, LIU L, LIU C. Multi-UAV autonomous collision avoidance based on PPO-GIC algorithm with CNN–LSTM fusion network[J]. Neural Networks, 2023, 162, 21- 33. doi: 10.1016/j.neunet.2023.02.027
28	CHRISTIANO P F, LEIKE J, BROWN T, et al. Deep reinforcement learning from human preferences[C]//Proc. of the 31st Annual Conference on Neural Information Processing Systems, 2017: 4299−4307.
29	YU K, LIANG X F, LI M Z, et al. USV path planning method with velocity variation and global optimization based on AIS service platform[J]. Ocean Engineering, 2021, 236 (15): 109560.
30	CUI Z W, GUAN W, ZHANG X K. Collision avoidance decision-making strategy for multiple USVs based on deep reinforcement learning algorithm[J]. Ocean Engineering, 2024, 308 (15): 118323.

参数	数值	参数	数值
折扣率$\gamma $	0.97	有效波高/m	0.35
策略网络学习率	0.0002	波浪周期/s	3.2
价值网络学习率	0.0002	平均风速/（m/s）	3.8
舵角阈值$\beta $	2	批量大小Batch	256
安全距离${d_{{\mathrm{safe}}}}$/m	50	直航奖励系数${\lambda _{\mathrm{s}}}$	2
LSTM单元数	64/32	GAE权衡参数$\lambda $	0.97
舵角正则化奖励系数${\lambda _{\mathrm{a}}}$	5	舵角变化率奖励系数${\lambda _{\mathrm{f}}}$	3
目标引导奖励系数${\lambda _{\text{g}}}$	8	路径曲率约束奖励系数${\lambda _{\mathrm{c}}}$	2

LSTM结构配置（LSTM_1/LSTM_2）	收敛步数	收敛值
单层LSTM（64）	1 940	6019
双层LSTM（32/16）	1770	6437
双层LSTM（64/32）	1690	6654
双层LSTM（128/64）	1730	6348

GAE权衡参数$\lambda $	收敛步数	收敛值
0.90	2100	6129
0.95	1 880	6246
0.97	1760	6438
0.99	1 810	6407

像素配置	平均决策时间/ms	成功率/%
64×64	12.5	82.3
128×128	18.3	98.8
256×256	34.6	96.2
512×512	72.9	84.6

实验场景	起始位置	目标位置
“U型”场景	（−500, −500）	（450, 450）
动态遮挡场景	（150, −550）	（150, 500）
狭窄场景	（−450, −450）	（450, 450）
密集障碍场景	（−350, −500）	（400, 450）

Cooperative navigation method for UAV/ USV based on the improved PPO algorithm

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 20

References 30

Related Articles 15

Recommended Articles

Metrics

Comments

实验场景	算法	路径长度/m	舵角变化率标准差/(°/s)
“U型”障碍场景	DDPG	1970.87	1.68
	PPO	1895.61	1.62
	改进PPO	1738.29	1.46
动态障碍遮挡场景	DDPG	1477.29	1.53
	PPO	1447.84	1.48
	改进PPO	1326.35	1.34
狭窄场景	A*+DWA	1709.14	1.21
	DDPG	1660.85	1.17
	PPO	1598.50	1.14
	改进PPO	1463.43	1.03
密集障碍场景	A*+DWA	1663.42	1.33
	DDPG	1613.34	1.27
	PPO	1551.63	1.23
	改进PPO	1418.57	1.12

[1]	Honghui BAN, Jifei PAN, Zheng WANG, Rui CUI. Trajectory deception error analysis based on multi-UAV collaboration [J]. Systems Engineering and Electronics, 2026, 48(7): 2195-2205.
[2]	Wenbin LIN, Chenguang SHI, Mu YAN, Fei WANG, Jianjiang ZHOU. Robust trajectory planning algorithm for UAVs swarm facing deceptive jamming networked radar [J]. Systems Engineering and Electronics, 2026, 48(5): 1551-1563.
[3]	Jie HU, Ruifeng CHU, Yixian ZHU, Ping CHEN, Fan BAO. Three-dimensional trajectory planning of UAV based on VPPSO algorithm [J]. Systems Engineering and Electronics, 2026, 48(5): 1738-1751.
[4]	Zhi YANG, Feng YU, Siying LIN, Zijun ZHOU. Swarm autonomous navigation technology based on visual collaboration [J]. Systems Engineering and Electronics, 2026, 48(4): 1396-1403.
[5]	Xuping GU, Xianjun SHI. Reconfigurability analysis and design of UAV based on structural analysis and tree-seed optimization algorithm [J]. Systems Engineering and Electronics, 2026, 48(3): 932-945.
[6]	Sen ZHANG, Yan PANG, Fuliang ZHOU. Three-dimensional path planning for UAV based on improved Informed-RRT^* algorithm [J]. Systems Engineering and Electronics, 2026, 48(2): 660-668.
[7]	Xu WANG, Guangbin CAI, Xiaoya YU, Ziqi YE, Bin SHAN. Attitude control of hypersonic vehicle based on dual-dynamic PPO algorithm [J]. Systems Engineering and Electronics, 2026, 48(2): 694-704.
[8]	Xuxin YANG, Wei JI. Optimization strategy for energy consumption in unmanned aerial vehicle assisted safe MEC systems [J]. Systems Engineering and Electronics, 2026, 48(2): 719-726.
[9]	Zeyu LU, Yao WANG, Weinan WU, Yiming SUN, Chunlin GONG. Two-layered path planning for unmanned aerial vehicle swarm in three-dimensional dynamic environment [J]. Systems Engineering and Electronics, 2026, 48(1): 132-143.
[10]	Qichen XU, Zhaohui ZHANG, Jing LI. Distributed adaptive multi-UAV collaborative path planning in unknown denial environments [J]. Systems Engineering and Electronics, 2026, 48(1): 172-184.
[11]	Xiaolong WEI, Yarong WU, Dengkai YAO, Guhao ZHAO. Hierarchical decision-making algorithm for UAV air combat maneuvering based on deep reinforcement learning [J]. Systems Engineering and Electronics, 2025, 47(9): 2993-3003.
[12]	Dapeng YANG, Zihao GONG, Xiaoye WANG, Zhengyu GUO, Delin LUO. Research on UAV cooperative interception maneuver decision-making based on multi-agent reinforcement learning [J]. Systems Engineering and Electronics, 2025, 47(9): 3076-3085.
[13]	Zhao YANG, Jinbiao HU, Yan WANG, Hongbiao QI. UAV coverage path planning for mountain patrol considering different takeoff and landing nests [J]. Systems Engineering and Electronics, 2025, 47(8): 2622-2631.
[14]	Xiaowei YAN, Chong LING, Shengbin SHI. Design and implementation of a rapid detection system for surface unexploded submunitions [J]. Systems Engineering and Electronics, 2025, 47(8): 2639-2645.
[15]	Xiaowei FU, Xinyi WANG, Zhe QIAO. Attack-defense confrontation strategy of multi-UAV based on APIQ algorithm [J]. Systems Engineering and Electronics, 2025, 47(7): 2205-2215.