基于改进PPO算法的无人机/无人船协同导航方法

doi:10.12305/j.issn.1001-506X.2026.07.29

摘要/Abstract

摘要：

针对激光雷达感知系统在“U型”障碍及遮挡场景下存在的感知局限性，以及复杂水域环境中无人船自主导航效率不足的问题，提出一种基于改进近端策略优化算法的无人机/无人船协同导航方法。该方法通过融合无人机的高空视觉感知能力与无人船的水面执行功能，弥补了激光雷达在特定场景下的感知缺陷，从而提升了无人船在复杂环境中的导航性能。在算法设计方面，采用长短期记忆网络来建模无人船运动状态的时序依赖特征，同时结合Ornstein-Uhlenbeck噪声和广义优势估计以增强策略探索效率并提高价值估计的稳定性。此外，通过设计一组辅助奖励函数，在优化路径长度的同时约束舵角波动，生成了更为平滑的航行轨迹。本研究为复杂水域环境下的无人机/无人船导航提供了理论研究基础。

关键词: 无人机, 无人船, 协同导航, 近端策略优化

Abstract:

Aiming at the problems of the perception limitations of light detection and ranging (LiDAR) sensing system in “U-shaped” obstacle and occlusion scenarios, as well as the insufficient autonomous navigation efficiency of unmanned surface vessel （USV） in complex water environments, a cooperative unmanned aerial vehicle （UAV）/USV navigation method based on an improved proximal policy optimization algorithm is proposed. By integrating the high-altitude visual perception capability of UAVs with the surface execution function of USVs, the proposed method compensates for the perceptual deficiencies of LiDAR in specific scenarios, thereby enhancing the navigation performance of USVs in complex environments. In terms of algorithm design, a long short-term memory network is adopted to model the temporal dependency features of the USV’s motion state, while incorporating Ornstein-Uhlenbeck noise and generalized advantage estimation to enhance policy exploration efficiency and improve the stability of value estimation. Furthermore, by designing a set of auxiliary reward functions, the method optimizes path length while constraining rudder angle fluctuations, resulting in smoother navigation trajectories. This study provides a theoretical foundation for UAV/USV navigation in complex water environments.

Key words: unmanned aerial vehicle （UAV）, unmanned surface vessel （USV）, cooperative navigation, proximal policy optimization （PPO）

中图分类号:

U 644.82

关巍, 胡彤博, 张显库, 罗春奇, 曲胜, 崔哲闻. 基于改进PPO算法的无人机/无人船协同导航方法[J]. 系统工程与电子技术, 2026, 48(7): 2457-2468.

Wei GUAN, Tongbo HU, Xianku ZHANG, Chunqi LUO, Sheng QU, Zhewen CUI. Cooperative navigation method for UAV/ USV based on the improved PPO algorithm[J]. Systems Engineering and Electronics, 2026, 48(7): 2457-2468.

图/表 20

图1

图2

图3

图4

图5

图6

图7

表1

表2

表3

表4

图8

图9

图10

表5

图11

表6

图12

图13

图14

参考文献 30

1	李远芳, 张莉, 陈钜龙. 水下无人船构建海洋环境监测系统研究[J]. 舰船科学技术, 2021, 43 (10): 166- 168. doi: 10.3404/j.issn.1672-7649.2021.5A.056
	LI Y F, ZHANG L, CHEN J L. Research on the construction of marine environmental monitoring system for unmanned underwater ships[J]. Ship Science and Technology, 2021, 43 (10): 166- 168. doi: 10.3404/j.issn.1672-7649.2021.5A.056
2	XIE Y C, NANLAL C, LIU Y C. Reliable LiDAR-based ship detection and tracking for autonomous surface vehicles in busy maritime environments[J]. Ocean Engineering, 2024, 312 (3): 119288.
3	SUN S, LYU H G, GAO Z J, et al. Grid map assisted radar target tracking in a detection occluded maritime environment[J]. IEEE Trans. on Instrumentation and Measurement, 2024, 73
4	WEI W, WANG J J, FANG Z R, et al. 3-U: joint design of UAV-USV-UUV networks for cooperative target hunting[J]. IEEE Trans. on Vehicular Technology, 2022, 72 (3): 4085- 4090.
5	REN Z G, ZHENG H, CHEN J, et al. Integrating UAV, UGV and UAV-UGV collaboration in future industrialized agriculture: analysis, opportunities and challenges[J]. Computers and Electronics in Agriculture, 2024, 227 (2): 109631.
6	YAN J, LIN J S, XIAN Y, et al. Cooperation detection and tracking of underwater target via aerial-surface-underwater vehicles[J]. IEEE Trans. on Automatic Control, 2025, 70 (2): 1068- 1083. doi: 10.1109/TAC.2024.3447976
7	WANG Y D, LIU W Z, LIU J, et al. Cooperative USV/UAV marine search and rescue with visual navigation and reinforcement learning based control[J]. ISA Transactions, 2023, 137, 222- 235. doi: 10.1016/j.isatra.2023.01.007
8	NIU H L, JI Z, PIETRO L, et al. Design, integration and sea trials of 3D printed unmanned aerial vehicle and unmanned surface vehicle for cooperative missions[C]// Proc. of the IEEE/SICE International Symposium on System Integration, 2021: 2747−2325.
9	SHAO G M, MA Y, REZA M, et al. A novel cooperative platform design for coupled USV-UAV systems[J]. IEEE Trans. on Industrial Informatics, 2019, 15 (9): 4913- 4922. doi: 10.1109/TII.2019.2912024
10	GUAN W, WANG K. Autonomous collision avoidance of unmanned surface vehicles based on improved a-star and dynamic window approach algorithms[J]. IEEE Intelligent Transportation Systems Magazine, 2023, 15 (3): 36- 50. doi: 10.1109/MITS.2022.3229109
11	ZHANG W L, SHAN L, CHANG L, et al. SVF-RRT: a stream-based VF-RRT for USVs path planning considering ocean currents[J]. IEEE Robotics and Automation Letters, 2023, 8 (4): 2413- 2420. doi: 10.1109/LRA.2023.3245409
12	李文刚, 汪流江, 方德翔, 等. 联合A*与动态窗口法的路径规划算法[J]. 系统工程与电子技术, 2021, 43 (12): 3694- 3702. doi: 10.12305/j.issn.1001-506X.2021.12.33
	LI W G, WANG L J, FANG D X, et al. A path planning algorithm combining A* and dynamic window method[J]. Systems Engineering and Electronics, 2021, 43 (12): 3694- 3702. doi: 10.12305/j.issn.1001-506X.2021.12.33
13	YANG C P, PAN J C, WEI K, et al. A novel unmanned surface vehicle path planning algorithm based on A* and artificial potential field in ocean currents[J]. Journal of Marine Science and Engineering, 2024, 12 (2): 285. doi: 10.3390/jmse12020285
14	CHEN Z H, YU J B, ZHAO Z Y, et al. A path planning method considering environmental disturbance based on VPF-RRT[J]. Drones, 2023, 7 (2): 145. doi: 10.3390/drones7020145
15	夏雨奇, 黄炎焱, 陈恰. 基于深度Q网络的无人车侦察路径规划[J]. 系统工程与电子技术, 2024, 46 (9): 3070- 3081.
	XIA Y Q, HUANG Y Y, CHEN Q. Unmanned vehicle reconnaissance path planning based on deep Q network[J]. Systems Engineering and Electronics, 2024, 46 (9): 3070- 3081.
16	关巍, 曲胜, 张显库, 等. 基于改进DQN算法的船舶全局路径规划研究[J]. 中国舰船研究, 2025, 20 (1): 107- 114.
	GUAN W, QU S, ZHANG X K, et al. Global path planning for ships based on improved DQN algorithm[J]. Chinese Journal of Ship Research, 2025, 20 (1): 107- 114.
17	ZHAO J, WANG P R, LI B Y, et al. A DDPG based USV path planning algorithm[J]. Applied Sciences, 2023, 13 (19): 10567. doi: 10.3390/app131910567
18	CUI Z, GUAN W, ZHANG X K, et al. Autonomous collision avoidance decision-making method for USV based on ATL-TD3 algorithm[J]. Ocean Engineering, 2024, 312 (3): 119297.
19	SUN P Q, YANG C X, ZHOU X J, et al. Path planning for unmanned surface vehicles with strong generalization ability based on improved proximal policy optimization[J]. Sensors, 2023, 23 (21): 8864. doi: 10.3390/s23218864
20	LI W Z, GE Y, GUAN Z H, et al. Synchronized motion based UAV–USV cooperative autonomous landing[J]. Journal of Marine Science and Engineering, 2022, 10 (9): 1214. doi: 10.3390/jmse10091214
21	LI W Z, GE Y, GUAN Z H, et al. NMPC-based UAV-USV cooperative tracking and landing[J]. Journal of the Franklin Institute, 2023, 360 (11): 7481- 7500. doi: 10.1016/j.jfranklin.2023.06.023
22	LUO W B, WANG X, HAN F, et al. Research on LSTM-PPO obstacle avoidance algorithm and training environment for unmanned surface vehicles[J]. Journal of Marine Science and Engineering, 2025, 13 (3): 479. doi: 10.3390/jmse13030479
23	LIU L S, CHEN J H, ZHANG Y Y, et al. Unmanned ground vehicle path planning based on improved DRL algorithm[J]. Electronics, 2024, 13 (13): 2479. doi: 10.3390/electronics13132479
24	王涛, 张卫华, 蒲亦非. 适用于强化学习惯性环境的分数阶改进OU噪声[J]. 四川大学学报（自然科学版）, 2023, 60 (2): 63- 69.
	WANG T, ZHANG W H, PU Y F, et al. Fractional order improved OU noise for reinforcement learning inertial environments[J]. Journal of Sichuan University （Natural Science Edition）, 2023, 60 (2): 63- 69.
25	CHEN Y A, YE R S, TAO Z Y, et al. Reinforcement learning for robot navigation with adaptive forward simulation time (AFST) in a semi-Markov model[C]//Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 2023: 3597−3604.
26	乔志洋, 余娅荣. 压缩感知理论在图像处理中的应用[J]. 云南大学学报（自然科学版）, 2017, 39 (S1): 63- 69.
	QIAO Z Y, YU Y R. Application of compressive sensing theory in image processing[J]. Journal of Yunnan University （Natural Science Edition）, 2017, 39 (S1): 63- 69.
27	LIANG C Q, LIU L, LIU C. Multi-UAV autonomous collision avoidance based on PPO-GIC algorithm with CNN–LSTM fusion network[J]. Neural Networks, 2023, 162, 21- 33. doi: 10.1016/j.neunet.2023.02.027
28	CHRISTIANO P F, LEIKE J, BROWN T, et al. Deep reinforcement learning from human preferences[C]//Proc. of the 31st Annual Conference on Neural Information Processing Systems, 2017: 4299−4307.
29	YU K, LIANG X F, LI M Z, et al. USV path planning method with velocity variation and global optimization based on AIS service platform[J]. Ocean Engineering, 2021, 236 (15): 109560.
30	CUI Z W, GUAN W, ZHANG X K. Collision avoidance decision-making strategy for multiple USVs based on deep reinforcement learning algorithm[J]. Ocean Engineering, 2024, 308 (15): 118323.

参数	数值	参数	数值
折扣率$\gamma $	0.97	有效波高/m	0.35
策略网络学习率	0.0002	波浪周期/s	3.2
价值网络学习率	0.0002	平均风速/（m/s）	3.8
舵角阈值$\beta $	2	批量大小Batch	256
安全距离${d_{{\mathrm{safe}}}}$/m	50	直航奖励系数${\lambda _{\mathrm{s}}}$	2
LSTM单元数	64/32	GAE权衡参数$\lambda $	0.97
舵角正则化奖励系数${\lambda _{\mathrm{a}}}$	5	舵角变化率奖励系数${\lambda _{\mathrm{f}}}$	3
目标引导奖励系数${\lambda _{\text{g}}}$	8	路径曲率约束奖励系数${\lambda _{\mathrm{c}}}$	2

LSTM结构配置（LSTM_1/LSTM_2）	收敛步数	收敛值
单层LSTM（64）	1 940	6019
双层LSTM（32/16）	1770	6437
双层LSTM（64/32）	1690	6654
双层LSTM（128/64）	1730	6348

GAE权衡参数$\lambda $	收敛步数	收敛值
0.90	2100	6129
0.95	1 880	6246
0.97	1760	6438
0.99	1 810	6407

像素配置	平均决策时间/ms	成功率/%
64×64	12.5	82.3
128×128	18.3	98.8
256×256	34.6	96.2
512×512	72.9	84.6

实验场景	起始位置	目标位置
“U型”场景	（−500, −500）	（450, 450）
动态遮挡场景	（150, −550）	（150, 500）
狭窄场景	（−450, −450）	（450, 450）
密集障碍场景	（−350, −500）	（400, 450）