动态环境下无人机路径规划的改进深度强化学习算法

doi:10.12305/j.issn.1001-506X.2026.11.22

摘要/Abstract

摘要：

针对复杂动态环境下无人机配送路径规划问题，构建同时考虑飞行成本、延误惩罚与碰撞风险的综合经济指标，引入环境动态变化率刻画障碍物时变程度，提出融合深度Q网络与多头注意力的改进深度强化学习(hybrid deep Q-network and multi-head attention deep reinforcement learning，HDMADRL)算法。仿真结果表明，HDMADRL能在高度动态边界情况下保持约75%的任务成功率。在中高动态场景中，相比基线方法，HDMADRL的平均成本降低8%~14%，平均碰撞次数减少20%~35%，任务成功率提高6%~15%。消融实验进一步验证，“位置–环境双路径编码+多头注意力”结构的优异性能。研究成果为无人机配送领域提供新型路径规划方法，对降低运营成本和提高飞行安全具有重要实践价值。

关键词: 动态环境, 无人机, 路径规划, 深度强化学习

Abstract:

In response to the problem of unmanned aerial vehicle （UAV） delivery path planning in complex dynamic environments, this paper constructs a comprehensive economic index that simultaneously considers flight costs, delay penalties, and collision risks. The dynamic change rate of the environment is introduced to characterize the time-varying degree of obstacles. A hybrid deep Q-network and multi-head attention deep reinforcement learning（HDMADRL） is proposed. The simulation results show that HDMADRL can maintain a task success rate of approximately 75% under highly dynamic boundary conditions. In medium to high dynamic scenarios, compared to baseline methods, HDMADRL reduces the average cost by 8%−14%, reduces the average collision frequency by 20%−35%, and increases the task success rate by 6%−15%. The ablation experiment further proves the excellent performances of the “position-environment dual path encoding+multi head attention” structure. The research results provide a new UAV delivery path planning method, which has important practical values in reducing operating costs and ensuring flight safety.

Key words: dynamic environment, unmanned aerial vehicle, path planning, deep reinforcement learning

中图分类号:

V 249

陈可嘉, 王朕卿, 司徒腾宽. 动态环境下无人机路径规划的改进深度强化学习算法[J]. 系统工程与电子技术, 2026, 48(11): 3918-3931.

Kejia Chen, Zhenqing Wang, Tengkuan Situ. Improved deep reinforcement learning algorithm for unmanned aerial vehicle path planning in dynamic environments[J]. Systems Engineering and Electronics, 2026, 48(11): 3918-3931.

图/表 11

图1

表1

表2

图2

表3

图3

图4

图5

表4

图6

图7

参考文献 41

1	Rahman M, Sarkar N I, Lutui R. A survey on multi-UAV path planning: classification, algorithms, open research problems, and future directions[J]. Drones, 2025, 9 (4): 263.
2	Zhou J H, Wang Y, Xie Q, et al. A UAV path planning method based on deep reinforcement learning with dense rewards[J]. IEEE Open Journal of Vehicular Technology, 2026, 7, 260. doi: 10.1109/OJVT.2025.3642719
3	Javed S, Ahmad R, Hassan A, et al. Critical node-aware UAV swarm path planning in disaster zones[J]. IEEE Internet of Things Journal, 2026, 13 (2): 2020. doi: 10.1109/JIOT.2025.3631566
4	Lu Y, Yan D, Wan Z Q, et al. Conflict-free 3D path planning for multi-UAV based on jump point search and incremental update[J]. Drones, 2025, 9 (10): 688.
5	张海阔, 孟秀云. 基于改进RRT*算法的无人机在线航迹规划[J]. 系统工程与电子技术, 2024, 46 (12): 4157.
6	Lee S, Ban T W, Lee H. Network-wide energy efficiency maximization in UAV-aided IoT networks: quasi-distributed deep reinforcement learning approach[J]. IEEE Internet of Things Journal, 2025, 12 (11): 15404.
7	Taher B H, Luo J, Qiao Y, et al. PAIR: a hybrid A* with PPO path planner for multi-UAV navigation in 2-D dynamic urban MEC environments[J]. Drones, 2026, 10 (1): 58.
8	陈际同, 周佳加, 吴迪, 等. 基于TD3-RRT的特殊环境下USV路径规划算法研究[J]. 系统仿真学报, 2025, 37 (11): 2888.
9	Fu J Y, Sun G H, Yao W R, et al. Dubins path planning of heterogeneous UAV collaborative data collection for IoT network[J]. IEEE Trans. on Intelligent Transportation Systems, 2026, 27 (2): 2214.
10	余子杰, 郑征, 李清东, 等. 基于深度强化学习的太阳能无人机航迹规划[J]. 航空学报, 2025, 46 (12): 280.
11	Huang H J, Yang Y C, Wang H, et al. Deep reinforcement learning for UAV navigation through massive MIMO technique[J]. IEEE Trans. on Vehicular Technology, 2020, 69 (1): 1117.
12	吕茂隆, 丁晨博, 韩浩然, 等. 基于深度强化学习的无人机自主感知-规划-控制策略[J]. 自动化学报, 2025, 51 (6): 1305.
13	Brauwers G, Frasincar F. A general survey on attention mechanisms in deep learning[J]. IEEE Trans. on Knowledge and Data Engineering, 2023, 35 (4): 3279.
14	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems, 2017.
15	Almahamid F, Grolinger K. Agile DQN: adaptive deep recurrent attention reinforcement learning for autonomous UAV obstacle avoidance[J]. Scientific Reports, 2025, 15 (1): 18043.
16	Li Z R, Zhao Y M, Zhang X H, et al. Word embedding factor based multi-head attention[J]. Artificial Intelligence Review, 2025, 58, 115.
17	Jarray R, Bouallègue S. Improved deep Q-network-based UAV path planning in partially observable environments[J]. Cluster Computing, 2025, 28 (16): 1046.
18	Zhang H, Gao Y, Wang X, et al. UAV swarm path planning for sensor data collection via double pre-partitioned deep Q-network[J]. IEEE Sensors Journal, 2025, 25 (17): 34091.
19	Wang Y Y, Xin B, Ding Y L, et al. Online path planning of messenger UAV in air-ground coordination over urban areas[J]. IEEE Trans. on Vehicular Technology, 2024, 74 (5): 7135.
20	Gu R J, Zhao Y F, Ren X H, et al. Integrating wind field analysis in UAV path planning: enhancing safety and energy efficiency for urban logistics[J]. Chinese Journal of Aeronautics, 2026, 39 (1): 103605.
21	Zhang J, Campbell J F, Sweeney D C, et al. Energy consumption models for delivery drones: a comparison and assessment[J]. Transportation Research Part D: Transport and Environment, 2021, 90, 102668.
22	Liang X, Meng G L, Xu Y M, et al. A geometrical path planning method for unmanned aerial vehicle in 2D/3D complex environment[J]. Intelligent Service Robotics, 2018, 11, 301.
23	袁昕旺, 苏金树, 夏雨生. 飞行自组网路由综述: 场景特性、多维分类与前景展望[J]. 计算机学报, 2025, 48 (12): 3000.
24	Ragi S, Chong E K P. UAV path planning in a dynamic environment via partially observable Markov decision process[J]. IEEE Trans. on Aerospace and Electronic Systems, 2013, 49 (4): 2397.
25	Millar R C, Hashemi L, Mahmoodi A, et al. Integrating unmanned and manned UAVs data network based on combined Bayesian belief network and multiobjective reinforcement learning algorithm[J]. Drone Systems and Applications, 2023, 11, 1.
26	Demir K, Tumen V, Kosunalp S, et al. A deep reinforcement learning algorithm for trajectory planning of swarm UAV fulfilling wildlife reconnaissance[J]. Electronics, 2024, 13 (13): 2568.
27	Chhikara P, Tekchandani R, Kumar N, et al. DCNN-GA: a deep neural net architecture for navigation of UAV in indoor environment[J]. IEEE Internet of Things Journal, 2020, 8 (6): 4448.
28	Kong F, Li J, Jiang B, et al. Trajectory optimization for drone logistics delivery via attention-based pointer network[J]. IEEE Trans. on Intelligent Transportation Systems, 2022, 24 (4): 4519.
29	Chen D Z, Qi Q, Zhuang Z R, et al. Mean field deep reinforcement learning for fair and efficient UAV control[J]. IEEE Internet of Things Journal, 2020, 8 (2): 813.
30	Hu Z J, Gao X G, Wan K F, et al. Asynchronous curriculum experience replay: a deep reinforcement learning approach for UAV autonomous motion control in unknown dynamic environments[J]. IEEE Trans. on Vehicular Technology, 2023, 72 (11): 13985.
31	Matter D, Potgieter L. Allocating epidemic response teams and vaccine deliveries by drone in generic network structures, according to expected prevented exposures[J]. PLoS One, 2021, 16 (3): e0248053.
32	Betti S F, Coro F, Das S K, et al. Greedy algorithms for scheduling package delivery with multiple drones[C]//23rd International Conference on Distributed Computing and Networking, 2022: 31.
33	Akshya J, Neelamegam G, Sureshkumar C, et al. Enhancing UAV path planning efficiency through Adam-optimized deep neural networks for area coverage missions[J]. Procedia Computer Science, 2024, 235, 2.
34	Mnih V, Kavukcuoglu K. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518 (7540): 529.
35	Kim M J, Kim J S, Ahn C W. Evolving population method for real-time reinforcement learning[J]. Expert Systems with Applications, 2023, 229, 120493.
36	Adnan M H, Zukarnain Z A, Subramaniam S K, et al. Efficient multi-UAV coordination via RL: joint path planning and task scheduling for post-disaster UAV-assisted MEC systems[J]. Computer Networks, 2026, 274, 111822.
37	Li B H, Chen B D. An adoptive rapidly-exploring random tree[J]. IEEE/CAA Journal of Automatica Sinica, 2021, 9 (2): 283.
38	Van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double Q-learning[C]//AAAI Conference on Artificial Intelligence, 2016.
39	Wang Z, Schaul T, Hessel M, et al. Dueling network architectures for deep reinforcement learning[C]//International Conference on Machine Learning, 2016: 1995.
40	Mnih V, Badia A P, Mirza M, et al. Asynchronous methods for deep reinforcement learning[C]//International Conference on Machine Learning, 2016: 1928.
41	Lei H, Ran H, Ansari I S, et al. DDPG-based aerial secure data collection[J]. IEEE Trans. on Communications, 2024, 72 (8): 5179.

类别	参数	取值/说明
环境设置	网格尺寸	8×8
	最大步数$ {T}_{\max } $	50 步/回合
	起点/终点	每回合随机采样，保证不重合
	初始障碍物	按固定比例随机生成禁飞单元
	环境动态性$ {p}_{\text{flip}} $	{0.00, 0.02, 0.04, 0.06, 0.08, 0.10, 0.12, 0.14, 0.16, 0.18, 0.20}
	动态更新机制	每个时间步，每个网格单元以概率$ {p}_{\text{flip}} $进行“障碍/自由”状态翻转
状态与动作空间	动作空间	5个离散动作：上/下/左/右/原地不动
状态与动作空间	状态向量维度	68 维：当前坐标（$ \dfrac{x}{8},\dfrac{y}{8}, $）、目标坐标（$ \dfrac{{g}_{x}}{8},\dfrac{{g}_{y}}{8}, $），加上展平后的8×8网格状态（64维）
训练与算法超参数	学习率	1×10⁻⁴
	经验回放容量$ \|\mathcal{D}\| $	10 000
	训练批大小	64
	探索策略	$ \varepsilon $-贪婪探索策略
	$ {\varepsilon }_{\text{init}}/{\varepsilon }_{\min } $	1.0/0.05
	衰减系数$ \varepsilon $	每回合按$ \varepsilon $← 0.995×$ \varepsilon $衰减
	目标网络更新	软更新，系数τ = 0.01（每若干步执行一次）

$ {p}_{\text{flip}} $	任务成功率/%	平均成本/元	平均碰撞次数
0	99.2	96.40	2.13
0.02	98.0	100.97	6.40
0.04	96.5	105.98	9.38
0.06	94.0	108.59	13.94
0.08	91.2	111.02	16.76
0.10	88.5	113.45	17.27
0.12	85.1	116.90	19.22
0.14	82.0	118.80	21.12
0.16	79.4	119.90	22.47
0.18	77.0	120.50	23.85
0.20	74.8	121.10	24.92

算法	$ {p}_{{\mathrm{flip}}} $	任务成功率/%	平均成本/元	平均碰撞次数
HDMADRL	0.02	98	100.97	6.4
Dueling DQN	0.02	98.3	101.2	6.9
Double DQN	0.02	97.8	101.8	7.5
A3C	0.02	98.5	102.3	7.8
DDPG	0.02	97	103	8.2
Plain-DQN	0.02	96.5	103.2	9.1
RRT	0.02	93	108.6	14.5
HDMADRL	0.1	88.5	113.45	17.27
Dueling DQN	0.1	86	118.7	20.4
Double DQN	0.1	85.2	120.3	21.1
A3C	0.1	84.5	121.5	21.9
DDPG	0.1	83.8	123	22.8
Plain-DQN	0.1	82	125.6	23.9
RRT	0.1	72.5	138.4	29.8
HDMADRL	0.2	74.8	121.1	24.92
Dueling DQN	0.2	68.3	131.4	24.4
Double DQN	0.2	66.5	133.2	30.1
A3C	0.2	65.2	134.9	31
DDPG	0.2	63.8	136.1	31.7
Plain-DQN	0.2	61	139.8	32.8
RRT	0.2	49	157.5	38.9

模型	$ {p}_{\text{flip}} $	任务成功率/%	平均成本/元	平均碰撞次数
HDMADRL	0.02	98	100.97	6.40
HDMADRL-Attn	0.02	97.5	102.50	7.40
HDMADRL-Dual	0.02	97	102.00	7.10
HDMADRL	0.10	88.5	113.45	17.27
HDMADRL-Attn	0.10	86.7	117.80	20.10
HDMADRL-Dual	0.10	84.8	119.10	19.30
HDMADRL	0.20	74.8	121.10	24.92
HDMADRL-Attn	0.20	71.5	128.10	29.20
HDMADRL-Dual	0.20	68.9	126.20	27.50

[1]	赵立阳, 吴金平, 杨静, 张瑞. 基于深度强化学习的多智能体集中式作战行动指挥决策方法[J]. 系统工程与电子技术, 2026, 48(9): 3089-3103.
[2]	王兆辰, 杨华东, 孙海文, 靳子荣. 基于霜冰优化算法的无人机路径优选CNN-LSTM-Attention模型[J]. 系统工程与电子技术, 2026, 48(9): 3104-3118.
[3]	张杰, 朱峰, 王超, 李东, 陈志群, 吕天启. 逻辑链构建下基于深度强化学习的多智能体协同算法优化[J]. 系统工程与电子技术, 2026, 48(9): 3126-3135.
[4]	张倬境, 盛永智, 龚孝龙. 深度强化学习驱动的多约束分数阶滑模制导律[J]. 系统工程与电子技术, 2026, 48(9): 3167-3178.
[5]	邢娜, 唐旭莹, 王月海, 王婕, 宁可庆. 基于A^*引导的高斯伪谱法无人机协同轨迹规划[J]. 系统工程与电子技术, 2026, 48(9): 3210-3219.
[6]	刘锡国, 刘敏, 毛忠阳, 徐盟飞, 马鹏. 基于信息调节LSTM的抗干扰决策算法[J]. 系统工程与电子技术, 2026, 48(9): 3266-3273.
[7]	黄敏, 李召瑞, 孙慧贤, 倪想. 基于动态加权均匀圆阵的无人机鲁棒测向方法[J]. 系统工程与电子技术, 2026, 48(8): 2520-2525.
[8]	班宏辉, 潘继飞, 王正, 崔瑞. 基于多无人机协同的航迹欺骗误差分析[J]. 系统工程与电子技术, 2026, 48(7): 2195-2205.
[9]	许贝佳, 刘学达, 禹明刚, 章乐贵. 基于多策略公共物品博弈的无人机集群弹药投放[J]. 系统工程与电子技术, 2026, 48(7): 2340-2351.
[10]	关巍, 胡彤博, 张显库, 罗春奇, 曲胜, 崔哲闻. 基于改进PPO算法的无人机/无人船协同导航方法[J]. 系统工程与电子技术, 2026, 48(7): 2457-2468.
[11]	马赞, 刘禹彬, 白杰, 陈勇, 孙淑光. 基于分层STPA-MC的无人机智能避让适航安全风险评估[J]. 系统工程与电子技术, 2026, 48(6): 2000-2013.
[12]	刘瑞航, 刘海颖, 刘宇辰, 陈晨, 李铁香. 基于多特征正交约束的无人机跨视角地理定位方法[J]. 系统工程与电子技术, 2026, 48(6): 2072-2080.
[13]	张国庆, 姚桂鹏, 李纪强. 无人机/船协同降落的鲁棒自适应模糊控制[J]. 系统工程与电子技术, 2026, 48(6): 2081-2088.
[14]	林文斌, 时晨光, 严牧, 汪飞, 周建江. 面向欺骗干扰组网雷达的无人机集群稳健航迹规划算法[J]. 系统工程与电子技术, 2026, 48(5): 1551-1563.
[15]	贾振帅, 肖冰, 钱寒雨, 张哲宇. 基于多策略学习的航天器在轨观测机动决策方法[J]. 系统工程与电子技术, 2026, 48(5): 1590-1598.