基于多智能体强化学习的无人机协同截击机动决策研究

doi:10.12305/j.issn.1001-506X.2025.09.29

系统工程与电子技术 ›› 2025, Vol. 47 ›› Issue (9): 3076-3085.doi: 10.12305/j.issn.1001-506X.2025.09.29

• 制导、导航与控制 • 上一篇

基于多智能体强化学习的无人机协同截击机动决策研究

杨大鹏¹^,², 龚资浩³, 王小也²^,*, 郭正玉⁴^,⁵, 罗德林³

1. 复旦大学航空航天系，上海 200433
2. 沈阳飞机设计研究所，辽宁沈阳 110035
3. 厦门大学航空航天学院，福建厦门 361102
4. 中国空空导弹研究院，河南洛阳，471000
5. 空基信息感知与融合全国重点实验室，河南洛阳，471000

收稿日期:2023-09-11 出版日期:2025-09-25 发布日期:2025-09-16
通讯作者: 王小也
作者简介:杨大鹏（1987—），男，高级工程师，硕士，主要研究方向为飞行器控制
龚资浩（1998—），男，硕士研究生，主要研究方向为无人机协同决策与控制
郭正玉（1982—），男，高级工程师，博士，主要研究方向为飞行器协同控制、信息感知与融合
罗德林（1968—），男，教授，博士，主要研究方向为飞行器制导与控制、无人机协同决策与控制、计算智能
基金资助:
空基信息感知与融合全国重点实验室与航空科学基金联合资助项目（20220001068001）资助课题

Research on UAV cooperative interception maneuver decision-making based on multi-agent reinforcement learning

Dapeng YANG¹^,², Zihao GONG³, Xiaoye WANG²^,*, Zhengyu GUO⁴^,⁵, Delin LUO³

1. Department of Aeronautics and Astronautics，Fudan University，Shanghai 200433，China
2. Shenyang Aircraft Design and Research Institute，Shenyang 110035，China
3. School of Aerospace Engineering，Xiamen University，Xiamen 361102，China
4. China Airborne Missile Academy，Luoyang 471000，China
5. National Key Laboratory of Air-based Information Perception and Fusion，Luoyang 471000，China

Received:2023-09-11 Online:2025-09-25 Published:2025-09-16
Contact: Xiaoye WANG

摘要/Abstract

摘要：

无人机智能化协同截击博弈对抗是未来空战的重要作战场景。针对无人机协同战术截击问题，构建基于多智能体强化学习的战术截击决策框架。首先，对截击空战过程中相对态势几何关系进行分析。随后，根据截击空战态势威胁模型设置截击空战奖励函数。最后，设置无人机独立动作价值网络、编队联合动作价值网络和状态价值网络，以形成无人机协同截击战术生成最优截击策略，并引入截击线评估该截击策略的有效性。仿真结果表明，面对动态博弈条件下的多目标拦截任务，该框架能自主进行拦截目标分配并且形成智能协同截击战术。

关键词: 多目标协同截击, 截击战术, 无人机, 多智能体强化学习

Abstract:

The intelligent cooperative interception and confrontation game involving unmanned aerial vehicle （UAV） are crucial combat scenarios for the future of air warfare. To address the problem of UAV cooperative tactical interception, a tactical interception decision-making framework is proposed based on multi-agent reinforcement learning. Firstly, the relative situation geometric relationship between the interception process and the air combat situation is analyzed to form the interception air combat state space. Subsequently, an interception air combat reward function is set according to the interception air combat situational threat model. Finally, the establishment of the independent action value network for UAV, the collective action value network for formations, and the state value network is employed to formulate the optimal interception strategy for cooperative UAV interception tactics. The effectiveness of this interception strategy is evaluated by introducing the concept of an interception boundary. Simulation results show that the framework can autonomously assign interception targets and form intelligent cooperative interception tactics when facing multi-target interception tasks under dynamic game conditions.

Key words: multi-target cooperative interception, interception tactics, unmanned aerial vehicle （UAV）, multi-agent reinforcement learning （MARL）

中图分类号:

TP 29

杨大鹏, 龚资浩, 王小也, 郭正玉, 罗德林. 基于多智能体强化学习的无人机协同截击机动决策研究[J]. 系统工程与电子技术, 2025, 47(9): 3076-3085.

Dapeng YANG, Zihao GONG, Xiaoye WANG, Zhengyu GUO, Delin LUO. Research on UAV cooperative interception maneuver decision-making based on multi-agent reinforcement learning[J]. Systems Engineering and Electronics, 2025, 47(9): 3076-3085.

图/表 11

图1

图2

图3

表1

图4

图5

表2

表3

图6

图7

图8

参考文献 31

1	SKOROBOGATOV G, BARRADO C, SALAMÍ E. Multiple UAV systems: a survey[J]. Unmanned Systems, 2020, 8 (2): 149- 169. doi: 10.1142/S2301385020500090
2	高树一, 林德福, 郑多, 等. 针对集群攻击的飞行器智能协同拦截策略研究[J]. 航空学报, 2023, 44 (18): 328301.
	GAO S Y, LIN D F, ZHENG D, et al. Intelligent cooperative interception strategy of aircraft against cluster attack[J]. Acta Aeronautica et Astronautica Sinica, 2023, 44 (18): 328301.
3	REZENDE M D, DE-LIMA B S P, GUIMARÃES S. A greedy ant colony system for defensive resource assignment problems[J]. Applied Artificial Intelligence, 2018, 32 (2): 138- 152. doi: 10.1080/08839514.2018.1451137
4	余敏建, 嵇慧明, 韩其松, 等. 基于合作协同进化的多机空战目标分配[J]. 系统工程与电子技术, 2020, 42 (6): 1290- 1300. doi: 10.3969/j.issn.1001-506X.2020.06.12
	YU M J, JI H M, HAN Q S, et al. Multi-aircraft air combat target allocation based on cooperative co-evolutionary[J]. Systems Engineering and Electronics, 2020, 42 (6): 1290- 1300. doi: 10.3969/j.issn.1001-506X.2020.06.12
5	郑都民. 集群空战中的群威胁评估及目标分配关键技术研究[D]. 西安: 西安电子科技大学, 2020.
	ZHENG D M. Key technology research of group threat assessment and target allocation in cluster air combat[D]. Xi’an: Xidian University, 2020.
6	SHARMA A, SHOVAL S, SHARMA A, et al. Path planning for multiple targets interception by the swarm of UAVs based on swarm intelligence algorithms: a review[J]. IEEE Technical Review, 2022, 39 (3): 675- 697. doi: 10.1080/02564602.2021.1894250
7	GUO D, LIANG Z X, JIANG P, et al. Weapon-target assignment for multi-to-multi interception with grouping constraint[J]. IEEE Access, 2019, 7, 34838- 34849. doi: 10.1109/ACCESS.2019.2898874
8	GUO J G, HU G J, GUO Z Y, et al. Evaluation model, intelligent assignment, and cooperative interception in multimissile and multitarget engagement[J]. IEEE Trans. on Aerospace Electronic Systems, 2022, 58 (4): 3104- 3115. doi: 10.1109/TAES.2022.3144111
9	ZHU R, SUN D, ZHOU Z Y. Cooperation strategy of unmanned air vehicles for multitarget interception[J]. Journal of Guidance, Control, and Dynamics, 2005, 28 (5): 1068- 1072.
10	徐鑫宇, 万路军, 陈平, 等. 防御性制空截击中警戒巡逻空域规划建模[J]. 系统工程与电子技术, 2022, 44 (5): 1589- 1599. doi: 10.12305/j.issn.1001-506X.2022.05.20
	XU X Y, WAN L J, CHEN P, et al. Modeling of alert patrol airspace planning in defensive counterair interception operation[J]. Systems Engineering and Electronics, 2022, 44 (5): 1589- 1599. doi: 10.12305/j.issn.1001-506X.2022.05.20
11	李俊, 林云, 韩玉龙, 等. 基于态势威胁的歼击机截击航路优化方法研究[J]. 舰船电子工程, 2016, 36 (10): 19- 22. doi: 10.3969/j.issn.1672-9730.2016.10.006
	LI J, LIN Y, HAN Y L, et al. Research on the optimization of air route in air-interception based on threat situation[J]. Ship Electronic Engineering, 2016, 36 (10): 19- 22. doi: 10.3969/j.issn.1672-9730.2016.10.006
12	罗木生, 张毅, 牛庆功, 等. 航空兵截击来袭敌机的起飞窗口计算方法[J]. 指挥控制与仿真, 2021, 43 (2): 123- 126.
	LUO M S, ZHANG Y, NIU Q G, et al. Method of take-off window calculation for aviation intercepting enemy aircraft[J]. Command Control and Simulation, 2021, 43 (2): 123- 126.
13	SINHA A, KUMAR S R. Supertwisting control-based cooperative salvo guidance using leader–follower approach[J]. IEEE Trans. on Aerospace Electronic Systems, 2020, 56 (5): 3556- 3565. doi: 10.1109/TAES.2020.2974044
14	ZHANG P, ZHANG X Y. Multiple missiles fixed-time cooperative guidance without measuring radial velocity for maneuvering targets interception[J]. ISA Transactions, 2022, 126, 388- 397. doi: 10.1016/j.isatra.2021.07.023
15	XU J W, ZHANG J, YANG L Y, et al. Autonomous decision-making for dogfights based on a tactical pursuit point approach[J]. Aerospace Science Technology, 2022, 129, 107857. doi: 10.1016/j.ast.2022.107857
16	DONG X F, REN Z. Impact angle constrained distributed cooperative guidance against maneuvering targets with undirected communication topologies[J]. IEEE Access, 2020, 8, 117867- 117876. doi: 10.1109/ACCESS.2020.3004128
17	WANG C C, WU A, HOU Y Q, et al. Optimal deployment of swarm positions in cooperative interception of multiple UAV swarms[J]. Digital Communications Networks, 2023, 9 (2): 567- 579. doi: 10.1016/j.dcan.2022.04.002
18	MOREIRA M, PAPP E, VENTURA R. Interception of non-cooperative UAVs[C]//Proc. of the IEEE International Symposium on Safety, Security, and Rescue Robotics, 2019: 120−125.
19	LI J G, XIE M Y, DONG Y F, et al. RTPN method for cooperative interception of maneuvering target by gun-launched UAV[J]. Mathematical Biosciences, 2022, 19 (5): 5190- 5206. doi: 10.3934/mbe.2022243
20	GHOSH S, DAVIS D T, CHUNG T H, et al. Development and testing of the intercept primitives for planar UAV engagement[C]//Proc. of the IEEE International Conference on Unmanned Aircraft Systems, 2017: 286−293.
21	TONG B D, LIU J, DUAN H B. Multi-UAV interception inspired by harris’ hawks cooperative hunting behavior[C]//Proc. of the IEEE International Conference on Robotics and Biomimetics, 2021: 1656−1661.
22	陈灿, 莫雳, 郑多, 等. 非对称机动能力多无人机智能协同攻防对抗[J]. 航空学报, 2020, 41 (12): 324152.
	CHEN C, MO L, ZHENG D, et al. Cooperative attack-defense game of multiple UAVs with asymmetric maneuverability[J]. Acta Aeronautica et Astronautica Sinica, 2020, 41 (12): 324152.
23	WANG B L, LI S G, GAO X Z, et al. UAV swarm confrontation using hierarchical multiagent reinforcement learning[J]. International Journal of Aerospace Engineering, 2021, 2021 (5): 3360116.
24	LEE S M, KIM H, MYUNG H, et al. Cooperative coevolutionary algorithm-based model predictive control guaranteeing stability of multirobot formation[J]. IEEE Trans. on Control Systems Technology, 2014, 23 (1): 37- 51.
25	LUO Y X, SONG J, ZHAO K, et al. UAV-cooperative penetration dynamic-tracking interceptor method based on DDPG[J]. Applied Sciences, 2022, 12 (3): 1618. doi: 10.3390/app12031618
26	LING H F, LUO H C, CHEN H, et al. Modelling and simulation of distributed UAV swarm cooperative planning and perception[J]. International Journal of Aerospace Engineering, 2021, 2021, 9977262. doi: 10.1155/2021/9977262
27	刘书元. 一种威胁指数加权的截击引导航线解算算法[J]. 系统仿真学报, 2023, 35 (5): 971- 978.
	LIU S Y. An algorithm used to obtain interception guidance solutions weighted by threat indexes[J]. Journal of System Simulation, 2023, 35 (5): 971- 978.
28	SON K, KIM D, KANG W J, et al. Qtran: learning to factorize with transformation for cooperative multi-agent reinforcement learning[C]// Proc. of the International Conference on Machine Learning, 2019: 5887−5896.
29	高东万. 基于协同强化学习的群体决策方法研究[D]. 大连: 大连理工大学, 2021.
	GAO D W. Research on group decision making based on collaborative reinforcement learning[D]. Dalian: Dalian University of Technology, 2021.
30	WANG Z, SCHAUL T, HESSEL M, et al. Dueling network architectures for deep reinforcement learning[C]//Proc. of the International Conference on Machine Learning, 2016: 1995−2003.
31	FAN J Q, WANG Z R, XIE Y C, et al. A theoretical analysis of deep Q-learning[C]//Proc. of the Learning for Dynamics and Control, 2020: 486−489.

编号	机动动作	切向过载${n_x}$	法向过载${n_n}$	滚转角$\mu $/rad
${a_1}$	匀速直飞	0	1	0
${a_2}$	加速直飞	2	1	0
${a_3}$	减速直飞	−2	1	0
${a_4}$	跃升	0	8	0
${a_5}$	俯冲	0	−8	0
${a_6}$	左转跃升	0	8	${{\text{π}}\mathord{\left/ {\vphantom {\pi 3}} \right. } 3}$
${a_7}$	右转俯冲	0	−8	${{ - {\text{π}} } \mathord{\left/ {\vphantom {{ - {\text{π}}} 3}} \right. } 3}$
${a_8}$	右转跃升	0	8	${{ -{\text{π}}} \mathord{\left/ {\vphantom {{ - \pi } 3}} \right. } 3}$
${a_9}$	左转俯冲	0	−8	${{\text{π}} \mathord{\left/ {\vphantom {\pi 3}} \right. } 3}$
${a_{10}}$	右转	0	2	${{ - {\text{π}}} \mathord{\left/ {\vphantom {{ - \pi } 3}} \right. } 3}$
${a_{11}}$	左转	0	2	${{\text{π}} \mathord{\left/ {\vphantom {{\text{π}}3}} \right. } 3}$

无人机	起始坐标/m	航速/（m/s）	俯仰角/rad	航向角/rad
$ {R_1} $	（0, 0, 5000）	250	0	0
${R_2}$	（0, 1000, 5000）	250	0	0
${R_3}$	（0, 2000, 5000）	250	0	0
${B_1}$	（30000, 0, 4800）	248	0	${\text{π}}$
${B_2}$	（30000, 2000, 4800）	248	0	${\text{π}} $

超参数	取值	超参数	取值
Memory_size	1000	Replace_target_iter	200
Batch_size	64	Episode_per_learning	2
学习率$\alpha $	0.0001	折扣因子$\gamma $	0.95
Max_episode_steps	150	Maximum_time_steps	20000000
权重系数${\lambda _{{\mathrm{opt}}}}$	1	权重系数${\lambda _{{\mathrm{nopt}}}}$	1

[1]	闻雯, 时晨光, 周建江. 多元威胁环境下无人机集群隐身航迹规划算法[J]. 系统工程与电子技术, 2025, 47(9): 2971-2984.
[2]	魏潇龙, 吴亚荣, 姚登凯, 赵顾颢. 基于深度强化学习的无人机空战机动分层决策算法[J]. 系统工程与电子技术, 2025, 47(9): 2993-3003.
[3]	张国庆, 徐轶晖, 李纪强, 张显库, 邱斌. 基于异步搜寻制导的机/船协同事件触发控制[J]. 系统工程与电子技术, 2025, 47(9): 3058-3065.
[4]	张欣悦, 吴晓莉, 王名珺, 晏彪, 武愈涵. 有/无人机协同操作界面的最佳交互方式评估[J]. 系统工程与电子技术, 2025, 47(8): 2600-2611.
[5]	李延通, 李子璠, 周姗姗, 张闯. 无人机光伏电站巡检双目标选址-路径问题研究[J]. 系统工程与电子技术, 2025, 47(8): 2612-2621.
[6]	羊钊, 胡锦标, 王艳, 齐洪彪. 考虑异巢起降的无人机山地巡检覆盖路径规划[J]. 系统工程与电子技术, 2025, 47(8): 2622-2631.
[7]	闫小伟, 凌冲, 石胜斌. 地表未爆子弹药快速检测系统设计与实现[J]. 系统工程与电子技术, 2025, 47(8): 2639-2645.
[8]	张晓璐, 陈亚洲, 赵敏. 无人机数据链带内电磁干扰效应预测模型与验证[J]. 系统工程与电子技术, 2025, 47(8): 2763-2773.
[9]	符小卫, 王辛夷, 乔哲. 基于APIQ算法的多无人机攻防对抗策略[J]. 系统工程与电子技术, 2025, 47(7): 2205-2215.
[10]	郑凯文, 杜承泽, 赵兴芳, 逄晓凡. 融合时空散列的三维RRT^*多编队航路规划[J]. 系统工程与电子技术, 2025, 47(7): 2256-2266.
[11]	吴北苹, 何晶, 党慧莹, 岳地久. 基于作战环的反无人机作战体系贡献率评估[J]. 系统工程与电子技术, 2025, 47(7): 2267-2274.
[12]	林思颖, 郁丰, 熊智, 吴方, 周紫君. 基于AHRS的GNSS间断拒止下低成本无人机导航方法[J]. 系统工程与电子技术, 2025, 47(7): 2329-2338.
[13]	唐俊超, 胡春鹤. 三维地形风场环境下无人机全覆盖路径规划[J]. 系统工程与电子技术, 2025, 47(7): 2349-2356.
[14]	何云风, 史贤俊, 卢建华, 赵超轮, 赵国荣. 故障条件下基于同步DMPC的多无人机分组编队控制[J]. 系统工程与电子技术, 2025, 47(7): 2357-2370.
[15]	符小卫, 王辛夷, 乔哲. 基于ASDDPG算法的多无人机对抗策略[J]. 系统工程与电子技术, 2025, 47(6): 1867-1879.

基于多智能体强化学习的无人机协同截击机动决策研究

Research on UAV cooperative interception maneuver decision-making based on multi-agent reinforcement learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 31

相关文章 15

编辑推荐

Metrics

本文评价