基于多智能体强化学习的无人机协同截击机动决策研究

doi:10.12305/j.issn.1001-506X.2025.09.29

Abstract

Abstract:

The intelligent cooperative interception and confrontation game involving unmanned aerial vehicle （UAV） are crucial combat scenarios for the future of air warfare. To address the problem of UAV cooperative tactical interception, a tactical interception decision-making framework is proposed based on multi-agent reinforcement learning. Firstly, the relative situation geometric relationship between the interception process and the air combat situation is analyzed to form the interception air combat state space. Subsequently, an interception air combat reward function is set according to the interception air combat situational threat model. Finally, the establishment of the independent action value network for UAV, the collective action value network for formations, and the state value network is employed to formulate the optimal interception strategy for cooperative UAV interception tactics. The effectiveness of this interception strategy is evaluated by introducing the concept of an interception boundary. Simulation results show that the framework can autonomously assign interception targets and form intelligent cooperative interception tactics when facing multi-target interception tasks under dynamic game conditions.

Key words: multi-target cooperative interception, interception tactics, unmanned aerial vehicle （UAV）, multi-agent reinforcement learning （MARL）

CLC Number:

TP 29

Dapeng YANG, Zihao GONG, Xiaoye WANG, Zhengyu GUO, Delin LUO. Research on UAV cooperative interception maneuver decision-making based on multi-agent reinforcement learning[J]. Systems Engineering and Electronics, 2025, 47(9): 3076-3085.

Figures/Tables 11

Fig.1

Fig.2

Fig.3

Table 1

Fig.4

Fig.5

Table 2

Table 3

Fig.6

Fig.7

Fig.8

References 31

1	SKOROBOGATOV G, BARRADO C, SALAMÍ E. Multiple UAV systems: a survey[J]. Unmanned Systems, 2020, 8 (2): 149- 169. doi: 10.1142/S2301385020500090
2	高树一, 林德福, 郑多, 等. 针对集群攻击的飞行器智能协同拦截策略研究[J]. 航空学报, 2023, 44 (18): 328301.
	GAO S Y, LIN D F, ZHENG D, et al. Intelligent cooperative interception strategy of aircraft against cluster attack[J]. Acta Aeronautica et Astronautica Sinica, 2023, 44 (18): 328301.
3	REZENDE M D, DE-LIMA B S P, GUIMARÃES S. A greedy ant colony system for defensive resource assignment problems[J]. Applied Artificial Intelligence, 2018, 32 (2): 138- 152. doi: 10.1080/08839514.2018.1451137
4	余敏建, 嵇慧明, 韩其松, 等. 基于合作协同进化的多机空战目标分配[J]. 系统工程与电子技术, 2020, 42 (6): 1290- 1300. doi: 10.3969/j.issn.1001-506X.2020.06.12
	YU M J, JI H M, HAN Q S, et al. Multi-aircraft air combat target allocation based on cooperative co-evolutionary[J]. Systems Engineering and Electronics, 2020, 42 (6): 1290- 1300. doi: 10.3969/j.issn.1001-506X.2020.06.12
5	郑都民. 集群空战中的群威胁评估及目标分配关键技术研究[D]. 西安: 西安电子科技大学, 2020.
	ZHENG D M. Key technology research of group threat assessment and target allocation in cluster air combat[D]. Xi’an: Xidian University, 2020.
6	SHARMA A, SHOVAL S, SHARMA A, et al. Path planning for multiple targets interception by the swarm of UAVs based on swarm intelligence algorithms: a review[J]. IEEE Technical Review, 2022, 39 (3): 675- 697. doi: 10.1080/02564602.2021.1894250
7	GUO D, LIANG Z X, JIANG P, et al. Weapon-target assignment for multi-to-multi interception with grouping constraint[J]. IEEE Access, 2019, 7, 34838- 34849. doi: 10.1109/ACCESS.2019.2898874
8	GUO J G, HU G J, GUO Z Y, et al. Evaluation model, intelligent assignment, and cooperative interception in multimissile and multitarget engagement[J]. IEEE Trans. on Aerospace Electronic Systems, 2022, 58 (4): 3104- 3115. doi: 10.1109/TAES.2022.3144111
9	ZHU R, SUN D, ZHOU Z Y. Cooperation strategy of unmanned air vehicles for multitarget interception[J]. Journal of Guidance, Control, and Dynamics, 2005, 28 (5): 1068- 1072.
10	徐鑫宇, 万路军, 陈平, 等. 防御性制空截击中警戒巡逻空域规划建模[J]. 系统工程与电子技术, 2022, 44 (5): 1589- 1599. doi: 10.12305/j.issn.1001-506X.2022.05.20
	XU X Y, WAN L J, CHEN P, et al. Modeling of alert patrol airspace planning in defensive counterair interception operation[J]. Systems Engineering and Electronics, 2022, 44 (5): 1589- 1599. doi: 10.12305/j.issn.1001-506X.2022.05.20
11	李俊, 林云, 韩玉龙, 等. 基于态势威胁的歼击机截击航路优化方法研究[J]. 舰船电子工程, 2016, 36 (10): 19- 22. doi: 10.3969/j.issn.1672-9730.2016.10.006
	LI J, LIN Y, HAN Y L, et al. Research on the optimization of air route in air-interception based on threat situation[J]. Ship Electronic Engineering, 2016, 36 (10): 19- 22. doi: 10.3969/j.issn.1672-9730.2016.10.006
12	罗木生, 张毅, 牛庆功, 等. 航空兵截击来袭敌机的起飞窗口计算方法[J]. 指挥控制与仿真, 2021, 43 (2): 123- 126.
	LUO M S, ZHANG Y, NIU Q G, et al. Method of take-off window calculation for aviation intercepting enemy aircraft[J]. Command Control and Simulation, 2021, 43 (2): 123- 126.
13	SINHA A, KUMAR S R. Supertwisting control-based cooperative salvo guidance using leader–follower approach[J]. IEEE Trans. on Aerospace Electronic Systems, 2020, 56 (5): 3556- 3565. doi: 10.1109/TAES.2020.2974044
14	ZHANG P, ZHANG X Y. Multiple missiles fixed-time cooperative guidance without measuring radial velocity for maneuvering targets interception[J]. ISA Transactions, 2022, 126, 388- 397. doi: 10.1016/j.isatra.2021.07.023
15	XU J W, ZHANG J, YANG L Y, et al. Autonomous decision-making for dogfights based on a tactical pursuit point approach[J]. Aerospace Science Technology, 2022, 129, 107857. doi: 10.1016/j.ast.2022.107857
16	DONG X F, REN Z. Impact angle constrained distributed cooperative guidance against maneuvering targets with undirected communication topologies[J]. IEEE Access, 2020, 8, 117867- 117876. doi: 10.1109/ACCESS.2020.3004128
17	WANG C C, WU A, HOU Y Q, et al. Optimal deployment of swarm positions in cooperative interception of multiple UAV swarms[J]. Digital Communications Networks, 2023, 9 (2): 567- 579. doi: 10.1016/j.dcan.2022.04.002
18	MOREIRA M, PAPP E, VENTURA R. Interception of non-cooperative UAVs[C]//Proc. of the IEEE International Symposium on Safety, Security, and Rescue Robotics, 2019: 120−125.
19	LI J G, XIE M Y, DONG Y F, et al. RTPN method for cooperative interception of maneuvering target by gun-launched UAV[J]. Mathematical Biosciences, 2022, 19 (5): 5190- 5206. doi: 10.3934/mbe.2022243
20	GHOSH S, DAVIS D T, CHUNG T H, et al. Development and testing of the intercept primitives for planar UAV engagement[C]//Proc. of the IEEE International Conference on Unmanned Aircraft Systems, 2017: 286−293.
21	TONG B D, LIU J, DUAN H B. Multi-UAV interception inspired by harris’ hawks cooperative hunting behavior[C]//Proc. of the IEEE International Conference on Robotics and Biomimetics, 2021: 1656−1661.
22	陈灿, 莫雳, 郑多, 等. 非对称机动能力多无人机智能协同攻防对抗[J]. 航空学报, 2020, 41 (12): 324152.
	CHEN C, MO L, ZHENG D, et al. Cooperative attack-defense game of multiple UAVs with asymmetric maneuverability[J]. Acta Aeronautica et Astronautica Sinica, 2020, 41 (12): 324152.
23	WANG B L, LI S G, GAO X Z, et al. UAV swarm confrontation using hierarchical multiagent reinforcement learning[J]. International Journal of Aerospace Engineering, 2021, 2021 (5): 3360116.
24	LEE S M, KIM H, MYUNG H, et al. Cooperative coevolutionary algorithm-based model predictive control guaranteeing stability of multirobot formation[J]. IEEE Trans. on Control Systems Technology, 2014, 23 (1): 37- 51.
25	LUO Y X, SONG J, ZHAO K, et al. UAV-cooperative penetration dynamic-tracking interceptor method based on DDPG[J]. Applied Sciences, 2022, 12 (3): 1618. doi: 10.3390/app12031618
26	LING H F, LUO H C, CHEN H, et al. Modelling and simulation of distributed UAV swarm cooperative planning and perception[J]. International Journal of Aerospace Engineering, 2021, 2021, 9977262. doi: 10.1155/2021/9977262
27	刘书元. 一种威胁指数加权的截击引导航线解算算法[J]. 系统仿真学报, 2023, 35 (5): 971- 978.
	LIU S Y. An algorithm used to obtain interception guidance solutions weighted by threat indexes[J]. Journal of System Simulation, 2023, 35 (5): 971- 978.
28	SON K, KIM D, KANG W J, et al. Qtran: learning to factorize with transformation for cooperative multi-agent reinforcement learning[C]// Proc. of the International Conference on Machine Learning, 2019: 5887−5896.
29	高东万. 基于协同强化学习的群体决策方法研究[D]. 大连: 大连理工大学, 2021.
	GAO D W. Research on group decision making based on collaborative reinforcement learning[D]. Dalian: Dalian University of Technology, 2021.
30	WANG Z, SCHAUL T, HESSEL M, et al. Dueling network architectures for deep reinforcement learning[C]//Proc. of the International Conference on Machine Learning, 2016: 1995−2003.
31	FAN J Q, WANG Z R, XIE Y C, et al. A theoretical analysis of deep Q-learning[C]//Proc. of the Learning for Dynamics and Control, 2020: 486−489.

编号	机动动作	切向过载${n_x}$	法向过载${n_n}$	滚转角$\mu $/rad
${a_1}$	匀速直飞	0	1	0
${a_2}$	加速直飞	2	1	0
${a_3}$	减速直飞	−2	1	0
${a_4}$	跃升	0	8	0
${a_5}$	俯冲	0	−8	0
${a_6}$	左转跃升	0	8	${{\text{π}}\mathord{\left/ {\vphantom {\pi 3}} \right. } 3}$
${a_7}$	右转俯冲	0	−8	${{ - {\text{π}} } \mathord{\left/ {\vphantom {{ - {\text{π}}} 3}} \right. } 3}$
${a_8}$	右转跃升	0	8	${{ -{\text{π}}} \mathord{\left/ {\vphantom {{ - \pi } 3}} \right. } 3}$
${a_9}$	左转俯冲	0	−8	${{\text{π}} \mathord{\left/ {\vphantom {\pi 3}} \right. } 3}$
${a_{10}}$	右转	0	2	${{ - {\text{π}}} \mathord{\left/ {\vphantom {{ - \pi } 3}} \right. } 3}$
${a_{11}}$	左转	0	2	${{\text{π}} \mathord{\left/ {\vphantom {{\text{π}}3}} \right. } 3}$

无人机	起始坐标/m	航速/（m/s）	俯仰角/rad	航向角/rad
$ {R_1} $	（0, 0, 5000）	250	0	0
${R_2}$	（0, 1000, 5000）	250	0	0
${R_3}$	（0, 2000, 5000）	250	0	0
${B_1}$	（30000, 0, 4800）	248	0	${\text{π}}$
${B_2}$	（30000, 2000, 4800）	248	0	${\text{π}} $

超参数	取值	超参数	取值
Memory_size	1000	Replace_target_iter	200
Batch_size	64	Episode_per_learning	2
学习率$\alpha $	0.0001	折扣因子$\gamma $	0.95
Max_episode_steps	150	Maximum_time_steps	20000000
权重系数${\lambda _{{\mathrm{opt}}}}$	1	权重系数${\lambda _{{\mathrm{nopt}}}}$	1

[1]	Xiaolong WEI, Yarong WU, Dengkai YAO, Guhao ZHAO. Hierarchical decision-making algorithm for UAV air combat maneuvering based on deep reinforcement learning [J]. Systems Engineering and Electronics, 2025, 47(9): 2993-3003.
[2]	Zhao YANG, Jinbiao HU, Yan WANG, Hongbiao QI. UAV coverage path planning for mountain patrol considering different takeoff and landing nests [J]. Systems Engineering and Electronics, 2025, 47(8): 2622-2631.
[3]	Xiaowei YAN, Chong LING, Shengbin SHI. Design and implementation of a rapid detection system for surface unexploded submunitions [J]. Systems Engineering and Electronics, 2025, 47(8): 2639-2645.
[4]	Xiaowei FU, Xinyi WANG, Zhe QIAO. Attack-defense confrontation strategy of multi-UAV based on APIQ algorithm [J]. Systems Engineering and Electronics, 2025, 47(7): 2205-2215.
[5]	Siying LIN, Feng YU, Zhi XIONG, Fang WU, Zijun ZHOU. Low-cost UAV navigation method based on AHRS for GNSS intermittent denial [J]. Systems Engineering and Electronics, 2025, 47(7): 2329-2338.
[6]	Junchao TANG, Chunhe HU. Complete coverage path planning for UAVs in 3D terrain and wind field environment [J]. Systems Engineering and Electronics, 2025, 47(7): 2349-2356.
[7]	Yunfeng HE, Xianjun SHI, Jianhua LU, Chaolun ZHAO, Guorong ZHAO. Multi-UAV grouping formation control based on synchronous DMPC under fault conditions [J]. Systems Engineering and Electronics, 2025, 47(7): 2357-2370.
[8]	Xiaowei FU, Xinyi WANG, Zhe QIAO. Confront strategy of multi-unmanned aerial vehicle based on ASDDPG algorithm [J]. Systems Engineering and Electronics, 2025, 47(6): 1867-1879.
[9]	Ruijing CUI, Jianbin SUN, GKewei YAN, Minghao LI. Construction method of equipment operational test indicator system based on UAF [J]. Systems Engineering and Electronics, 2025, 47(5): 1536-1550.
[10]	Wei CHEN, Congqing WANG, Qiang ZENG, Zhan LI. UAV coverage path planning for aircraft surface visual inspection [J]. Systems Engineering and Electronics, 2025, 47(4): 1206-1213.
[11]	Ze GENG, Yanyan HUANG, Han ZHANG. UAV swarm anti-artillery search path planning based on artillery transfer path prediction [J]. Systems Engineering and Electronics, 2025, 47(4): 1222-1234.
[12]	Zeyang YIN, Hao LIANG, Yuxin LIAO, Xiaofang CHEN, Yongfang XIE. UAV formation path planning based on reachable envelope analysis of dynamic obstacle [J]. Systems Engineering and Electronics, 2025, 47(4): 1275-1284.
[13]	Shipei ZOU, Yuhui WANG, Hongrui LIU. Multi-round attack and defense game decision-making of UAVs based on RF-XGBoost algorithm [J]. Systems Engineering and Electronics, 2025, 47(2): 518-526.
[14]	Zhongjie YIN, Bo HOU, Xiaolong JIN, Zhiliang FAN, Haiyang WANG. Covert spoofing method for anti-jamming UAV with array antenna [J]. Systems Engineering and Electronics, 2025, 47(2): 633-640.
[15]	Jinxin LIU, Wei SHENG, Yumin ZHANG. Optimization algorithm for UAV formation consensus based on motion constraint [J]. Systems Engineering and Electronics, 2025, 47(1): 230-243.

Research on UAV cooperative interception maneuver decision-making based on multi-agent reinforcement learning

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 31

Related Articles 15

Recommended Articles

Metrics

Comments