基于PPO算法的集群多目标火力规划方法

doi:10.12305/j.issn.1001-506X.2024.11.18

Abstract

Abstract:

To solve the problem of multi-target firepower planning in defensive combat scenarios under high dynamic battlefield situation, a firepower planning method based on the proximal strategy optimization algorithm is proposed. With the goal of maximizing combat effectiveness, the reinforcement learning reward function is designed from four aspects: ammunition consumption, combat effect, combat cost and combat time. Considering the influence of historical decision sequence on the current planning, the neural network is designed based on the Actor-Critic framework with the long short-term memory network (LSTM) as the core. The network is trained by the proximal strategy optimization algorithm, and the trained reinforcement learning agent is used for sequential decision-making. A series of coherent fire planning schemes are generated in real time according to the situation of multiple decision-making stages. Simulation results show that the agent can realize multi-target firepower planning under high dynamic situation, and its computational efficiency has more obvious advantages than other algorithms.

Key words: multi-target firepower planning, proximal strategy optimization algorithm, long short-term memory network (LSTM), sequential decision-making

CLC Number:

TP273

Hucheng QIN, Yanyan HUANG, Tiande CHEN, Han ZHANG. Cluster multi-target fire planning method based on PPO algorithm[J]. Systems Engineering and Electronics, 2024, 46(11): 3764-3773.

Figures/Tables 11

Fig.1

Fig.2

Fig.3

Table 1

Table 2

Table 3

Table 4

Fig.4

Table 5

Table 6

Table 7

References 34

1	CHENG X , SHI W P , CAI W L , et al. Communication-efficient coordinated RSS-based distributed passive localization via drone cluster[J]. IEEE Trans.on Vehicular Technology, 2022, 71 (1): 1072- 1076. doi: 10.1109/TVT.2021.3125361
2	张阳, 司光亚, 王艳正. 无人机集群网电攻击行动协同目标分配建模[J]. 系统工程与电子技术, 2019, 41 (9): 2025- 2033. doi: 10.3969/j.issn.1001-506X.2019.09.15
	ZHANG Y , SI G Y , WANG Y Z . Modeling of cooperative target allocation of the UAV swarm cyberspace attack action[J]. Systems Engineering and Electronics, 2019, 41 (9): 2025- 2033. doi: 10.3969/j.issn.1001-506X.2019.09.15
3	柳强, 何明, 刘锦涛, 等. 无人机"蜂群"的蜂拥涌现行为识别与抑制机理[J]. 电子学报, 2019, 47 (2): 374- 381. doi: 10.3969/j.issn.0372-2112.2019.02.017
	LIU Q , HE M , LIU J T , et al. A mechanism for identifying and suppressing the emergent flocking behaviors of UAV swarms[J]. Acta Electonica Sinica, 2019, 47 (2): 374- 381. doi: 10.3969/j.issn.0372-2112.2019.02.017
4	WANG H C , DING G R , CHEN J , et al. UAV anti-jamming communications with power and mobility control[J]. IEEE Trans.on Wireless Communications, 2023, 22 (7): 4729- 4744. doi: 10.1109/TWC.2022.3228265
5	GAO N , QIN Z J , JING X J , et al. Anti-intelligent UAV jamming strategy via deep Q-networks[J]. IEEE Trans.on Communications, 2020, 68 (1): 569- 581. doi: 10.1109/TCOMM.2019.2947918
6	LAYEB A , BENAYAD , ZEYNEB . A novel firefly algorithm based ant colony optimization for solving combinatorial optimization problems[J]. International Journal of Computer Science and Applications, 2014, 11 (2): 19- 37.
7	GHORBANI M K , AFSHAR A , HAMIDIFAR H , et al. A fuzzy multi-objective multiple-pollutant model for rivers using an ant colony algorithm[J]. Proceedings of the Institution of Civil Engineers: Water Management, 2022, 175 (4): 190- 205. doi: 10.1680/jwama.20.00081
8	ARRAM A , AYOB M . A novel multi-parent order crossover in genetic algorithm for combinatorial optimization problems[J]. Compu-ters and Industrial Engineering, 2019, 133 (8): 267- 274.
9	ZHANG H G , LIU Y A , ZHOU J . Balanced-evolution genetic algorithm for combinatorial optimization problems: the general outline and implementation of balanced-evolution strategy based on linear diversity index[J]. Natural Computing, 2018, 17 (3): 611- 639. doi: 10.1007/s11047-018-9670-5
10	KIM J W , KIM S K . Fitness switching genetic algorithm for solving combinatorial optimization problems with rare feasible solutions[J]. Journal of Supercomputing, 2016, 72 (9): 235- 248.
11	HSIEH F S , GUO Y H . A discrete cooperatively coevolving particle swarm optimization algorithm for combinatorial double auctions[J]. Applied Intelligence, 2019, 49 (11): 3845- 3863. doi: 10.1007/s10489-019-01556-8
12	GENG R M , JI R X , ZI S J . Research on task allocation of UAV cluster based on particle swarm quantization algorithm[J]. Mathematical Biosciences and Engineering, 2023, 20 (1): 18- 33.
13	汪民乐, 范阳涛. 基于效果的常规导弹火力规划模型智能求解算法[J]. 系统工程与电子技术, 2017, 39 (11): 2509- 2514. doi: 10.3969/j.issn.1001-506X.2017.11.17
	WANG M L , FAN Y T . Intelligent solving algorithm for effects-based firepower allocation model of conventional missiles[J]. Systems Engineering and Electronics, 2017, 39 (11): 2509- 2514. doi: 10.3969/j.issn.1001-506X.2017.11.17
14	孙海文, 谢晓方, 孙涛, 等. 改进型布谷鸟搜索算法的防空火力优化规划模型求解[J]. 兵工学报, 2019, 40 (1): 189- 197. doi: 10.3969/j.issn.1000-1093.2019.01.022
	SUN H W , XIE X F , SUN T , et al. Improved cuckoo search algorithm for solving antiaircraft weapon-target optimal assignment model[J]. Acta Armamentarii, 2019, 40 (1): 189- 197. doi: 10.3969/j.issn.1000-1093.2019.01.022
15	孙海文, 谢晓方, 庞威, 等. 基于改进火力规划模型的综合防空火力智能优化规划[J]. 控制与决策, 2020, 35 (5): 1102- 1112.
	SUN H W , XIE X F , PANG W , et al. Integrated air defense firepower intelligence optimal assignment based on improved firepower assignment model[J]. Control and Decision, 2020, 35 (5): 1102- 1112.
16	KALLESTAD J , HASIBI R , HEMMATI A , et al. A general deep reinforcement learning hyper heuristic framework for solving combinatorial optimization problems[J]. European Journal of Operational Research, 2023, 309 (1): 446- 468. doi: 10.1016/j.ejor.2023.01.017
17	WANG H L , WU H J , LAI G M . WagerWin: an efficient reinforcement learning framework for gambling games[J]. IEEE Trans.on Games, 2023, 15 (3): 483- 491. doi: 10.1109/TG.2022.3226526
18	INSEOK O , SEUNGEUN M , SANGBIN M , et al. Creating pro-level AI for a real-time fighting game using deep reinforcement learning[J]. IEEE Trans.on Games, 2022, 14 (2): 212- 220. doi: 10.1109/TG.2021.3049539
19	LI X J , LIU H S , DONG M H . A general framework of motion planning for redundant robot manipulator based on deep reinforcement learning[J]. IEEE Trans.on Industrial Informatics, 2022, 18 (8): 5253- 5263.
20	FAN F , XU G L , FENG N , et al. Spatiotemporal path tracking via deep reinforcement learning of robot for manufacturing internal logistics[J]. Journal of Manufacturing Systems, 2023, 69 (31): 150- 169.
21	DENG Y , LI Y L , DING B L , et al. Leveraging long short-term user preference in conversational recommendation via multi-agent reinforcement learning[J]. IEEE Trans.on Know-ledge and Data Engineering, 2023, 35 (11): 11541- 11555.
22	YANG Y C , CHRN C T , LU T Y , et al. Hierarchical reinforcement learning for conversational recommendation with knowledge graph reasoning and heterogeneous questions[J]. IEEE Trans.on Services Computing, 2023, 16 (5): 3439- 3452.
23	OUNOUGHI C , OUNOUGHI D , BEN Y S . EcoLight+: a novel multi-modal data fusion for enhanced eco-friendly traffic signal control driven by urban traffic noise prediction[J]. Knowledge and Information Systems, 2023, 65 (12): 5309- 5329.
24	HOU Y P , HE H S , JIANG X F , et al. Deep-reinforcement-learning-aided loss-tolerant congestion control for 6LoWPAN networks[J]. IEEE Internet of Things Journal, 2023, 10 (21): 19125- 19140.
25	WU Y Q , LIAO S Q , LIU X , et al. Deep reinforcement learning on autonomous driving policy with auxiliary critic network[J]. IEEE Trans.on Neural Networks and Learning Systems, 2023, 34 (7): 2680- 3690.
26	CUI J P , YUAN L , HE L , et al. Multi-input autonomous driving based on deep reinforcement learning with double bias experience replay[J]. IEEE Sensors Journal, 2023, 23 (11): 11253- 11261.
27	LIEN S Y , DENG D J . Intelligent session management for URLLC in 5G open radio access network: a deep reinforcement learning approach[J]. IEEE Trans.on Industrial Informatics, 2023, 19 (2): 1844- 1853.
28	ZANGOOEI M , SAHA N , GOLKARIFARD M , et al. Reinforcement learning for radio resource management in RAN slicing: a survey[J]. IEEE Communications Magazine, 2023, 61 (2): 118- 124.
29	朱建文, 赵长见, 李小平, 等. 基于强化学习的集群多目标分配与智能决策方法[J]. 兵工学报, 2021, 42(9): 2040-2048.
	ZHU J W, ZHAO C J, LI X P, et al. Multi-target assignment and intelligent decision based on reinforcement learning[J]. 2021, 42(9): 2040-2048.
30	黄亭飞, 程光权, 黄魁华, 等. 基于DQN的多类型拦截装备复合式反无人机任务分配方法[J]. 控制与决策, 2022, 37 (1): 142- 150.
	HUANG T F , CHENG G Q , HUANG K H , et al. Task assignment method of compound anti-drone based on DQN for multi type interception equipment[J]. Control and Decision, 2022, 37 (1): 142- 150.
31	SHOAIB M , UMAR M S . Phishing detection model using feline finch optimisation-based LSTM classifier[J]. International Journal of Sensor Networks, 2023, 42 (4): 205- 220.
32	XIE G L , ZHANG W , HU Z , et al. Upper confident bound advantage function proximal policy optimization[J]. Cluster Computing, 2023, 26 (3): 2001- 2010.
33	TAO C Q , LIN K , HUANG Z Q , et al. CRAM: code recommendation with programming context based on self- attention mechanism[J]. IEEE Trans.on Reliability, 2023, 72 (1): 302- 316.
34	LIOU T S , WANG M J . Ranking fuzzy numbers with integral value[J]. Fuzzy Sets and Systems, 1992, 50 (3): 247- 255.

装备类别	可使用弹药类别
w₁	s₃、s₅
w₂	s₁、s₃、s₅、s₆
w₃	s₂、s₃
w₄	s₄
w₅	s₁、s₂、s₅
w₆	s₂、s₃、s₄

变量	值
p_akb	rand(0.3, 0.6)
t_akb/s	rand(300, 800)
各类弹药数量	randint(50, 101)
各类无人机目标作战价值	rand(0, 0.8)
各类无人机目标期望毁伤下界	rand(0.7, 0.9)

反无人机装备类型	弹药类型	无人机目标单发毁伤概率
反无人机装备类型	弹药类型	bt₁	bt₂	bt₃	bt₄	bt₅
w₁	s₃	0.521	0.49	0.503	0.566	0.566
w₁	s₅	0.517	0.546	0.588	0.406	0.372
w₂	s₁	0.503	0.472	0.556	0.31	0.596
w₂	s₃	0.336	0.473	0.514	0.571	0.587
w₂	s₅	0.337	0.571	0.326	0.437	0.355
w₂	s₆	0.467	0.47	0.451	0.403	0.478
w₃	s₂	0.483	0.57	0.479	0.342	0.48
w₃	s₃	0.472	0.373	0.309	0.578	0.587
w₄	s₄	0.423	0.322	0.379	0.534	0.535
w₅	s₁	0.583	0.315	0.459	0.418	0.332
w₅	s₂	0.447	0.584	0.518	0.405	0.546
w₅	s₅	0.58	0.549	0.4	0.441	0.363
w₆	s₂	0.309	0.307	0.467	0.436	0.547
w₆	s₃	0.471	0.405	0.489	0.454	0.505
w₆	s₄	0.538	0.482	0.451	0.579	0.456

超参数名称	取值
Actor网络学习率	0.000 001
Critic网络学习率	0.000 05
γ	0.99
batch_size	32
经验回放池容量	1 000
每轮更新频率	10
迭代轮数	10 000
c₁	0.5
c₂	0.05
ε	0.2

红方装备	装备类型	弹药类型	目标1	目标2	目标3	目标4	目标5	目标6	目标7
红方装备	装备类型	弹药类型	bt₁	bt₂	bt₄	bt₅	bt₂	bt₅	bt₃
装备1	w₁	s₅	0	0	0	0	0	0	1
装备2	w₂	s₁	0	0	0	1	0	0	0
装备3	w₂	s₆	0	1	0	0	0	0	0
装备4	w₃	s₃	0	0	0	0	0	1	0
装备5	w₄	s₄	0	0	1	0	0	0	0
装备6	w₅	s₂	1	0	0	0	0	0	0
装备7	w₆	s₃	0	0	0	0	1	0	0

Cluster multi-target fire planning method based on PPO algorithm

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 34

Related Articles 2

Recommended Articles

Metrics

Comments

分配次数	本文方法	文献[29]算法	文献[30]算法	遗传算法	拍卖算法
1	0.243	0.340	0.347	0.331	0.193
2	0.237	0.329	0.343	0.335	0.193
3	0.241	0.332	0.342	0.331	0.193

分配数量	本文方法规划耗时	文献[29]算法训练耗时	文献[29]算法规划耗时	文献[30]耗时	遗传算法规划耗时	拍卖算法规划耗时
10	0.117	196.694	0.196	0.406	0.743	0.004
100	0.509	880.873	0.851	2.918	4.379	0.119
500	2.158	4 938.145	2.684	23.066	36.990	6.599

[1]	Ruiguan LIN, Huawei WANG, Changchang CHE, Xiaomei NI, Minglan XIONG. Predictive maintenance model of aeroengine based on LSTM classifier [J]. Systems Engineering and Electronics, 2022, 44(3): 1052-1059.
[2]	Junlong ZHAO, Wei LI, Honglin WANG, Teng HUANG, Yifu GAN, Ye WANG. Waveform design of radar based on long short-term memory network [J]. Systems Engineering and Electronics, 2021, 43(2): 376-382.