基于强化学习的装备体系韧性优化方法

doi:10.12305/j.issn.1001-506X.2025.07.15

Abstract

Abstract:

The equipment system-of-systems (ESoS) inevitably is affected by disturbance events such as external attacks and internal failures in actual operation, causing multiple equipment node failures. How to scientifically and rationally formulate recovery strategies to quickly restore system capabilities and enhance the resilience of the ESoS has important military value and significance. Based on this, this paper proposes an ESoS resilience optimization method based on reinforcement learning. Firstly, the ESoS resilience measurement index is established by integrating network topology and network performance parameters. Secondly, a reinforcement learning algorithm based on Q-Learning node recovery sequence is proposed, and different disturbance scenarios are used to test the change of resilience. Finally, combined with typical cases to verify the feasibility and effectiveness of the proposed algorithm. Through comparative experiments with empirical recovery strategies and genetic algorithm, the results show that with deliberate attacks, the toughness value obtained based on reinforcement learning is 37.46% higher than that based on node ability importance priority recovery strategy, degree priority recovery strategy and random recovery strategy 52.28% and 85.65%; compared with the genetic algorithm, the resilience value obtained after optimization increased by 28.72%. The above analysis effectively shows the superiority of the proposed method and model.

Key words: equipment system-of-systems (ESoS), reinforcement learning, resilience optimization, recovery strategy

CLC Number:

E917

Jiahao LIU, Renjie XU, Maotong SUN, Jiuyao JIANG, Jichao LI, Kewei YANG. Reinforcement learning-based resilience optimization method of equipment system-of-systems[J]. Systems Engineering and Electronics, 2025, 47(7): 2216-2223.

Figures/Tables 10

Fig.1

Fig.2

Fig.3

Table 1

Fig.4

Table 2

Fig.5

Fig.6

Table 3

Fig.7

References 30

1	杨晓光, 高自友, 盛昭瀚, 等. 复杂系统管理是中国特色管理学体系的重要组成部分[J]. 管理世界, 2022, 38 (10): 1- 24.
	YANG X G , GAO Z Y , SHENG Z H , et al. Complex system management is an important part of the management system with Chinese characteristics[J]. Journal of Management World, 2022, 38 (10): 1- 24.
2	GRIFFITH G P , HOP H , VIHTAKARI M , et al. Ecological resilience of arctic marine food webs to climate change[J]. Nature Climate Change, 2019, 9 (11): 868- 872.
3	HOLLING C S . Resilience and stability of ecological systems[J]. Annual Review of Ecology and Systematics, 1973, 4 (1): 1- 23.
4	MADNI A M , JACKSON S . Towards a conceptual framework for resilience engineering[J]. IEEE Systems Journal, 2009, 3 (2): 181- 191.
5	Federal Financial Institutions Examination Council. Business continuity planning[R]. Washington: Federal Financial Institutions Examination Council, 2015.
6	U.S. Department of Defense. Space policy[R]. Washington: U.S. Department of Defense, 2012.
7	MATTIS J. Summary of the 2018 national defense strategy of the United States of America[R]. Washington: U.S. Department of Defense, 2018.
8	DEPTULA D, HEATHER P, LAWRENCE S, et al. Restoring America's military competitiveness: mosaic warfare[R]. Arlington: Mitchell Institute for Aerospace Studies, 2019.
9	荣明, 胡晓峰, 杨镜宇. 基于试验床的作战体系弹性评估研究[J]. 系统仿真学报, 2018, 30 (12): 4711- 4717.
	RONG M , HU X F , YANG J Y . Research on assessment of operation system's resilience based on test bed[J]. Journal of System Simulation, 2018, 30 (12): 4711- 4717.
10	王宏宇, 吴纬, 魏艳艳. 基于超网络模型武器装备体系抗毁性分析[J]. 系统工程与电子技术, 2017, 39 (8): 1782- 1787. doi: 10.3969/j.issn.1001-506X.2017.08.17
	WANG H Y , WU W , WEI Y Y , et al. Weapon system- of-systems invulnerability analysis based on super network model[J]. Systems Engineering and Electronics, 2017, 39 (8): 1782- 1787. doi: 10.3969/j.issn.1001-506X.2017.08.17
11	荣明, 胡晓峰, 杨镜宇. 基于动态超网的作战体系结构弹性分析评估研究[J]. 系统仿真学报, 2019, 31 (6): 1055- 1061.
	RONG M , HU X F , YANG J Y . Resilience analysis and evaluation of combat system of systems architecture based on dynamic super network[J]. Journal of System Simulation, 2019, 31 (6): 1055- 1061.
12	陈志伟, 王靖, 谷长超, 等. 考虑动态重构的装备体系可用性及弹性分析[J]. 系统工程与电子技术, 2021, 43 (8): 2347- 2354. doi: 10.12305/j.issn.1001-506X.2021.08.38
	CHEN Z W , WANG J , GU C C , et al. Performance availability and resilience analysis of weapon system of systems considering dynamic reconfiguration[J]. Systems Engineering and Electronics, 2021, 43 (8): 2347- 2354. doi: 10.12305/j.issn.1001-506X.2021.08.38
13	PAN X , WANG H X , YANG Y J , et al. Resilience based importance measure analysis for SoS[J]. Journal of Systems Engineering and Electronics, 2019, 30 (5): 920- 930.
14	李际超, 吴俊, 谭跃进, 等. 基于有向自然连通度的作战网络抗毁性研究[J]. 复杂系统与复杂性科学, 2015, 12 (4): 25- 31.
	LI J C , WU J , TAN Y J , et al. Robustness of combat networks based on directed natural connectivity[J]. Complex Systems and Complexity Science, 2015, 12 (4): 25- 31.
15	赵丹玲, 谭跃进, 李际超, 等. 基于作战环的武器装备体系贡献度评估[J]. 系统工程与电子技术, 2017, 39 (10): 2239- 2247. doi: 10.3969/j.issn.1001-506X.2017.10.13
	ZHAO D L , TAN Y J , LI J C , et al. Armament system of systems contribution evaluation based on operation loop[J]. Systems Engineering and Electronics, 2017, 39 (10): 2239- 2247. doi: 10.3969/j.issn.1001-506X.2017.10.13
16	JIANG J Y , LI J C , YANG K W . Weapon system portfolio selection based on structural robustness[J]. Journal of Systems Engineering and Electronics, 2020, 31 (6): 1216- 1229.
17	LI J C , JIANG J , YANG K W , et al. Research on functional robustness of heterogeneous combat networks[J]. IEEE Systems Journal, 2019, 13 (2): 1487- 1495.
18	徐任杰, 宫琳, 谢剑, 等. 基于装备体系韧性的作战网络链路重要度评估及恢复策略[J]. 系统工程与电子技术, 2023, 45 (1): 139- 147. doi: 10.12305/j.issn.1001-506X.2023.01.17
	XU R J , GONG L , XIE J , et al. Operation network link importance evaluation and recovery strategy based on equipment system-of-system resilience[J]. Systems Engineering and Electronics, 2023, 45 (1): 139- 147. doi: 10.12305/j.issn.1001-506X.2023.01.17
19	CHEN Z W , HONG D P , CUI W W , et al. Resilience evaluation and optimal design for weapon system of systems with dynamic reconfiguration[J]. Reliability Engineering & System Safety, 2023, 237, 109409.
20	SUN Q , LI H X , WANG Y Z , et al. Multi-swarm-based cooperative reconfiguration model for resilient unmanned weapon system-of-systems[J]. Reliability Engineering & System Safety, 2022, 222, 108426.
21	HAN Q , PANG B , LI S , et al. Evaluation method and optimization strategies of resilience for air & space defense system of systems based on kill network theory and improved self-information quantity[J]. Defense Technology, 2023, 21 (3): 219- 239.
22	潘星, 张国忠, 张跃东, 等. 工程弹性系统与系统弹性理论研究综述[J]. 系统工程与电子技术, 2019, 41 (9): 2006- 2015. doi: 10.3969/j.issn.1001-506X.2019.09.13
	PAN X , ZHANG G Z , ZHANG Y D , et al. Review of engineered resilient systems and system resilience theory[J]. Systems Engineering and Electronics, 2019, 41 (9): 2006- 2015. doi: 10.3969/j.issn.1001-506X.2019.09.13
23	PAN X , DANG Y H , WANG H X , et al. Resilience model and recovery strategy of transportation network based on travel OD-grid analysis[J]. Reliability Engineering & System Safety, 2022, 223, 108483.
24	CARES J R . Distributed networked operations the foundations of network centric warfare[M]. Newport: Alidade Press, 2004.
25	CARES J R. An information age combat model[C]//Proc. of the 9th International Command and Control Research and Technology Symposium, 2004.
26	谭跃进, 张小可, 杨克巍. 武器装备体系网络化描述与建模方法[J]. 系统管理学报, 2012, 21 (6): 781- 786.
	TAN Y J , ZHANG X K , YANG K W . Research on networked description and modeling methods of armament system-of-systems[J]. Journal of Systems & Management, 2012, 21 (6): 781- 786.
27	SUGIYAMA T , SCHWEIGHOFER N , IZAWA J . Reinforcement learning establishes a minimal metacognitive process to monitor and control motor learning performance[J]. Nature Communications, 2023, 1 (1): 3988.
28	李凯文, 张涛, 王锐, 等. 基于深度强化学习的组合优化研究进展[J]. 自动化学报, 2021, 47 (11): 2521- 2537.
	LI K W , ZHANG T , WANG R , et al. Research reviews of combinatorial optimization methods based on deep reinforcement learning[J]. Acta Automatica Sinica, 2021, 47 (11): 2521- 2537.
29	ZENG C Y , LIU H F , LU L N , et al. SHATTER: searching heterogeneous combat network attack sequences through network embedding and reinforcement learning[J]. IEEE Systems Journal, 2023, 17 (3): 4497- 4508.
30	GARG S , HU J , FORTINO G , et al. Editorial deep reinforcement learning for next-generation IoT networks[J]. Computer Networks, 2023, 228, 109760.

节点	作战能力
S1	0.5
S2	0.3
S3	0.2
S4	0.7
D1	0.8
D2	0.9
D3	0.7
I1	0.4
I2	0.3
I3	0.6
I4	0.2
I5	0.6
I6	0.1
I7	0.7
T	0.6

同时恢复节点数	韧性值
同时恢复1个装备节点	0.466
同时恢复2个装备节点	0.722
同时恢复3个装备节点	0.790
同时恢复4个装备节点	0.844
同时恢复5个装备节点	0.875
同时恢复6个装备节点	0.896
同时恢复7个装备节点	0.925
同时恢复8个装备节点	0.937

扰动场景	恢复策略	节点恢复顺序	韧性值	提升比例/%
扰动场景1	基于强化学习的恢复策略	D1→I3→I1→D2→S1→I4→S2→I6	0.569	71.38
	基于节点能力重要度优先恢复策略	D2→D1→I3→S1→I1→S2→I4→I6	0.519	56.32
	基于度优先恢复策略	D2→D1→I4→I6→I3→S1→S2→I1	0.431	29.81
	随机恢复策略	S2→I4→I3→S1→D1→I6→I1	0.332	0.00
扰动场景2	基于强化学习的恢复策略	S4→I5→S3→D3→I3→I4→S2→I6	0.528	67.61
	基于节点能力重要度优先恢复策略	S4→D3→I5→I3→S2→S3→I4→I6	0.447	41.90
	基于度优先恢复策略	D3→S4→I5→S3→I4→I6→I3→S2	0.475	50.79
	随机恢复策略	I4→I5→D3→I6→S3→S4→I3→S2	0.315	0.00
扰动场景3	基于强化学习的恢复策略	S4→D1→S3→I5→D3→S1→D2→I4	0.466	85.65
	基于节点能力重要度优先恢复策略	D2→D1→S4→D3→I5→S1→S3→I4	0.339	35.05
	基于度优先恢复策略	D2→D1→D3→S3→S4→I4→I5→S1	0.306	21.91
	随机恢复策略	D3→S1→I4→S3→D1→D2→I5	0.251	0.00

[1]	Xiaowei FU, Xinyi WANG, Zhe QIAO. Attack-defense confrontation strategy of multi-UAV based on APIQ algorithm [J]. Systems Engineering and Electronics, 2025, 47(7): 2205-2215.
[2]	Xiaowei FU, Xinyi WANG, Zhe QIAO. Confront strategy of multi-unmanned aerial vehicle based on ASDDPG algorithm [J]. Systems Engineering and Electronics, 2025, 47(6): 1867-1879.
[3]	Linzhi MENG, Xiaojuan SUN, Yuxin HU, Bin GAO, Guoqing SUN, Wenhao MU. Reinforcement learning task scheduling algorithm for satellite on-orbit processing [J]. Systems Engineering and Electronics, 2025, 47(6): 1917-1929.
[4]	Kangjie ZHENG, Xinyu ZHANG, Weisong WANG, Zhensheng LIU. Intelligent ship dynamic autonomous obstacle avoidance decision based on DQN and rule [J]. Systems Engineering and Electronics, 2025, 47(6): 1994-2001.
[5]	Shuhan LIU, Tong LI, Fuqiang LI, Chungang YANG. Intent and situation-dual driven anti-jamming communication mechanism for data link [J]. Systems Engineering and Electronics, 2025, 47(6): 2055-2064.
[6]	Zhikang LIN, Longfei SHI, Jialei LIU, Jiazhi MA. Scintillation detection scheduling method of netted radar based on deep Q-learning [J]. Systems Engineering and Electronics, 2025, 47(5): 1443-1452.
[7]	Ziyi WANG, Xiongjun FU, Jian DONG, Cheng FENG. Optimization of radar collaborative anti-jamming strategies based on hierarchical multi-agent reinforcement learning [J]. Systems Engineering and Electronics, 2025, 47(4): 1108-1114.
[8]	Wei XIONG, Dong ZHANG, Zhi REN, Shuheng YANG. Research on intelligent decision-making methods for coordinated attack by manned aerial vehicles and unmanned aerial vehicles [J]. Systems Engineering and Electronics, 2025, 47(4): 1285-1299.
[9]	Peng MA, Rui JIANG, Bin WANG, Mengfei XU, Changbo HOU. Strategy reconstruction for resilience against intelligence jamming based on implicit opponent modeling [J]. Systems Engineering and Electronics, 2025, 47(4): 1355-1363.
[10]	Kaiqiang TANG, Huiqiao FU, Jiasheng LIU, Guizhou DENG, Chunlin CHEN. Hierarchical optimization research of constrained vehicle routing based on deep reinforcement learning [J]. Systems Engineering and Electronics, 2025, 47(3): 827-841.
[11]	Xiarong CHEN, Jichao LI, Gang CHEN, Peng LIU, Jiang JIANG. Portfolio of weapon system-of-systems based on heterogeneous information networks [J]. Systems Engineering and Electronics, 2025, 47(3): 855-861.
[12]	Ke FU, Hao CHEN, Yu WANG, Quan LIU, Jian HUANG. Uncertainty-based Bayesian policy reuse method [J]. Systems Engineering and Electronics, 2025, 47(2): 535-543.
[13]	Xiaolin LIU, Mengjiao GUO, Zhuo LI. Adaptive graph convolutional recurrent network prediction method for flight delay based on Dueling DQN optimization [J]. Systems Engineering and Electronics, 2025, 47(2): 568-579.
[14]	Xunliang YAN, Kuan WANG, Zijian ZHANG, Peichen WANG. Reentry guidance method based on LSTM-DDPG [J]. Systems Engineering and Electronics, 2025, 47(1): 268-279.
[15]	Tingyu ZHANG, Ying ZENG, Nan LI, Hongzhong HUANG. Spacecraft power-signal composite network optimization algorithm based on DRL [J]. Systems Engineering and Electronics, 2024, 46(9): 3060-3069.

Reinforcement learning-based resilience optimization method of equipment system-of-systems

RichHTML

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 30

Related Articles 15

Recommended Articles

Metrics

Comments