基于强化学习的装备体系韧性优化方法

doi:10.12305/j.issn.1001-506X.2025.07.15

系统工程与电子技术 ›› 2025, Vol. 47 ›› Issue (7): 2216-2223.doi: 10.12305/j.issn.1001-506X.2025.07.15

• 系统工程 • 上一篇

基于强化学习的装备体系韧性优化方法

柳佳豪¹, 徐任杰¹^,²^,*, 孙茂桐², 姜九瑶¹, 李际超¹, 杨克巍¹

1. 国防科技大学系统工程学院, 湖南长沙 410073
2. 慕尼黑工业大学管理学院, 德国海尔布隆 74076

收稿日期:2023-05-29 出版日期:2025-07-16 发布日期:2025-07-22
通讯作者: 徐任杰
作者简介:柳佳豪 (2002—), 男, 博士研究生, 主要研究方向为强化学习、体系韧性优化
徐任杰 (1998—), 男, 博士研究生, 主要研究方向为管理科学与复杂系统管理、体系韧性评估及优化、复杂系统与复杂网络
孙茂桐 (1997—), 男, 博士研究生, 主要研究方向为管理科学与医疗健康管理、强化学习应用
姜九瑶 (1998—), 女, 博士研究生, 主要研究方向为复杂网络、作战体系韧性评估
李际超 (1990—), 男, 副教授, 博士研究生导师, 博士, 主要研究方向为异质信息网络建模及分析、军事复杂系统智能决策
杨克巍 (1977—), 男, 教授, 博士研究生导师, 博士, 主要研究方向为体系需求建模、体系结构设计与优化、系统建模与仿真

Reinforcement learning-based resilience optimization method of equipment system-of-systems

Jiahao LIU¹, Renjie XU¹^,²^,*, Maotong SUN², Jiuyao JIANG¹, Jichao LI¹, Kewei YANG¹

1. College of Systems Engineering, National University of Defense Technology, Changsha 410073, China
2. School of Management, Technical University of Munich, Heilbronn 74076, Germany

Received:2023-05-29 Online:2025-07-16 Published:2025-07-22
Contact: Renjie XU

摘要/Abstract

摘要：

装备体系在实际运行下不可避免会受到外部攻击和内部故障等扰动事件的影响, 引起多个装备节点失效, 如何科学合理制定恢复策略迅速恢复体系能力, 增强装备体系韧性具有重要军事价值及意义。基于此, 提出一种基于强化学习的装备体系韧性优化方法。首先, 综合网络拓扑和网络性能参数建立装备体系韧性度量指标。其次, 提出基于Q-Learning节点恢复顺序的强化学习算法, 并采用不同的扰动场景来测试韧性的变化。最后, 结合典型案例验证提出算法的可行性及有效性。通过与经验性恢复策略、遗传算法的对比实验, 结果表明蓄意攻击下, 基于强化学习获得的韧性值相比于基于节点能力重要度优先恢复策略、基于度优先恢复策略和随机恢复策略高37.46%、52.28%和85.65%;与遗传算法相比, 优化后获得的韧性值提高28.72%。上述分析有效表明了所提方法和模型的优越性。

关键词: 装备体系, 强化学习, 韧性优化, 恢复策略

Abstract:

The equipment system-of-systems (ESoS) inevitably is affected by disturbance events such as external attacks and internal failures in actual operation, causing multiple equipment node failures. How to scientifically and rationally formulate recovery strategies to quickly restore system capabilities and enhance the resilience of the ESoS has important military value and significance. Based on this, this paper proposes an ESoS resilience optimization method based on reinforcement learning. Firstly, the ESoS resilience measurement index is established by integrating network topology and network performance parameters. Secondly, a reinforcement learning algorithm based on Q-Learning node recovery sequence is proposed, and different disturbance scenarios are used to test the change of resilience. Finally, combined with typical cases to verify the feasibility and effectiveness of the proposed algorithm. Through comparative experiments with empirical recovery strategies and genetic algorithm, the results show that with deliberate attacks, the toughness value obtained based on reinforcement learning is 37.46% higher than that based on node ability importance priority recovery strategy, degree priority recovery strategy and random recovery strategy 52.28% and 85.65%; compared with the genetic algorithm, the resilience value obtained after optimization increased by 28.72%. The above analysis effectively shows the superiority of the proposed method and model.

Key words: equipment system-of-systems (ESoS), reinforcement learning, resilience optimization, recovery strategy

中图分类号:

E917

柳佳豪, 徐任杰, 孙茂桐, 姜九瑶, 李际超, 杨克巍. 基于强化学习的装备体系韧性优化方法[J]. 系统工程与电子技术, 2025, 47(7): 2216-2223.

Jiahao LIU, Renjie XU, Maotong SUN, Jiuyao JIANG, Jichao LI, Kewei YANG. Reinforcement learning-based resilience optimization method of equipment system-of-systems[J]. Systems Engineering and Electronics, 2025, 47(7): 2216-2223.

图/表 10

图1

图2

图3

表1

图4

表2

图5

图6

表3

图7

参考文献 30

1	杨晓光, 高自友, 盛昭瀚, 等. 复杂系统管理是中国特色管理学体系的重要组成部分[J]. 管理世界, 2022, 38 (10): 1- 24.
	YANG X G , GAO Z Y , SHENG Z H , et al. Complex system management is an important part of the management system with Chinese characteristics[J]. Journal of Management World, 2022, 38 (10): 1- 24.
2	GRIFFITH G P , HOP H , VIHTAKARI M , et al. Ecological resilience of arctic marine food webs to climate change[J]. Nature Climate Change, 2019, 9 (11): 868- 872.
3	HOLLING C S . Resilience and stability of ecological systems[J]. Annual Review of Ecology and Systematics, 1973, 4 (1): 1- 23.
4	MADNI A M , JACKSON S . Towards a conceptual framework for resilience engineering[J]. IEEE Systems Journal, 2009, 3 (2): 181- 191.
5	Federal Financial Institutions Examination Council. Business continuity planning[R]. Washington: Federal Financial Institutions Examination Council, 2015.
6	U.S. Department of Defense. Space policy[R]. Washington: U.S. Department of Defense, 2012.
7	MATTIS J. Summary of the 2018 national defense strategy of the United States of America[R]. Washington: U.S. Department of Defense, 2018.
8	DEPTULA D, HEATHER P, LAWRENCE S, et al. Restoring America's military competitiveness: mosaic warfare[R]. Arlington: Mitchell Institute for Aerospace Studies, 2019.
9	荣明, 胡晓峰, 杨镜宇. 基于试验床的作战体系弹性评估研究[J]. 系统仿真学报, 2018, 30 (12): 4711- 4717.
	RONG M , HU X F , YANG J Y . Research on assessment of operation system's resilience based on test bed[J]. Journal of System Simulation, 2018, 30 (12): 4711- 4717.
10	王宏宇, 吴纬, 魏艳艳. 基于超网络模型武器装备体系抗毁性分析[J]. 系统工程与电子技术, 2017, 39 (8): 1782- 1787. doi: 10.3969/j.issn.1001-506X.2017.08.17
	WANG H Y , WU W , WEI Y Y , et al. Weapon system- of-systems invulnerability analysis based on super network model[J]. Systems Engineering and Electronics, 2017, 39 (8): 1782- 1787. doi: 10.3969/j.issn.1001-506X.2017.08.17
11	荣明, 胡晓峰, 杨镜宇. 基于动态超网的作战体系结构弹性分析评估研究[J]. 系统仿真学报, 2019, 31 (6): 1055- 1061.
	RONG M , HU X F , YANG J Y . Resilience analysis and evaluation of combat system of systems architecture based on dynamic super network[J]. Journal of System Simulation, 2019, 31 (6): 1055- 1061.
12	陈志伟, 王靖, 谷长超, 等. 考虑动态重构的装备体系可用性及弹性分析[J]. 系统工程与电子技术, 2021, 43 (8): 2347- 2354. doi: 10.12305/j.issn.1001-506X.2021.08.38
	CHEN Z W , WANG J , GU C C , et al. Performance availability and resilience analysis of weapon system of systems considering dynamic reconfiguration[J]. Systems Engineering and Electronics, 2021, 43 (8): 2347- 2354. doi: 10.12305/j.issn.1001-506X.2021.08.38
13	PAN X , WANG H X , YANG Y J , et al. Resilience based importance measure analysis for SoS[J]. Journal of Systems Engineering and Electronics, 2019, 30 (5): 920- 930.
14	李际超, 吴俊, 谭跃进, 等. 基于有向自然连通度的作战网络抗毁性研究[J]. 复杂系统与复杂性科学, 2015, 12 (4): 25- 31.
	LI J C , WU J , TAN Y J , et al. Robustness of combat networks based on directed natural connectivity[J]. Complex Systems and Complexity Science, 2015, 12 (4): 25- 31.
15	赵丹玲, 谭跃进, 李际超, 等. 基于作战环的武器装备体系贡献度评估[J]. 系统工程与电子技术, 2017, 39 (10): 2239- 2247. doi: 10.3969/j.issn.1001-506X.2017.10.13
	ZHAO D L , TAN Y J , LI J C , et al. Armament system of systems contribution evaluation based on operation loop[J]. Systems Engineering and Electronics, 2017, 39 (10): 2239- 2247. doi: 10.3969/j.issn.1001-506X.2017.10.13
16	JIANG J Y , LI J C , YANG K W . Weapon system portfolio selection based on structural robustness[J]. Journal of Systems Engineering and Electronics, 2020, 31 (6): 1216- 1229.
17	LI J C , JIANG J , YANG K W , et al. Research on functional robustness of heterogeneous combat networks[J]. IEEE Systems Journal, 2019, 13 (2): 1487- 1495.
18	徐任杰, 宫琳, 谢剑, 等. 基于装备体系韧性的作战网络链路重要度评估及恢复策略[J]. 系统工程与电子技术, 2023, 45 (1): 139- 147. doi: 10.12305/j.issn.1001-506X.2023.01.17
	XU R J , GONG L , XIE J , et al. Operation network link importance evaluation and recovery strategy based on equipment system-of-system resilience[J]. Systems Engineering and Electronics, 2023, 45 (1): 139- 147. doi: 10.12305/j.issn.1001-506X.2023.01.17
19	CHEN Z W , HONG D P , CUI W W , et al. Resilience evaluation and optimal design for weapon system of systems with dynamic reconfiguration[J]. Reliability Engineering & System Safety, 2023, 237, 109409.
20	SUN Q , LI H X , WANG Y Z , et al. Multi-swarm-based cooperative reconfiguration model for resilient unmanned weapon system-of-systems[J]. Reliability Engineering & System Safety, 2022, 222, 108426.
21	HAN Q , PANG B , LI S , et al. Evaluation method and optimization strategies of resilience for air & space defense system of systems based on kill network theory and improved self-information quantity[J]. Defense Technology, 2023, 21 (3): 219- 239.
22	潘星, 张国忠, 张跃东, 等. 工程弹性系统与系统弹性理论研究综述[J]. 系统工程与电子技术, 2019, 41 (9): 2006- 2015. doi: 10.3969/j.issn.1001-506X.2019.09.13
	PAN X , ZHANG G Z , ZHANG Y D , et al. Review of engineered resilient systems and system resilience theory[J]. Systems Engineering and Electronics, 2019, 41 (9): 2006- 2015. doi: 10.3969/j.issn.1001-506X.2019.09.13
23	PAN X , DANG Y H , WANG H X , et al. Resilience model and recovery strategy of transportation network based on travel OD-grid analysis[J]. Reliability Engineering & System Safety, 2022, 223, 108483.
24	CARES J R . Distributed networked operations the foundations of network centric warfare[M]. Newport: Alidade Press, 2004.
25	CARES J R. An information age combat model[C]//Proc. of the 9th International Command and Control Research and Technology Symposium, 2004.
26	谭跃进, 张小可, 杨克巍. 武器装备体系网络化描述与建模方法[J]. 系统管理学报, 2012, 21 (6): 781- 786.
	TAN Y J , ZHANG X K , YANG K W . Research on networked description and modeling methods of armament system-of-systems[J]. Journal of Systems & Management, 2012, 21 (6): 781- 786.
27	SUGIYAMA T , SCHWEIGHOFER N , IZAWA J . Reinforcement learning establishes a minimal metacognitive process to monitor and control motor learning performance[J]. Nature Communications, 2023, 1 (1): 3988.
28	李凯文, 张涛, 王锐, 等. 基于深度强化学习的组合优化研究进展[J]. 自动化学报, 2021, 47 (11): 2521- 2537.
	LI K W , ZHANG T , WANG R , et al. Research reviews of combinatorial optimization methods based on deep reinforcement learning[J]. Acta Automatica Sinica, 2021, 47 (11): 2521- 2537.
29	ZENG C Y , LIU H F , LU L N , et al. SHATTER: searching heterogeneous combat network attack sequences through network embedding and reinforcement learning[J]. IEEE Systems Journal, 2023, 17 (3): 4497- 4508.
30	GARG S , HU J , FORTINO G , et al. Editorial deep reinforcement learning for next-generation IoT networks[J]. Computer Networks, 2023, 228, 109760.

节点	作战能力
S1	0.5
S2	0.3
S3	0.2
S4	0.7
D1	0.8
D2	0.9
D3	0.7
I1	0.4
I2	0.3
I3	0.6
I4	0.2
I5	0.6
I6	0.1
I7	0.7
T	0.6

同时恢复节点数	韧性值
同时恢复1个装备节点	0.466
同时恢复2个装备节点	0.722
同时恢复3个装备节点	0.790
同时恢复4个装备节点	0.844
同时恢复5个装备节点	0.875
同时恢复6个装备节点	0.896
同时恢复7个装备节点	0.925
同时恢复8个装备节点	0.937

扰动场景	恢复策略	节点恢复顺序	韧性值	提升比例/%
扰动场景1	基于强化学习的恢复策略	D1→I3→I1→D2→S1→I4→S2→I6	0.569	71.38
	基于节点能力重要度优先恢复策略	D2→D1→I3→S1→I1→S2→I4→I6	0.519	56.32
	基于度优先恢复策略	D2→D1→I4→I6→I3→S1→S2→I1	0.431	29.81
	随机恢复策略	S2→I4→I3→S1→D1→I6→I1	0.332	0.00
扰动场景2	基于强化学习的恢复策略	S4→I5→S3→D3→I3→I4→S2→I6	0.528	67.61
	基于节点能力重要度优先恢复策略	S4→D3→I5→I3→S2→S3→I4→I6	0.447	41.90
	基于度优先恢复策略	D3→S4→I5→S3→I4→I6→I3→S2	0.475	50.79
	随机恢复策略	I4→I5→D3→I6→S3→S4→I3→S2	0.315	0.00
扰动场景3	基于强化学习的恢复策略	S4→D1→S3→I5→D3→S1→D2→I4	0.466	85.65
	基于节点能力重要度优先恢复策略	D2→D1→S4→D3→I5→S1→S3→I4	0.339	35.05
	基于度优先恢复策略	D2→D1→D3→S3→S4→I4→I5→S1	0.306	21.91
	随机恢复策略	D3→S1→I4→S3→D1→D2→I5	0.251	0.00

[1]	符小卫, 王辛夷, 乔哲. 基于APIQ算法的多无人机攻防对抗策略[J]. 系统工程与电子技术, 2025, 47(7): 2205-2215.
[2]	符小卫, 王辛夷, 乔哲. 基于ASDDPG算法的多无人机对抗策略[J]. 系统工程与电子技术, 2025, 47(6): 1867-1879.
[3]	孟麟芝, 孙小涓, 胡玉新, 高斌, 孙国庆, 牟文浩. 面向卫星在轨处理的强化学习任务调度算法[J]. 系统工程与电子技术, 2025, 47(6): 1917-1929.
[4]	郑康洁, 张新宇, 王伟菘, 刘震生. DQN与规则结合的智能船舶动态自主避障决策[J]. 系统工程与电子技术, 2025, 47(6): 1994-2001.
[5]	刘书含, 李彤, 李富强, 杨春刚. 意图态势双驱动的数据链抗干扰通信机制[J]. 系统工程与电子技术, 2025, 47(6): 2055-2064.
[6]	林志康, 施龙飞, 刘甲磊, 马佳智. 基于深度Q学习的组网雷达闪烁探测调度方法[J]. 系统工程与电子技术, 2025, 47(5): 1443-1452.
[7]	尚晓凡, 薛奇, 高欣, 刘同. 面向能力需求的装备组合配置分析方法[J]. 系统工程与电子技术, 2025, 47(5): 1575-1581.
[8]	王子怡, 傅雄军, 董健, 冯程. 基于分层多智能体强化学习的雷达协同抗干扰策略优化[J]. 系统工程与电子技术, 2025, 47(4): 1108-1114.
[9]	熊威, 张栋, 任智, 杨书恒. 面向有人/无人机协同打击的智能决策方法研究[J]. 系统工程与电子技术, 2025, 47(4): 1285-1299.
[10]	马鹏, 蒋睿, 王斌, 徐盟飞, 侯长波. 基于隐式对手建模的策略重构抗智能干扰方法[J]. 系统工程与电子技术, 2025, 47(4): 1355-1363.
[11]	唐开强, 傅汇乔, 刘佳生, 邓归洲, 陈春林. 基于深度强化学习的带约束车辆路径分层优化研究[J]. 系统工程与电子技术, 2025, 47(3): 827-841.
[12]	陈夏瑢, 李际超, 陈刚, 刘鹏, 姜江. 基于异质网络的装备体系组合发展规划问题[J]. 系统工程与电子技术, 2025, 47(3): 855-861.
[13]	付可, 陈浩, 王宇, 刘权, 黄健. 基于不确定性的贝叶斯策略重用方法[J]. 系统工程与电子技术, 2025, 47(2): 535-543.
[14]	刘晓琳, 郭梦娇, 李卓. Dueling DQN优化下的航班延误自适应图卷积循环网络预测方法[J]. 系统工程与电子技术, 2025, 47(2): 568-579.
[15]	赵万兵, 夏元清, 戴荔, 张元. 弱通信下无人潜航器事件触发一致性协同控制[J]. 系统工程与电子技术, 2025, 47(2): 591-599.

基于强化学习的装备体系韧性优化方法

Reinforcement learning-based resilience optimization method of equipment system-of-systems

RichHTML

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 30

相关文章 15

编辑推荐

Metrics

本文评价