基于深度强化学习的航天器功率-信号复合网络优化算法

doi:10.12305/j.issn.1001-506X.2024.09.18

Abstract

Abstract:

To maximize the utilization of limited energy and achieve flexible and efficient grid connection for spacecraft power supply systems, a composite grid topology optimization model for power transmission and signal communication is proposed based on deep reinforcement learning (DRL). Various interpretable component models are employed based on knowledge distillation principles to analyze the optimization mechanism. Firstly, the transformation law of the control domain of the spacecraft bus voltage regulation in the on-orbit operation stage is analyzed, and the composite network topology model of power transmission and signal communication is established by combining the node propagation parameters. Secondly, asynchronous advantage actor-critic (A3C) is utilized to adaptively optimize potential operational reliability risks in routing distribution and topology of the electrical signal transmission network. Finally, various interpretable components are used to perform knowledge distillation on the trained DRL model, forming an interpretable quantitative analysis method. The proposed method theoretically predicts optimal grid-connected processes of space power supply under random shadow effects, providing theoretical support and reference for designing space power supply controllers under higher task requirements and complex environments.

Key words: space power system, complex network theory, deep reinforcement learning (DRL), reliability optimization, interpretable analysis

CLC Number:

V423

Tingyu ZHANG, Ying ZENG, Nan LI, Hongzhong HUANG. Spacecraft power-signal composite network optimization algorithm based on DRL[J]. Systems Engineering and Electronics, 2024, 46(9): 3060-3069.

Figures/Tables 10

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Table 1

Fig.9

References 36

1	王文龙, 杨建中. 航天器对接与捕获技术综述[J]. 机械工程学报, 2021, 57 (20): 215- 231.
	WANG W L , YANG J Z . Spacecraft docking & capture technology: review[J]. Journal of Mechanical Engineering, 2021, 57 (20): 215- 231.
2	李孝鹏, 黄洪钟, 李福秋. 基于PRA的复杂航天多阶段任务系统可靠性分析[J]. 系统工程与电子技术, 2019, 41 (9): 2141- 2147. doi: 10.3969/j.issn.1001-506X.2019.09.30
	LI X P , HUANG H Z , LI F Q . PRA based reliability analysis of complex space phased-mission system[J]. Systems Engineering and Electronics, 2019, 41 (9): 2141- 2147. doi: 10.3969/j.issn.1001-506X.2019.09.30
3	JASEM K , MOHSEN H , KEYHAN S . Modeling and control of quasi Z-source inverters for parallel operation of battery energy storage systems: application to micro grids[J]. Electric Power Systems Research, 2015, 125, 164- 173. doi: 10.1016/j.epsr.2015.04.004
4	RYAN M, KENNETH A. The use of software agents for autonomous control of a DC space power system[C]//Proc. of the 12th International Energy Conversion Engineering Conference, 2014.
5	OKAYA S. Advanced concept of the space electric power system integrated with the propulsion[C]//Proc. of the 13th International Energy Conversion Engineering Conference, 2015.
6	RICHARD C, BRENT G. Modular power standard for space explorations missions[C]//Proc. of the AIAA Space Conferences and Exposition, 2016.
7	何雄, 陈永刚, 王力. 适用于分布式宇航电源系统的电源控制器研究[J]. 电源学报, 2022, 20 (5): 5- 13.
	HE X , CHEN Y G , WANG L . Research on power conditioning unit for distributed aerospace power supply system[J]. Journal of Power Supply, 2022, 20 (5): 5- 13.
8	钟丹华, 唐筱, 舒斌, 等. 载人飞船电源系统并网供电特性研究[J]. 航天器工程, 2020, 29 (1): 29- 33. doi: 10.3969/j.issn.1673-8748.2020.01.005
	ZHONG D H , TANG X , SHU B , et al. Characteristic of parallel power supply technology for manned spacecraft power system[J]. Spacecraft Engineering, 2020, 29 (1): 29- 33. doi: 10.3969/j.issn.1673-8748.2020.01.005
9	周新顺, 王蓓蓓, 郭晓峰. 航天器大功率并网控制技术研究[J]. 中国空间科学技术, 2018, 38 (6): 59- 66.
	ZHOU X S , WANG B B , GUO X F . Research on high power bus interconnection control technology for spacecraft[J]. Chinese Space Science and Technology, 2018, 38 (6): 59- 66.
10	蒋冀, 王宏佳, 徐志伟. 一种航天器直流供电并网系统控制方法[J]. 电源技术, 2018, 42 (9): 1383- 1386. doi: 10.3969/j.issn.1002-087X.2018.09.041
	JIANG J , WANG H J , XU Z W . A kind of control methods of spacecraft DC grid-connected power supply system[J]. Chinese Journal of Power Sources, 2018, 42 (9): 1383- 1386. doi: 10.3969/j.issn.1002-087X.2018.09.041
11	张大鹏, 孟宪会. 一种航天器间并网供电方案的研究[J]. 航天器工程, 2009, 18 (5): 101- 107. doi: 10.3969/j.issn.1673-8748.2009.05.018
	ZHANG D P , MENG X H . Research on parallel operation between power support systems of different spacecrafts[J]. Spacecraft Engineering, 2009, 18 (5): 101- 107. doi: 10.3969/j.issn.1673-8748.2009.05.018
12	SALAMEH H, KHASAWNEH H. Spectrum-time availability-aware routing mechanism for software-defined networks with out-of-band full-duplex capabilities[C]//Proc. of the 17th International Conference on Software Defined Systems, 2020: 24-28.
13	SUURBALLE J . Disjoint paths in a network[J]. Networks, 1974, 4 (2): 125- 145. doi: 10.1002/net.3230040204
14	VASS B , TAPOLCAI J , BERCZI-KOVACS E . Enumerating maximal shared risk link groups of circular disk failures hitting k nodes[J]. IEEE/ACM Trans.on Networking, 2021, 29 (4): 1648- 1661. doi: 10.1109/TNET.2021.3070100
15	YE Z Y, CHEN Z Y, NI P C. Reliability analysis and optimization algorithm of power communication network based on resource association features[C]//Proc. of the International Wireless Communications and Mobile Computing, 2020: 116-119.
16	ROSATO V , ISSACHAROFF L , TIRITICCO F . Modelling interdependent infrastructures using interacting dynamical mo-dels[J]. International Journal of Critical Infrastructure, 2008, 4 (1/2): 63- 79. doi: 10.1504/IJCIS.2008.016092
17	CHEN Y, MILANOVIC J V. Hybrid modelling of interconnected electric power and ICT system for reliability analysis[C]//Proc. of the IEEE Belgrade PowerTech, 2023.
18	LIU N , HU X J , MA L . Vulnerability assessment for coupled network consisting of power grid and EV traffic network[J]. IEEE Trans.on Smart Grid, 2021, 13 (1): 589- 598.
19	KAELBLING L P , LITTMAN M L , MOORE A W . Reinforcement learning: a survey[J]. Journal of Artificia Intelligence Research, 1996, 4, 237- 285. doi: 10.1613/jair.301
20	PAPADIMITRIOU C H , TSITSIKLIS J N . The complexity of Markov decision processes[J]. Mathematics of Operations Research, 1987, 12 (3): 441- 450. doi: 10.1287/moor.12.3.441
21	RIEDMILLER M. Neural fitted Q iteration-first experiences with a data efficient neural reinforcement learning method[C]//Proc. of the 16th European Conference on Machine Learning, 2005: 317-328.
22	LANGE S, RIEDMILLER M. Deep auto-encoder neural networks in reinforcement learning[C]//Proc. of the International Joint Conference on Neural Networks, 2010.
23	孟泠宇, 郭秉礼, 杨雯, 等. 基于深度强化学习的网络路由优化方法[J]. 系统工程与电子技术, 2022, 44 (7): 2311- 2318. doi: 10.12305/j.issn.1001-506X.2022.07.28
	MENG L Y , GUO B L , YANG W , et al. Network routing optimization approach based on deep reinforcement learning[J]. Systems Engineering and Electronics, 2022, 44 (7): 2311- 2318. doi: 10.12305/j.issn.1001-506X.2022.07.28
24	MUHATI E, RAWAT D. Asynchronous advantage actor-critic (A3C) learning for cognitive network security[C]//Proc. of the International Conference on Trust, Privacy and Security in Intelligent Systems and Applications, 2021: 106-113.
25	FUSEINI M , ALHASSAN M . Improving deep learning with prior knowledge and cognitive models: a survey on enhancing interpretability, adversarial robustness and zero-shot learning[J]. Cognitive Systems Research, 2023, 30, 101188.
26	WÜRFEL M, HAN Q, KAISER M. Online advertising revenue forecasting: an interpretable deep learning approach[C]//Proc. of the IEEE International Conference on Big Data, 2021: 1980-1989.
27	ALEJANDRO B . Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI[J]. Information Fusion, 2020, 58, 82- 115. doi: 10.1016/j.inffus.2019.12.012
28	ZHANG Z L , LI Y , YANG S . Code-aware fault localization with pre-training and interpretable machine learning[J]. Expert Systems with Applications, 2024, 238, 121689. doi: 10.1016/j.eswa.2023.121689
29	周志杰, 曹友, 胡昌华. 基于规则的建模方法的可解释性及其发展[J]. 自动化学报, 2020, 47 (6): 1201- 1216.
	ZHOU Z J , CAO Y , HU C H . The interpretability of rule-based modeling approach and its development[J]. Acta Automatica Sinica, 2020, 47 (6): 1201- 1216.
30	BEVEN K . Deep learning, hydrological processes and the uniqueness[J]. Hydrological Processes, 2020, 34 (16): 3608- 3613. doi: 10.1002/hyp.13805
31	马季军, 何小斌, 涂浡. 我国载人航天电源系统的技术发展成就及趋势[J]. 上海航天, 2021, 38 (3): 207- 218.
	MA J J , HE X B , TU B . Technical development achievements and trends of manned spaceflight power system in China[J]. Aerospace Shanghai, 2021, 38 (3): 207- 218.
32	PATEL M R . Spacecraft power systems[M]. Florida: CRC Press, 2004.
33	ZHANG T Y , HUANG H Z , LI Y F . Hierarchical fault propagation of command and control system[J]. Smart Structures and Systems, 2022, 29 (6): 791- 797.
34	GAO S , HUANG Y F , ZHANG S . Short-term runoff prediction with GRU and LSTM networks without requiring time step optimization during sample generation[J]. Journal of Hydrology, 2020, 589, 125188. doi: 10.1016/j.jhydrol.2020.125188
35	GARBAY T, CHUQUIMIA O, PINNA A. Distilling the knowledge in CNN for WCE screening tool[C]//Proc. of the Conference on Design and Architectures for Signal and Image Processing, 2019: 19-22.
36	YIM J, JOO D, BAE J. A gift from knowledge distillation: fast optimization, network minimization and transfer learning[C]//Proc. of the Conference on Computer Vision and Pattern Recognition, 2017: 7130-7138.

奖励所涉及前后节点V_E, V_C	当前转移状态s_t→s_t+1	当前时刻奖励\|R(s_t+1, a)\|≥γ(0.6)	长期累积折扣奖励G_t
			
{v_{E, 21}, v_{E, 24}, v_{E, 29}, v_{E, 35}}	78	0.69	52.31
{v_{E, 33}, v_{E, 47}, v_{E, 48}, v_{E, 79}, v_{E, 80}}	164	0.62	61.24
			
{v_{E, 33}, v_{E, 47}, v_{E, 48}, v_{E, 49}, v_{E, 52}}	420	-0.8	77.17
			
{v_{E, 29}, v_{E, 35}, v_{E, 70}, v_{E, 79}}	633	0.77	124.95
{v_{E, 29}, v_{E, 35}, v_{E, 70}, v_{E, 79}}	633	0.77	124.95

[1]	Mengyu ZHANG, Yajie DOU, Ziyi CHEN, Jiang JIANG, Kewei YANG, Bingfeng GE. Review of deep reinforcement learning and its applications in military field [J]. Systems Engineering and Electronics, 2024, 46(4): 1297-1308.
[2]	Fengguo WU, Wei TAO, Hui LI, Jianwei ZHANG, Chengchen ZHENG. UAV intelligent avoidance decisions based on deep reinforcement learning algorithm [J]. Systems Engineering and Electronics, 2023, 45(6): 1702-1711.
[3]	Jin TANG, Yangang LIANG, Zhihui BAI, Kebo LI. Landing control algorithm of rotor UAV based on DQN [J]. Systems Engineering and Electronics, 2023, 45(5): 1451-1460.
[4]	XIE Hao, GUO Aihuang, SONG Chunlin, JIAO Runze. eNB selection for LTE-V using deep reinforcement learning [J]. Systems Engineering and Electronics, 2019, 41(7): 1652-1657.
[5]	CAO Shi-xin, JIN Xin. Ranking model on different importance of military satellite information supporting to operations [J]. Journal of Systems Engineering and Electronics, 2009, 31(7): 1672-1676.

Spacecraft power-signal composite network optimization algorithm based on DRL

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 36

Related Articles 5

Recommended Articles

Metrics

Comments