深度强化学习在天基信息网络中的应用——现状与前景

doi:10.12305/j.issn.1001-506X.2023.03.31

Abstract

Abstract:

Space information network (SIN) will face challenges from its development trend of complex structure, dynamic environment, and diverse types of emerging applications. In this context, the data-driven deep reinforcement learning (DRL) methods are introduced into SIN field as one of the promising solutions to cope with the aforementioned challenges. This paper firstly reviewed commonly used basic DRL methods in SIN field, with a comprehensive literature review of DRL-based SIN methods. Then, considering the relay selection in satellite-terrestrial network as an example, we propose an algorithm based on mean field DRL to address the large-scale issue. A model transfer mechanism based on finetune is proposed to solve the problem of data difference between simulation environment and real environment. Simulation result demonstrates that the proposed method can optimize network performance with acceptable computational complexity and time efficiency. Then the limitations and challenges of DRL in the field of SIN are summarized. Finally, based on frontier hotspots of DRL, we further provide insights into several future research directions in the context of DRL-based SIN methods.

Key words: space information network(SIN), deep reinforcement learning(DRL), relay selection, network performance optimization

CLC Number:

TN929.5

Siqi TANG, Zhisong PAN, Guyu HU, Yang WU, Yunbo LI. Application of deep reinforcement learning in space information network——status quo and prospects[J]. Systems Engineering and Electronics, 2023, 45(3): 886-901.

Figures/Tables 13

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Table 1

Table 2

Feasibility analysis for SIN's research direction with DRL available"

研究方向	计算能力	实时性要求	算法效果	综合收益	进展
资源分配	有限, 星上计算	较低, 可根据算法速度设定动态调整资源的时隙	较好, 可大幅提高资源利用率	较高, 资源紧缺是SIN面临的重要问题	NASA已进行星上验证^[30]
跳波束	有限, 星上计算	较低, 可根据算法速度设定跳波束时隙	基于MADRL的方法效果较好, DRL方法面临维度灾难	较高, 解决了流量空间分布不均匀对资源的浪费	理论研究
接入网络选择	终端分布式决策, 不需星上计算	较低, 可根据算法速度设定接入调整频率	较好, 但需要收集多层异构网络的信息	较高, 优化了空天地一体化网络中的接入决策	理论研究
拥塞控制^[31]	终端分布式决策, 不需星上计算	较低, 可根据算法速度设定窗口调整频率	较好, 问题简单直接, 且决策空间有限	较高, 但需要考虑网络设备更换的代价	理论研究
计算卸载	较高, 终端分布式决策, 不需星上计算	高, 但对卫星无要求, 对终端能力和算法时效性有要求	有待提高, 其与通信过程的资源分配问题耦合, 考虑因素多, 决策维度高, DRL训练难度大	目前有限, 但在计算任务日益增加、边缘能力日益增强的未来场景^[32]中较有前景	理论研究
卫星切换	终端分布式决策, 不需星上计算	较低, LEO卫星过顶时间为分钟级, DRL算法使用阶段的决策时间为毫秒级	有待提高, 现有方法没能与资源分配结合, 因此效果有待优化	现阶段收益有限, 对未来超大规模星座^[33]有意义	理论研究
路由选择	有限, 星上计算	高, 数据包转发对时效性要求高	在拥塞或者受干扰的网络中性能优于其他方法	较低, 路由决策无法牺牲时间代价	理论研究
接入协议优化^[34]	终端分布式决策, 不需星上计算	高, 数据包流量大	MARL效果较好, 而DRL在节点规模增大时, 收敛效果变差	较低, 每个发送数据包需要承受DRL决策的时间代价	理论研究
缓存	有限, 星上计算	高, 内容访问请求流量大	有待提高, 内容数量多, 缓存决策动作空间大	较低, 卫星缓存资源有限, 优化缓存策略取得的收益有限	理论研究

Table 2

Fig.6

Table 3

Fig.7

Table 4

Fig.8

Fig.9

References 55

1	LIU J J , SHI Y P , FADLULLAH Z M , et al. Space-air-ground integrated network: a survey[J]. IEEE Communications Surveys & Tutorials, 2018, 20 (4): 2714- 2741.
2	ARULKUMARAN K , DEISENROTH M P , BRUNDAGE M , et al. A brief survey of deep reinforcement learning[J]. IEEE Signal Processing Magazine, 2017, 34 (6): 26- 38. doi: 10.1109/MSP.2017.2743240
3	张沛, 刘帅军, 马治国, 等. 基于深度增强学习和多目标优化改进的卫星资源分配算法[J]. 通信学报, 2020, 41 (6): 51- 60.
	ZHANG P , LIU S J , MA Z G , et al. Improved satellite resource allocation algorithm based on DRL and MOP[J]. Journal on Communications, 2020, 41 (6): 51- 60.
4	LIU S J , HU X , WANG W D . Deep reinforcement learning based dynamic channel allocation algorithm in multibeam satellite systems[J]. IEEE Access, 2018, 6, 15733- 15742. doi: 10.1109/ACCESS.2018.2809581
5	刘建业, 王华, 周晚萌. 基于GA-SA的低轨星座传感器资源调度算法[J]. 系统工程与电子技术, 2018, 40 (11): 2476- 2481. doi: 10.3969/j.issn.1001-506X.2018.11.13
	LIU J Y , WANG H , ZHOU W M . LEO constellation sensor resources scheduling algorithm based on genetic and simulated annealing[J]. Systems Engineering and Electronics, 2018, 40 (11): 2476- 2481. doi: 10.3969/j.issn.1001-506X.2018.11.13
6	ZHAO B K , LIU J H , WEI Z L , et al. A deep reinforcement learning based approach for energy-efficient channel allocation in satellite Internet of things[J]. IEEE Access, 2020, 8, 62197- 62206. doi: 10.1109/ACCESS.2020.2983437
7	HU X , LIAO X L , LIU Z J , et al. Multi-agent deep reinforcement learning-based flexible satellite payload for mobile terminals[J]. IEEE Trans.on Vehicular Technology, 2020, 69 (9): 9849- 9865. doi: 10.1109/TVT.2020.3002983
8	LUIS J J G, GUERSTER M, DEL PORTILLO I, et al. Deep reinforcement learning for continuous power allocation in flexible high throughput satellites[C]//Proc. of the 2nd IEEE Cognitive Communications for Aerospace Applications Workshop, 2019.
9	LUIS J J G, PACHLER N, GUERSTER M, et al. Artificial intelligence algorithms for power allocation in high throughput sate- llites: a comparison[C]//Proc. of the IEEE Aerospace Conference, 2020.
10	FERREIRA P V R , PAFFENROTH R , WYGLINSKI A M , et al. Multi-objective reinforcement learning for cognitive sate-llite communications using deep neural network ensembles[J]. IEEE Journal on Selected Areas in Communications, 2018, 36 (5): 1030- 1041. doi: 10.1109/JSAC.2018.2832820
11	HU X , LIU S J , WANG Y P , et al. Deep reinforcement learning-based beam Hopping algorithm in multibeam satellite systems[J]. IET Communications, 2019, 13 (16): 2485- 2491. doi: 10.1049/iet-com.2018.5774
12	HU X , ZHANG Y C , LIAO X L , et al. Dynamic beam hopping method based on multi-objective deep reinforcement learning for next generation satellite broadband systems[J]. IEEE Trans.on Broadcasting, 2020, 66 (3): 630- 646. doi: 10.1109/TBC.2019.2960940
13	TANG Q Q , FEI Z S , LI B , et al. Computation offloading in LEO satellite networks with hybrid cloud and edge computing[J]. IEEE Internet of Things Journal, 2021, 8 (11): 9164- 9176. doi: 10.1109/JIOT.2021.3056569
14	ZHOU C H , WU W , HE H L , et al. Deep reinforcement learning for delay-oriented IoT task scheduling in space-air-ground integrated network[J]. IEEE Trans.on Wireless Communications, 2020, 20 (2): 911- 925.
15	CUI G F , LONG Y T , XU L X , et al. Joint offloading and resource allocation for satellite assisted vehicle-to-vehicle communication[J]. IEEE Systems Journal, 2020, 15 (3): 3958- 3969.
16	QIU C , YAO H P , YU F R , et al. Deep Q-learning aided networking, caching, and computing resources allocation in software-defined satellite-terrestrial networks[J]. IEEE Trans.on Vehicular Technology, 2019, 68 (6): 5871- 5883. doi: 10.1109/TVT.2019.2907682
17	MENG X L , WU L D , YU S B . Research on resource allocation method of space information networks based on deep reinforcement learning[J]. Remote Sensing, 2019, 11 (4): 448. doi: 10.3390/rs11040448
18	朱立东, 张勇, 贾高一. 卫星互联网路由技术现状及展望[J]. 通信学报, 2021, 42 (8): 33- 42.
	ZHU L D , ZHANG Y , JIA G Y . Current status and future prospects of routing technologies for satellite Internet[J]. Journal on Communications, 2021, 42 (8): 33- 42.
19	WANG C , WANG H W , WANG W D . A two-hops state-aware routing strategy based on deep reinforcement learning for LEO satellite networks[J]. Electronics, 2019, 8 (9): 920. doi: 10.3390/electronics8090920
20	TU Z, ZHOU H C, LI K, et al. A routing optimization method for software-defined SGIN based on deep reinforcement learning[C]//Proc. of the IEEE Global Communications Conference Workshops, 2019.
21	LIU J H , ZHAO B K , XIN Q , et al. DRL-ER: an intelligent energy-aware routing protocol with guaranteed delay bounds in satellite mega-constellations[J]. IEEE Trans.on Network Science and Engineering, 2020, 8 (4): 2872- 2884.
22	HAN C , HUO L Y , TONG X H , et al. Spatial anti-jamming scheme for internet of satellites based on the deep reinforcement learning and stackelberg game[J]. IEEE Trans.on Vehi-cular Technology, 2020, 69 (5): 5331- 5342. doi: 10.1109/TVT.2020.2982672
23	杨斌, 何锋, 靳瑾, 等. LEO卫星通信系统覆盖时间和切换次数分析[J]. 电子与信息学报, 2014, 36 (4): 804- 809.
	YANG B , HE F , JIN J , et al. Analysis of coverage time and handoff number on LEO satellite communication systems[J]. Journal of Electronics & Information Technology, 2014, 36 (4): 804- 809.
24	XU H H , LI D S , LIU M L , et al. QoE-driven intelligent hand- over for user-centric mobile satellite networks[J]. IEEE Trans.on Vehicular Technology, 2020, 69 (9): 10127- 10139. doi: 10.1109/TVT.2020.3000908
25	HE S X, WANG T Y, WANG S W. Load-aware satellite handover strategy based on multi-agent reinforcement learning[C]//Proc. of the IEEE Global Communications Conference, 2020.
26	CAO Y , LIEN S Y , LIANG Y C . Deep reinforcement learning for multi-user access control in non-terrestrial networks[J]. IEEE Trans.on Communications, 2020, 69 (3): 1605- 1619.
27	LEE J H, PARK J, BENNIS M, et al. Integrating LEO sate-llite and UAV relaying via reinforcement learning for non-terrestrial networks[C]//Proc. of the IEEE Global Communications Conference, 2020.
28	LI X N, ZHANG H J, LI W, et al. Multi-agent DRL for user association and power control in terrestrial-satellite network[C]//Proc. of the IEEE Global Communications Conference, 2021.
29	FERIANI A , HOSSAIN E . Single and multi-agent deep reinforcement learning for AI-enabled wireless networks: a tutorial[J]. IEEE Communications Surveys & Tutorials, 2021, 23 (2): 1226- 1252.
30	FERREIRA P V R , PAFFENROTH R , WYGLINSKI A M , et al. Reinforcement learning for satellite communications: From LEO to deep space operations[J]. IEEE Communications Magazine, 2019, 57 (5): 70- 75.
31	MAI T, YAO H P, JING Y Q, et al. Self-learning congestion control of MPTCP in satellites communications[C]//Proc. of the 15th International Wireless Communications & Mobile Computing Conference, 2019: 775-780.
32	XIE R C , TANG Q Q , WANG Q N , et al. Satellite-terrestrial integrated edge computing networks: architecture, challenges, and open issues[J]. IEEE Network, 2020, 34 (3): 224- 231.
33	HASSAN N U L , HUANG C W , YUEN C , et al. Dense small satellite networks for modern terrestrial communication systems: benefits, infrastructure, and technologies[J]. IEEE Wireless Communications, 2020, 27 (5): 96- 103.
34	ZHAO B , REN G L , DONG X D , et al. Distributed Q-learning based joint relay selection and access control scheme for IoT-oriented satellite terrestrial relay networks[J]. IEEE Communications Letters, 2021, 25 (6): 1901- 1905.
35	MNIH V , KAVUKCUOGLU K , SILVER D , et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518 (7540): 529- 533.
36	YANG Y D, LUO R, LI M N, et al. Mean field multi-agent reinforcement learning[C]//Proc. of the 35th International Conference on Machine Learning, 2018: 5571-5580.
37	ARTI M K . Channel estimation and detection in satellite communication systems[J]. IEEE Trans.on Vehicular Technology, 2016, 65 (12): 10173- 10179.
38	BANKEY V , UPADHYAY P K , DA COSTA D B , et al. Performance analysis of multi-antenna multiuser hybrid satellite-terrestrial relay systems for mobile services delivery[J]. IEEE Access, 2018, 6, 24729- 24745.
39	PACHECO F , EXPOSITO E , GINESTE M . A framework to classify heterogeneous Internet traffic with machine learning and deep learning techniques for satellite communications[J]. Computer Networks, 2020, 173, 107213.
40	RAO S K . Advanced antenna technologies for satellite communications payloads[J]. IEEE Trans.on Antennas and Propagation, 2015, 63 (4): 1205- 1217.
41	万里鹏, 兰旭光, 张翰博, 等. 深度强化学习理论及其应用综述[J]. 模式识别与人工能, 2019, 32 (1): 67- 81.
	WAN L P , LAN X G , ZHANG H B , et al. A review of deep reinforcement learning theory and application[J]. Pattern Re-cognition and Artificial Intelligence, 2019, 32 (1): 67- 81.
42	ARULKUMARAN K , DEISENROTH M P , BRUNDAGE M , et al. Deep reinforcement learning: a brief survey[J]. IEEE Signal Processing Magazine, 2017, 34 (6): 26- 38.
43	NG A Y, RUSSELL S J. Algorithms for inverse reinforcement learning[C]//Proc. of the 17th International Conference on Machine Learning, 2000: 663-670.
44	ZHU Z D, LIN K X, ZHOU J Y. Transfer learning in deep reinforcement learning: a survey[EB/OL]. [2021-08-23]. https://arxiv.org/abs/2009.07888.
45	谭晓阳, 张哲. 元强化学习综述[J]. 南京航空航天大学学报, 2021, 53 (5): 653- 663.
	TAN X Y , ZHANG Z . Review on meta reinforcement learning[J]. Journal of Nanjing University of Aeromautics and Astronautics, 2021, 53 (5): 653- 663.
46	周文吉, 俞扬. 分层强化学习综述[J]. 智能系统学报, 2017, 12 (5): 590- 594.
	ZHOU W J , YU Y . Summarize of hierarchical reinforcement learning[J]. CAAI Transactions on Intelligent Systems, 2017, 12 (5): 590- 594.
47	GENG Y Z , LIU E W , WANG R , et al. Hierarchical reinforcement learning for relay selection and power optimization in two-hop cooperative relay network[J]. IEEE Trans.on Communications, 2021, 70 (1): 171- 184.
48	YANG R, SUN X, NARASIMHAN K. A generalized algorithm for multi-objective reinforcement learning and policy ada-ptation[C]//Proc. of the Conference and Workshop on Neural Information Processing Systems, 2019.
49	HOCHREITER S , SCHMIDHUBER J . Long short-term memory[J]. Neural Computation, 1997, 9 (8): 1735- 1780.
50	GOODFELLOW I , POUGET-ABADIE J , MIRZA M , et al. Generative adversarial nets[J]. Advances in Neural Information Processing Systems, 2014, 27, 1- 9.
51	YAN S, XIONG Y, LIN D. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Proc. of the 32ed Association for the Advance of Artificial Intelligence, 2018.
52	NURVITADHI E, SIM J, SHEFFIELD D, et al. Accelerating recurrent neural networks in analytics servers: Comparison of FPGA, CPU, GPU, and ASIC[C]// Proc. of the IEEE 26th International Conference on Field Programmable Logic and App- lications, 2016.
53	李德仁, 沈欣, 李迪龙, 等. 论军民融合的卫星通信、遥感、导航一体天基信息实时服务系统[J]. 武汉大学学报(信息科学版), 2017, 42 (11): 1501- 1505.
	LI D R , SHEN X , LI D L , et al. On civil-military integrated space-based real-time information service system[J]. Geomatics and Information Science of Wuhan University, 2017, 42 (11): 1501- 1505.
54	HINTON G, VINYALS O, DEAN J. Distilling the knowledge in a neural network[EB/OL]. [2021-08-23]. https://arxiv.org/abs/1503.02531.
55	LIU Z, LI J, SHEN Z, et al. Learning efficient convolutional networks through network slimming[C]//Proc. of the IEEE International Conference on Computer Vision, 2017: 2736-2744.

问题	时机	目标
卫星切换	当前卫星无法继续服务	保持连续服务
接入选择	每一时刻	优化网络效能

领域	文献	场景	针对问题	优化目标	DRL方法
资源分配	[3]	多波束GEO卫星网络	用户时隙分配	用户满意度、能量和频谱利用率	DQN
	[4]	多波束GEO卫星网络	用户信道分配	呼通率	DQN
	[6]	LEO卫星物联网	用户信道分配	能量利用率	DQN
	[7]	多波束GEO卫星网络	波束带宽分配	公平性、流量满足程度	MARL
	[8]	多波束GEO卫星网络	波束功率分配	功率消耗、流量满足程度	DDPG
	[10]	GEO卫星网络	配置链路参数	吞吐量、误码率、功耗、带宽稳定	DQN
跳波束	[11]	多波束GEO卫星	波束点亮方案	传输时延	DQN
跳波束	[12]	多波束GEO卫星	波束点亮方案	实时服务时延, 非实时服务吞吐量, 公平性	双环DQN
计算卸载与缓存	[14]	空天地一体化网络	任务卸载位置决策	平均处理时延	DQN
	[15]	GEO卫星辅助车联网	任务卸载、计算和通信资源联合分配	时延	优化、DQN
	[16]	天地一体化网络	通信、缓存和计算资源联合分配	通信、缓存和计算开销	DQN
	[17]	多层卫星网络	缓存策略、计算卸载、接入选择联合决策	缓存和计算开销	A3C
路由选择	[19]	LEO卫星星座	下一跳路由选择	跳数、丢包率、拥塞避免	Double DQN
	[20]	天地一体化网络	下一跳路由选择	时延、丢包率、吞吐量	DDPG
	[21]	LEO卫星星座	下一跳路由选择	时延、卫星电池能量寿命	DQN
	[22]	LEO卫星星座	抗干扰路径集合计算	集合中链路不受干扰	近似策略优化
卫星切换	[24]	LEO卫星星座	切换选择	QoE	DQN
卫星切换	[25]	LEO卫星星座	切换选择	切换次数	MARL
接入选择	[26]	空天地一体化网络	接入选择	吞吐量	DQN
接入选择	[27]	空天地一体化网络	接入选择与航迹调整	吞吐量	DQN

方法	吞吐量/Mbps		运算时间/min
方法	N=30	N=120	N=30	N=120
GA	19.8	—	375.3	—
MRP	16.3	36.8	—	—
D-DRL	18.4	40.5	11.41	55.67
MF-MADRL	20.1	46.7	4.15	8.27

[1]	Guangda ZHANG, Qinghua REN, Zhikai FAN. Security transmission scheme for multi-relay system with a friendly jammer-assisted based on energy harvesting [J]. Systems Engineering and Electronics, 2023, 45(3): 876-885.
[2]	Yin LU, Gaoyu WANG, Chuying YANG, Lingqing YANG, Kun ZHAO, Hongbo ZHU. NOMA relay selection and cooperative transmission based on single source optimal path [J]. Systems Engineering and Electronics, 2022, 44(1): 292-298.
[3]	Hong ZOU, Bin WAN, Dapeng WU. Edge cooperative relay mechanism with cooperative willingness aware [J]. Systems Engineering and Electronics, 2020, 42(5): 1173-1181.
[4]	Huaicong KONG, Min LIN, Xiaoyu LIU, Xuewen WU, Jinyuan WANG. Performance analysis of hybrid satellite-terrestrial cooperative transmission system based on relay selection [J]. Systems Engineering and Electronics, 2020, 42(1): 198-205.
[5]	LI Min, WANG Pingshan, WANG Kaili. Selection cooperation based on packet aggregation in multi-source multi-destination networks [J]. Systems Engineering and Electronics, 2019, 41(5): 1149-1155.
[6]	WANG Heng, WEI Xinyu, LI Min. Relay selection methods for multi-source and multi-destination cooperative networks [J]. Systems Engineering and Electronics, 2017, 39(6): 1358-1365.
[7]	FU Xiao-mei, YAO Xiao-ming, ZONG Qun. Relay selection and power optimization in cooperative coalitions [J]. Systems Engineering and Electronics, 2016, 38(5): 1176-.
[8]	XIAO Hai-lin, WANG Li-yuan, YAN Kun, LI Min-zheng. Hybrid-decode-amplify-and-forward cooperative communication based on best relay selection algorithm [J]. Systems Engineering and Electronics, 2015, 37(2): 400-405.

Application of deep reinforcement learning in space information network——status quo and prospects

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 13

References 55

Related Articles 8

Recommended Articles

Metrics

Comments