基于核函数强化学习的抗干扰频点分配

doi:10.12305/j.issn.1001-506X.2021.06.12

Abstract

Abstract:

Aiming at the problem of learning unknown dynamic interference patterns, a frequency point cooperation algorithm for radar and communication anti-interference based on kernel function reinforcement learning is proposed. On the contrary, the proposed algorithm can optimize the anti-interference frequency allocation strategy by using the usage of the intermediate frequency points in the past slots. Firstly, the problem of curse of dimensions is solved by reinforcement learning of kernel function. Secondly, the online kernel sparsity method based on approximate linear correlation ensures the sparsity of anti-interference frequency assignment algorithm. Finally, simulation results verify the effectiveness of the proposed algorithm. Due to the learning of sparse codewords for the dynamic characteristics of the system, compared with the traditional anti-interference frequency assignment algorithm based on Q-learning, the proposed algorithm has shorter convergence time and can quickly avoid the interference of external unknown interference sources.

Key words: anti-interference, reinforcement learning, kernel method, Q-learning

CLC Number:

TN97

Zhiwei JIANG, Yang HUANG, Qihui WU. Anti-interference frequency allocation based on kernel reinforcement learning[J]. Systems Engineering and Electronics, 2021, 43(6): 1547-1556.

Figures/Tables 19

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Table 1

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Fig.13

Fig.14

Fig.15

Fig.16

Fig.17

Fig.18

References 27

1	TARIQ F , KHANDAKER M R A , WONG K K , et al. A speculative study on 6G[J]. IEEE Wireless Communications, 2020, 27 (4): 118- 125. doi: 10.1109/MWC.001.1900488
2	Federal Communications Commission . Facilitating opportunities for flexible, efficient, and reliable spectrum use employing cognitive radio technologies[J]. Et Docket, 2003, 3 (108): 5- 57.
3	LETAIEF K B , CHEN W , SHI Y , et al. The roadmap to 6G: AI empowered wireless networks[J]. IEEE Communications Magazine, 2019, 57 (8): 84- 90. doi: 10.1109/MCOM.2019.1900271
4	ZOU Y L , WANG X B , HANZO L , et al. A survey on wireless security: technical challenges, recent advances, and future trends[J]. Proceedings of the IEEE, 2016, 104 (9): 1727- 1765. doi: 10.1109/JPROC.2016.2558521
5	姚富强. 通信抗干扰工程与实践[M]. 第2版. 北京: 电子工业出版社, 2012.
	YAO F Q . Communication anti-jamming engineering and practice[M]. 2ed. Beijing: Publishing House of Electronics Industry, 2012.
6	ZHANG L Y , GUAN Z Y , MELODIA T . United against the enemy: anti-jamming based on cross-layer cooperation in wireless networks[J]. IEEE Trans.on Wireless Communications, 2016, 15 (8): 5733- 5747. doi: 10.1109/TWC.2016.2569083
7	LIU X , XU Y H , JIA L L , et al. Anti-jamming communications using spectrum waterfall: a deep reinforcement learning approach[J]. IEEE Communications Letters, 2018, 22 (5): 998- 1001. doi: 10.1109/LCOMM.2018.2815018
8	WU Y , WANG B , LIU K J , et al. Anti-jamming games in multi-channel cognitive radio networks[J]. IEEE Journal on Selected Areas in Communications, 2011, 30 (1): 4- 15.
9	YANG D J , XUE G L , ZHANG J , et al. Coping with a smart jammer in wireless networks: a Stackelberg game approach[J]. IEEE Trans.on Wireless Communications, 2013, 12 (8): 4038- 4047. doi: 10.1109/TWC.2013.071913121570
10	SAGDUYU Y E , BERRY R A , EPHREMIDES A . Jamming games in wireless networks with incomplete information[J]. IEEE Communications Magazine, 2011, 49 (8): 112- 118. doi: 10.1109/MCOM.2011.5978424
11	ZHU H J , FANG C L H , LIU Y , et al. You can jam but you cannot hide: defending against jamming attacks for geo-location database driven spectrum sharing[J]. IEEE Journal on Selected Areas in Communications, 2016, 34 (10): 2723- 2737. doi: 10.1109/JSAC.2016.2605799
12	SUTTON R S , BARTO A G . Reinforcement learning: an introduction[M]. Massachusetts: MIT Press, 2018.
13	WATKINS C J C H , DAYAN P . Q-learning[J]. Machine Learning, 1992, 8 (3/4): 279- 292. doi: 10.1023/A:1022676722315
14	MACHUZAK S, JAYAWEERA S K. Reinforcement learning based anti-jamming with wideband autonomous cognitive radios[C]//Proc. of the IEEE/CIC International Conference on Communications in China, 2016.
15	SINGH S, TRIVEDI A. Anti-jamming in cognitive radio networks using reinforcement learning algorithms[C]//Proc. of the 9th International Conference on Wireless and Optical Communications Networks, 2012.
16	AREF M A, JAYAWEERA S K, MACHUZAK S. Multi-agent reinforcement learning based cognitive anti-jamming[C]//Proc. of the IEEE Wireless Communications and Networking Conference, 2017.
17	POWELL W B . Approximate dynamic programming: solving the curses of dimensionality[M]. New Jersey: Wiley, 2007.
18	BROWNE D W. Predicting communications activity in the radio spectrum[C]//Proc. of the Conference Record of the 46th Asilomar Conference on Signals, Systems and Computers, 2012: 1069-1073.
19	TUMULURU V K , WANG P , NIYATO D . Channel status prediction for cognitive radio networks[J]. Wireless Communications and Mobile Computing, 2012, 12 (10): 862- 874. doi: 10.1002/wcm.1017
20	NGUYEN A, YOSINSKI J, CLUNE J. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 427-436.
21	PAPERNOT N, MCDANIEL P, JHA S, et al. The limitations of deep learning in adversarial settings[C]//Proc. of the IEEE European Symposium on Security and Privacy, 2016: 372-387.
22	李潮, 张巨泉. 雷达频域主要抗干扰技术及其效果度量[J]. 舰船电子对抗, 2005, 28 (1): 16- 20.
	LI C , ZHANG J Q . Chief Anti-jamming technologies of radar in frequency domain and their effectiveness measurement[J]. Shipboard Electronic Countermeasure, 2005, 28 (1): 16- 20.
23	贲德, 王峰, 雷志勇. 基于认知原理的机载雷达抗干扰技术研究[J]. 中国电子科学研究院学报, 2013, 8 (4): 368- 372.
	BEN D , WANG F , LEI Z Y . Key anti-jamming technique of airborne radar based on cognition[J]. Journal of China Academy of Electronics and Information Technology, 2013, 8 (4): 368- 372.
24	周红平. 雷达信号处理若干关键技术的研究[D]. 合肥: 中国科学技术大学, 2010.
	ZHOU H P. Research on some key technologies of radar signal processing[D]. Hefei: University of Science and Technology of China, 2010.
25	RICHARDS M A . Fundamentals of radar signal processing[M]. New York: McGraw-Hill Education, 2005.
26	TSILIGKARIDIS T, ROMERO D. Reinforcement learning with budget-constrained nonparametric function approximation for opportunistic spectrum access[C]//Proc. of the IEEE Global Conference on Signal and Information Processing, 2018: 579-583.
27	XU X , HU D W , LU X C . Kernel-based least squares policy iteration for reinforcement learning[J]. IEEE Trans.on Neural Networks, 2007, 18 (4): 973- 992. doi: 10.1109/TNN.2007.899161

参数	值	描述
y	2	历史状态数
σ_s	1.2	核函数参数
σ_a	2	核函数参数
μ	0.82	近似精度阈值
λ	0.01	折扣系数
γ	0.02	学习速率

[1]	Bakun ZHU, Weigang ZHU, Wei LI, Ying YANG, Tianhao GAO. Research on decision-making modeling of cognitive jamming for multi-functional radar based on Markov [J]. Systems Engineering and Electronics, 2022, 44(8): 2488-2497.
[2]	Guan WANG, Haizhong RU, Dali ZHANG, Guangcheng MA, Hongwei XIA. Design of intelligent control system for flexible hypersonic vehicle [J]. Systems Engineering and Electronics, 2022, 44(7): 2276-2285.
[3]	Lingyu MENG, Bingli GUO, Wen YANG, Xinwei ZHANG, Zuoqing ZHAO, Shanguo HUANG. Network routing optimization approach based on deep reinforcement learning [J]. Systems Engineering and Electronics, 2022, 44(7): 2311-2318.
[4]	Dongzi GUO, Rong HUANG, Hechuan XU, Liwei SUN, Naigang CUI. Research on deep deterministic policy gradient guidance method for reentry vehicle [J]. Systems Engineering and Electronics, 2022, 44(6): 1942-1949.
[5]	Mingren HAN, Yufeng WANG. Optimization method for orbit transfer of all-electric propulsion satellite based on reinforcement learning [J]. Systems Engineering and Electronics, 2022, 44(5): 1652-1661.
[6]	Shihan TAN, Fenglin JIN, Congying DUN. Task assignment strategy for space-air-ground integrated vehicular networks oriented to user demand [J]. Systems Engineering and Electronics, 2022, 44(5): 1717-1727.
[7]	Zhiwei YANG, Xuexin XIE, Shuwan LI. Analysis of coherent processing capability of pulse width-FMpolarity agile waveform [J]. Systems Engineering and Electronics, 2022, 44(4): 1139-1147.
[8]	Li HE, Liang SHEN, Hui LI, Zhuang WANG, Wenquan TANG. Survey on policy reuse in reinforcement learning [J]. Systems Engineering and Electronics, 2022, 44(3): 884-899.
[9]	Bakun ZHU, Weigang ZHU, Wei LI, Ying YANG, Tianhao GAO. Multi-function radar intelligent jamming decision method based on prior knowledge [J]. Systems Engineering and Electronics, 2022, 44(12): 3685-3695.
[10]	Siwei TAN, Bo TANG, Linsen ZHANG, Sen ZHANG. Interference sensing technical solution for frequency hopping electromagnetic fuze [J]. Systems Engineering and Electronics, 2022, 44(11): 3330-3337.
[11]	Qingqing YANG, Yingying GAO, Yu GUO, Boyuan XIA, Kewei YANG. Target search path planning for naval battle field based on deep reinforcement learning [J]. Systems Engineering and Electronics, 2022, 44(11): 3486-3495.
[12]	Bin ZENG, Hongqiang ZHANG, Houpu LI. Research on anti-submarine strategy for unmanned undersea vehicles [J]. Systems Engineering and Electronics, 2022, 44(10): 3174-3181.
[13]	Qitian WAN, Baogang LU, Yaxin ZHAO, Qiuqiu WEN. Autopilot parameter rapid tuning method based on deep reinforcement learning [J]. Systems Engineering and Electronics, 2022, 44(10): 3190-3199.
[14]	Bin ZENG, Rui WANG, Houpu LI, Xu FAN. Scheduling strategies research based on reinforcement learning for wartime support force [J]. Systems Engineering and Electronics, 2022, 44(1): 199-208.
[15]	Xiaoge WANG, Hui CHEN, Mengyu NI, Liuliu NI, Bingbing LI. Radar anti-false target jamming method based on phase modulation [J]. Systems Engineering and Electronics, 2021, 43(9): 2476-2483.

Anti-interference frequency allocation based on kernel reinforcement learning

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 19

References 27

Related Articles 15

Recommended Articles

Metrics

Comments