基于隐式对手建模的策略重构抗智能干扰方法

doi:10.12305/j.issn.1001-506X.2025.04.32

Abstract

Abstract:

With the continuous advancement of artificial intelligence technology, intelligent jamming seri-ously threatens wireless signal transmission, and traditional anti-jamming algorithms are insufficient. Based on the above issue, using reinforcement learning algorithms as foundation, implicit opponent modeling techniques are introduced, encoding the jamming agent's strategy implicitly in the neural network input and determining communication frequencies through neural network decisions. In response to the non-stationary nature of intelligent jamming strategies, profit trends are monitored to identify whether jamming strategies are switching and strategy reconstruction technology is proposed. Multi-scale window detection is utilized to identify the start point of experiential failure and discard failed experiences. Learning rate is simultaneously reset to accelerate the convergence speed of the neural network. Experimental results demonstrate that during the relative convergence phase, the proposed method's transmission success rate is increased by over 25% compared to the deep Q-network anti-jamming method.

Key words: anti-jamming, intelligent jamming, deep reinforcement learning, trend detection, implicit opponent modeling

CLC Number:

TN795

Peng MA, Rui JIANG, Bin WANG, Mengfei XU, Changbo HOU. Strategy reconstruction for resilience against intelligence jamming based on implicit opponent modeling[J]. Systems Engineering and Electronics, 2025, 47(4): 1355-1363.

Figures/Tables 10

Fig.1

Fig.2

Fig.3

Table 1

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

References 31

1	WANGX M,WANGJ L,XUY H,et al.Dynamic spectrum anti-jamming communications: challenges and opportunities[J].IEEE Communications Magazine,2020,58(2):79-85. doi: 10.1109/MCOM.001.1900530
2	ZHOUQ,NIUY T,XIANGP,et al.Intra-domain knowledge reuse assisted reinforcement learning for fast anti-jamming communication[J].IEEE Trans.on Information Forensics and Security,2023,18,4707-4720. doi: 10.1109/TIFS.2023.3284611
3	RUBYR,YANGH L,WUK S.Anti-jamming strategy for federated learning in Internet of medical things: a game approach[J].IEEE Journal of Biomedical and Health Informatics,2022,27(2):888-899.
4	JAITLY S, MALHOTRA H, BHUSHAN B. Security vulnerabilities and countermeasures against jamming attacks in wireless sensor networks: a survey[C]//Proc. of the International Conference on Computer, Communications and Electronics, 2017: 559-564.
5	MACHUZAK S, JAYAWEERA S K. Reinforcement learning based anti-jamming with wideband autonomous cognitive radios[C]//Proc. of the IEEE/CIC International Conference on Communications in China, 2016.
6	潘筱茜,张姣,刘琰,等.基于深度强化学习的多域联合干扰规避[J].信号处理,2022,38(12):2572-2581.
	PANX Q,ZHANGJ,LIUY,et al.Multi domain joint interference avoidance based on deep reinforcement learning[J].Signal Processing,2022,38(12):2572-2581.
7	SLIMENIF,CHTOUROUZ,SCHEERSB,et al.Cooperative Q-learning based channel selection for cognitive radio networks[J].Wireless Networks,2019,25,4161-4171. doi: 10.1007/s11276-018-1737-9
8	LIUX,XUY H,JIAL L,et al.Anti-jamming communications using spectrum waterfall: a deep reinforcement learning approach[J].IEEE Communications Letters,2018,22(5):998-1001. doi: 10.1109/LCOMM.2018.2815018
9	吴志娟,林艳,张一晋,等.基于多智能体协同的无人机簇群多域节能抗干扰通信[J].中国科学: 信息科学,2023,53(12):2511-2526.
	WUZ J,LINY,ZHANGY J,et al.Multi-agent collaboration based UAV clusters multi-domain energy-saving anti-jamming communication[J].SCIENTIA SINICA Informationis,2023,53(12):2511-2526.
10	SILVERD,HUANGA,MADDISONC J,et al.Mastering the game of go with deep neural networks and tree search[J].Nature,2016,529(7587):484-489. doi: 10.1038/nature16961
11	NGUYEN P K H, NGUYEN V H, DO V L. A deep double-Q learning-based scheme for anti-jamming communications[C]//Proc. of the 28th European Signal Processing Conference, 2021: 1566-1570.
12	VANH N,HOANGD T,NGUYEND N,et al.DeepFake: deep dueling-based deception strategy to defeat reactive jammers[J].IEEE Trans.on Wireless Communications,2021,20(10)):6898-6914.
13	XIAOL,ZHANGH L,XIAOY L,et al.Reinforcement learning-based downlink interference control for ultra-dense small cells[J].EEE Trans.on Wireless Communications,2020,19(1):423-434. doi: 10.1109/TWC.2019.2945951
14	YANGH L,ZHAOJ,LAMK Y,et al.Distributed deep reinforcement learning-based spectrum and power allocation for heterogeneous networks[J].IEEE Trans.on Wireless Communications,2022,21(9):6935-6948. doi: 10.1109/TWC.2022.3153175
15	XIAOL,DINGY Z,HUANGJ H,et al.UAV anti-jamming video transmissions with QoE guarantee: a reinforcement learning-based approach[J].IEEE Trans.on Communications,2021,69(9):5933-5947. doi: 10.1109/TCOMM.2021.3087787
16	LUX Z,XIAOL,DAIC H,et al.UAV-aided cellular communications with deep reinforcement learning against jamming[J].IEEE Wireless Communications,2020,27(4):48-53. doi: 10.1109/MWC.001.1900207
17	FANC Q,LIUH Y,LIB,et al.Adversarial game against hybrid attacks in UAV communications with partial information[J].IEEE Trans.on Vehicular Technolog,2021,71(2):2204-2208.
18	NOORIH,SADEGHIV S.Jamming and anti-jamming in interference channels: a stochastic game approach[J].IET Communications,2020,14(4):682-692. doi: 10.1049/iet-com.2019.0637
19	WANGB B,WUY L,LIUK J R,et al.An anti-jamming stochastic game for cognitive radio networks[J].IEEE Journal on Selected Areas in Communications,2011,29(4):877-889. doi: 10.1109/JSAC.2011.110418
20	LIW,XUY H,CHENJ,et al.Know the enemy: an opponent modeling-based anti-intelligent jamming strategy beyond equilibrium solutions[J].IEEE Wireless Communications Letters,2022,12(2):217-221.
21	AL-SHEDIVAT M, BANSAL T, BURDA Y, et al. Continuous adaptation via meta-learning in nonstationary and competitive environments[EB/OL]. [2014-02-01]. https://arxiv.org/abs/1710.03641.
22	THUENTE D, ACHARYA M. Intelligent jamming in wireless networks with applications to 802.11 b and other networks[C]// Proc. of the IEEE Military Communications Conference, 2006.
23	GLEAVE A, DENNIS M, WILD C, et al. Adversarial policies: attacking deep rein-forcement learning[EB/OL]. [2014-02-01]. https://arxiv.org/abs/1905.10615.
24	LIY Y,WANGX M,LIUD X,et al.On the performance of deep reinforcement learning-based anti-jamming method confronting intelligent jammer[J].Applied Sciences,2019,9(7):1361. doi: 10.3390/app9071361
25	MNIH V, KAVUKCUOGLU K, SILVER D, et al. Playing atari with deep reinforcement learning[EB/OL]. [2014-02-01]. https://arxiv.org/abs/1312.5602.
26	VAN H H, GUEZ A, SILVER D. Deep reinforcement learning with double Q -learning[C]//Proc. of the AAAI Conference on Artificial Intelligence, 2016.
27	WANG Z Y, SCHAUL T, HESSEL M, et al. Dueling network architectures for deep reinforcement learning[C]//Proc. of the International Conference on Machine Learning, 2016: 1995-2003.
28	HE H, BOYD-GRABER J, KWOK K, et al. Opponent modeling in deep reinforcement learning[C]//Proc. of the International Conference on Machine Learning, 2016: 1804-1813.
29	赵天昊. 动态博弈环境下的隐式对手建模方法[D]. 大连: 大连理工大学, 2021.
	ZHAO T H. A method for modeling implicit opponents in dynamic game environments[D]. Dalian: Dalian University of Technology, 2021.
30	EYRE-WALKERA.Changing effective population size and the McDonald-Kreitman test[J].Genetics,2002,162(4):2017-2024. doi: 10.1093/genetics/162.4.2017
31	SLIMENIF,SCHEERSB,CHTOUROUZ,et al.Cognitive radio jamming mitigation using Markov decision process and reinforcement learning[J].Procedia Computer Science,2015,73,199-208.

参数名称	数值
经验抽取数量	64
最大学习率	6e-3
最小学习率	2e-3
折扣因子γ	0.965
探索因子ε	0.07
目标网络更新轮数	50
全连接层1神经元数	32
全连接层2神经元数	32
全连接层3神经元数	10
全连接层4神经元数	1

[1]	Ziyi WANG, Xiongjun FU, Jian DONG, Cheng FENG. Optimization of radar collaborative anti-jamming strategies based on hierarchical multi-agent reinforcement learning [J]. Systems Engineering and Electronics, 2025, 47(4): 1108-1114.
[2]	Ziyi WANG, Xiongjun FU, Jian DONG, Cheng FENG. Optimization of radar collaborative anti-jamming strategies based on hierarchical multi-agent reinforcement learning [J]. Systems Engineering and Electronics, 2025, 47(4): 1108-1114.
[3]	Wei XIONG, Dong ZHANG, Zhi REN, Shuheng YANG. Research on intelligent decision-making methods for coordinated attack by manned aerial vehicles and unmanned aerial vehicles [J]. Systems Engineering and Electronics, 2025, 47(4): 1285-1299.
[4]	Kaiqiang TANG, Huiqiao FU, Jiasheng LIU, Guizhou DENG, Chunlin CHEN. Hierarchical optimization research of constrained vehicle routing based on deep reinforcement learning [J]. Systems Engineering and Electronics, 2025, 47(3): 827-841.
[5]	Xiarong CHEN, Jichao LI, Gang CHEN, Peng LIU, Jiang JIANG. Portfolio of weapon system-of-systems based on heterogeneous information networks [J]. Systems Engineering and Electronics, 2025, 47(3): 855-861.
[6]	Lei YU, Yipin LIU, Yinsheng WEI. Anti-mainlobe jamming method via blind extraction based on maximizing SINR [J]. Systems Engineering and Electronics, 2024, 46(9): 2968-2979.
[7]	Tingyu ZHANG, Ying ZENG, Nan LI, Hongzhong HUANG. Spacecraft power-signal composite network optimization algorithm based on DRL [J]. Systems Engineering and Electronics, 2024, 46(9): 3060-3069.
[8]	Yuqi XIA, Yanyan HUANG, Qia CHEN. Path planning for unmanned vehicle reconnaissance based on deep Q-network [J]. Systems Engineering and Electronics, 2024, 46(9): 3070-3081.
[9]	Zhipeng YANG, Zihao CHEN, Chang ZENG, Song LIN, Jindi MAO, Kai ZHANG. Online route planning decision-making method of aircraft in complex environment [J]. Systems Engineering and Electronics, 2024, 46(9): 3166-3175.
[10]	Hongda GUO, Jingtao LOU, Youchun XU, Peng YE, Yongle LI, Jinsheng CHEN. Event-triggered communication of multiple unmanned ground vehicles collaborative based on MADDPG [J]. Systems Engineering and Electronics, 2024, 46(7): 2525-2533.
[11]	Mengyu ZHANG, Yajie DOU, Ziyi CHEN, Jiang JIANG, Kewei YANG, Bingfeng GE. Review of deep reinforcement learning and its applications in military field [J]. Systems Engineering and Electronics, 2024, 46(4): 1297-1308.
[12]	Yanling LI, Feizhou LUO, Zhilei GE. Robust observer-based deep reinforcement learning for attitude stabilization of vertical takeoff and landing vehicle [J]. Systems Engineering and Electronics, 2024, 46(3): 1038-1047.
[13]	Jiahao ZHANG, Wei LI, Wantian WANG, Hengfeng WANG, Yaxing LI, Ji MENG. Radar interference cancellation method and performance analysis under condition of incomplete array degree of freedom [J]. Systems Engineering and Electronics, 2024, 46(11): 3639-3647.
[14]	Jingwen HAN, Yong YANG, Jing LIAN, Guoqing WU, Xuesong WANG. Identification method of corner reflector based on polarization and HRRP feature fusion for radar seeker [J]. Systems Engineering and Electronics, 2024, 46(11): 3658-3670.
[15]	Hongzhang GAO, Songhu GE, Yu GUO, Rang LIU, Jin MENG. Analysis and verification of cancellation blind area with alternate polarization array [J]. Systems Engineering and Electronics, 2024, 46(1): 51-61.

Strategy reconstruction for resilience against intelligence jamming based on implicit opponent modeling

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 31

Related Articles 15

Recommended Articles

Metrics

Comments