基于纯自注意力机制的毫米波雷达手势识别

doi:10.12305/j.issn.1001-506X.2024.03.11

Abstract

Abstract:

In the context of building intelligent control and the internet of everything, remote control of devices through hand gestures for human-computer interaction has gradually become a research hotspot. For this, a gesture recognition method based on pure self-attention mechanism model with millimeter-wave radar as sensor is proposed. Firstly, the time sequence echo data of 13 kinds of gestures from the front-view direction is collected. Then, three-dimension fast Fourier transform (3D-FFT), moving target indication (MTI) and constant false alarm rate (CFAR) detector operations are carried out on the data and fixed type Feature extraction. These features are introduced into radar feature transformer (RFT) based network. Finally, based on the measured data, the steps of data feature extraction, network training, gesture recognition and so on are completed. The experimental results show that the accuracy rate of the proposed method in the test dataset is 95.38%. Moreover, it has the characteristics of short metwork training time, low model complexity and good generalization, which provides a new research idea for the existing research.

Key words: millimeter-wave radar, gesture recognition, self-attention mechanism, noise suppression

CLC Number:

TN958.94

Chunjie ZHANG, Guanbo WANG, Qi CHEN, Zhi'an DENG. Gesture recognition based on millimeter-wave radar with pure self-attention mechanism[J]. Systems Engineering and Electronics, 2024, 46(3): 859-867.

Figures/Tables 19

Fig.1

Table 1

Fig.2

Table 2

Fig.3

Table 3

Fig.4

Fig.5

Fig.6

Table 4

Fig.7

Fig.8

Fig.9

Table 5

Fig.10

Fig.11

Fig.12

Fig.13

Table 6

References 32

1	王俊, 郑彤, 雷鹏, 等. 基于卷积神经网络的手势动作雷达识别方法[J]. 北京航空航天大学学报, 2018, 44 (6): 1117- 1123.
	WANG J , ZHENG T , LEI P , et al. Hand gesture recognition method by radar based on convolutional neural network[J]. Journal of Beijing University of Aeronautics and Astronautics, 2018, 44 (6): 1117- 1123.
2	王勇, 王沙沙, 田增山, 等. 基于FMCW雷达的双流融合神经网络手势识别方法[J]. 电子学报, 2019, 47 (7): 1408- 1415. doi: 10.3969/j.issn.0372-2112.2019.07.003
	WANG Y , WANG S S , TIAN Z S , et al. Two-stream fusion neural network approach for hand gesture recognition based on FMCW radar[J]. Acta Electronica Sinica, 2019, 47 (7): 1408- 1415. doi: 10.3969/j.issn.0372-2112.2019.07.003
3	夏朝阳, 周成龙, 介钧誉, 等. 基于多通道调频连续波毫米波雷达的微动手势识别[J]. 电子与信息学报, 2020, 42 (1): 164- 172.
	XIA C Y , ZHOU C L , JIE J Y , et al. Micromotion gesture recognition based on multi-channel frequency modulated continuous wave millimeter wave radar[J]. Journal of Electronics & Information Technology, 2020, 42 (1): 164- 172.
4	靳标, 彭宇, 邝晓飞, 等. 基于串联式一维神经网络的毫米波雷达动态手势识别方法[J]. 电子与信息学报, 2021, 43 (9): 2743- 2750.
	JIN B , PENG Y , KUANG X F , et al. Dynamic gesture recognition method based on millimeter-wave radar by one-dimensional series neural network[J]. Journal of Electronics & Information Technology, 2021, 43 (9): 2743- 2750.
5	冯翔, 刘涛, 崔文卿, 等. 基于双视角时序特征融合的毫米波雷达手势数字识别研究[J]. 电子与信息学报, 2023, 45 (6): 2134- 2143.
	FENG X , LIU T , CUI W Q , et al. Hand-writing number recognition based on millimeter-wave radar with dual-view feature fusion network[J]. Journal of Electronics & Information Technology, 2023, 45 (6): 2134- 2143.
6	TOHIDI E , RADMARD M , MAJD M N , et al. Compressive sensing MTI processing in distributed MIMO radars[J]. IET Signal Processing, 2018, 12 (3): 327- 334. doi: 10.1049/iet-spr.2016.0597
7	GARCIA F D A , RODRIGUEZ A C F , F-RAIDENRAICH G , et al. CA-CFAR detection performance in homogeneous Weibull clutter[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 16 (6): 887- 891. doi: 10.1109/LGRS.2018.2885451
8	QIU X P , SUN T X , XU Y G , et al. Pretrained models for natural language proces-sing: a survey[J]. Science China Technological Sciences, 2020, 63, 1872- 1897. doi: 10.1007/s11431-020-1647-3
9	ALVESPINTO A , DEMUS C , SPRANGERM , et al. Iterative named entity recognition with conditional random fields[J]. Ap-plied Sciences, 2022, 12 (1): 330.
10	SAWAI R , PAIK I , KUWANA A . Sentence augmentation for language translation using GPT-2[J]. Electronics, 2021, 10 (24): 3082. doi: 10.3390/electronics10243082
11	SHOEIBI N , SHOEIBI N , HERNÁNDEZG , et al. AI-crime hunter: an AI mixture of experts for crime discovery on twitter[J]. Electronics, 2021, 10 (24): 3081. doi: 10.3390/electronics10243081
12	KIM G , CHOI L , LI Q L , et al. A CNN-based advertisement recommendation through real-time user face recognition[J]. Applied Sciences, 2021, 11 (20): 9705. doi: 10.3390/app11209705
13	ARSHAGHI A , ASHOURIN M , GHABE-LI L . Detection and classification of potato diseases potato using a new convolution neural network architecture[J]. Traitement du Signal: Signal Image Parole, 2021, 38 (6): 1783- 1791. doi: 10.18280/ts.380622
14	CHOW B H Y , REYES-ALDASORO C C . Automatic gemstone classification using Computer Vision[J]. Minerals, 2021, 12 (1): 60.
15	FALTINGS U , BETTINGER T , BARTH S , et al. Impact on inference model performance for ml tasks using real-life training data and synthetic training data from GANs[J]. Information, 2021, 13 (1): 9. doi: 10.3390/info13010009
16	HUANG J X , LIU Q T , ZHENG Y X , et al. Chinese comma disambiguation in math word problems using SMOTE and random forests[J]. AI, 2021, 2 (4): 738- 755. doi: 10.3390/ai2040044
17	BAO T , REN N , LUO R , et al. A BERT-based hybrid short text classification model incorporating CNN and attention-based BiGRU[J]. Journal of Organizational and End User Computing, 2021, 33 (4): 1485- 1505.
18	GE S G , RUM S N M . Key points' location in infrared images of the human body based on MSCF-ResNet[J]. Future Internet, 2021, 14 (1): 15. doi: 10.3390/fi14010015
19	CAMPS O , AL-CHAWA M M , STAVRINIDES S G , et al. Stochastic computing emulation of memristor cellular nonlinear networks[J]. Micromachines, 2021, 13 (1): 67. doi: 10.3390/mi13010067
20	LI H W , MAO H Y , WANG J Z . Part-of-speech tagging with rulebased data preprocessing and transformer[J]. Electronics, 2022, 11 (1): 56.
21	ALTURAYEIF N , LUQMAN H . Fine-grained sentiment analysis of arabic COVID-19 tweets using BERT-based transfor-mers and dynamically weighted loss function[J]. Applied Sciences, 2021, 11 (22): 10694. doi: 10.3390/app112210694
22	BOMBIERI M , ROSPOCHER M , DALL'ALBA D , et al. Automatic detection of procedural knowledge in robotic-assisted surgical texts[J]. International Journal of Computer Assisted Radiology and Surgery, 2021, 16 (8): 1287- 1295. doi: 10.1007/s11548-021-02370-9
23	YANG T J , BAI X H , CUI X J , et al. TransDIR: deformable imaging registration network based on transformer to improve the feature extraction ability[J]. Medical physics, 2021, 49 (2): 952- 965.
24	PANBOONYUEN T , THONGBAI S , WO-NGWEERANIMIT W , et al. Object detection of road assets using transformer-based YOLOX with feature pyramid decoder on thai highway panorama[J]. Information, 2021, 13 (1): 5. doi: 10.3390/info13010005
25	ILIADIS L A , NIKOLAIDIS S , SARIGI-ANNIDIS P , et al. Artwork style recognition using vision transformers and MLPMixer[J]. Technologies, 2021, 10 (1): 2. doi: 10.3390/technologies10010002
26	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[EB/OL]. [2022-11-09]. https://arxiv.org/abs/1706.03762.
27	SUN H , LIU J Q , CHAI S R , et al. Multimodal adaptive fusion transformer network for the estimation of depression level[J]. Sensors, 2021, 21 (14): 4764. doi: 10.3390/s21144764
28	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognitionat scale[EB/OL]. [2022-11-09]. https://arxiv.org/abs/2010.11929.
29	LIU Z, LIN Y T, CAO Y, et al. Swin transformer: hierarchical vision transformerusing shifted windows[C]//Proc. of the IEEE/CVF International Conference on Computer Vision, 2021: 9992-10002.
30	XU X K , FENG Z J , CAO C Q , et al. An improved swin transformer-based model for remote sensing object detectionand instance segmentation[J]. Remote Sensing, 2021, 13 (23): 4779. doi: 10.3390/rs13234779
31	LEE J W , LEE S J , CHO W K , et al. Vision transformer-based tailing detection in videos[J]. Applied Sciences, 2021, 11 (24): 11591. doi: 10.3390/app112411591
32	DEVLIN J, CHANG M W, LEE K, et al. Bert: pretraining of deep bidirectional transformers for language understanding[EB/OL]. [2022-11-09]. https://arxiv.org/abs/1810.04805.

手势特征提取方法	手势种类	准确率/%	所需操作	模块说明
文献[1]方法	4	87.8	2D-FFT	-
			卷积层	复杂度O(M²·K²·Cin·Cout)
			池化层(Avg Pool)	区域元素求平均
文献[2]方法	10	92.06	2D-FFT	-
			mat矩阵转JPG图像(2组)	仿真软件内置函数
			MUSIC算法(含谱峰搜索)	-
			卷积层	复杂度O(M²·K²·Cin·Cout)
			池化层	区域元素求平均
			特征融合(数据拼接)	-
文献[4]方法	5	96.01	卷积层	复杂度O(M²·K²·Cin·Cout)
			平均池化层	区域元素求平均
			最大池化层	区域元素求最大值
			特征融合(数据拼接)	-
本文方法	13	95.38	3D-FFT(含谱峰搜索)	对2D-FFT结果在天线维做FFT
			MTI	-
			CA-CFAR	-
			特征融合(数据拼接)	-

手势特征	单位
目标距离	m
速度	m/s
水平角度	rad
竖直角度	rad
水平角度随速度变化	(rad·s)/m
竖直角度随时间变化	(rad·s)/m

层类型	每层复杂度	顺序计算步骤	两数据点最大计算步长
自注意力层	O(n²·d)	O(1)	O(1)
循环层	O(n·d²)	O(n)	O(n)
卷积层	O(k·n·d²)	O(1)	O(log_kn)

参数	数值
发送天线/个	1
接收天线/个	4
起始频/GHz	60
带宽/GHz	4
调频斜率/(MHz/μs)	85
采样/ksps	2 457
帧周期/ms	100
采样点/个	96
Chrip数/个	32
帧数/个	16

类型	输入尺寸	输出尺寸	说明
添加分类向量	(1, 6, 16)	(1, 6, 17)	-
添加位置编码	(1, 6, 17)	(1, 17, 6)	编码可学习
LayerNorm1	(1, 17, 6)	(1, 17, 6)	Epsilon=1e-6
注意力机制模块	(1, 17, 6)	(1, 17, 6)	缩放点积的多头注意力
LayerNorm2	(1, 17, 6)	(1, 17, 6)	Epsilon=1e-6
多层感知机模块	(1, 17, 6)	(1, 17, 6)	Activation=Gelu
提取分类向量	(1, 17, 6)	(1, 6)	-
分类Dense层	(1, 6)	(1, 13)	Activation=Softmax

Gesture recognition based on millimeter-wave radar with pure self-attention mechanism

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 19

References 32

Related Articles 4

Recommended Articles

Metrics

Comments

网络结构	手势类别数	准确率/%
文献[1]	4	87.80
文献[2]	10	92.06
文献[3]	7	87.71
文献[4]	5	96.01
文献[5]	10	95.00
本文RFT模型	13	95.38

[1]	Jingyue ZHENG, Peilun WU, Jiahui CHEN, Shisheng GUO, Guolong CUI. Multipath false target analysis and elimination method for automotive millimeter wave radar [J]. Systems Engineering and Electronics, 2024, 46(1): 88-96.
[2]	Qingyuan ZHAO, Zhiqiang ZHAO, Chunmao YE, Yaobing LU. Micro-motion fusion recognition of double band early warning radar based on self-attention mechanism [J]. Systems Engineering and Electronics, 2023, 45(3): 708-716.
[3]	Yangliang WAN, Xingdong LIANG, Yanlei LI. Interference suppression method based on azimuth frequency domain filtering for airport millimeter-wave radar [J]. Systems Engineering and Electronics, 2022, 44(11): 3357-3363.
[4]	TAN Xiao-heng, ZHOU Shuai， HUANG Zhen-lin. New ranging method for 24 GHz LFMCW radar based on wavelet packet [J]. Journal of Systems Engineering and Electronics, 2013, 35(3): 522-526.