基于CNN-BiLSTM-MHA时空融合框架的毫米波雷达人体姿态估计

doi:10.12305/j.issn.1001-506X.2026.05.06

Abstract

Abstract:

Human pose estimation has many applications in human-computer interaction, activity recognition, and health monitoring. Traditional methods based on optical sensors are often limited by lighting conditions and privacy leakage risks, while wearable device-based technologies face issues such as cumbersome usage and discomfort during long-term wear. To address these challenges, a millimeter-wave radar human pose estimation method is proposed that leverages a spatio-temporal fusion framework combining convolutional neural network （CNN）, bidirectional long short-term memory （BiLSTM） network and multi-head attention （MHA）. High-quality point cloud data are generated using self-developed millimeter-wave radar equipment, and a sliding window mechanism is introduced to expand single-frame point clouds into multi-frame time series datas. Spatial features are extracted through CNN, and time-series modeling is performed using BiLSTM, and further optimization of global feature expression through MHA. This spatio-temporal information fusion framework, based on multi-frame point cloud datas, effectively exploits spatio-temporal features, mitigates the radar point cloud sparsity issue, and significantly enhances the accuracy and robustness of pose estimation. Experimental results show that, compared to existing methods, the proposed method successfully localizes all 25 skeletal joints, with average localization errors of 2.69 cm, 2.49 cm, and 2.98 cm along the x, y, and z axes, respectively. This provides a solution for millimeter-wave radar human pose estimation and demonstrates strong practical application potential.

Key words: millimeter-wave radar, human pose estimation, convolutional neural network（CNN）, bidirectional long short-term memory（BiLSTM） network, multi-head attention（MHA）

CLC Number:

TN 957

Yuquan LUO, Yuqiang HE, Yaxin LI, Song LIANG, Jun WANG. Millimeter-wave radar human pose estimation based on CNN-BiLSTM-MHA spatio-temporal fusion framework[J]. Systems Engineering and Electronics, 2026, 48(5): 1502-1514.

Figures/Tables 23

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Fig.13

Table 1

Table 2

Table 3

Fig.14

Table 4

Table 5

Table 6

Fig.15

Fig.16

Table 7

References 32

1	SONG Y X, ZHAO Y, YU H, et al. mmWave radar-based WPT/VMD noncontact repetitive motion counter[J]. IEEE Sensors Journal, 2023, 23 (19): 23145- 23157. doi: 10.1109/JSEN.2023.3307390
2	FUKUSHIMA T, BLAUBERGER P, RUSSOMANNO T G, et al. The potential of human pose estimation for motion capture in sports: a validation study[J]. Sports Engineering, 2024, 27 (1): 19. doi: 10.1007/s12283-024-00460-w
3	ZHOU Y J, HUANG H, YUAN S, et al. MetaFi++: WiFi-enabled transformer-based human pose estimation for metaverse avatar simulation[J]. IEEE Internet of Things Journal, 2023, 10 (16): 14128- 14136. doi: 10.1109/JIOT.2023.3262940
4	张春杰, 王冠博, 陈奇, 等. 基于纯自注意力机制的毫米波雷达手势识别[J]. 系统工程与电子技术, 2024, 46 (3): 859- 867.
	ZHANG C J, WANG G B, CHEN Q, et al. Gesture recognition based on millimeter-wave radar with pure self-attention mechanism[J]. Systems Engineering and Electronics, 2024, 46 (3): 859- 867.
5	杨德贵, 许道峰. 基于时频域特征融合的IR-UWB穿墙雷达人体行为识别方法[J]. 系统工程与电子技术, 2024, 46 (3): 849- 858.
	YANG D G, XU D F. Human behavior recognition method of IR-UWB through wall radar based on time-frequency domain feature fusion[J]. Systems Engineering and Electronics, 2024, 46 (3): 849- 858.
6	CHEN J, CHEN D Y, JIANG H X, et al. Skeleton-based 3D human pose estimation with low-resolution infrared array sensor using attention-based CNN-BiGRU[J]. International Journal of Machine Learning and Cybernetics, 2024, 15 (5): 2049- 2062. doi: 10.1007/s13042-023-02015-0
7	NANDAGOPAL S, KARTHY G, OLIVER A S, et al. Optimal deep convolutional neural network with pose estimation for human activity recognition[J]. Computer Systems Science and Engineering, 2023, 44 (2): 1719- 1733. doi: 10.32604/csse.2023.028003
8	BOUDLAL H, SERRHINI M, TAHIRI A, et al. A novel approach for simultaneous human activity recognition and pose estimation via skeleton-based leveraging WiFi CSI with YOLOv8 and mediapipe frameworks[J]. Signal, Image and Video Processing, 2024, 18 (4): 3673- 3689. doi: 10.1007/s11760-024-03031-5
9	CHOI J Y, HA E, SON M, et al. Human joint angle estimation using deep learning-based three-dimensional human pose estimation for application in a real environment[J]. Sensors, 2024, 24 (12): 3823. doi: 10.3390/s24123823
10	WU X, ZHANG H Y, KONG C X, et al. LiDAR-based 3-D human pose estimation and action recognition for medical scenes[J]. IEEE Sensors Journal, 2024, 24 (9): 15531- 15539. doi: 10.1109/JSEN.2024.3373192
11	LAN S C, YE L, ZHANG K. Applying mmWave radar sensors to vocabulary-level dynamic chinese sign language recognition for the community with deafness and hearing loss[J]. IEEE Sensors Journal, 2023, 23 (22): 27273- 27283. doi: 10.1109/JSEN.2023.3324369
12	ZHU Y A, XIAO M R, XIE Y J, et al. In-bed human pose estimation using multi-source information fusion for health monitoring in real-world scenarios[J]. Information Fusion, 2024, 105, 102209. doi: 10.1016/j.inffus.2023.102209
13	SHIAO Y, CHEN G Y, HOANG T. Three-dimensional human posture recognition by extremity angle estimation with minimal IMU sensor[J]. Sensors, 2024, 24 (13): 4306. doi: 10.3390/s24134306
14	YU S, ZHAI D H, XIA Y, et al. Synthetic depth image-based category-level object pose estimation with effective pose decoupling and shape optimization[J]. IEEE Trans. on Instrumentation and Measurement, 2024, 73, 5026718.
15	LI D, MU Q, YUAN Y L, et al. 6D pose estimation based on 3D edge binocular reprojection optimization for robotic assembly[J]. IEEE Robotics and Automation Letters, 2023, 8 (12): 8319- 8326. doi: 10.1109/LRA.2023.3327933
16	LI X, ZHANG D, LI M, et al. Accurate head pose estimation using image rectification and a lightweight convolutional neural network[J]. IEEE Trans. on Multimedia, 2023, 25, 2239- 2251. doi: 10.1109/TMM.2022.3144893
17	LIU S N, ZHAO L, YANG X, et al. Remote drowsiness detection based on the mmWave FMCW radar[J]. IEEE Sensors Journal, 2022, 22 (15): 15222- 15234. doi: 10.1109/JSEN.2022.3186486
18	SKARIA S, HENDY N, AL-HOURANI A, et al. Machine-learning methods for material identification using mmWave radar sensor[J]. IEEE Sensors Journal, 2023, 23 (2): 1471- 1478. doi: 10.1109/JSEN.2022.3227207
19	蔡嘉怡, 初萍, 庄伦涛, 等. 基于空间属性特征的毫米波雷达身体干扰识别[J]. 系统工程与电子技术, 2024, 46 (10): 3365- 3374.
	CAI J Y, CHU P, ZHUANG L T, et al. Millimeter-wave radar body interference recognition based on spatial attribute features[J]. Systems Engineering and Electronics, 2024, 46 (10): 3365- 3374.
20	YU C X, XU Z, YAN K, et al. Noninvasive human activity recognition using millimeter-wave radar[J]. IEEE Systems Journal, 2022, 16 (2): 3036- 3047. doi: 10.1109/JSYST.2022.3140546
21	HUANG Y C, LI W, DOU Z, et al. Activity recognition based on millimeter-wave radar by fusing point cloud and range-Doppler information[J]. Signals, 2022, 3 (2): 266- 283. doi: 10.3390/signals3020017
22	MOON J Y, KIM B K, KANG J, et al. Fixed point cloud normalization and none-sequential modeling for hand gesture recognition based on short-range mmWave radar sensor’s sparse time-series point cloud[J]. IEEE Sensors Journal, 2024, 24 (7): 10656- 10668. doi: 10.1109/JSEN.2024.3362473
23	WANG Z M, JIANG D, SUN B, et al. A data augmentation method for human activity recognition based on mmWave radar point cloud[J]. IEEE Sensors Letters, 2023, 7 (5): 6002404.
24	SENGUPTA A, JIN F, ZHANG R Y, et al. mm-pose: real-time human skeletal posture estimation using mmWave radars and CNNs[J]. IEEE Sensors Journal, 2020, 20 (17): 10032- 10044. doi: 10.1109/JSEN.2020.2991741
25	LI G Z, ZHANG Z, YANG H M, et al. Capturing human pose using mmWave radar[C]//Proc. of the IEEE International Conference on Pervasive Computing and Communications Workshops, 2020.
26	AN S Z, OGRAS U Y. MARS: mmWave-based assistive rehabilitation system for smart healthcare[J]. ACM Transaction on Embedded Computing Systems, 2021, 20 (5s): 72.
27	AN S Z, OGRAS U Y. Fast and scalable human pose estimation using mmWave point cloud[C]//Proc. of the 59th ACM/IEEE Design Automation Conference, 2022: 889−894.
28	SENGUPTA A, CAO S Y. MmPose-NLP: a natural language processing approach to precise skeletal pose estimation using mmWave radars[J]. IEEE Trans. on Neural Networks and Learning Systems, 2023, 34 (11): 8418- 8429. doi: 10.1109/TNNLS.2022.3151101
29	HU S T, CAO S Y, TOOSIZADEH N, et al. MmPose-FK: a forward kinematics approach to dynamic skeletal pose estimation using mmWave radars[J]. IEEE Sensors Journal, 2024, 24 (5): 6469- 6481. doi: 10.1109/JSEN.2023.3348199
30	舒月, 傅东宁, 陈展野, 等. 基于 RD-ANM 的毫米波雷达动目标超分辨 DOA 估计方法[J]. 雷达学报, 2023, 12 (5): 986- 999.
	SHU Y, FU D N, CHEN Z Y, et al. Super-resolution DOA estimation method for a moving target equipped with a millimeter-wave radar based on RD-ANM[J]. Journal of Radar, 2023, 12 (5): 986- 999.
31	LEE K M, LEE I S, SHIN H S, et al. Reconstruction of range-Doppler map corrupted by FMCW radar asynchronization[J]. Sensors, 2023, 23 (12): 5605. doi: 10.3390/s23125605
32	ZHAO L F, LIN Z X, SUN R Y, et al. A review of state-of-the-art methodologies and applications in action recognition[J]. Electronics, 2024, 13 (23): 4733. doi: 10.3390/electronics13234733

数据集	模型	x		y		z		平均值
数据集	模型	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE
文献[26]	MARS	9.25	12.26	5.15	6.92	9.63	12.91	8.01	10.70
	mmPose	8.55	12.24	5.20	6.94	8.69	12.55	7.48	10.58
	FUSE	6.70	8.87	3.74	5.02	6.95	9.33	5.79	7.74
	mmPose-NLP	4.00	6.86	2.94	4.38	4.11	7.82	3.68	6.25
	CNN-BiLSTM-MHA	3.20	5.85	2.49	3.81	3.40	6.42	3.13	5.45

数据集	模型	x		y		z		平均值
数据集	模型	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE
自建数据集	MARS	5.90	7.99	3.69	4.89	5.98	8.47	5.19	7.12
	mmPose	5.30	7.92	3.52	4.83	5.21	8.40	4.68	7.05
	FUSE	4.00	5.43	2.99	4.14	4.06	5.88	3.69	5.15
	mmPose-NLP	4.13	6.17	2.94	4.08	4.06	6.54	3.71	5.59
	CNN-BiLSTM-MHA	2.61	4.34	2.37	3.59	2.90	4.72	2.63	4.22

点云特征	x	y	z	平均值
xyz	3.12	2.94	3.33	3.13
xyz+v	2.61	2.37	2.90	2.63
xyz+I	4.08	3.23	4.34	3.88
xyz+v+I	4.20	3.24	4.20	3.88

CNN结构	x	y	z	平均值
BN× MP×	2.94	2.86	3.10	2.97
BN√ MP×	2.61	2.37	2.90	2.63
BN× MP√	3.28	2.88	3.21	3.12
BN√ MP√	4.64	4.82	4.44	4.63

模块	x	y	z	平均
BiLSTM×	5.83	4.78	6.20	5.60
LSTM	3.09	2.74	3.30	3.04
MLP	3.90	3.28	4.06	3.75
Transformer	3.28	2.77	3.46	3.17
BiLSTM	2.61	2.37	2.90	2.63

Millimeter-wave radar human pose estimation based on CNN-BiLSTM-MHA spatio-temporal fusion framework

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 23

References 32

Related Articles 9

Recommended Articles

Metrics

Comments

模型配置	x	y	z	平均
无MHA	2.69	2.49	2.98	2.72
有MHA	2.61	2.37	2.90	2.63

实验场景	x	y	z	平均值
有遮挡	3.14	2.89	3.66	3.23
无遮挡	2.88	2.58	3.28	2.91

[1]	Qi WANG, Ziyao WANG, Junfeng ZHENG. Simulation model for radar system considering multi-source noise and signal transmission [J]. Systems Engineering and Electronics, 2025, 47(3): 768-778.
[2]	Ruixiang FANG, Xiaojin SHI, Yunhua ZHANG. MIAA based three-dimensional millimeter-wave imaging algorithm with sparse trajectory scanning [J]. Systems Engineering and Electronics, 2025, 47(11): 3612-3625.
[3]	Chunjie ZHANG, Guanbo WANG, Qi CHEN, Zhi'an DENG. Gesture recognition based on millimeter-wave radar with pure self-attention mechanism [J]. Systems Engineering and Electronics, 2024, 46(3): 859-867.
[4]	Jiayi CAI, Ping CHU, Luntao ZHUANG, Zhaocheng YANG. Millimeter-wave radar body interference recognition based on spatial attribute features [J]. Systems Engineering and Electronics, 2024, 46(10): 3365-3374.
[5]	Jingyue ZHENG, Peilun WU, Jiahui CHEN, Shisheng GUO, Guolong CUI. Multipath false target analysis and elimination method for automotive millimeter wave radar [J]. Systems Engineering and Electronics, 2024, 46(1): 88-96.
[6]	Yangliang WAN, Xingdong LIANG, Yanlei LI. Interference suppression method based on azimuth frequency domain filtering for airport millimeter-wave radar [J]. Systems Engineering and Electronics, 2022, 44(11): 3357-3363.
[7]	Ruochen ZHAO, Jingdong WANG, Siyu LIN, Dongze GU. Small building detection algorithm based on convolutional neural network [J]. Systems Engineering and Electronics, 2021, 43(11): 3098-3106.
[8]	Juan SU, Long YANG, Hua HUANG, Guodong JIN. Improved SSD algorithm for small-sized SAR ship detection [J]. Systems Engineering and Electronics, 2020, 42(5): 1026-1034.
[9]	TAN Xiao-heng, ZHOU Shuai， HUANG Zhen-lin. New ranging method for 24 GHz LFMCW radar based on wavelet packet [J]. Journal of Systems Engineering and Electronics, 2013, 35(3): 522-526.