基于跨模态注意力与门控融合的声场景分类

doi:10.12305/j.issn.1001-506X.2025.11.03

Abstract

Abstract:

Aiming at the problems of insufficient acquisition of correlation between modes and inefficient feature fusion in acoustic scene classification task, a acoustic scene classification model based on cross-modal attention and gating fusion is proposed. This model enables bidirectional interaction between acoustic and visual modalities via a cross-modal attention module, dynamically capturing their correlation. Meanwhile, the gating fusion module is designed to dynamically adjust the weights of acoustic and visual modes, realize the adaptive fusion of features, and the residual enhancement and dual-path pooling strategy are introduced to boost the robustness of features. The proposed model and the excellent methods with the same task are evaluated across accuracy, frame rate and model parameters. The simulation results show that the overall classification effect of the proposed model outperforms other methods while maintaining high accuracy, which proves its effectiveness and practicability.

Key words: acoustic scene classification, cross-modal attention, dynamic gating, adaptive fusion

CLC Number:

Juan WEI, Huiwen ZHOU, Fangli NING. Acoustic scene classification based on cross-modal attention and gating fusion[J]. Systems Engineering and Electronics, 2025, 47(11): 3543-3550.

Figures/Tables 9

Fig.1

Fig.2

Fig.3

Table 1

Table 2

Table 3

Table 4

Table 5

Fig.4

References 30

1	MULIMANI M, VENKATESH S, KOOLAGUDI S G. Acoustic event and scene classification: a review[J]. SN Computer Science, 2024, 6 (1): 54. doi: 10.1007/s42979-024-03592-9
2	HUANG R, XIE Y, JIANG P X. Local time-frequency feature fusion using cross-attention for acoustic scene classification[J]. Symmetry, 2024, 17 (1): 49. doi: 10.3390/sym17010049
3	KHAN S, ULLAH I, ALI F, et al. Deep learning-based marine big data fusion for ocean environment monitoring: towards shape optimization and salient objects detection[J]. Frontiers in Marine Science, 2023, 9, 1094915. doi: 10.3389/fmars.2022.1094915
4	刘立芳, 杨海霞, 齐小刚. 基于线性判别分析的时频域特征提取算法[J]. 系统工程与电子技术, 2019, 41 (10): 2184- 2190.
	LIU L F, YANG H X, QI X G. Time-frequency domain feature extraction algorithm based on linear discriminant analysis[J]. Systems Engineering and Electronics, 2019, 41 (10): 2184- 2190.
5	LQBAL U, LI D L, DU Z Z, et al. Augmenting aquaculture efficiency through involutional neural networks and self-attention for oplegnathus punctatus feeding intensity classification from log mel spectrograms[J]. Animals, 2024, 14 (11): 1690. doi: 10.3390/ani14111690
6	PAUL B, PHADIKAR S, BERA S, et al. Isolated word recognition based on a hyper-tuned cross-validated CNN-BiLSTM from Mel frequency cepstral coefficients[J]. Multimedia Tools and Applications, 2025, 84, 17309- 17328.
7	HONG Z, LYU T Q, ZHAO D, et al. Improvement of pipeline leak detection method: integration of spectral entropy and sample entropy for better description of complexity features[J]. Applied Acoustics, 2025, 231, 110458. doi: 10.1016/j.apacoust.2024.110458
8	吕勤哲, 全英汇, 沙明辉, 等. 基于集成深度学习的有源干扰智能分类[J]. 系统工程与电子技术, 2022, 44 (12): 3595- 3602.
	LYU Q Z, QUAN Y H, SHA M H, et al. Intelligent classification of active jamming based on integrated deep learning[J]. Systems Engineering and Electronics, 2022, 44 (12): 3595- 3602.
9	ZHANG T, LIANG J H, DING B Y. Acoustic scene classification using deep CNN with fine-resolution feature[J]. Expert Systems with Applications, 2020, 143, 113067. doi: 10.1016/j.eswa.2019.113067
10	RYBCZAK M, KOZAKIEWICZ K. Deep machine learning of mobileNet, efficient, and Inception models[J]. Algorithms, 2024, 17 (3): 96. doi: 10.3390/a17030096
11	WANG H L, ZOU Y X, CHONG D D, et al. Environmental sound classification with parallel temporal-spectral attention[EB/OL]. [2025-04-10]. https://arxiv.org/abs/1912.06808.
12	SPOORTHY V, KOOLAGUDI S G. Bi-level acoustic scene classification using lightweight deep learning model[J]. Circuits, Systems, and Signal Processing, 2024, 43 (1): 388- 407. doi: 10.1007/s00034-023-02478-0
13	XIE J, ZHU M Y. Investigation of acoustic and visual features for acoustic scene classification[J]. Expert Systems with Applications, 2019, 126, 20- 29. doi: 10.1016/j.eswa.2019.01.085
14	XIAO B, SHI D Y, BI X L, et al. CS-CoLBP: cross-scale co-occurrence local binary pattern for image classification[J]. International Journal of Computer Vision, 2025, 133, 2327- 2344. doi: 10.1007/s11263-024-02297-z
15	SHU H M, DAWOD A Y, DONG L J. Recognition and classification of microseismic event waveforms based on histogram of oriented gradients and shallow machine learning approach[J]. Journal of Applied Geophysics, 2024, 230, 105551. doi: 10.1016/j.jappgeo.2024.105551
16	WANG M, CHEN C X, XIE Y, et al. Audio-visual scene classification using transfer learning and hybrid fusion strategy[R]. Barcelona: Detection and Classification of Acoustic Scenes and Events Challenge, 2021.
17	LIU Y, NEOPHYTOU A, SENGUPTA S, et al. Cross-modal spectrum transformation network for acoustic scene classification[C]// Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2021: 830−834.
18	常月, 侯元波, 谭奕舟, 等. 基于自注意力机制的多模态场景分类[J]. 复旦学报（自然科学版）, 2023, 62 (1): 46- 52.
	CHANG Y, HOU Y B, TAN Y Z, et al. Multi-modal scene classification based on self-attention mechanism[J]. Journal of Fudan University （Natural Science）, 2023, 62 (1): 46- 52.
19	GONG R, ZHANG Y, ZHANG Y H, et al. Demsasa: micro-video scene classification based on denoising multi-shots association self-attention[J]. Pattern Analysis and Applications, 2024, 27 (4): 155. doi: 10.1007/s10044-024-01378-6
20	黄天阳, 侯元波, 李圣辰, 等. 互编码器辅助视频的多模态场景分类[J]. 南京邮电大学学报（自然科学版）, 2023, 43 (1): 104- 110.
	HUANG T Y, HOU Y B, LI S C, et al. Multimodal scene classification for encoder-assisted videos[J]. Journal of Nanjing University of Posts and Telecommunications （Natural Science）, 2023, 43 (1): 104- 110.
21	ZHOU L G, QI X N, HU J J, et al. Feature pyramid attention network for audio-visual scene classification[J]. CAAI Transactions on Intelligence Technology, 2025, 10(2): 359−374.
22	GEMMEKE J F, ELLIS D P W, FREEDMAN D, et al. Audio set: an ontology and human-labeled dataset for audio events[C]// Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2017: 776−780.
23	CRAMER A L, WU H H, SALAMON J, et al. Look, listen, and learn more: design choices for deep audio embeddings[C]// Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2019: 3852−3856.
24	MASCARENHAS S, AGARWAL M. A comparison between VGG16, VGG19 and ResNet50 architecture frameworks for image classification[C]//Proc. of the International Conference on Disruptive Technologies for Multi-Disciplinary Research and Applications, 2021.
25	HOU Y B, TAN Y Z, CHANG Y, et al. CNN-based dual-stream network for audio-visual scene classification[R]. Barcelona: Detection and Classification of Acoustic Scenes and Events Challenge, 2021.
26	WANG S S, MESAROS A, HEITTOLA T, et al. A curated dataset of urban scenes for audio-visual scene analysis[C]// Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2021: 626−630.
27	SHAHADE A K, WALSE K H, THAKARE V M. Deep learning approach-based hybrid fine-tuned Smith algorithm with Adam optimiser for multilingual opinion mining[J]. International Journal of Computer Applications in Technology, 2023, 73 (1): 50- 65. doi: 10.1504/IJCAT.2023.134080
28	SHIM J W. Enhancing cross entropy with a linearly adaptive loss function for optimized classification performance[J]. Scientific Reports, 2024, 14 (1): 27405. doi: 10.1038/s41598-024-78858-6
29	ZHOU L G, QI X N, HU J J, et al. Attentional graph convolutional network for structure-aware audiovisual scene classification[J]. IEEE Trans. on Instrumentation and Measurement, 2023, 72, 9600515.
30	WANG Z Q, WU Y K, WANG Y F, et al. Audio-visual scene recognition using attention-based graph convolutional model[J]. Multimedia Tools and Applications, 2025, 84, 14915- 14939.

数据集	训练集	验证集	测试集
TAU Urban Audio Visual Scenes 2021	8506	2430	1215

名称	具体配置
CPU	Intel（R） Core（TM） i5-10400F CPU
GPU	Nvidia GeForce RTX 3060 GPU
框架平台	Pytorch 1.11.0
编程语言	Python 3.8
优化器	Adam
初始学习率	0.0001
训练轮数	50
批次大小	16

场景	ASC	DCASE 2021基线	特征级融合	决策级融合	声学辅助视觉	视觉辅助声学	CMA-GF
机场	79.74	66.80	80.58	76.42	76.45	71.45	84.58
公共汽车	96.08	85.90	96.04	94.12	79.47	93.24	97.20
地铁	65.15	80.40	100.00	93.94	80.30	95.25	100.00
地铁站	44.71	80.80	84.61	98.82	93.41	94.12	98.48
公园	75.25	77.20	98.02	95.32	97.30	81.19	94.06
公共广场	47.72	71.10	83.49	70.41	81.44	79.38	93.41
商场	72.22	72.60	76.44	78.61	73.33	77.78	79.54
步行街	85.71	72.70	79.86	71.73	98.57	97.14	96.71
交通街道	91.30	89.60	96.12	86.96	94.20	97.10	97.10
有轨电车	35.62	73.10	75.64	60.27	84.93	60.00	89.92
均值	69.35	77.02	87.08	82.66	85.94	84.67	93.10

方法	准确率/%	FPS/（frame/s）	模型参数量（×10⁶）
DCASE2021基线	77.02	2088.89	0.79
文献[29]的方法	91.60	953.69	28.50
文献[30]的方法	92.85	478.45	89.40
文献[21]的方法	94.10	1215.34	74.43
文献[16]的方法	95.20	326.50	110.39
CMA-GF	93.10	842.79	30.86

实验	CMAM	GFM-DG	GFM-RE	GFM-DP	准确率/%	FPS/（frame/s）	模型参数量（×10⁶）
1	−	−	−	−	80.34	905.64	29.91
2	√	−	−	−	85.66	880.92	30.30
3	−	√	−	−	82.14	894.75	29.94
4	−	√	√	−	83.72	881.54	30.47
5	−	√	√	√	85.01	872.43	30.47
6	√	√	−	−	87.68	865.01	30.33
7	√	√	√	−	90.12	853.76	30.86
8	√	√	√	√	93.10	842.79	30.86

Acoustic scene classification based on cross-modal attention and gating fusion

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 9

References 30

Related Articles 3

Recommended Articles

Metrics

Comments

[1]	Juan WEI, Dehua HE, Fangli NING. Acoustic scene classification based on adaptive multi-branch convolution [J]. Systems Engineering and Electronics, 2025, 47(10): 3148-3154.
[2]	Shufeng GONG, Weijun LONG, De BEN, Minghai PAN. Adaptive fuzzy CFAR detection fusion algorithm for netted radar [J]. Systems Engineering and Electronics, 2022, 44(1): 100-107.
[3]	Juan WEI, Zhikai DING, Fangli NING. Spectrogram extraction method for acoustic scene data based on neural network [J]. Systems Engineering and Electronics, 2021, 43(12): 3462-3469.