二阶逐层特征融合网络的图像超分辨重建

doi:10.12305/j.issn.1001-506X.2024.02.03

摘要/Abstract

摘要：

针对一些超分辨网络忽略了对网络各层次特征的复用以及融合的问题, 构建了具有较强特征复用和融合能力的二阶逐层特征融合超分辨网络, 以获得具有高分辨率、高保真度的重建图像。网络的核心是逐层特征融合模块, 该模块通过特征融合操作增强特征的重用。此外, 还提出了二阶特征融合机制, 该机制在网络的局部和全局层次上采用逐层特征融合方法进行特征融合。实验结果表明该网络的重建图像在线条和轮廓上更清晰, 并且在峰值信噪比和结构相似度上也取得了更好的结果。例如当缩放尺度因子为2时, 各测试集上的峰值信噪比/结构相似度依次为38.20 dB/0.961 2、33.81 dB/0.919 5、32.28 dB/0.901 0、32.65 dB/0.932 4、39.11 dB/0.977 9, 相比其他模型有一定提升, 从客观标准和主观角度证明了二阶逐层特征融合超分辨网络具有一定的优越性。

关键词: 超分辨重建, 卷积神经网络, 特征融合, 二阶特征融合机制

Abstract:

Some super-resolution networks ignore the reuse of the features of different levels, and there is no fusing of the features. In order to solve those problems, a second-order progressive feature fusion super-resolution network with strong feature reuse and fuse ability is constructed to realize the reconstructed image with high resolution and high fidelity. The core of the network is progressive feature fusion block. Progressive feature fusion block enhances the reuse of features through feature fusion operation. In addition, a second-order feature fusion mechanism is proposed, which adopts progressive feature fusion method for feature fusion at the local and global levels of the network. The experimental results show that the reconstructed image of the network is clearer than that of other networks on line and contour, and better results are obtained in peak signal to noise ratio (SNR) and structural similarity. For example, when the scaling factor is 2, the peak SNR/structure similarity on each test set is 38.20 dB/0.961 2, 33.81 dB/0.919 5, 32.28 dB/0.901 0, 32.65 dB/0.932 4, and 39.11 dB/0.977 9 respectively, which proves that the proposed model acheives improvent compared to other models. The advantages of the second-order progressive feature fusion super-resolution network is proven from the objective standard and subjective point of view.

Key words: super-resolution reconstruction, convolutional neural network (CNN), feature fusion, second-order feature fusion mechanism

中图分类号:

TN911.73

于蕾, 邓秋月, 郑丽颖, 吴昊宇. 二阶逐层特征融合网络的图像超分辨重建[J]. 系统工程与电子技术, 2024, 46(2): 391-400.

Lei YU, Qiuyue DENG, Liying ZHENG, Haoyu WU. Second-order progressive feature fusion network for image super-resolution reconstruction[J]. Systems Engineering and Electronics, 2024, 46(2): 391-400.

图/表 19

图1

图2

图3

图4

图5

表1

表2

图6

表3

表4

表5

表6

表7

图7

图8

图9

图10

图11

图12

参考文献 25

1	LIU A R , LIU Y H , GU J J , et al. Blind image super-resolution: a survey and beyond[J]. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2022, 45 (5): 5461- 5480.
2	TIAN C W, ZHANG X Y, LIN J C W, et al. Generative adversarial networks for image super-resolution: a survey[EB/OL]. [2023-04-09]. https://arxiv.org/abs/2204.13620.
3	毕笃彦, 王世平, 刘坤, 等. 基于并行映射卷积网络的超分辨率重建算法[J]. 系统工程与电子技术, 2018, 40 (8): 1873- 1880.
	BI D Y , WANG S P , LIU K , et al. Super-resolution algorithm based on parallel mapping convolution network[J]. Systems Engi- neering and Electronics, 2018, 40 (8): 1873- 1880.
4	TAO H J , LU X B . Contour-based smoky vehicle detection from surveillance video for alarm systems[J]. Signal, Image and Video Processing, 2019, 13 (2): 217- 225. doi: 10.1007/s11760-018-1348-z
5	王钢, 周若飞, 邹昳琨. 基于压缩感知理论的图像优化技术[J]. 电子与信息学报, 2020, 42 (1): 222- 233.
	WANG G , ZHOU R F , ZOU Y K . Research on image optimization technology based on compressed sensing[J]. Journal of Electronics &Information Technology, 2020, 42 (1): 222- 233.
6	陈书贞, 曹世鹏, 崔美玥, 等. 基于深度多级小波变换的图像盲去模糊算法[J]. 电子与信息学报, 2021, 43 (1): 154- 161.
	CHEN S Z , CAO S P , CUI M Y , et al. Image blind deblurring algorithm based on deep multi-level wavelet transform[J]. Journal of Electronics &Information Technology, 2021, 43 (1): 154- 161.
7	CHEN H G , HE X H , QING L B , et al. Real-world single image super resolution: a brief review[J]. Information Fusion, 2022, 79, 124- 145. doi: 10.1016/j.inffus.2021.09.005
8	BASHIR S M A , WANG Y , KHAN M , et al. A comprehensive review of deep learning based single image super-resolution[J]. PeerJ Computer Science, 2021, 7, e621.
9	ZHU X B , LI Z Z , LOU J G , et al. Video super-resolution based on a spatio-temporal matching network[J]. Pattern Re-cognition, 2021, 110 (2): 107619.
10	DONG C, LOY C C, HE K M, et al. Learning a deep convolutional network for image super-resolution[C]//Proc. of the 13th European Conference on Computer Vision,, 2014: 184-199.
11	DONG C, LOY C C, TANG X O. Accelerating the super-resolution convolutional neural network[C]//Proc. of the 14th European Conference on Computer Vision, 2016: 391-407.
12	SHI W Z, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1874-1883.
13	KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1646-1654.
14	AHN N, KANG B, SOHN K A. Fast, accurate, and lightweight super-resolution with cascading residual network[C]//Proc. of the 15th European Conference on Computer Vision, 2018: 256-272.
15	TAI Y, YANG J, LIU X M. Image super-resolution via deep recursive residual network[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 2790-2798.
16	TAI Y, YANG J, LIU X M, et al. MemNet: a persistent memory network for image restoration[C]//Proc. of the IEEE International Conference on Computer Vision, 2017: 4549-4557.
17	程德强, 郭昕, 陈亮亮, 等. 多通道递归残差网络的图像超分辨率重建[J]. 中国图象图形学报, 2021, 26 (3): 605- 618.
	CHENG D Q , GUO X , CHEN L L , et al. Image super-resolution reconstruction from multi-channel recursive residual network[J]. Journal of Image and Graphics, 2021, 26 (3): 605- 618.
18	QIN J H , HUANG Y J , WEN W S . Multi-scale feature fusion residual network for single image super-resolution[J]. Neurocomputing, 2020, 379, 334- 342.
19	LI J C, FANG F M, MEI K F, et al. Multi-scale residual network for image super-resolution[C]//Proc. of the 15th European Conference on Computer Vision, 2018: 517-532.
20	MUQEET A , IQBAL M T B , BAE S H . Hybrid residual attention network for single image super resolution[J]. IEEE Access, 2019, 137020- 137029.
21	KEYS R . Cubic convolution interpolation for digital image processing[J]. IEEE Trans. on Acoustics, Speech, and Signal Processing, 1981, 29 (6): 1153- 1160.
22	KIM J, LEE J K, LEE K M. Deeply-recursive convolutional network for image super-resolution[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1637-1645.
23	LAI W S, HUANG J B, AHUJA N, et al. Deep Laplacian pyramid networks for fast and accurate super-resolution[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 5838-5843.
24	HUI Z, WANG X M, GAO X B. Fast and accurate single image super-resolution via information distillation network[C]//Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 723-731.
25	ZHANG K, ZUO W M, ZHANG L. Learning a single convolutional super-resolution network for multiple degradations[C]//Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 3262-3271.

特征融合方法	PSNR/SSIM
特征融合方法	Set5	Set14	BSD100	Urban100	Manga109
HFF	38.17/0.961 1	33.78/0.919 2	32.25/0.900 5	32.48/0.931 0	38.95/0.977 8
BFF	38.16/0.961 0	33.78/0.918 7	32.25/0.900 4	32.47/0.930 9	38.96/0.977 8
IFF	38.16/0.961 0	33.81/0.918 9	32.24/0.900 2	32.49/0.931 1	38.95/0.977 7
PFF	38.17/0.961 1	33.78/0.919 3	32.26/0.900 5	32.52/0.931 4	39.07/0.977 8

模块	PSNR/SSIM
模块	Set5	Set14	BSD100	Urban100	Manga109
BRB	38.14/0.960 9	33.75/0.918 9	32.24/0.900 4	32.39/0.930 2	38.94/0.977 7
PFFB	38.17/0.961 1	33.78/0.919 3	32.26/0.900 5	32.52/0.931 4	39.07/0.977 8

二阶特征融合机制	PSNR/SSIM
二阶特征融合机制	Set5	Set14	BSD100	Urban100	Manga109
无	38.16/0.960 9	33.77/0.918 9	32.25/0.900 5	32.47/0.930 6	39.03/0.977 7
有	38.17/0.961 1	33.78/0.919 3	32.26/0.900 5	32.52/0.931 4	39.07/0.977 8

模块数量	PSNR/SSIM
模块数量	Set5	Set14	BSD100	Urban100	Manga109
n=6	38.17/0.961 1	33.78/0.919 3	32.26/0.900 5	32.52/0.931 4	39.07/0.977 8
n=8	38.20↑/0.961 2↑	33.81↑/0.919 5↑	32.28↑/0.901 0↑	32.65↑/0.932 4↑	39.11↑/0.977 9↑
n=10	38.21↑/0.961 2	33.81/0.919 2↓	32.29↑/0.900 9↓	32.66↑/0.932 5↑	39.08↓/0.977 9

模型	PSNR/SSIM
模型	Set5	Set14	BSD100	Urban100	Manga109
Bicubic	33.66/0.929 9	30.24/0.868 8	29.56/0.843 1	26.88/0.840 3	30.80/0.933 9
SRCNN	36.66/0.954 2	32.45/0.906 7	31.36/0.887 9	29.50/0.894 6	35.60/0.966 3
FSRCNN	37.00/0.955 8	32.63/0.908 8	31.53/0.892 0	29.88/0.902 0	36.67/0.971 0
VDSR	37.53/0.958 7	33.03/0.912 4	31.90/0.896 0	30.76/0.914 0	37.22/0.975 0
DRCN	37.63/0.958 8	33.04/0.911 8	31.85/0.894 2	30.75/0.913 3	37.55/0.973 2
LapSRN	37.52/0.959 1	32.99/0.912 4	31.80/0.895 2	30.41/0.910 3	37.27/0.974 0
IDN	37.83/0.960 0	33.30/0.914 8	32.08/0.898 5	31.27/0.919 6	38.01/0.974 9
CARN	37.76/0.959 0	33.52/0.916 6	32.09/0.897 8	31.92/0.925 6	38.36/0.976 5
MCSR	38.03/0.960 0	33.58/0.917 0	32.18/0.899 0	31.94/0.926 0	-/-
SPFFSR	38.20/0.961 2	33.81/0.919 5	32.28/0.901 0	32.65/0.932 4	39.11/0.977 9