融合深度图像先验的全变差图像着色算法

doi:10.12305/j.issn.1001-506X.2022.02.04

Abstract

Abstract:

In this paper, we propose a total variation (TV) model with depth image priors for image colorization. Under the plug-and-play (PnP) framework, we design the numerical algorithm to solve the model by incorporating the alternating direction method of multipliers (ADMM), and give the convergence result of the algorithm. The experimental results show that the model can effectively integrate the edge capture function of coupled TV and the detail capture function of convolutional neural network (CNN), and also can achieve a large scale effective coloring for structural images and multi textures detailed images.

Key words: image coloring, coupled total variation (TV), convolutional neural network (CNN), plug-and-play (PnP) framework, alternating direction method of multipliers (ADMM)

CLC Number:

TP751

Xi ZHANG, Zhengmeng JIN, Yaqin JIANG. Total variation algorithm with depth image priors for image colorization[J]. Systems Engineering and Electronics, 2022, 44(2): 385-393.

Figures/Tables 12

Fig.1

Table 1

Fig.2

Fig.3

Table 2

Table 3

Fig.4

Fig.5

Fig.6

Table 4

Table 5

Table 6

References 31

1	PAUL S , BHATTACHARYA S , GUPTA S . Spatiotemporal colorization of video using 3D steerable pyramids[J]. IEEE Trans.on Circuits and Systems for Video Technology, 2017, 27 (8): 1605- 1619. doi: 10.1109/TCSVT.2016.2539539
2	汪冬宇, 汪磊. 基于MRI医学图像的脑肿瘤区域着色方法[J]. 养生保健指南, 2018, (5): 311. doi: 10.3969/j.issn.1006-6845.2018.05.304
	WANG D Y , WANG L . Brain tumor region coloring method based on MRI medical image[J]. Health Guide, 2018, (5): 311. doi: 10.3969/j.issn.1006-6845.2018.05.304
3	CHENG Z Z, YANG Q X, SHENG B. Deep colorization[C]// Proc. of the IEEE International Conference on Computer Vision, 2015: 415-423.
4	DESHPANDE A, ROCK J, FORSYTH D. Learning large-scale automatic image colorization[C]//Proc. of the IEEE International Conference on Computer Vision, 2015: 567-575.
5	LARSSON G, MAIRE M, SHAKHNAROVICH G. Learning representations for automatic colorization[C]//Proc. of the Computer Vision-ECCV Springer International Publishing, 2016: 577-593.
6	ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 1125-1137.
7	ⅡZUKA S , SIMO -SERRA E , ISHIKAWA H . Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification[J]. ACM Trans.on Graphics, 2016, 35 (4): 110.
8	ZHANG R , ZHU J Y , ISOLA P , et al. Real-time user-guided image colorization with learned deep priors[J]. ACM Trans.on Graphics, 2017, 36 (4): 119.
9	徐中辉, 吕维帅. 基于卷积神经网络的图像着色[J]. 电子技术应用, 2018, 44 (10): 19- 22.
	XU Z H , LYU W S . Image coloring based on convolution neural network[J]. Application of Electronic Technique, 2018, 44 (10): 19- 22.
10	SAPIRO G. Inpainting the colors[C]//Proc. of the IEEE International Conference on Image Processing, 2005: 698-701.
11	YATZIV L , SAPIRO G . Fast image and video colorization using chrominance blending[J]. IEEE Trans.on Image Processing, 2006, 15 (5): 1120- 1129. doi: 10.1109/TIP.2005.864231
12	滕升华, 谌安军, 邹谋炎. 一种基于拉普拉斯方程的图像彩色化方法[J]. 中国图象图形学报, 2006, 11 (4): 545- 548. doi: 10.3969/j.issn.1006-8961.2006.04.017
	TENG S H , CHEN A J , ZOU M Y . Colorization using laplace equation[J]. Journal of Image and Graphics, 2006, 11 (4): 545- 548. doi: 10.3969/j.issn.1006-8961.2006.04.017
13	KANG S H , MARCH R . Variational models for image colorization via chromaticity and brightness decomposition[J]. IEEE Trans.on Image Processing, 2007, 16 (9): 2251- 2261. doi: 10.1109/TIP.2007.903257
14	CASELLES V , FACCIOLO G , MEINHARDT E . Anisotropic Cheeger sets and applications[J]. SIAM Journal on Imaging Sciences, 2009, 2 (4): 1211- 1254. doi: 10.1137/08073696X
15	LI F , BAO Z , LIU R H , et al. Fast image inpainting and colorization by Chambolle's dual method[J]. Journal of Visual Communication and Image Representation, 2011, 22 (6): 529- 542. doi: 10.1016/j.jvcir.2011.06.006
16	金正猛, 周晨. 基于耦合全变差的快速图像着色算法[J]. 电子学报, 2016, 44 (10): 2364- 2369. doi: 10.3969/j.issn.0372-2112.2016.10.012
	JIN Z M , ZHOU C . A fast coupled total variation algorithm for image colorization[J]. Acta Electronica Sinica, 2016, 44 (10): 2364- 2369. doi: 10.3969/j.issn.0372-2112.2016.10.012
17	JIN Z M , ZHOU C , NG M K . A coupled total variation model with curvature driven for image colorization[J]. Inverse Problems and Imaging, 2017, 10 (4): 1037- 1055.
18	MIN L H , LI Z H , JIN Z M , et al. Color edge preserving image colorization with a coupled natural vectorial total variation[J]. Computer Vision and Image Understanding, 2020, 196, 102981. doi: 10.1016/j.cviu.2020.102981
19	ZHAO X L , XU W H , JIANG T X , et al. Deep plug-and-play prior for low-rank tensor completion[J]. Neurocomputing, 2020, 400, 137- 149. doi: 10.1016/j.neucom.2020.03.018
20	SREEHARI S , VENKATAKRISHNAN S V , WOHLBERG B , et al. Plug-and-play priors for bright field electron tomography and sparse interpolation[J]. IEEE Trans.on Computational Imaging, 2016, 2 (4): 408- 423.
21	VENKATAKRISHNAN S V, BOUMAN C A, WOHLBERG B. Plug-and-play priors for model based reconstruction[C]//Proc. of the IEEE Global Conference on Signal and Information Processing, 2013: 945-948.
22	CHAN S H , WANG X , ELGENDY O A . Plug-and-play ADMM for image restoration: fixed-point convergence and applications[J]. IEEE Trans.on Computer Imaging, 2006, 3 (1): 84- 98.
23	KOJU R , JOSHI S R . Comparative analysis of color image watermarking technique in RGB, YUV, and YCbCr color channels[J]. Nepal Journal of Science and Technology, 2015, 15 (2): 133- 140. doi: 10.3126/njst.v15i2.12130
24	CHELALI F Z, CHERABIT N, DJERADI A. Face recognition system using skin detection in RGB and YCbCr color space[C]// Proc. of the World Symposium on Web Application & Networking, 2015.
25	SAPIRO G . Geometric partial differential equations and image analysis[M]. Cambridge: Cambridge University Press, 2006.
26	ZHANG K , ZUO W M , ZHANG L . FFDNet: toward a fast and flexible solution for CNN based image denoising[J]. IEEE Trans.on Image Processing, 2018, 27 (9): 4608- 4622. doi: 10.1109/TIP.2018.2839891
27	GABAY D , MERCIER B . A dual algorithm for the solution of nonlinear variational problems via finite-element approximations[J]. Computer and Mathematics with Applications, 1976, 2 (1): 17- 40. doi: 10.1016/0898-1221(76)90003-1
28	BOYD S , PARIKH N , CHU E , et al. Distributed optimization and statistical learning via the alternating direction method of multip-liers[J]. Foundations and Trends in Machine Learning, 2010, 3 (1): 1- 122. doi: 10.1561/2200000016
29	AMIR K , ORLY Y P . Quaternion structural similarity: a new quality index for color images[J]. IEEE Trans.on Image Processing, 2012, 21 (4): 1526- 1536. doi: 10.1109/TIP.2011.2181522
30	CHAN S H , WANG X , ELGENDY O A . Plug-and-play ADMM for image restoration: fixed-point convergence and applications[J]. IEEE Trans.on Computer Imaging, 2016, 3 (1): 84- 98.
31	RYU E, LIU J, WANG S, et al. Plug-and-play methods provably converge with properly trained denoisers[C]//Proc. of the 36th International Conference on Machine Learning, 2019: 5546-5557.

参数	图像1	图像2	图像3	图像4	图像5	图像6	图像7	图像8	图像9	图像10
λ₁	0.1	0.5	0.1	0.5	0.5	0.2	0.8	1	0.5	1
λ₂	120	200	120	50	80	80	100	150	250	10

图像	评价指标	Larsson模型	Iizuka模型	Kang模型	Jin模型	Min模型	本文模型
图像1	PSNR MSE QSSIM	18.94 0.012 8 0.934 5	17.85 0.016 4 0.942 9	20.56 0.012 5 0.909 2	25.42 0.002 9 0.944 0	24.01 0.004 0 0.941 0	25.52 0.002 8 0.944 2
图像2	PSNR MSE QSSIM	14.33 0.014 9 0.957 0	15.15 0.012 3 0.965 5	26.90 0.000 8 0.974 4	24.34 0.001 5 0.954 2	24.59 0.001 4 0.938 5	25.40 0.001 2 0.961 9
图像3	PSNR MSE QSSIM	14.59 0.034 7 0.913 1	13.89 0.040 8 0.918 4	21.23 0.007 5 0.882 7	24.86 0.003 3 0.925 5	24.77 0.003 3 0.921 1	24.64 0.003 4 0.922 6
图像4	PSNR MSE QSSIM	13.36 0.046 1 0.886 2	12.67 0.054 0 0.892 3	18.44 0.014 3 0.858 0	22.17 0.006 1 0.857 5	21.78 0.006 6 0.851 8	22.49 0.005 6 0.857 7

评价指标	Larsson模型	Iizuka模型	Kang模型	Jin模型	Min模型	本文模型
平均PSNR 平均MSE 平均QSSIM	15.31 0.027 1 0.922 7	14.89 0.030 9 0.929 8	21.78 0.008 8 0.906 1	24.20 0.003 5 0.920 3	23.79 0.003 8 0.913 1	24.51 0.003 3 0.921 6

图像	评价指标	Lrasson模型	Iizuka模型	Kang模型	Jin模型	Min模型	本文模型
	PSNR	24.59	25.49	19.03	28.37	28.42	29.63
图像5	MSE QSSIM	0.003 5 0.988 9	0.002 8 0.989 1	0.012 5 0.962 4	0.001 5 0.983 1	0.001 4 0.983 0	0.001 1 0.983 9
图像6	PSNR MSE QSSIM	24.82 0.003 3 0.985 8	23.27 0.004 7 0.988 0	24.68 0.003 4 0.986 4	28.44 0.001 4 0.972 6	29.07 0.001 2 0.938 2	29.20 0.001 2 0.975 3
图像7	PSNR MSE QSSIM	15.08 0.029 3 0.957 1	24.21 0.003 6 0.980 7	23.21 0.004 5 0.977 9	26.94 0.001 9 0.972 6	28.86 0.001 2 0.968 4	29.44 0.001 1 0.975 3
图像8	PSNR MSE QSSIM	28.12 0.001 5 0.983 2	29.10 0.001 2 0.985 7	24.52 0.003 5 0.983 3	26.74 0.002 1 0.974 8	29.22 0.001 2 0.972 7	29.85 0.001 0 0.980 5
图像9	PSNR MSE QSSIM	19.83 0.010 4 0.951 6	24.82 0.003 3 0.948 2	27.75 0.001 7 0.958 5	27.55 0.001 8 0.952 6	27.41 0.001 8 0.953 6	29.52 0.001 1 0.960 6
图像10	PSNR MSE QSSIM	16.63 0.021 7 0.946 7	17.57 0.017 5 0.941 7	23.12 0.004 9 0.939 5	24.08 0.003 9 0.939 8	24.24 0.003 8 0.940 9	25.07 0.003 1 0.950 0

评价指标	Larsson模型	Iizuka模型	Kang模型	Jin模型	Min模型	本文模型
平均PSNR 平均MSE 平均QSSIM	21.51 0.011 6 0.968 9	24.08 0.005 5 0.972 2	23.72 0.005 1 0.968 0	27.02 0.002 1 0.965 9	27.87 0.001 8 0.959 5	28.79 0.001 4 0.970 9

Total variation algorithm with depth image priors for image colorization

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 31

Related Articles 15

Recommended Articles

Metrics

Comments

评价指标	Larsson 模型	Iizuka 模型	kang 模型	Jin 模型	Min 模型	本文模型
平均运行时间	3.2	3.7	420.7	81.3	882.5	230.2

[1]	Caiyun WANG, Yida WU, Jianing WANG, Lu MA, Huanyue ZHAO. SAR image target recognition based on combinatorial optimization convolutional neural network [J]. Systems Engineering and Electronics, 2022, 44(8): 2483-2487.
[2]	Dong CHEN, Yanwei JU. Ship object detection SAR images based on semantic segmentation [J]. Systems Engineering and Electronics, 2022, 44(4): 1195-1201.
[3]	Jingming SUN, Shengkang YU, Jun SUN. Pose sensitivity analysis of HRRP recognition based on deep learning [J]. Systems Engineering and Electronics, 2022, 44(3): 802-807.
[4]	Lei YANG, Su ZHANG, Minghui GAI, Cheng FANG. High-resolution SAR imagery with enhancement of directional structure feature [J]. Systems Engineering and Electronics, 2022, 44(3): 808-818.
[5]	Hengyan LIU, Limin ZHANG, Wenjun YAN, Zhaogen ZHONG, Qing LING, Xiaojun LIANG. LDPC decoding based on WBP-CNN algorithm [J]. Systems Engineering and Electronics, 2022, 44(3): 1030-1035.
[6]	Kai SHAO, Miaomiao ZHU, Guangyu WANG. Modulation recognition method based on generative adversarial andconvolutional neural network [J]. Systems Engineering and Electronics, 2022, 44(3): 1036-1043.
[7]	Cheng FANG, Huijuan LI, Wen LU, Yumeng SONG, Lei YANG. Multi-feature enhancement algorithm for high resolution SAR based on morphological auto-blocking [J]. Systems Engineering and Electronics, 2022, 44(2): 470-479.
[8]	Qinzhe LYU, Yinghui QUAN, Minghui SHA, Shuxian DONG, Mengdao XING. Ensemble deep learning-based intelligent classification of active jamming [J]. Systems Engineering and Electronics, 2022, 44(12): 3595-3602.
[9]	Bo DAN, Zhequan FU, Shan GAO, Tao JIAN. Full-polarization high resolution range profile recognition technology for sea surface target based on convolutional neural network [J]. Systems Engineering and Electronics, 2022, 44(1): 108-116.
[10]	Ziyan LIU, Shanshan MA, Jing LIANG, Mingcheng ZHU, Lei YUAN. Attention mechanism based CNN channel estimation algorithm in millimeter-wave massive MIMO system [J]. Systems Engineering and Electronics, 2022, 44(1): 307-312.
[11]	Lei YANG, Huijuan LI, Bo HUANG, Wei LIU, Pucheng LI. High resolution SAR imagery with structural feature enhancement under two-layer sparse group Lasso [J]. Systems Engineering and Electronics, 2021, 43(2): 351-362.
[12]	Shiyu SHEN, Xiaodong YE, Hao WANG, Shifei TAO. SAR image speckle suppression method based on muti-scale interactive structure convolutional neural network [J]. Systems Engineering and Electronics, 2021, 43(12): 3526-3532.
[13]	Yuyuan ZHNAG, Wenjun YAN, Limin ZHANG, Yuan ZHANG. Blind recognition algorithm of serial space-time block code based on convolutional neural network [J]. Systems Engineering and Electronics, 2021, 43(11): 3360-3370.
[14]	Yunfei MA, Xisheng JIA, Huajun BAI, Chiming GUO, Shuangchuan WANG. Fault diagnosis of compressed vibration signal based on 1-dimensional CNN with optimized parameters [J]. Systems Engineering and Electronics, 2020, 42(9): 1911-1919.
[15]	Zhenzhen YANG, Jun LE, Yongpeng YANG, Lu FAN. Object detection algorithm of nonconvex motion-assisted low rank and sparse decomposition [J]. Systems Engineering and Electronics, 2020, 42(6): 1218-1225.