# American Institute of Mathematical Sciences

2016, 10(4): 1037-1055. doi: 10.3934/ipi.2016031

## A coupled total variation model with curvature driven for image colorization

 1 School of Science, Nanjing University of Posts and Telecommunications, Nanjing, China, China 2 Centre for Mathematical Imaging and Vision and Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Hong Kong

Received  September 2015 Revised  December 2015 Published  October 2016

In this paper, we study the problem of image colorization based on the propagation from given color pixels to the other grey-level pixels in grayscale images. We propose to use a coupled total variation model with curvature information of luminance channel to control the colorization process. There are two distinct advantages of the proposed model: (i) the involved optimization problem is convex and it is not sensitive to initial guess of colorization procedure; (ii) the proposed model makes use of curvature information to control the color diffusion process which is more effectively than that by using the gradient information. The existence of the minimizer of the proposed model can be shown, and the numerical solver based on convex programming techniques can be developed to solve the resulting model very efficiently. Experimental results are reported to demonstrate that the performance of the proposed model is better than those of the other color propagation models, especially when we deal with large regions of grayscale images for colorization.
Citation: Zhengmeng Jin, Chen Zhou, Michael K. Ng. A coupled total variation model with curvature driven for image colorization. Inverse Problems & Imaging, 2016, 10 (4) : 1037-1055. doi: 10.3934/ipi.2016031
##### References:
 [1] L. Ambrosio, N. Fusco and D. Pallara, Functions of Bounded Variation and Free Discontinuity Problems,, Oxford University Press, (2000). [2] X. Bresson and T. F. Chan, Fast dual minimization of the vectorical total variation norm and application to color image processing,, Inverse Problems and Imaging, 2 (2008), 455. doi: 10.3934/ipi.2008.2.455. [3] A. Bugeau, V.-T. Ta and N. Papadakis, Variational exemplar-based image colorization,, IEEE Trans. Image Process., 23 (2014), 298. doi: 10.1109/TIP.2013.2288929. [4] V. Caselles, G. Facciolo and E. Meinhardt, Anisotropic Cheeger sets and applications,, SIAM J. Imaging Sci., 2 (2009), 1211. doi: 10.1137/08073696X. [5] R. H. Chan, J. Yang and X. Yuan, Alternating Direction Method for Image Inpainting in Wavelet Domains,, SIAM J. Imaging Sci., 4 (2011), 807. doi: 10.1137/100807247. [6] Y. Chen and T. Wunderli, Adaptive total variation for image restoration in BV space,, J. Math. Anal. Appl., 272 (2002), 117. doi: 10.1016/S0022-247X(02)00141-5. [7] Z. M. Jin, F. Li and M. K. Ng, A variational approach for image decolorization by variance maximization,, SIAM J. Imaging Sci., 7 (2014), 944. doi: 10.1137/130935197. [8] S. H. Kang and R. March, Variational models for image colorization via chromaticity and brightness decomposition,, IEEE Transactions On Image Processing, 16 (2007), 2251. doi: 10.1109/TIP.2007.903257. [9] A. Levin, D. Lischinski and Y. Weiss, Colorization using optimization,, in Proc. SIGGRAPH Conf., 23 (2004), 689. doi: 10.1145/1186562.1015780. [10] F. Pierre, J.-F. Aujol, A. Bugeau, N. Papadakis and V.-T. Ta, Luminance-chrominance model for image colorization,, SIAM J. Imaging Sci., 8 (2015), 536. doi: 10.1137/140979368. [11] M. H. Quang, S. H. Kang and T. M. Le, Image and video colorization using vector-valued reproducing kernel Hilbert spaces,, Journal of Mathematical Imaging and Vision, 37 (2010), 49. doi: 10.1007/s10851-010-0192-8. [12] M. L. Song, D. C. Tao and C. Chen, Color to gray: Visual cue preservation,, IEEE Transactions on Pattern Analysis and Machine Intelligence, 32 (2012), 1537. [13] J. Eckstein and D. Bertsekas, On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators,, Math. Programming, 55 (1992), 293. doi: 10.1007/BF01581204. [14] D. Gabay and B. Mercier, A dual algorithm for the solution of nonlinear variational problems via finite-element approximations,, Comp. Math. Appl., 2 (1976), 17. doi: 10.1016/0898-1221(76)90003-1. [15] T. Goldstein and S. Osher, The split Bregman method for $L^1$ regularized problems,, SIAM Journal on Imaging Sciences, 2 (2009), 323. doi: 10.1137/080725891. [16] X. D. Hou, J. Harel and C. Koch, Image signature: Highlighting sparse salient regions,, IEEE Trans. Pattern Anal. Mach. Intell., 34 (2012), 194. [17] R. Hunter, Photoelectric color difference meter,, Journal of the Optical Society of America, 48 (2008), 985. doi: 10.1364/JOSA.48.000985. [18] R. Irony, D. Cohen and D. Lischinski, Colorization by example, in Eurographics conference on Rendering Techniques,, Eurographics Association, (2005), 201. [19] W. K. Pratt, Ed., Digital Image Processing, Wilely-Interscience publication,, New York, (2001). [20] K. Ito and K. Kunisch, An augmented Lagrangian technique for variational inequalities,, Appl. Math. Optim., 21 (1990), 223. doi: 10.1007/BF01445164. [21] L. Itti, C. Koch and E. Niebur, A model of saliency-based visual attention for rapid scene analysis,, IEEE Transactions on Pattern Analysis and Machine Intelligence, 20 (1998), 1254. doi: 10.1109/34.730558. [22] M. K. Ng, P. Weiss and X. Yuan, Solving constrained total-variation image restoration and reconstruction problems via alternating direction methods,, SIAM J. Sci. Comput., 32 (2010), 2710. doi: 10.1137/090774823. [23] F. Pierre, J.-F. Aujol, A. Bugeau, N. Papadakis and V.-T. Ta, Luminance-chrominance model for image colorization,, SIAM J. Imaging Sciences, 8 (2015), 536. doi: 10.1137/140979368. [24] F. Pierre, J.-F. Aujol, A. Bugeau, N. Papadakis and V.-T. Ta, A unified model for image colorization, computer vision - ECCV 2014 workshops,, Lecture Notes in Computer Science, 8927 (2015), 297. [25] G. Sapiro, Inpainting the colors,, in Proc. IEEE Int. Conf. Image Processing, 2 (2005), 698. doi: 10.1109/ICIP.2005.1530151. [26] G. Sapiro, Geometric Partial Differential Equations and Image Analysis,, Cambridge university press, (2006). [27] A. A. Shah, M. Gandhi and K. M. Shah, Medical image colorization uisng optimization technique,, International Journal of Scientific and Research Publications, 3 (2013), 1. [28] T. Welsh, M. Ashikhmin and K. Mueller, Transferring color to grayscale images,, ACM Transactions on Graphics, 21 (2002), 277. [29] L. Yatziv and G. Sapiro, Fast image and video colorization using chrominance blending,, IEEE Trans. Image Process., 15 (2006), 1120. doi: 10.1109/TIP.2005.864231.

show all references

##### References:
 [1] L. Ambrosio, N. Fusco and D. Pallara, Functions of Bounded Variation and Free Discontinuity Problems,, Oxford University Press, (2000). [2] X. Bresson and T. F. Chan, Fast dual minimization of the vectorical total variation norm and application to color image processing,, Inverse Problems and Imaging, 2 (2008), 455. doi: 10.3934/ipi.2008.2.455. [3] A. Bugeau, V.-T. Ta and N. Papadakis, Variational exemplar-based image colorization,, IEEE Trans. Image Process., 23 (2014), 298. doi: 10.1109/TIP.2013.2288929. [4] V. Caselles, G. Facciolo and E. Meinhardt, Anisotropic Cheeger sets and applications,, SIAM J. Imaging Sci., 2 (2009), 1211. doi: 10.1137/08073696X. [5] R. H. Chan, J. Yang and X. Yuan, Alternating Direction Method for Image Inpainting in Wavelet Domains,, SIAM J. Imaging Sci., 4 (2011), 807. doi: 10.1137/100807247. [6] Y. Chen and T. Wunderli, Adaptive total variation for image restoration in BV space,, J. Math. Anal. Appl., 272 (2002), 117. doi: 10.1016/S0022-247X(02)00141-5. [7] Z. M. Jin, F. Li and M. K. Ng, A variational approach for image decolorization by variance maximization,, SIAM J. Imaging Sci., 7 (2014), 944. doi: 10.1137/130935197. [8] S. H. Kang and R. March, Variational models for image colorization via chromaticity and brightness decomposition,, IEEE Transactions On Image Processing, 16 (2007), 2251. doi: 10.1109/TIP.2007.903257. [9] A. Levin, D. Lischinski and Y. Weiss, Colorization using optimization,, in Proc. SIGGRAPH Conf., 23 (2004), 689. doi: 10.1145/1186562.1015780. [10] F. Pierre, J.-F. Aujol, A. Bugeau, N. Papadakis and V.-T. Ta, Luminance-chrominance model for image colorization,, SIAM J. Imaging Sci., 8 (2015), 536. doi: 10.1137/140979368. [11] M. H. Quang, S. H. Kang and T. M. Le, Image and video colorization using vector-valued reproducing kernel Hilbert spaces,, Journal of Mathematical Imaging and Vision, 37 (2010), 49. doi: 10.1007/s10851-010-0192-8. [12] M. L. Song, D. C. Tao and C. Chen, Color to gray: Visual cue preservation,, IEEE Transactions on Pattern Analysis and Machine Intelligence, 32 (2012), 1537. [13] J. Eckstein and D. Bertsekas, On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators,, Math. Programming, 55 (1992), 293. doi: 10.1007/BF01581204. [14] D. Gabay and B. Mercier, A dual algorithm for the solution of nonlinear variational problems via finite-element approximations,, Comp. Math. Appl., 2 (1976), 17. doi: 10.1016/0898-1221(76)90003-1. [15] T. Goldstein and S. Osher, The split Bregman method for $L^1$ regularized problems,, SIAM Journal on Imaging Sciences, 2 (2009), 323. doi: 10.1137/080725891. [16] X. D. Hou, J. Harel and C. Koch, Image signature: Highlighting sparse salient regions,, IEEE Trans. Pattern Anal. Mach. Intell., 34 (2012), 194. [17] R. Hunter, Photoelectric color difference meter,, Journal of the Optical Society of America, 48 (2008), 985. doi: 10.1364/JOSA.48.000985. [18] R. Irony, D. Cohen and D. Lischinski, Colorization by example, in Eurographics conference on Rendering Techniques,, Eurographics Association, (2005), 201. [19] W. K. Pratt, Ed., Digital Image Processing, Wilely-Interscience publication,, New York, (2001). [20] K. Ito and K. Kunisch, An augmented Lagrangian technique for variational inequalities,, Appl. Math. Optim., 21 (1990), 223. doi: 10.1007/BF01445164. [21] L. Itti, C. Koch and E. Niebur, A model of saliency-based visual attention for rapid scene analysis,, IEEE Transactions on Pattern Analysis and Machine Intelligence, 20 (1998), 1254. doi: 10.1109/34.730558. [22] M. K. Ng, P. Weiss and X. Yuan, Solving constrained total-variation image restoration and reconstruction problems via alternating direction methods,, SIAM J. Sci. Comput., 32 (2010), 2710. doi: 10.1137/090774823. [23] F. Pierre, J.-F. Aujol, A. Bugeau, N. Papadakis and V.-T. Ta, Luminance-chrominance model for image colorization,, SIAM J. Imaging Sciences, 8 (2015), 536. doi: 10.1137/140979368. [24] F. Pierre, J.-F. Aujol, A. Bugeau, N. Papadakis and V.-T. Ta, A unified model for image colorization, computer vision - ECCV 2014 workshops,, Lecture Notes in Computer Science, 8927 (2015), 297. [25] G. Sapiro, Inpainting the colors,, in Proc. IEEE Int. Conf. Image Processing, 2 (2005), 698. doi: 10.1109/ICIP.2005.1530151. [26] G. Sapiro, Geometric Partial Differential Equations and Image Analysis,, Cambridge university press, (2006). [27] A. A. Shah, M. Gandhi and K. M. Shah, Medical image colorization uisng optimization technique,, International Journal of Scientific and Research Publications, 3 (2013), 1. [28] T. Welsh, M. Ashikhmin and K. Mueller, Transferring color to grayscale images,, ACM Transactions on Graphics, 21 (2002), 277. [29] L. Yatziv and G. Sapiro, Fast image and video colorization using chrominance blending,, IEEE Trans. Image Process., 15 (2006), 1120. doi: 10.1109/TIP.2005.864231.
 [1] Yuan Shen, Lei Ji. Partial convolution for total variation deblurring and denoising by new linearized alternating direction method of multipliers with extension step. Journal of Industrial & Management Optimization, 2017, 13 (5) : 1-17. doi: 10.3934/jimo.2018037 [2] Russell E. Warren, Stanley J. Osher. Hyperspectral unmixing by the alternating direction method of multipliers. Inverse Problems & Imaging, 2015, 9 (3) : 917-933. doi: 10.3934/ipi.2015.9.917 [3] Zhongming Wu, Zhongming Wu, Zhongming Wu. Linearized block-wise alternating direction method of multipliers for multiple-block convex programming. Journal of Industrial & Management Optimization, 2017, 13 (4) : 1-23. doi: 10.3934/jimo.2017078 [4] Bingsheng He, Xiaoming Yuan. Linearized alternating direction method of multipliers with Gaussian back substitution for separable convex programming. Numerical Algebra, Control & Optimization, 2013, 3 (2) : 247-260. doi: 10.3934/naco.2013.3.247 [5] Yunhai Xiao, Junfeng Yang, Xiaoming Yuan. Alternating algorithms for total variation image reconstruction from random projections. Inverse Problems & Imaging, 2012, 6 (3) : 547-563. doi: 10.3934/ipi.2012.6.547 [6] Yue Lu, Ying-En Ge, Li-Wei Zhang. An alternating direction method for solving a class of inverse semi-definite quadratic programming problems. Journal of Industrial & Management Optimization, 2016, 12 (1) : 317-336. doi: 10.3934/jimo.2016.12.317 [7] Liyan Ma, Lionel Moisan, Jian Yu, Tieyong Zeng. A stable method solving the total variation dictionary model with $L^\infty$ constraints. Inverse Problems & Imaging, 2014, 8 (2) : 507-535. doi: 10.3934/ipi.2014.8.507 [8] Chunlin Wu, Juyong Zhang, Xue-Cheng Tai. Augmented Lagrangian method for total variation restoration with non-quadratic fidelity. Inverse Problems & Imaging, 2011, 5 (1) : 237-261. doi: 10.3934/ipi.2011.5.237 [9] Johnathan M. Bardsley. An efficient computational method for total variation-penalized Poisson likelihood estimation. Inverse Problems & Imaging, 2008, 2 (2) : 167-185. doi: 10.3934/ipi.2008.2.167 [10] Yunhai Xiao, Soon-Yi Wu, Bing-Sheng He. A proximal alternating direction method for $\ell_{2,1}$-norm least squares problem in multi-task feature learning. Journal of Industrial & Management Optimization, 2012, 8 (4) : 1057-1069. doi: 10.3934/jimo.2012.8.1057 [11] Xiaoqun Zhang, Tony F. Chan. Wavelet inpainting by nonlocal total variation. Inverse Problems & Imaging, 2010, 4 (1) : 191-210. doi: 10.3934/ipi.2010.4.191 [12] Rinaldo M. Colombo, Francesca Monti. Solutions with large total variation to nonconservative hyperbolic systems. Communications on Pure & Applied Analysis, 2010, 9 (1) : 47-60. doi: 10.3934/cpaa.2010.9.47 [13] Juan C. Moreno, V. B. Surya Prasath, João C. Neves. Color image processing by vectorial total variation with gradient channels coupling. Inverse Problems & Imaging, 2016, 10 (2) : 461-497. doi: 10.3934/ipi.2016008 [14] Florian Krügel. Some properties of minimizers of a variational problem involving the total variation functional. Communications on Pure & Applied Analysis, 2015, 14 (1) : 341-360. doi: 10.3934/cpaa.2015.14.341 [15] Adriana González, Laurent Jacques, Christophe De Vleeschouwer, Philippe Antoine. Compressive optical deflectometric tomography: A constrained total-variation minimization approach. Inverse Problems & Imaging, 2014, 8 (2) : 421-457. doi: 10.3934/ipi.2014.8.421 [16] Yuyuan Ouyang, Yunmei Chen, Ying Wu. Total variation and wavelet regularization of orientation distribution functions in diffusion MRI. Inverse Problems & Imaging, 2013, 7 (2) : 565-583. doi: 10.3934/ipi.2013.7.565 [17] Konstantinos Papafitsoros, Kristian Bredies. A study of the one dimensional total generalised variation regularisation problem. Inverse Problems & Imaging, 2015, 9 (2) : 511-550. doi: 10.3934/ipi.2015.9.511 [18] Yunho Kim, Paul M. Thompson, Luminita A. Vese. HARDI data denoising using vectorial total variation and logarithmic barrier. Inverse Problems & Imaging, 2010, 4 (2) : 273-310. doi: 10.3934/ipi.2010.4.273 [19] Sören Bartels, Marijo Milicevic. Iterative finite element solution of a constrained total variation regularized model problem. Discrete & Continuous Dynamical Systems - S, 2017, 10 (6) : 1207-1232. doi: 10.3934/dcdss.2017066 [20] Matteo Cozzi. On the variation of the fractional mean curvature under the effect of $C^{1, \alpha}$ perturbations. Discrete & Continuous Dynamical Systems - A, 2015, 35 (12) : 5769-5786. doi: 10.3934/dcds.2015.35.5769

2016 Impact Factor: 1.094