结合绘画先验的线稿上色方法

doi:10.11896/jsjkx.210300140

摘要/Abstract

摘要： 自动线稿上色近年来已成为计算机视觉领域的研究热点之一。现有方法旨在通过改进网络架构或图像生成流程来提升上色的质量,但生成结果多存在色相集中、饱和度和明度分布不合理等现象。为此,提出一种结合绘画先验的线稿上色方法。该方法受插画师上色过程的启发,学习其广泛使用的绘画先验知识(如色相变化、饱和度对比和明暗对比)来提升自动线稿上色的质量。具体来讲,该方法在HSV色彩空间上增加了像素级损失,引导网络生成异常纹理较少的结果。同时,提出的三项启发式损失函数分别引入了色相变化、饱和度对比和明暗对比等绘画先验,引导网络生成具有合理色彩组成的上色结果。在真实线稿构建的测试数据集上,采用FID和MOS两项指标对所提方法和现有方法从生成结果与真实数据的分布相似度及视觉质量方面进行了比较。实验结果表明,相比性能第二的模型,所提方法的FID指标降低了21.00,MOS指标提高了0.96,因此所提线稿上色方法有效提升了自动线稿上色的视觉质量。

关键词: HSV色彩空间, 对抗生成网络(GAN), 绘画先验, 深度学习, 自动线稿上色

Abstract: Automatic sketch colorization has become an important research topic in computer vision.Previous methods intent to improve the colorization quality with advanced network architecture or innovative pipeline.However, they usually generate results with concentrated hue, unreasonable saturation and gray distribution.To alleviate these problems, this paper proposes a sketch colorization method with drawing priors.Inspired by the actual coloring process, this method learns the widely used drawing priors (such as hue variation, saturation contrast, and gray contrast) to improve the quality of automatic sketch colorization.Speci-fically, it incorporates pixel-level loss in the HSV color space to gain more natural results with less artifacts.Meanwhile, three heuristic loss functions that introduce the drawing priors such as hue variation, saturation and gray contrast are used to train our method to generate results with harmonious color composition.We compare our method with current state-of-the-art methods on test dataset constructed by real sketch images.Fréchet inception distance (FID) and mean opinion score (MOS) are adopted to measure the similarity between the distribution of real and generated images and the visual quality, respectively.Compared to the second-best method, the experimental results show that the FID of our method decreases by 21.00 and the MOS increases by 0.96, respectively.All the experimental results prove that the proposed method effectively improves the visual quality of automa-tic sketch colorization.

Key words: Automatic sketch colorization, Deep learning, Drawing prior, Generative adversarial networks(GAN), HSV color space

中图分类号:

TP391

窦智, 王宁, 王世杰, 王智慧, 李豪杰. 结合绘画先验的线稿上色方法[J]. 计算机科学, 2022, 49(4): 195-202. https://doi.org/10.11896/jsjkx.210300140

DOU Zhi, WANG Ning, WANG Shi-jie, WANG Zhi-hui, LI Hao-jie. Sketch Colorization Method with Drawing Prior[J]. Computer Science, 2022, 49(4): 195-202. https://doi.org/10.11896/jsjkx.210300140

参考文献

[1] QU Y G,WONG T T,HENG P A.Manga colorization[J].ACM Transactions on Graphics,2006,25(3):1214-1220.
[2] SYKORA D,DINGLIANA J,COLLINS S.LazyBrush:Flexible Painting Tool for Hand-drawn Cartoons[J].Computer Graphics Forum,2009,28(2):599-608.
[3] GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial net-works[C]//Advances in Neural Information Processing Systems.2014:2672-2680.
[4] TAIZAN.Paintschainer canna[EB/OL].https://petalica-pa-int.pixiv.dev/index_en.html.
[5] TAIZAN.Paintschainer tanpopo[EB/OL].https://petalica-pa-int.pixiv.dev/index_en.html.
[6] TAIZAN.Paintschainer satsuki[EB/OL].https://petalica-pa-int.pixiv.dev/index_en.html.
[7] ZHANG L,LI C,WONG T T,et al.Two-stage sketch colorization[J].ACM Transactions on Graphics,2018,37(6):1-14.
[8] CI Y Z,MA X Z,WANG Z H,et al.User-guided deep anime line art colorization with conditional adversarial networks[C]//ACM Multimedia Conference on Multimedia Conference.2018:1536-1544.
[9] YOO S J,BAHNG H J,CHUNG S H,et al.Coloring with limi-ted data:Few-shot colorization via memory augmented networks[C]//IEEE Conference on Computer Vision and Pattern Recognition.Long Beach,CA,USA,2019:11283-11292.
[10] SMITH A R.Color gamut transform pairs[C]//Proceedings of the 5th Annual Conference on Computer Graphics and Interactive Techniques,SIGGRAPH 1978.Atlanta,GA,USA,1978:12-19.
[11] KIM E J,SUK H J.Hue extraction and Tone match:Genera-ting a Theme Color to Enhance the Emotional Quality of an Image[J/OL].ACM Siggraph.https://dl.acm.org/doi/abs/10.1145/2787626.2787657.
[12] LYNCH D K,LIVINGSTON W.Color and light in nature[J].Optometry and Vision,2001,74(6):1342-1343.
[13] JAMES G.Color and Light:A Guide for the Realist Painter[M].Andrews Mcmeel Publishing,2010.
[14] HUANG Y C,TUNG Y S,CHEN J C,et al.An adaptive edge detection based colorization algorithm and its applica-tions[C]//Proceedings of the 13th Annual ACM International Conference on Multimedia.ACM,2005:351-354.
[15] LEVIN A,LISCHINSKI D,WEISS Y.Colorization using optimization[J].ACM Transactions on Graphics,2004,23:689-694.
[16] ISOLA P,ZHU J Y,ZHOU T H,et al.Image-to-Image Translation with Conditional Adversarial Networks[C]//IEEE Confe-rence on Computer Vision and Pattern Recognition.Honolulu,HI,2017:5967-5976.
[17] WANG T C,LIU M Y,ZHU J Y,et al.High-resolution image synthesis and semantic manipulation with conditional gans[C]//IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City,UT,USA,2018:8798-8807.
[18] CHEN W,HAYS J.SketchyGAN:Towards Diverse and Realistic Sketch to Image Synthesis[C]//IEEE Conference on Computer Vision and Pattern Recognition,2018.Salt Lake City,UT,USA:9416-9425.
[19] ZHU J Y,PARK T,ISOLA P,et al.Unpaired image-to-imagetranslation using cycle-consistent adversarial networks[C]//IEEE International Conference on Computer Vision(ICCV 2017).Venice,Italy,2017:2242-2251.
[20] YI Z,ZHANG H,TAN P.Dualgan:unsupervised dual learning for image-to-image translation[C]//IEEE International Confe-rence on Computer Vision(ICCV 2017).Venice,Italy,2017:2868-2876.
[21] FURUSAWA C,HIROSHIBA K,OGAKI K,et al.Comi-colorization:Semi-Automatic Manga Colorization[J].SIGGRAPH Asia 2017 Technical Briefs,2017,12:1-4.
[22] FRANS K.Outline colorization through tandem adversarial networks[J].arXiv:1704.08834,2017.
[23] LIU Y F,QIN Z C,WAN T,et al.Auto-painter:cartoon image generation from sketch by using conditional wasserstein generative adversarial networks[J].Neurocomputing,2018,311:78-87.
[24] MEHDI M,SIMON O.Conditional Generative AdversarialNets[J].arXiv:1411.1784,2014.
[25] HYUNSU K,HO Y J,EUNHYEOK P,et al.Tag2Pix:LineArt Colorization Using Text Tag With SECat and Changing Loss[C]//2019 IEEE/CVF International Conference on Computer Vision.Seoul,Korea (South),2019:9055-9064.
[26] GULRAJANI I,AHMED F,ARJOVSKY M,et al.Improvedtraining of wasserstein gans[C]//Advances in Neural Information Processing Systems 30:Annual Conference on Neural Information Processing Systems,2017.Long Beach,CA,USA:5767-5777.
[27] XIE S,GIRSHICK R,DOLLÁR P,et al.Aggregated ResidualTransformations for Deep Neural Networks[C]//IEEE Confe-rence on Computer Vision and Pattern Recognition.Honolulu,HI,USA,2017:5987-5995.
[28] YU F,KOLTUN V,FUNKHOUSER T.Dilated residual net-works[C]//IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,HI,USA,2017:5987-5995.
[29] SHI W,CABALLERO J,FERENC H,et al.Real-time singleimage and video super-resolution using an efficient sub-pixel con-volutional neural network[C]//IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,2016:1874-1883.
[30] SAITO M,MATSUI Y.Illustration2Vec:a semantic vector re-presentation of illustrations[C]//SIGGRAPH Asia 2015 Technical Briefs.Kobe,Japan,2015,5:1-4.
[31] LEDIG C,THEIS L,HUSZAR F,et al.Cunningham A andAcosta A.Photo-realistic single image super-resolution using a generative adversarial network[C]//IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,HI,USA,2016:105-114.
[32] KARRAS T,AILA T,LAINE S,et al.Progressive growing of gans for improved quality,stability,and variation[J].arXiv:1710.10196.
[33] OTSU N.A threshold selection method from gray-level histograms[J].IEEE Transactions on Systems,Man,and Cyberne-tics,1979,9(1):62-66.
[34] JUSTIN J,ALEXANDRE A,LI F F.Perceptual losses for real-time style transfer and super-resolution[C]//Computer Vision(ECCV 2016).14th European Conference.Amsterdam,The Netherlands,2016:694-711.
[35] WINNEMOELLER H,KYPRIANIDIS J E,OLSEN S C.Xdog:an extended difference-of-gaussians compendium including advanced image stylization[J].Computers and Graphics,2012,36(6):740-753.
[36] HEUSEL M,RAMSAUER H,UNTERTHINER T,et al.Gans trained by a two time-scale update rule converge to a local nash equilibrium[C]//Advances in Neural Information Processing Systems 30:Annual Conference on Neural Information Proces-sing Systems,2017.Long Beach,CA,USA:6626-6637.

相关文章 15

[1]	徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺. 时序知识图谱表示学习 Temporal Knowledge Graph Representation Learning 计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204
[2]	饶志双, 贾真, 张凡, 李天瑞. 基于Key-Value关联记忆网络的知识图谱问答方法 Key-Value Relational Memory Networks for Question Answering over Knowledge Graph 计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277
[3]	汤凌韬, 王迪, 张鲁飞, 刘盛云. 基于安全多方计算和差分隐私的联邦学习方案 Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy 计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108
[4]	孙奇, 吉根林, 张杰. 基于非局部注意力生成对抗网络的视频异常事件检测方法 Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection 计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[5]	王剑, 彭雨琦, 赵宇斐, 杨健. 基于深度学习的社交网络舆情信息抽取方法综述 Survey of Social Network Public Opinion Information Extraction Based on Deep Learning 计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099
[6]	郝志荣, 陈龙, 黄嘉成. 面向文本分类的类别区分式通用对抗攻击方法 Class Discriminative Universal Adversarial Attack for Text Classification 计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077
[7]	姜梦函, 李邵梅, 郑洪浩, 张建朋. 基于改进位置编码的谣言检测模型 Rumor Detection Model Based on Improved Position Embedding 计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046
[8]	侯钰涛, 阿布都克力木·阿布力孜, 哈里旦木·阿布都克里木. 中文预训练模型研究进展 Advances in Chinese Pre-training Models 计算机科学, 2022, 49(7): 148-163. https://doi.org/10.11896/jsjkx.211200018
[9]	周慧, 施皓晨, 屠要峰, 黄圣君. 基于主动采样的深度鲁棒神经网络学习 Robust Deep Neural Network Learning Based on Active Sampling 计算机科学, 2022, 49(7): 164-169. https://doi.org/10.11896/jsjkx.210600044
[10]	苏丹宁, 曹桂涛, 王燕楠, 王宏, 任赫. 小样本雷达辐射源识别的深度学习方法综述 Survey of Deep Learning for Radar Emitter Identification Based on Small Sample 计算机科学, 2022, 49(7): 226-235. https://doi.org/10.11896/jsjkx.210600138
[11]	胡艳羽, 赵龙, 董祥军. 一种用于癌症分类的两阶段深度特征选择提取算法 Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification 计算机科学, 2022, 49(7): 73-78. https://doi.org/10.11896/jsjkx.210500092
[12]	程成, 降爱莲. 基于多路径特征提取的实时语义分割方法 Real-time Semantic Segmentation Method Based on Multi-path Feature Extraction 计算机科学, 2022, 49(7): 120-126. https://doi.org/10.11896/jsjkx.210500157
[13]	祝文韬, 兰先超, 罗唤霖, 岳彬, 汪洋. 改进Faster R-CNN的光学遥感飞机目标检测 Remote Sensing Aircraft Target Detection Based on Improved Faster R-CNN 计算机科学, 2022, 49(6A): 378-383. https://doi.org/10.11896/jsjkx.210300121
[14]	王建明, 陈响育, 杨自忠, 史晨阳, 张宇航, 钱正坤. 不同数据增强方法对模型识别精度的影响 Influence of Different Data Augmentation Methods on Model Recognition Accuracy 计算机科学, 2022, 49(6A): 418-423. https://doi.org/10.11896/jsjkx.210700210
[15]	毛典辉, 黄晖煜, 赵爽. 符合监管合规性的自动合成新闻检测方法研究 Study on Automatic Synthetic News Detection Method Complying with Regulatory Compliance 计算机科学, 2022, 49(6A): 523-530. https://doi.org/10.11896/jsjkx.210300083

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed