利用生成对抗网络的人脸图像分步补全法

doi:10.11896/jsjkx.200800014

计算机科学 ›› 2021, Vol. 48 ›› Issue (9): 174-180.doi: 10.11896/jsjkx.200800014

• 计算机图形学&多媒体 • 上一篇下一篇

利用生成对抗网络的人脸图像分步补全法

林椹尠¹, 张梦凯², 吴成茂³, 郑兴宁²

1 西安邮电大学理学院西安710121
2 西安邮电大学通信与信息工程学院西安710121
3 西安邮电大学电子工程学院西安710121

收稿日期:2020-08-03 修回日期:2020-10-29 出版日期:2021-09-15 发布日期:2021-09-10
通讯作者: 张梦凯(zmkdyx@163.com)
作者简介:lzhx126@126.com
基金资助:
国家自然科学基金(61671377)

Face Image Inpainting with Generative Adversarial Network

LIN Zhen-xian¹, ZHANG Meng-kai², WU Cheng-mao³, ZHENG Xing-ning²

1 School of Science,Xi'an University of Posts & Telecommunications,Xi'an 710121,China
2 School of Communication and Information Engineering,Xi'an University of Posts & Telecommunications,Xi'an 710121,China
3 School of Electronic Engineering,Xi'an University of Posts & Telecommunications,Xi'an 710121,China

Received:2020-08-03 Revised:2020-10-29 Online:2021-09-15 Published:2021-09-10
About author:LIN Zhen-xian,born in 1969,Ph.D,professor.Her main research interests include wavelet theory and its application in signal and image.
ZHANG Meng-kai,born in 1995,postgraduate.His main research interests include deep learning and computer vision.
Supported by:
National Natural Science Foundation of China(61671377)

摘要/Abstract

摘要： 人脸图像修复技术是近年来图像处理领域的研究热点,而人脸图像大面积缺失导致损失语义信息过多,一直是该领域的重点难点问题。针对这一问题,文中提出了一种基于生成对抗网络的图像分步补全算法。将人脸图像修复问题分为两步,设计两个串联的生成对抗网络,首先残缺图像通过预补全网络进行图像的预补全,预补全图像进入增强网络进行特征增强;判别器分别判断预补全图像和增强图像与理想图像的差异性;采用长短时记忆单元连接两部分的信息流,增强信息的传递。然后使用内容损失、对抗损失和全变分损失相结合的损失函数,提高网络的修复效果。最后在CelebA数据集上进行实验,结果显示,所提算法相较于对比算法在峰值信噪比指标上提高了16.84%～22.85%,在结构相似性指标上提高了10%～12.82%。

关键词: 生成对抗网络, 人脸图像, 图像补全, 长短时记忆, 深度学习, 缺失区域, 跳跃连接

Abstract: Face image inpainting is a hot topic of image processing research in recent years.Due to the loss of excessive sematic information,it is a difficult problem to inpaint large area missing of face images.Aiming at the problem of inpainting face images,a step-by-step image inpainting algorithm based on generative adversarial network is proposed.Face images inpainting task is divided into two steps.Firstly,face images are completed through the pre-completion network,and pre-completion images is enhanced feature through the enhancement network.The discriminator judges the difference between the pre-completion images,the enhanced images and the ideal image respectively.The long-term memory unit is used to connect the information flow of two parts.Secondly,the adversarial loss,content loss and total variation loss are combined to improve the effectively.Experiments are conducted on CelebA dataset,and this algorithm has an improvement of 16.84%~22.85% in PSNR and 10%～12.82% in SSIM compared with others typical image inpainting algorithms

Key words: Generative adversarial networks, Face images, Image inpainting, Long short term memory, Deep learning, Loss areas, Skip connection

中图分类号:

TP183

林椹尠, 张梦凯, 吴成茂, 郑兴宁. 利用生成对抗网络的人脸图像分步补全法[J]. 计算机科学, 2021, 48(9): 174-180.

LIN Zhen-xian, ZHANG Meng-kai, WU Cheng-mao, ZHENG Xing-ning. Face Image Inpainting with Generative Adversarial Network[J]. Computer Science, 2021, 48(9): 174-180.

参考文献

[1]HE Y T,TANG X H,ZHANG Y,et al.Improved Criminisi algorithm based on structure tensor[J].Journal of Image and Graphics,2018,23(10):1492-1507.
[2]LAN X L,LIU H X,YAO H B.Improved image inpainting algorithm based on texture blocks and gradient feature[J].Computer Engineering and Applications,2018,54(20):172-177.
[3]KRHENBÜHL P,KOLTUN V.Efficient Inference in FullyConnected CRFs with Gaussian Edge Potentials[J].Advances in Neural Information Processing Systems,2012,24(2011):109-117.
[4]XU L M,WU Y J,ZHANG B.Image Inpainting AlgorithmBased on Adaptive High Order Variation in Eight Neighbors[J].Journal of Graphics,2017,38(4):558-565.
[5]BERTALMIO M,SAPIRO G,CASELLES V,et al.Image in-painting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques.ACM Press/Addison-Wesley Publishing Co.,2000:417-424.
[6]LEVIN A,ZOMET A,WEISS Y.Learning how to inpaint from global mage statistics[C]//IEEE In International Conference on Computer Vision.2003:305-312.
[7]SHEN J,CHAN T F.Mathematical Models for Local Nontexture Inpaintings[J].SIAM Journal on Applied Mathematics,2001,62:1019-1043.
[8]SHEN J,KANG S H,CHAN T F.Euler's Elastica and Curvature-Based Inpainting[J].SIAM Journal on AppliedMathema-tics,2002,63(2):564-592.
[9]BOUREAU Y L,BACH F,LECUN Y,et al.Learning mid-level features for recognition[C]//2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.IEEE,2010,2559-2566.
[10]LI S,ZHAO M.Image inpainting with salient structure completion and texture propagation[J].Pattern Recognition Letters,2011,32(9):1256-1266.
[11]EFROS A A,LEUNG T K.Texture synthesis by non-parame-tric sampling[C]//Proceedings of the Seventh IEEE Internatio-nal Conference on Computer Vision.1999,2:1033-1038.
[12]CRIMINISI A,PÉREZ P,TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing,2004,13(9):1200-1212.
[13]BARNES C,SHECHTMAN E,FINKELSTEIN A,et al.PatchMatch:A randomized correspondence algorithm for structural image editing[C]//ACM Transactions on Graphics (ToG).ACM,2009:1-11.
[14]WEI Y,LIU S.Domain-based structure-aware image inpainting[J].Signal Image & Video Processing,2016,10(5):911-919.
[15]GIRSHICK R,DONAHUE J,DARRELL T,et al.Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2014:580-587.
[16]LONG J,SHELHAMER E,DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:3431-3440.
[17]XIE J,XU L,CHEN E.Image denoising and inpainting withdeep neural networks[C]//Advances in Neural Information Processing Systems.2012:341-349.
[18]KÖHLER R,SCHULER C,SCHÖLKOPF B,et al.Mask.specific inpainting with deep neural networks[C]//German Confe-rence on Pattern Recognition.Cham:Springer,2014:523-534.
[19]GOODFELLOW I,POUGET-ABADIE J,MIRZAM,et al.Gene-rative adversarial nets[C]//Advances in Neural Information Processing Systems.2014:2672-2680.
[20]PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context encoders:Feature learning by inpainting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2536-2544.
[21]YANG C,LU X,LIN Z,et al.High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:6721-6729.
[22]IIZUKA S S,SERRA E,SHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Gra-phics (TOG),2017,36(4):107:1-2.
[23]YU J H,ZHE L,YANG J M,et al.Generative Image Inpainting with Contextual Attention [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.2018:5505-5514.
[24]LIU G,REDA F A,SHIH K J,et al.Image inpainting for irre-gular holes using partial convolutions [C]//Proceedings of the European Conference on Computer Vision (ECCV).2018:85-100.
[25]HOCHREITER S,SCHMIDHUBER J.Long Short-Term Me-mory[J].Neural Computation,1997,9(8):1735-1780.
[26]YEH R A,CHEN C,LIM T Y,et al.Semantic Image Inpainting with Deep Generative Models [C]//CVPR 2016.2016:5485-5493.
[27]ISOLA P,ZHU J Y,ZHOU T H,et al.Image-to-image translation with conditional adversarial networks[C]//IEEEConfe-rence on Computer Vision and Pattern Recognition (CVPR).2017.
[28]RONNEBERGER O,FISCHER P,BROX T.U-Net:Convolu-tional Networks for Biomedical Image Segmentation[C]//International Conference on Medical Image Computing and Computer-assisted Intervention.Cham:Springer,2015:234-241.
[29]RADFORD A,METZ L,CHINTALA S.Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks [C]//Proceedings of the International Confe-rence on Learning Representations (ICLR).2016.
[30]ZHAO H,GALLO O,FROSIO I,et al.Loss functions for image restoration with neural networks[J].IEEE Transactions on Computational Imaging,2016,3(1):47-57.
[31]WHYTE O,SIVIC J,ZISSERMAN A,et al.Non-uniform deblurring for shaken images[J].International Journal of Compu-ter Vision,2012,98(2):168-186.
[32]RUDIN L I,OSHER S,FATEMI E.Nonlinear total variation based noise removal algorithms[J].Physica D Nonlinear Phenomena,1992,60(1/2/3/4):259-268.
[33]NAH S,KIM T H,LEE K M.Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:3883-3891.
[34]TONG Y B,ZHANG Q S,QI Y P.Image Quality Assessing by Combining PSNR with SSIM[J].Journal of Image and Gra-phics,2006,11(12),1758-1763.
[35]WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality as-sessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[36]ZHENG C X,CHAM T J,CAI J.Pluralistic Image Completion [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).2019:1438-1447.
[37]CIMPOI M,MAJI S,KOKKINOS I,et al.Describing Textures in the Wild[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2014:3606-3613.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

利用生成对抗网络的人脸图像分步补全法

Face Image Inpainting with Generative Adversarial Network

PDF (PC)

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

Metrics

本文评价

推荐阅读 10

[1]	董晓梅, 王蕊, 邹欣开. 面向推荐应用的差分隐私方案综述[J]. 计算机科学, 2021, 48(9): 21-35.
[2]	周新民, 胡宜桂, 刘文洁, 孙荣俊. 基于多模态多层级数据融合方法的城市功能识别研究[J]. 计算机科学, 2021, 48(9): 50-58.
[3]	钱梦薇, 过弋. 融合偏置深度学习的距离分解Top-N推荐算法[J]. 计算机科学, 2021, 48(9): 103-109.
[4]	徐涛, 田崇阳, 刘才华. 基于深度学习的人群异常行为检测综述[J]. 计算机科学, 2021, 48(9): 125-134.
[5]	张新峰, 宋博. 一种基于改进三元组损失和特征融合的行人重识别方法[J]. 计算机科学, 2021, 48(9): 146-152.
[6]	黄晓生, 徐静. 基于PCANet的非下采样剪切波域多聚焦图像融合[J]. 计算机科学, 2021, 48(9): 181-186.
[7]	刘立波, 苟婷婷. 融合深度典型相关分析和对抗学习的跨模态检索[J]. 计算机科学, 2021, 48(9): 200-207.
[8]	田野, 陈宏巍, 王法胜, 陈兴文. 室内移动机器人的SLAM算法综述[J]. 计算机科学, 2021, 48(9): 223-234.
[9]	谢良旭, 李峰, 谢建平, 许晓军. 基于融合神经网络模型的药物分子性质预测[J]. 计算机科学, 2021, 48(9): 251-256.
[10]	冯霞, 胡志毅, 刘才华. 跨模态检索研究进展综述[J]. 计算机科学, 2021, 48(8): 13-23.
[11]	周文辉, 石敏, 朱登明, 周军. 基于残差注意力网络的地震数据超分辨率方法[J]. 计算机科学, 2021, 48(8): 24-31.
[12]	王立梅, 朱旭光, 汪德嘉, 张勇, 邢春晓. 基于深度学习的民事案件判决结果分类方法研究[J]. 计算机科学, 2021, 48(8): 80-85.
[13]	郭琳, 李晨, 陈晨, 赵睿, 范仕霖, 徐星雨. 基于通道注意递归残差网络的图像超分辨率重建[J]. 计算机科学, 2021, 48(8): 139-144.
[14]	刘帅, 芮挺, 胡育成, 杨成松, 王东. 基于深度学习SuperGlue算法的单目视觉里程计[J]. 计算机科学, 2021, 48(8): 157-161.
[15]	王施云, 杨帆. 基于U-Net特征融合优化策略的遥感影像语义分割方法[J]. 计算机科学, 2021, 48(8): 162-168.