计算机科学 ›› 2021, Vol. 48 ›› Issue (9): 174-180.doi: 10.11896/jsjkx.200800014

• 计算机图形学&多媒体 • 上一篇    下一篇

利用生成对抗网络的人脸图像分步补全法

林椹尠1, 张梦凯2, 吴成茂3, 郑兴宁2   

  1. 1 西安邮电大学理学院 西安710121
    2 西安邮电大学通信与信息工程学院 西安710121
    3 西安邮电大学电子工程学院 西安710121
  • 收稿日期:2020-08-03 修回日期:2020-10-29 出版日期:2021-09-15 发布日期:2021-09-10
  • 通讯作者: 张梦凯(zmkdyx@163.com)
  • 作者简介:lzhx126@126.com
  • 基金资助:
    国家自然科学基金(61671377)

Face Image Inpainting with Generative Adversarial Network

LIN Zhen-xian1, ZHANG Meng-kai2, WU Cheng-mao3, ZHENG Xing-ning2   

  1. 1 School of Science,Xi'an University of Posts & Telecommunications,Xi'an 710121,China
    2 School of Communication and Information Engineering,Xi'an University of Posts & Telecommunications,Xi'an 710121,China
    3 School of Electronic Engineering,Xi'an University of Posts & Telecommunications,Xi'an 710121,China
  • Received:2020-08-03 Revised:2020-10-29 Online:2021-09-15 Published:2021-09-10
  • About author:LIN Zhen-xian,born in 1969,Ph.D,professor.Her main research interests include wavelet theory and its application in signal and image.
    ZHANG Meng-kai,born in 1995,postgraduate.His main research interests include deep learning and computer vision.
  • Supported by:
    National Natural Science Foundation of China(61671377)

摘要: 人脸图像修复技术是近年来图像处理领域的研究热点,而人脸图像大面积缺失导致损失语义信息过多,一直是该领域的重点难点问题。针对这一问题,文中提出了一种基于生成对抗网络的图像分步补全算法。将人脸图像修复问题分为两步,设计两个串联的生成对抗网络,首先残缺图像通过预补全网络进行图像的预补全,预补全图像进入增强网络进行特征增强;判别器分别判断预补全图像和增强图像与理想图像的差异性;采用长短时记忆单元连接两部分的信息流,增强信息的传递。然后使用内容损失、对抗损失和全变分损失相结合的损失函数,提高网络的修复效果。最后在CelebA数据集上进行实验,结果显示,所提算法相较于对比算法在峰值信噪比指标上提高了16.84%~22.85%,在结构相似性指标上提高了10%~12.82%。

关键词: 生成对抗网络, 人脸图像, 图像补全, 长短时记忆, 深度学习, 缺失区域, 跳跃连接

Abstract: Face image inpainting is a hot topic of image processing research in recent years.Due to the loss of excessive sematic information,it is a difficult problem to inpaint large area missing of face images.Aiming at the problem of inpainting face images,a step-by-step image inpainting algorithm based on generative adversarial network is proposed.Face images inpainting task is divided into two steps.Firstly,face images are completed through the pre-completion network,and pre-completion images is enhanced feature through the enhancement network.The discriminator judges the difference between the pre-completion images,the enhanced images and the ideal image respectively.The long-term memory unit is used to connect the information flow of two parts.Secondly,the adversarial loss,content loss and total variation loss are combined to improve the effectively.Experiments are conducted on CelebA dataset,and this algorithm has an improvement of 16.84%~22.85% in PSNR and 10%~12.82% in SSIM compared with others typical image inpainting algorithms

Key words: Generative adversarial networks, Face images, Image inpainting, Long short term memory, Deep learning, Loss areas, Skip connection

中图分类号: 

  • TP183
[1]HE Y T,TANG X H,ZHANG Y,et al.Improved Criminisi algorithm based on structure tensor[J].Journal of Image and Graphics,2018,23(10):1492-1507.
[2]LAN X L,LIU H X,YAO H B.Improved image inpainting algorithm based on texture blocks and gradient feature[J].Computer Engineering and Applications,2018,54(20):172-177.
[3]KRHENBÜHL P,KOLTUN V.Efficient Inference in FullyConnected CRFs with Gaussian Edge Potentials[J].Advances in Neural Information Processing Systems,2012,24(2011):109-117.
[4]XU L M,WU Y J,ZHANG B.Image Inpainting AlgorithmBased on Adaptive High Order Variation in Eight Neighbors[J].Journal of Graphics,2017,38(4):558-565.
[5]BERTALMIO M,SAPIRO G,CASELLES V,et al.Image in-painting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques.ACM Press/Addison-Wesley Publishing Co.,2000:417-424.
[6]LEVIN A,ZOMET A,WEISS Y.Learning how to inpaint from global mage statistics[C]//IEEE In International Conference on Computer Vision.2003:305-312.
[7]SHEN J,CHAN T F.Mathematical Models for Local Nontexture Inpaintings[J].SIAM Journal on Applied Mathematics,2001,62:1019-1043.
[8]SHEN J,KANG S H,CHAN T F.Euler's Elastica and Curvature-Based Inpainting[J].SIAM Journal on AppliedMathema-tics,2002,63(2):564-592.
[9]BOUREAU Y L,BACH F,LECUN Y,et al.Learning mid-level features for recognition[C]//2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.IEEE,2010,2559-2566.
[10]LI S,ZHAO M.Image inpainting with salient structure completion and texture propagation[J].Pattern Recognition Letters,2011,32(9):1256-1266.
[11]EFROS A A,LEUNG T K.Texture synthesis by non-parame-tric sampling[C]//Proceedings of the Seventh IEEE Internatio-nal Conference on Computer Vision.1999,2:1033-1038.
[12]CRIMINISI A,PÉREZ P,TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing,2004,13(9):1200-1212.
[13]BARNES C,SHECHTMAN E,FINKELSTEIN A,et al.PatchMatch:A randomized correspondence algorithm for structural image editing[C]//ACM Transactions on Graphics (ToG).ACM,2009:1-11.
[14]WEI Y,LIU S.Domain-based structure-aware image inpainting[J].Signal Image & Video Processing,2016,10(5):911-919.
[15]GIRSHICK R,DONAHUE J,DARRELL T,et al.Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2014:580-587.
[16]LONG J,SHELHAMER E,DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:3431-3440.
[17]XIE J,XU L,CHEN E.Image denoising and inpainting withdeep neural networks[C]//Advances in Neural Information Processing Systems.2012:341-349.
[18]KÖHLER R,SCHULER C,SCHÖLKOPF B,et al.Mask.specific inpainting with deep neural networks[C]//German Confe-rence on Pattern Recognition.Cham:Springer,2014:523-534.
[19]GOODFELLOW I,POUGET-ABADIE J,MIRZAM,et al.Gene-rative adversarial nets[C]//Advances in Neural Information Processing Systems.2014:2672-2680.
[20]PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context encoders:Feature learning by inpainting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2536-2544.
[21]YANG C,LU X,LIN Z,et al.High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:6721-6729.
[22]IIZUKA S S,SERRA E,SHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Gra-phics (TOG),2017,36(4):107:1-2.
[23]YU J H,ZHE L,YANG J M,et al.Generative Image Inpainting with Contextual Attention [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.2018:5505-5514.
[24]LIU G,REDA F A,SHIH K J,et al.Image inpainting for irre-gular holes using partial convolutions [C]//Proceedings of the European Conference on Computer Vision (ECCV).2018:85-100.
[25]HOCHREITER S,SCHMIDHUBER J.Long Short-Term Me-mory[J].Neural Computation,1997,9(8):1735-1780.
[26]YEH R A,CHEN C,LIM T Y,et al.Semantic Image Inpainting with Deep Generative Models [C]//CVPR 2016.2016:5485-5493.
[27]ISOLA P,ZHU J Y,ZHOU T H,et al.Image-to-image translation with conditional adversarial networks[C]//IEEEConfe-rence on Computer Vision and Pattern Recognition (CVPR).2017.
[28]RONNEBERGER O,FISCHER P,BROX T.U-Net:Convolu-tional Networks for Biomedical Image Segmentation[C]//International Conference on Medical Image Computing and Computer-assisted Intervention.Cham:Springer,2015:234-241.
[29]RADFORD A,METZ L,CHINTALA S.Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks [C]//Proceedings of the International Confe-rence on Learning Representations (ICLR).2016.
[30]ZHAO H,GALLO O,FROSIO I,et al.Loss functions for image restoration with neural networks[J].IEEE Transactions on Computational Imaging,2016,3(1):47-57.
[31]WHYTE O,SIVIC J,ZISSERMAN A,et al.Non-uniform deblurring for shaken images[J].International Journal of Compu-ter Vision,2012,98(2):168-186.
[32]RUDIN L I,OSHER S,FATEMI E.Nonlinear total variation based noise removal algorithms[J].Physica D Nonlinear Phenomena,1992,60(1/2/3/4):259-268.
[33]NAH S,KIM T H,LEE K M.Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:3883-3891.
[34]TONG Y B,ZHANG Q S,QI Y P.Image Quality Assessing by Combining PSNR with SSIM[J].Journal of Image and Gra-phics,2006,11(12),1758-1763.
[35]WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality as-sessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[36]ZHENG C X,CHAM T J,CAI J.Pluralistic Image Completion [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).2019:1438-1447.
[37]CIMPOI M,MAJI S,KOKKINOS I,et al.Describing Textures in the Wild[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2014:3606-3613.
[1] 董晓梅, 王蕊, 邹欣开. 面向推荐应用的差分隐私方案综述[J]. 计算机科学, 2021, 48(9): 21-35.
[2] 周新民, 胡宜桂, 刘文洁, 孙荣俊. 基于多模态多层级数据融合方法的城市功能识别研究[J]. 计算机科学, 2021, 48(9): 50-58.
[3] 钱梦薇, 过弋. 融合偏置深度学习的距离分解Top-N推荐算法[J]. 计算机科学, 2021, 48(9): 103-109.
[4] 徐涛, 田崇阳, 刘才华. 基于深度学习的人群异常行为检测综述[J]. 计算机科学, 2021, 48(9): 125-134.
[5] 张新峰, 宋博. 一种基于改进三元组损失和特征融合的行人重识别方法[J]. 计算机科学, 2021, 48(9): 146-152.
[6] 黄晓生, 徐静. 基于PCANet的非下采样剪切波域多聚焦图像融合[J]. 计算机科学, 2021, 48(9): 181-186.
[7] 刘立波, 苟婷婷. 融合深度典型相关分析和对抗学习的跨模态检索[J]. 计算机科学, 2021, 48(9): 200-207.
[8] 田野, 陈宏巍, 王法胜, 陈兴文. 室内移动机器人的SLAM算法综述[J]. 计算机科学, 2021, 48(9): 223-234.
[9] 谢良旭, 李峰, 谢建平, 许晓军. 基于融合神经网络模型的药物分子性质预测[J]. 计算机科学, 2021, 48(9): 251-256.
[10] 冯霞, 胡志毅, 刘才华. 跨模态检索研究进展综述[J]. 计算机科学, 2021, 48(8): 13-23.
[11] 周文辉, 石敏, 朱登明, 周军. 基于残差注意力网络的地震数据超分辨率方法[J]. 计算机科学, 2021, 48(8): 24-31.
[12] 王立梅, 朱旭光, 汪德嘉, 张勇, 邢春晓. 基于深度学习的民事案件判决结果分类方法研究[J]. 计算机科学, 2021, 48(8): 80-85.
[13] 郭琳, 李晨, 陈晨, 赵睿, 范仕霖, 徐星雨. 基于通道注意递归残差网络的图像超分辨率重建[J]. 计算机科学, 2021, 48(8): 139-144.
[14] 刘帅, 芮挺, 胡育成, 杨成松, 王东. 基于深度学习SuperGlue算法的单目视觉里程计[J]. 计算机科学, 2021, 48(8): 157-161.
[15] 王施云, 杨帆. 基于U-Net特征融合优化策略的遥感影像语义分割方法[J]. 计算机科学, 2021, 48(8): 162-168.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] 雷丽晖,王静. 可能性测度下的LTL模型检测并行化研究[J]. 计算机科学, 2018, 45(4): 71 -75 .
[2] 廖湖声, 黄珊珊, 徐俊刚, 刘仁峰. Spark性能优化技术研究综述[J]. 计算机科学, 2018, 45(7): 7 -15 .
[3] 李佳艺, 赵宇, 王莉. 一种多结构及文本融合的网络表征方法[J]. 计算机科学, 2018, 45(7): 38 -41 .
[4] 王慧研,张腾飞,马福民. 基于空间距离自适应权重度量的粗糙K-means算法[J]. 计算机科学, 2018, 45(7): 190 -196 .
[5] 王飞,易绵竹,谭新. 基于本体语义网络的语言理解模型[J]. 计算机科学, 2018, 45(6A): 101 -105 .
[6] 宫海彦,耿生玲. 复杂障碍空间中基于移动对象运动规律的不确定轨迹预测[J]. 计算机科学, 2018, 45(6A): 130 -134 .
[7] 张旋,周乐,侯爱华. 一种适用于MLC闪存的CCI噪声均衡化算法[J]. 计算机科学, 2018, 45(6A): 541 -544 .
[8] 毛峡, 王岚, 李建军. 基于中心一致性敏感直方图的图像联合分割算法[J]. 计算机科学, 2018, 45(8): 28 -35 .
[9] 张新明, 程金凤, 康强, 王霞. 改进的混合蛙跳算法及其在多阈值图像分割中的应用[J]. 计算机科学, 2018, 45(8): 54 -62 .
[10] 赵艺琳, 姜麟, 米允龙, 李金海. 基于加权粒度和优势关系的程度多粒度粗糙集近似集的动态并行更新算法[J]. 计算机科学, 2018, 45(10): 11 -20 .