计算机科学 ›› 2021, Vol. 48 ›› Issue (9): 174-180.doi: 10.11896/jsjkx.200800014

• 计算机图形学&多媒体 • 上一篇    下一篇

利用生成对抗网络的人脸图像分步补全法

林椹尠1, 张梦凯2, 吴成茂3, 郑兴宁2   

  1. 1 西安邮电大学理学院 西安710121
    2 西安邮电大学通信与信息工程学院 西安710121
    3 西安邮电大学电子工程学院 西安710121
  • 收稿日期:2020-08-03 修回日期:2020-10-29 出版日期:2021-09-15 发布日期:2021-09-10
  • 通讯作者: 张梦凯(zmkdyx@163.com)
  • 作者简介:lzhx126@126.com
  • 基金资助:
    国家自然科学基金(61671377)

Face Image Inpainting with Generative Adversarial Network

LIN Zhen-xian1, ZHANG Meng-kai2, WU Cheng-mao3, ZHENG Xing-ning2   

  1. 1 School of Science,Xi'an University of Posts & Telecommunications,Xi'an 710121,China
    2 School of Communication and Information Engineering,Xi'an University of Posts & Telecommunications,Xi'an 710121,China
    3 School of Electronic Engineering,Xi'an University of Posts & Telecommunications,Xi'an 710121,China
  • Received:2020-08-03 Revised:2020-10-29 Online:2021-09-15 Published:2021-09-10
  • About author:LIN Zhen-xian,born in 1969,Ph.D,professor.Her main research interests include wavelet theory and its application in signal and image.
    ZHANG Meng-kai,born in 1995,postgraduate.His main research interests include deep learning and computer vision.
  • Supported by:
    National Natural Science Foundation of China(61671377)

摘要: 人脸图像修复技术是近年来图像处理领域的研究热点,而人脸图像大面积缺失导致损失语义信息过多,一直是该领域的重点难点问题。针对这一问题,文中提出了一种基于生成对抗网络的图像分步补全算法。将人脸图像修复问题分为两步,设计两个串联的生成对抗网络,首先残缺图像通过预补全网络进行图像的预补全,预补全图像进入增强网络进行特征增强;判别器分别判断预补全图像和增强图像与理想图像的差异性;采用长短时记忆单元连接两部分的信息流,增强信息的传递。然后使用内容损失、对抗损失和全变分损失相结合的损失函数,提高网络的修复效果。最后在CelebA数据集上进行实验,结果显示,所提算法相较于对比算法在峰值信噪比指标上提高了16.84%~22.85%,在结构相似性指标上提高了10%~12.82%。

关键词: 长短时记忆, 缺失区域, 人脸图像, 深度学习, 生成对抗网络, 跳跃连接, 图像补全

Abstract: Face image inpainting is a hot topic of image processing research in recent years.Due to the loss of excessive sematic information,it is a difficult problem to inpaint large area missing of face images.Aiming at the problem of inpainting face images,a step-by-step image inpainting algorithm based on generative adversarial network is proposed.Face images inpainting task is divided into two steps.Firstly,face images are completed through the pre-completion network,and pre-completion images is enhanced feature through the enhancement network.The discriminator judges the difference between the pre-completion images,the enhanced images and the ideal image respectively.The long-term memory unit is used to connect the information flow of two parts.Secondly,the adversarial loss,content loss and total variation loss are combined to improve the effectively.Experiments are conducted on CelebA dataset,and this algorithm has an improvement of 16.84%~22.85% in PSNR and 10%~12.82% in SSIM compared with others typical image inpainting algorithms

Key words: Deep learning, Face images, Generative adversarial networks, Image inpainting, Long short term memory, Loss areas, Skip connection

中图分类号: 

  • TP183
[1]HE Y T,TANG X H,ZHANG Y,et al.Improved Criminisi algorithm based on structure tensor[J].Journal of Image and Graphics,2018,23(10):1492-1507.
[2]LAN X L,LIU H X,YAO H B.Improved image inpainting algorithm based on texture blocks and gradient feature[J].Computer Engineering and Applications,2018,54(20):172-177.
[3]KRHENBÜHL P,KOLTUN V.Efficient Inference in FullyConnected CRFs with Gaussian Edge Potentials[J].Advances in Neural Information Processing Systems,2012,24(2011):109-117.
[4]XU L M,WU Y J,ZHANG B.Image Inpainting AlgorithmBased on Adaptive High Order Variation in Eight Neighbors[J].Journal of Graphics,2017,38(4):558-565.
[5]BERTALMIO M,SAPIRO G,CASELLES V,et al.Image in-painting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques.ACM Press/Addison-Wesley Publishing Co.,2000:417-424.
[6]LEVIN A,ZOMET A,WEISS Y.Learning how to inpaint from global mage statistics[C]//IEEE In International Conference on Computer Vision.2003:305-312.
[7]SHEN J,CHAN T F.Mathematical Models for Local Nontexture Inpaintings[J].SIAM Journal on Applied Mathematics,2001,62:1019-1043.
[8]SHEN J,KANG S H,CHAN T F.Euler's Elastica and Curvature-Based Inpainting[J].SIAM Journal on AppliedMathema-tics,2002,63(2):564-592.
[9]BOUREAU Y L,BACH F,LECUN Y,et al.Learning mid-level features for recognition[C]//2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.IEEE,2010,2559-2566.
[10]LI S,ZHAO M.Image inpainting with salient structure completion and texture propagation[J].Pattern Recognition Letters,2011,32(9):1256-1266.
[11]EFROS A A,LEUNG T K.Texture synthesis by non-parame-tric sampling[C]//Proceedings of the Seventh IEEE Internatio-nal Conference on Computer Vision.1999,2:1033-1038.
[12]CRIMINISI A,PÉREZ P,TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing,2004,13(9):1200-1212.
[13]BARNES C,SHECHTMAN E,FINKELSTEIN A,et al.PatchMatch:A randomized correspondence algorithm for structural image editing[C]//ACM Transactions on Graphics (ToG).ACM,2009:1-11.
[14]WEI Y,LIU S.Domain-based structure-aware image inpainting[J].Signal Image & Video Processing,2016,10(5):911-919.
[15]GIRSHICK R,DONAHUE J,DARRELL T,et al.Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2014:580-587.
[16]LONG J,SHELHAMER E,DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:3431-3440.
[17]XIE J,XU L,CHEN E.Image denoising and inpainting withdeep neural networks[C]//Advances in Neural Information Processing Systems.2012:341-349.
[18]KÖHLER R,SCHULER C,SCHÖLKOPF B,et al.Mask.specific inpainting with deep neural networks[C]//German Confe-rence on Pattern Recognition.Cham:Springer,2014:523-534.
[19]GOODFELLOW I,POUGET-ABADIE J,MIRZAM,et al.Gene-rative adversarial nets[C]//Advances in Neural Information Processing Systems.2014:2672-2680.
[20]PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context encoders:Feature learning by inpainting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2536-2544.
[21]YANG C,LU X,LIN Z,et al.High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:6721-6729.
[22]IIZUKA S S,SERRA E,SHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Gra-phics (TOG),2017,36(4):107:1-2.
[23]YU J H,ZHE L,YANG J M,et al.Generative Image Inpainting with Contextual Attention [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.2018:5505-5514.
[24]LIU G,REDA F A,SHIH K J,et al.Image inpainting for irre-gular holes using partial convolutions [C]//Proceedings of the European Conference on Computer Vision (ECCV).2018:85-100.
[25]HOCHREITER S,SCHMIDHUBER J.Long Short-Term Me-mory[J].Neural Computation,1997,9(8):1735-1780.
[26]YEH R A,CHEN C,LIM T Y,et al.Semantic Image Inpainting with Deep Generative Models [C]//CVPR 2016.2016:5485-5493.
[27]ISOLA P,ZHU J Y,ZHOU T H,et al.Image-to-image translation with conditional adversarial networks[C]//IEEEConfe-rence on Computer Vision and Pattern Recognition (CVPR).2017.
[28]RONNEBERGER O,FISCHER P,BROX T.U-Net:Convolu-tional Networks for Biomedical Image Segmentation[C]//International Conference on Medical Image Computing and Computer-assisted Intervention.Cham:Springer,2015:234-241.
[29]RADFORD A,METZ L,CHINTALA S.Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks [C]//Proceedings of the International Confe-rence on Learning Representations (ICLR).2016.
[30]ZHAO H,GALLO O,FROSIO I,et al.Loss functions for image restoration with neural networks[J].IEEE Transactions on Computational Imaging,2016,3(1):47-57.
[31]WHYTE O,SIVIC J,ZISSERMAN A,et al.Non-uniform deblurring for shaken images[J].International Journal of Compu-ter Vision,2012,98(2):168-186.
[32]RUDIN L I,OSHER S,FATEMI E.Nonlinear total variation based noise removal algorithms[J].Physica D Nonlinear Phenomena,1992,60(1/2/3/4):259-268.
[33]NAH S,KIM T H,LEE K M.Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:3883-3891.
[34]TONG Y B,ZHANG Q S,QI Y P.Image Quality Assessing by Combining PSNR with SSIM[J].Journal of Image and Gra-phics,2006,11(12),1758-1763.
[35]WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality as-sessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[36]ZHENG C X,CHAM T J,CAI J.Pluralistic Image Completion [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).2019:1438-1447.
[37]CIMPOI M,MAJI S,KOKKINOS I,et al.Describing Textures in the Wild[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2014:3606-3613.
[1] 饶志双, 贾真, 张凡, 李天瑞.
基于Key-Value关联记忆网络的知识图谱问答方法
Key-Value Relational Memory Networks for Question Answering over Knowledge Graph
计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277
[2] 汤凌韬, 王迪, 张鲁飞, 刘盛云.
基于安全多方计算和差分隐私的联邦学习方案
Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy
计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108
[3] 张佳, 董守斌.
基于评论方面级用户偏好迁移的跨领域推荐算法
Cross-domain Recommendation Based on Review Aspect-level User Preference Transfer
计算机科学, 2022, 49(9): 41-47. https://doi.org/10.11896/jsjkx.220200131
[4] 徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺.
时序知识图谱表示学习
Temporal Knowledge Graph Representation Learning
计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204
[5] 王剑, 彭雨琦, 赵宇斐, 杨健.
基于深度学习的社交网络舆情信息抽取方法综述
Survey of Social Network Public Opinion Information Extraction Based on Deep Learning
计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099
[6] 郝志荣, 陈龙, 黄嘉成.
面向文本分类的类别区分式通用对抗攻击方法
Class Discriminative Universal Adversarial Attack for Text Classification
计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077
[7] 姜梦函, 李邵梅, 郑洪浩, 张建朋.
基于改进位置编码的谣言检测模型
Rumor Detection Model Based on Improved Position Embedding
计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046
[8] 孙奇, 吉根林, 张杰.
基于非局部注意力生成对抗网络的视频异常事件检测方法
Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection
计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[9] 胡艳羽, 赵龙, 董祥军.
一种用于癌症分类的两阶段深度特征选择提取算法
Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification
计算机科学, 2022, 49(7): 73-78. https://doi.org/10.11896/jsjkx.210500092
[10] 戴朝霞, 李锦欣, 张向东, 徐旭, 梅林, 张亮.
基于DNGAN的磁共振图像超分辨率重建算法
Super-resolution Reconstruction of MRI Based on DNGAN
计算机科学, 2022, 49(7): 113-119. https://doi.org/10.11896/jsjkx.210600105
[11] 程成, 降爱莲.
基于多路径特征提取的实时语义分割方法
Real-time Semantic Segmentation Method Based on Multi-path Feature Extraction
计算机科学, 2022, 49(7): 120-126. https://doi.org/10.11896/jsjkx.210500157
[12] 侯钰涛, 阿布都克力木·阿布力孜, 哈里旦木·阿布都克里木.
中文预训练模型研究进展
Advances in Chinese Pre-training Models
计算机科学, 2022, 49(7): 148-163. https://doi.org/10.11896/jsjkx.211200018
[13] 周慧, 施皓晨, 屠要峰, 黄圣君.
基于主动采样的深度鲁棒神经网络学习
Robust Deep Neural Network Learning Based on Active Sampling
计算机科学, 2022, 49(7): 164-169. https://doi.org/10.11896/jsjkx.210600044
[14] 金方焱, 王秀利.
融合RACNN和BiLSTM的金融领域事件隐式因果关系抽取
Implicit Causality Extraction of Financial Events Integrating RACNN and BiLSTM
计算机科学, 2022, 49(7): 179-186. https://doi.org/10.11896/jsjkx.210500190
[15] 苏丹宁, 曹桂涛, 王燕楠, 王宏, 任赫.
小样本雷达辐射源识别的深度学习方法综述
Survey of Deep Learning for Radar Emitter Identification Based on Small Sample
计算机科学, 2022, 49(7): 226-235. https://doi.org/10.11896/jsjkx.210600138
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!