计算机科学 ›› 2018, Vol. 45 ›› Issue (12): 229-234.doi: 10.11896/j.issn.1002-137X.2018.12.038

• 图形图像与模式识别 • 上一篇    下一篇

基于生成对抗网络的图像修复

孙全, 曾晓勤   

  1. (河海大学计算机与信息学院智能科学与技术研究所 南京 211100)
  • 收稿日期:2017-11-01 出版日期:2018-12-15 发布日期:2019-02-25
  • 作者简介:孙 全(1993-),男,硕士生,主要研究方向为神经网络、图像处理,E-mail:905152434@qq.com;曾晓勤(1957-),男,博士,教授,主要研究方向为神经网络、机器学习,E-mail:xzeng@hhu.edu.cn(通信作者)。
  • 基金资助:
    本文受国家重点研发计划项目(2017YFC0405805)资助。

Image Inpainting Based on Generative Adversarial Networks

SUN Quan, ZENG Xiao-qin   

  1. (Institute of Intelligence Science and Technology,College of Computer and Information,Hohai University,Nanjing 211100,China)
  • Received:2017-11-01 Online:2018-12-15 Published:2019-02-25

摘要: 针对现有图像修复算法存在受损区域的形状和大小受限以及修复痕迹明显、修复边缘不连续的问题,文中提出一种基于生成对抗网络的图像修复方法。该方法采用生成对抗网络(Generative Adversarial Networks,GAN)这种新的生成模型作为基本架构,结合Wasserstein距离,同时融入条件对抗网络(CGAN)的思想;以破损图像作为附加条件信息,采用对抗损失与内容损失相结合的方式来训练网络模型,以修复破损区域。此方法能够修复大多数破损情况下的图像。在CelebA和LFW两个数据集上的实验结果表明,所提方法能够取得很好的修复效果。

关键词: Wasserstein距离, 对抗学习, 生成对抗网络, 图像修复

Abstract: Aiming at the problems of the restricted shape and size of the damaged area,the obvious inpainting tracks and the discontinuous inpainting edge in the existing image impainting algorithms,this paper proposed an image inpainting method based on generative adversarial networks.In this method,a new generative model named generative adversarial networks(GAN) is used as the basic framework with combining Wasserstein distance and the idea of conditional genera-tive adversarial networks(CGAN).The network receives the damaged image as additional conditional information and combines adversarial loss with content loss to train the network model for restoring pixels of missing areas.This method can be used to repair most of the damages in images.The experimental results on two datasets of CelebA and LFW suggest the capability of this method to obtain good performance.

Key words: Adversarial learning, Generative adversarial networks(GAN), Image inpainting, Wasserstein distance

中图分类号: 

  • TP391.41
[1]CHAN T F,SHEN J.Mathematical Models for Local Nontexture Inpaintings[J].Siam Journal on Applied Mathematics,2002,62(3):1019-1043.
[2]BARNES C,SHECHTMAN E,FINKELSTEIN A,et al.PatchMatch:a randomized correspondence algorithm for structural image editing[J].Acm Transactions on Graphics,2009,28(3):1-11.
[3]PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context Encoders:Feature Learning by Inpainting[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas:IEEE Computer Society,2016:2536-2544.
[4]GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al.Generative Adversarial Networks[C]∥Proceedings of the Conference on Advances in Neural Information Processing Systems.Montreal,Canada:Curran Associates,2014:2672-2680.
[5]DENTON E,GROSS S,FERGUS R.Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks[J].arXiv preprint arXiv:1611.06430,2016.
[6]SIMONYAN K,ZISSERMAN A.Very Deep Convolutional Networks for Large-Scale Image Recognition[J].arXiv preprint arXiv:1409.1556,2014.
[7]YEH R A,CHEN C,LIM T Y,et al.Semantic Image Inpainting with Deep Generative Models[C]∥Proceedings of IEEE Confe-rence on Computer Vision and Pattern Recognition.Honolulu,HI,USA:IEEE Computer Society,2017:6882-6890.
[8]RADFORD A,METZ L,CHINTALA S.Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[J].arXiv preprint arXiv:1511.06434,2015.
[9]ARJOVSKY M,CHINTALA S,BOTTOU L.Wasserstein gan[J].arXiv preprint arXiv:1701.07875,2017.
[10]RONNEBERGER O,FISCHER P,BROX T.U-Net:Convolu-tional Networks for Biomedical Image Segmentation[C]∥Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention.Munich:Springer Cham,2015:234-241.
[11]WANG K F,GOU C,DUAN Y J,et al.Generative adversarial networks:the state of the art and beyond[J].Acta Automatica Sinica,2017,43(3):321-332.(in Chinese)
王坤峰,苟超,段艳杰,等.生成式对抗网络GAN 的研究进展与展望[J].自动化学报,2017,43(3):321-332.
[12]RATLIFF L J,BURDEN S A,SASTRY S S.Characterization and computation of local Nash equilibria in continuous games[C]∥Proceedings of the 2013 51st Annual Allerton Conference on Communication,Control,and Computing.Allerton:IEEE Computer Society,2013:917-924.
[13]MIRZA M,OSINDERO S.Conditional Generative AdversarialNets[J].arXiv preprint arXiv:1411.1784,2014.
[14]IOFFE S,SZEGEDY C.Batch Normalization:Accelerating Deep Network Training by Reducing Internal Covariate Shift[C]∥Proceedings of the 32nd International Conference on Machine Learning.Lille,France:International Machine Learning Society,2015:448-456.
[15]TREFNY J,MATAS J.Extended Set of Local Binary Patterns for Rapid Object Detection[C]∥Proceedings of the 15th Computer Vision Winter Workshop.Nove Hrady,Czech Republic:CVWW,2010:1-7.
[16]VIOLA P,JONES M.Rapid object detection using a boostedcascade of simple features[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Kauai:IEEE Computer Society,2001:511-518.
[17]GULRAJANI I,AHMED F,ARJOVSKY M,et al.ImprovedTraining of Wasserstein GANs[C]∥Proceedings of Annual Conference on Neural Information Processing Systems.Long Beach,CA,USA,2017:5769-5779.
[18]KINGMA D P,BA J.Adam:A Method for Stochastic Optimization[J].arXiv preprint arXiv:1412.6980,2014.
[19]GANGNET M,BLAKE A.Poisson image editing[C]∥Procee-dings of the ACM SIGGRAPH.New York:Association for Computing Machinery,2003:313-318.
[1] 张佳, 董守斌.
基于评论方面级用户偏好迁移的跨领域推荐算法
Cross-domain Recommendation Based on Review Aspect-level User Preference Transfer
计算机科学, 2022, 49(9): 41-47. https://doi.org/10.11896/jsjkx.220200131
[2] 曹晓雯, 梁美玉, 鲁康康.
基于细粒度语义推理的跨媒体双路对抗哈希学习模型
Fine-grained Semantic Reasoning Based Cross-media Dual-way Adversarial Hashing Learning Model
计算机科学, 2022, 49(9): 123-131. https://doi.org/10.11896/jsjkx.220600011
[3] 孙奇, 吉根林, 张杰.
基于非局部注意力生成对抗网络的视频异常事件检测方法
Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection
计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[4] 戴朝霞, 李锦欣, 张向东, 徐旭, 梅林, 张亮.
基于DNGAN的磁共振图像超分辨率重建算法
Super-resolution Reconstruction of MRI Based on DNGAN
计算机科学, 2022, 49(7): 113-119. https://doi.org/10.11896/jsjkx.210600105
[5] 尹文兵, 高戈, 曾邦, 王霄, 陈怡.
基于时频域生成对抗网络的语音增强算法
Speech Enhancement Based on Time-Frequency Domain GAN
计算机科学, 2022, 49(6): 187-192. https://doi.org/10.11896/jsjkx.210500114
[6] 徐辉, 康金梦, 张加万.
基于特征感知的数字壁画复原方法
Digital Mural Inpainting Method Based on Feature Perception
计算机科学, 2022, 49(6): 217-223. https://doi.org/10.11896/jsjkx.210500105
[7] 高志宇, 王天荆, 汪悦, 沈航, 白光伟.
基于生成对抗网络的5G网络流量预测方法
Traffic Prediction Method for 5G Network Based on Generative Adversarial Network
计算机科学, 2022, 49(4): 321-328. https://doi.org/10.11896/jsjkx.210300240
[8] 黎思泉, 万永菁, 蒋翠玲.
基于生成对抗网络去影像的多基频估计算法
Multiple Fundamental Frequency Estimation Algorithm Based on Generative Adversarial Networks for Image Removal
计算机科学, 2022, 49(3): 179-184. https://doi.org/10.11896/jsjkx.201200081
[9] 石达, 芦天亮, 杜彦辉, 张建岭, 暴雨轩.
基于改进CycleGAN的人脸性别伪造图像生成模型
Generation Model of Gender-forged Face Image Based on Improved CycleGAN
计算机科学, 2022, 49(2): 31-39. https://doi.org/10.11896/jsjkx.210600012
[10] 唐雨潇, 王斌君.
基于深度生成模型的人脸编辑研究进展
Research Progress of Face Editing Based on Deep Generative Model
计算机科学, 2022, 49(2): 51-61. https://doi.org/10.11896/jsjkx.210400108
[11] 李建, 郭延明, 于天元, 武与伦, 王翔汉, 老松杨.
基于生成对抗网络的多目标类别对抗样本生成算法
Multi-target Category Adversarial Example Generating Algorithm Based on GAN
计算机科学, 2022, 49(2): 83-91. https://doi.org/10.11896/jsjkx.210800130
[12] 谈馨悦, 何小海, 王正勇, 罗晓东, 卿粼波.
基于Transformer交叉注意力的文本生成图像技术
Text-to-Image Generation Technology Based on Transformer Cross Attention
计算机科学, 2022, 49(2): 107-115. https://doi.org/10.11896/jsjkx.210600085
[13] 陈贵强, 何军.
自然场景下遥感图像超分辨率重建算法研究
Study on Super-resolution Reconstruction Algorithm of Remote Sensing Images in Natural Scene
计算机科学, 2022, 49(2): 116-122. https://doi.org/10.11896/jsjkx.210700095
[14] 侯宏旭, 孙硕, 乌尼尔.
蒙汉神经机器翻译研究综述
Survey of Mongolian-Chinese Neural Machine Translation
计算机科学, 2022, 49(1): 31-40. https://doi.org/10.11896/jsjkx.210900006
[15] 蒋宗礼, 樊珂, 张津丽.
基于生成对抗网络和元路径的异质网络表示学习
Generative Adversarial Network and Meta-path Based Heterogeneous Network Representation Learning
计算机科学, 2022, 49(1): 133-139. https://doi.org/10.11896/jsjkx.201000179
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!