Computer Science ›› 2020, Vol. 47 ›› Issue (8): 202-207.doi: 10.11896/jsjkx.190700017

Previous Articles     Next Articles

Dense Convolution Generative Adversarial Networks Based Image Inpainting

MENG Li-sha, REN Kun, FAN Chun-qi, HUANG Long   

  1. 1 Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
    2 Engineering Research Center of Digital Community, Ministry of Education, Beijing 100124, China
    3 Beijing Laboratory for Urban Mass Transit, Beijing 100124, China
    4 Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing 100124, China
  • Online:2020-08-15 Published:2020-08-10
  • About author:MENG Li-sha, born in 1992, postgra-duate.Her main research interests include deep learning and computer vision.
    REN Kun, born in 1973, Ph.D, lecturer.Her main research interests include deep learning and computer vision.
  • Supported by:
    This work was supported by the National Natural Science Foundation of China (61803005, 61640312, 61763037, 1305026), Beijing Natural Science Foundation (4192011, 4172007) and Science Foundation from the Education Commission of Beijing (KM20130005006).

Abstract: Image inpainting is one technique of reconstruction defect areas by inferring information from the known context of defect images.For semantic image inpainting of large areas, there are still many problems in inpainting algorithms based on generation models, such as blur, artifacts, and poor visual similarity, especially for the complex background images and small datasets.To solve this problem, an image inpainting algorithm based on dense convolution generative adversarial networks is proposed.The generated adversarial network is the basic framework.Firstly, dense convolutional blocks are used to enhance image feature extraction, improve image repair capability, and avoid the problem of gradient disappearance caused by the network depth increa-sing in the generator network.Secondly, skip connection between the encoding and decoding structures is involved to avoid information transmission lost problems between network layers.After that, a total loss function, composed of the reconstruction loss, adversarial loss and TV loss, is used to optimize the network and enhance network stability.Finally, the proposed algorithm is validated on the CelebA dataset and Car dataset respectively, compared with three typical image inpainting algorithms.The effectiveness of the algorithm is proved in visual perception, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

Key words: Dense convolutional block, Generative adversarial networks, Image inpainting, Loss function, Skip connection

CLC Number: 

  • TP391
[1]BERTALMIO M, SAPIRO G, CASELLESV, et al.Image in-painting[C]∥Conference on Computer Graphics and Interactive Techniques.ACM Press/Addison-Wesley Publishing Company, 2000:417-424.
[2]PETERSON I.Filling in Blanks:Automating the Inpainting of a Picture's Missing Pieces [J].Science News, 2002, 161(19):299-300.
[3]CRIMINISI A, PEREZ P, TOYAMA K.Region Filling and Object Removal by Exemplar-based Image Inpainting [J].IEEE Transactions on Image Processing, 2004, 13(9):1200-1212.
[4]GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al.Gene-rative Adversarial Nets[C]∥International Conference on Neural Information Procession Systems.2014:2672-2680.
[5]PATHAK D, KRAHENBUHL P, DONAHUE J, et al.Context Encoders:Feature Learning by Inpainting[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2536-2544.
[6]YEH R A, CHEN C, LIM T Y, et al.Semantic Image Inpainting with Deep Generative Models[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:6882-6890。
[7]ZHAO G, LIU J, JIANG J, et al.A Deep Cascade of Neural Networks for Image Inpainting, Deblurring and Denoising [J].Multimedia Tools and Applications, 2018, 77(22):29589-29604.
[8]ALTINEL F, OZAY M, OKATANI T.Deep Structured Energy-Based Image Inpainting [C]∥2018 24th International Confe-rence on Pattern Recognition (ICPR).Beijing, 2018:423-428.
[9]YU J, LIN Z, YANG J, et al.Generative Image Inpainting with Contextual Attention[C]∥2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).IEEE, 2018:5505-5514
[10]RONNEBERGER O, FISCHER P, BROX T.U-Net:Convolu-tional Networks for Biomedical Image Segmentation[C]∥Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham:Sprin-ger, 2015:234-241.
[11]DOSOVISKIY A, BROX T.Generating Images with Perceptual Similarity Metrics based on Deep Networks [C]∥Advances in Neural Information Processing Systems.2016:658-666.
[12]ISOLA P, ZHU J Y, ZHOU T, et al.Image-to-Image Translation with Conditional Adversarial Networks [C]∥Computer Vision and Pattern Recognition.2016:5967-5976.
[13]LI C, WAND M.Precomputed Real-Time Texture Synthesiswith Markovian Generative Adversarial Networks[C]∥Euro-pean Conference on Computer Vision.2016:702-716.
[14]LEDIG C, THEIS L, HUSZAR F, CABALLERO J, et al.Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network [C]∥2017 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR).Honolulu, HI, 2017:105-114.
[15]HUANG G, LIU Z, MAATEN L V D, et al.Densely Connected Convolutional Networks[C]∥IEEE Conference on Computer Vision and Pattern Recognition.IEEE Computer Society, 2017:2261-2269
[16]KINGMA D P, WELLING M.Auto-Encoding Variational Bayes [C]∥Proceedings of the International Conference on Learning Representations (ICLR).2014.
[17]AKAIKE H.Fitting Autoregressive Models for Prediction [J].Annals of the Institute of Statistical Mathematics, 1969, 21(1):243-247.
[18]HINTON G E, OSINDEROS S, TEH Y W.A Fast Learning Al-gorithm for Deep Belief Nets [J].Neural Computation, 2006, 18(7):1527-1554.
[19]RADFORD A, METZ L, CHINTALA S.Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks [J].arXiv:1511.06434, 2015.
[20]JAVANMARDI M, SAJJADI M, LIU T, et al.UnsupervisedTotal Variation Loss for Semi-supervised Deep Learning of Semantic Segmentation [J].arXiv:1605.01368v3, 2016
[1] MENG Yue-bo, MU Si-rong, LIU Guang-hui, XU Sheng-jun, HAN Jiu-qiang. Person Re-identification Method Based on GoogLeNet-GMP Based on Vector Attention Mechanism [J]. Computer Science, 2022, 49(7): 142-147.
[2] GAO Rong-hua, BAI Qiang, WANG Rong, WU Hua-rui, SUN Xiang. Multi-tree Network Multi-crop Early Disease Recognition Method Based on Improved Attention Mechanism [J]. Computer Science, 2022, 49(6A): 363-369.
[3] XU Guo-ning, CHEN Yi-peng, CHEN Yi-ming, CHEN Jin-yin, WEN Hao. Data Debiasing Method Based on Constrained Optimized Generative Adversarial Networks [J]. Computer Science, 2022, 49(6A): 184-190.
[4] XU Hui, KANG Jin-meng, ZHANG Jia-wan. Digital Mural Inpainting Method Based on Feature Perception [J]. Computer Science, 2022, 49(6): 217-223.
[5] DOU Zhi, WANG Ning, WANG Shi-jie, WANG Zhi-hui, LI Hao-jie. Sketch Colorization Method with Drawing Prior [J]. Computer Science, 2022, 49(4): 195-202.
[6] GAO Zhi-yu, WANG Tian-jing, WANG Yue, SHEN Hang, BAI Guang-wei. Traffic Prediction Method for 5G Network Based on Generative Adversarial Network [J]. Computer Science, 2022, 49(4): 321-328.
[7] LI Si-quan, WAN Yong-jing, JIANG Cui-ling. Multiple Fundamental Frequency Estimation Algorithm Based on Generative Adversarial Networks for Image Removal [J]. Computer Science, 2022, 49(3): 179-184.
[8] TAN Xin-yue, HE Xiao-hai, WANG Zheng-yong, LUO Xiao-dong, QING Lin-bo. Text-to-Image Generation Technology Based on Transformer Cross Attention [J]. Computer Science, 2022, 49(2): 107-115.
[9] ZHANG Wei-qi, TANG Yi-feng, LI Lin-yan, HU Fu-yuan. Image Stream From Paragraph Method Based on Scene Graph [J]. Computer Science, 2022, 49(1): 233-240.
[10] LIN Zhen-xian, ZHANG Meng-kai, WU Cheng-mao, ZHENG Xing-ning. Face Image Inpainting with Generative Adversarial Network [J]. Computer Science, 2021, 48(9): 174-180.
[11] ZHANG Xiao-yu, WANG Bin, AN Wei-chao, YAN Ting, XIANG Jie. Glioma Segmentation Network Based on 3D U-Net++ with Fusion Loss Function [J]. Computer Science, 2021, 48(9): 187-193.
[12] HUANG Ying-qi, CHEN Hong-mei. Cost-sensitive Convolutional Neural Network Based Hybrid Method for Imbalanced Data Classification [J]. Computer Science, 2021, 48(9): 77-85.
[13] XU Tao, TIAN Chong-yang, LIU Cai-hua. Deep Learning for Abnormal Crowd Behavior Detection:A Review [J]. Computer Science, 2021, 48(9): 125-134.
[14] GUO Lin, LI Chen, CHEN Chen, ZHAO Rui, FAN Shi-lin, XU Xing-yu. Image Super-resolution Reconstruction Using Recursive ResidualNetwork Based on ChannelAttention [J]. Computer Science, 2021, 48(8): 139-144.
[15] PAN Xiao-qin, LU Tian-liang, DU Yan-hui, TONG Xin. Overview of Speech Synthesis and Voice Conversion Technology Based on Deep Learning [J]. Computer Science, 2021, 48(8): 200-208.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!