计算机科学 ›› 2019, Vol. 46 ›› Issue (3): 119-124.doi: 10.11896/j.issn.1002-137X.2019.03.017

• 2018 中国多媒体大会 • 上一篇    下一篇

深度卷积先验引导的鲁棒图像层分离方法及其应用

姜智颖,刘日升   

  1. (大连理工大学-立命馆大学国际信息与软件学院 辽宁 大连 116621)
    (辽宁省泛在网络与服务软件重点实验室 辽宁 大连 116621)
  • 收稿日期:2018-07-22 修回日期:2018-09-25 出版日期:2019-03-15 发布日期:2019-03-22
  • 通讯作者: 刘日升(1984-),男,博士,副教授,CCF会员,主要研究方向为机器学习、优化方法、计算机视觉和多媒体技术,E-mail:rsliu@dlut.edu.cn(通信作者)。
  • 作者简介:姜智颖(1995-),女,硕士生,主要研究方向为图像处理,E-mail:zyjiang@mail.dlut.edu.cn
  • 基金资助:
    国家自然科学基金(61672125,61733002,61572096,61432003,61632019)资助

Deep Convolutional Prior Guided Robust Image Separation Method and Its Applications

JIANG Zhi-ying, LIU Ri-sheng   

  1. (Dalian University of Technology-Ritsumeikan University International School of Information Science & Engineering,Dalian,Liaoning 116621,China)
    (Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province,Dalian,Liaoning 116621,China)
  • Received:2018-07-22 Revised:2018-09-25 Online:2019-03-15 Published:2019-03-22

摘要: 图像层分离是根据任务需要将观测图像分解成两个独立且具有实际意义的组成成分。图像恢复领域中的很多问题在本质上都可以被理解为两个不同层的分离,如自然图像去雨、本质图像分解、反射层去除等。因此,做好图像层分解工作对解决这些问题有极大的推动作用。由于这个问题的求解具有病态性,已有的方法大多都是通过设计一个复杂的模型先验来约束所需要的两层。然而,复杂的先验会导致目标函数难以被优化求解,算法的有效性也不能很好地发挥出来。更重要的是,这些方法只能针对其中某一个特定的任务,不能迁移到其他应用上,泛化能力不强。为了弥补上述不足,文中提出了一个自适应的灵活优化框架,将深度网络整合到图像层分离迭代过程中。近年来,深度学习在特征提取上的优势逐步体现,在低级图像处理领域也越来越多地被采用。因此,该算法使用深度结构替代传统模型先验来刻画不同层的特征,同时,为了减少网络对训练数据的依赖并提升算法的有效性,将深度信息与传统优化框架相结合。具体来讲,首先基于MAP(最大后验概率)建立能量函数。然后使用ADMM(交替方向乘子法)将该模型分解成3个子问题。前两个子问题用来估计分离的两层,其中鉴于卷积操作在特征刻画上的优势,使用其作为隐式先验刻画问题属性;最后一个子问题通过优化的手段得到最终的精确结果。通过深度卷积先验引导优化迭代,既保持了深度结构对特征的刻画优势,又保留了传统模型优化的稳定性。最后,将所提方法应用到多种图像恢复问题上,包括自然图像去雨和反射层去除。与同类方法的定性与定量比较表明,所提方法在可视效果和数值结果上均表现出了极大的优势,证明了该方法具有强大的泛化能力和有效性。

关键词: 残差网络, 层分离, 反射层去除, 自然图像去雨, 最大后验概率估计

Abstract: Single image layer separation aims to divide the observed image into two independent and practical components based on the requirement of tasks.Many tasks in computer vision can be understood as the separation of two different layers essentially,such as single image rain streak removal,intrinsic image decomposition and reflection removal.Therefore,an excellent image layer decomposition method would promote the solution of these problems greatly.Since there is only one known variable,two variables need to be recovered.This problem is fundamentally ill-posed.Most exis-ting approaches tend to design complex priors according to the different characteristics between the two separated layers.However,loss function with complex prior regularization is hard to be optimized.Performance is also compromised by the fixed iteration schemes and less data fitting ability.More importantly,these conventional prior based methods can only be applied to one specific task as they are weak in generalization.To partially mitigate the limitations mentioned above,this paper developed a flexible optimization technique to incorporate deep architectures into optimization iterations for adaptive image layer separation.As we all know,the convolutional neural network is a network structure composed of convolutions and other non-linear operations.

第3期姜智颖,等:深度卷积先验引导的鲁棒图像层分离方法及其应用
The convolution operation uses different convolution kernels to extract different features for a given image,so the convolution kernel has very strong capabilities for feature extraction.Recently,the advantages of deep learning in feature extraction have been gradually reflected and are increasingly used in the low-level image processing.Therefore,the proposed method uses deep convolutional prior instead of traditional model prior to characterize different layers.At the same time,in order to reduce the network’s dependence on training data and improve the effectiveness of the algorithm on different tasks,deep information is combined with traditional optimization framework.Specifically,energy function using MAP (Maximum A Posteriori) is built and then the model is transfered to three subproblems based on ADMM (Alternating Direction Method of Multipliers).The first two subproblems are to estimate two approximate separated layers,and the other subproblem is to solve the final result.In other words,deep convolutional networks are used to guide the process of model optimization.In this way,the proposed method not only retains the advantage of feature extraction in deep structure,but also maintains the stability of traditional model optimization and improves the effectiveness of networks.Finally,this method is applied to a variety of ima-ge restoration tasks,including single image rain streak removal and reflection removal.By comparing this method with several tasks-specific methods including conventional model methods and deep learning methods respectively,this me-thod shows great advantages in both visual effects and numerical results.It reveals that this method has a strong genera-lization in multi-tasks and outperforms other methods in each task.

Key words: Layer separation, Maximum a posteriori estimate, Rain streak removal, Reflection removal, Residual network

中图分类号: 

  • TP391
[1]GU S H,MENG D Y,ZUO W M,et al.Joint Convolutional
Analysis and Synthesis Sparse Representation for Single Image Layer Separation[C]∥Proceedings of the IEEE International Conference on Computer Vision.Venice,Italy,2017:1717-1725.
[2]LIANG Z T,XU J,ZHANG D,et al.A Hybrid l1l0 Layer Decomposition Model for Tone Mapping[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Salt Lake City,USA,2018.
[3]KANG L W,LIN C W,FU Y H.Automatic single-image-based rain streaks removal via image decomposition[J].IEEE Transactions on Image Processing,2012,21(4):1742-1755.
[4]HUANG D A,KANG L W,YANG M C,et al.Context-Aware
Single Image Rain Removal[C]∥Proceedings of the IEEE International Conference on Multimedia and Expo.Melbourne,Australia,2012:164-169.
[5]LUO Y,XU Y,JI H.Removing rain from a single image via discriminative sparse coding[C]∥Proceedings of the IEEE International Conference on Computer Vision.Santiago,Chile,2015:3397-3405.
[6]LI Y,TAN R T,GUO X J,et al.Rain streak removal using layer priors[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,USA,2016:2736-2744.
[7]FU X Y,HUANG J B,DING X H,et al.Clearing the Skies:A deep network architecture for single-image rain removal[J].IEEE Transactions on Image Processing,2017,26(6):2944-2956.
[8]FU X Y,HUANG J B,ZENG D L,et al.Removing Rain from Single Images via a Deep Detail Network[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,Hawaii,2017:3855-3863.
[9]YANG W H,TAN R T,FENG J S,et al.DeepJoint Rain Detection and Removal from a Single Image[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,Hawaii,2017:1357-1366.
[10]LI Y,BROWN M S.Single Image Layer Separation Using Relative Smoothness[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Columbus,USA,2014:2752-2759.
[11]SHIH Y C,KRISHNAN D,DURAND F,et al.Reflection re-
moval using ghosting cues[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Boston,USA,2015:3193-3201.
[12]SPRINGER O,WEISS Y.Reflection separation using guided annotation[C]∥Proceedings of the IEEE International Conference on Image Processing.Beijing,China,2017:1192-1196.
[13]ZHANG K,ZUO W M,GU S H,et al.Learning deep CNN denoiser prior for image restoration[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,Hawaii,2017.
[14]BOYD S,PARIKH N,CHU E,et al.Distributed optimization
and statistical learning via the alternating direction method of multipliers[J].Foundations and Trends in Machine Learning,2011,3(1):1-122.
[15]ZHU C,BYRD R H,LU P,et al.Algorithm 778:L-BFGS-B:Fortran subroutines for large-scale bound-constrained optimization[J].ACM Transactions on Mathematical Software,1997,23(4):550-560.
[16]MARTIN D,FOWLKES C,TAL D,et al.A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]∥Proceedings of the IEEE International Conference on Computer Vision.Vancouver,Canada,2001:416-423.
[17]GARG K,NAYAR S K.Photorealistic rendering of rain streaks[J].ACM Transactions on Graphics,2006,25(3):996-1002.
[18]LI Y,BROWN M S.Exploiting reflection change for automatic reflection removal[C]∥Proceedings of the IEEE International Conference on Computer Vision.Sydney,Australia,2013:2432-2439.
[1] 王馨彤, 王璇, 孙知信.
基于多尺度记忆残差网络的网络流量异常检测模型
Network Traffic Anomaly Detection Method Based on Multi-scale Memory Residual Network
计算机科学, 2022, 49(8): 314-322. https://doi.org/10.11896/jsjkx.220200011
[2] 高荣华, 白强, 王荣, 吴华瑞, 孙想.
改进注意力机制的多叉树网络多作物早期病害识别方法
Multi-tree Network Multi-crop Early Disease Recognition Method Based on Improved Attention Mechanism
计算机科学, 2022, 49(6A): 363-369. https://doi.org/10.11896/jsjkx.210500044
[3] 赵人行, 徐频捷, 刘瑶.
基于深度卷积残差网络的心电单导联房颤检测方法
ECG-based Atrial Fibrillation Detection Based on Deep Convolutional Residual Neural Network
计算机科学, 2022, 49(5): 186-193. https://doi.org/10.11896/jsjkx.220200002
[4] 韩红旗, 冉亚鑫, 张运良, 桂婕, 高雄, 易梦琳.
基于共同子空间分类学习的跨媒体检索研究
Study on Cross-media Information Retrieval Based on Common Subspace Classification Learning
计算机科学, 2022, 49(5): 33-42. https://doi.org/10.11896/jsjkx.210200157
[5] 高心悦, 田汉民.
基于改进U-Net网络的液滴分割方法
Droplet Segmentation Method Based on Improved U-Net Network
计算机科学, 2022, 49(4): 227-232. https://doi.org/10.11896/jsjkx.210300193
[6] 张红民, 李萍萍, 房晓冰, 刘宏.
改进YOLOv3网络模型的人体异常行为检测方法
Human Abnormal Behavior Detection Method Based on Improved YOLOv3 Network Model
计算机科学, 2022, 49(4): 233-238. https://doi.org/10.11896/jsjkx.210300251
[7] 瞿中, 陈雯.
基于空洞卷积和多特征融合的混凝土路面裂缝检测
Concrete Pavement Crack Detection Based on Dilated Convolution and Multi-features Fusion
计算机科学, 2022, 49(3): 192-196. https://doi.org/10.11896/jsjkx.210100164
[8] 郭琳, 李晨, 陈晨, 赵睿, 范仕霖, 徐星雨.
基于通道注意递归残差网络的图像超分辨率重建
Image Super-resolution Reconstruction Using Recursive ResidualNetwork Based on ChannelAttention
计算机科学, 2021, 48(8): 139-144. https://doi.org/10.11896/jsjkx.200500150
[9] 许华杰, 张晨强, 苏国韶.
基于深层卷积残差网络的航拍图建筑物精确分割方法
Accurate Segmentation Method of Aerial Photography Buildings Based on Deep Convolutional Residual Network
计算机科学, 2021, 48(8): 169-174. https://doi.org/10.11896/jsjkx.200500096
[10] 暴雨轩, 芦天亮, 杜彦辉, 石达.
基于i_ResNet34模型和数据增强的深度伪造视频检测方法
Deepfake Videos Detection Method Based on i_ResNet34 Model and Data Augmentation
计算机科学, 2021, 48(7): 77-85. https://doi.org/10.11896/jsjkx.210300258
[11] 王建明, 黎向锋, 叶磊, 左敦稳, 张丽萍.
基于信道注意结构的生成对抗网络医学图像去模糊
Medical Image Deblur Using Generative Adversarial Networks with Channel Attention
计算机科学, 2021, 48(6A): 101-106. https://doi.org/10.11896/jsjkx.200600144
[12] 牛康力, 谌雨章, 张龚平, 谭前程, 王绎冲, 罗美琪.
基于深度学习的无人机航拍车流量监测
Vehicle Flow Measuring of UVA Based on Deep Learning
计算机科学, 2021, 48(6A): 275-280. https://doi.org/10.11896/jsjkx.200900149
[13] 鲍志强, 陈卫东.
基于最大后验估计的谣言源定位器
Rumor Source Detection in Social Networks via Maximum-a-Posteriori Estimation
计算机科学, 2021, 48(4): 243-248. https://doi.org/10.11896/jsjkx.200400053
[14] 龚航, 刘培顺.
夜间行驶车辆远光灯检测方法
Detection Method of High Beam in Night Driving Vehicle
计算机科学, 2021, 48(12): 256-263. https://doi.org/10.11896/jsjkx.200700026
[15] 柴冰, 李冬冬, 王喆, 高大启.
融合频率和通道卷积注意的脑电(EEG)情感识别
EEG Emotion Recognition Based on Frequency and Channel Convolutional Attention
计算机科学, 2021, 48(12): 312-318. https://doi.org/10.11896/jsjkx.201000141
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!