计算机科学

计算机科学 ›› 2022, Vol. 49 ›› Issue (8): 120-126.doi: 10.11896/jsjkx.220200179

• 计算机图形学& 多媒体 • 上一篇    下一篇

基于重参数化多尺度融合网络的高效极暗光原始图像降噪

魏恺轩, 付莹   

  1. 北京理工大学计算机学院 北京 100081
  • 收稿日期:2022-02-27 修回日期:2022-03-27 发布日期:2022-08-02
  • 通讯作者: 付莹(fuying@bit.edu.cn)
  • 作者简介:(kaixuan_wei@outlook.com)
  • 基金资助:
    国家自然科学基金(62171038,61827901,62088101)

Re-parameterized Multi-scale Fusion Network for Efficient Extreme Low-light Raw Denoising

WEI Kai-xuan, FU Ying   

  1. School of Computer Science and Technology,Beijing Institute of Technology,Beijing 100081,China
  • Received:2022-02-27 Revised:2022-03-27 Published:2022-08-02
  • About author:WEI Kai-xuan,born in 1996,postgra-duate.His main research interests include computer vision,computational photography and computational imaging.
    FU Ying,born in 1987,Ph.D,professor.Her main research interests include physics-based vision,image and video processing,and computational photo-graphy.
  • Supported by:
    National Natural Science Foundation of China(62171038,61827901,62088101).

PDF (PC)

631

摘要： 实用的暗光降噪增强解决方案往往需要具备计算速度快、内存效率高、能够实现视觉上高质量的降噪等优点。现有方法大多以提升降噪质量为目标,因此在速度和内存要求上有所折中,这在很大程度上限制了其实用性。文中提出了一种新的深度降噪网络——重参数化多尺度融合网络,用于极暗光单张原始图像降噪,在不损失降噪性能的同时加快模型的推断速度并降低内存开销。具体地,在多尺度空间提取图像特征,利用轻量级的空间通道并行注意力模块动态自适应地聚焦于空间及通道中的核心特征;同时使用重参数化的卷积单元,在不增加任何推断计算量的情况下进一步丰富模型的表征能力。该模型在常见CPU上(如Intel i7-7700K)可以在1s左右恢复超高清4K分辨率图像,在普通GPU(如NVIDIA GTX 1080Ti)上以24帧率的速度运行,在几乎4倍快于现有先进方法(如UNet)的同时仍保持具有竞争力的恢复质量。

关键词: 多尺度融合, 极暗光图像降噪, 空间通道并行注意力模块, 重参数化卷积单元

Abstract: Practical low-light denoising/enhancement solutions often require fast computation,high memory efficiency,and can achieve visually high-quality restoration results.Most existing methods aim to restore quality but compromise on speed and memory requirements,which limits their usefulness to a large extent.This paper proposes a new deep denoising architecture,a re-parameterized multi-scale fusion network for extreme low-light raw denoising,which greatly improves the inference speed without losing high-quality denoising performance.Specifically,image features are extracted in multi-scale space,and a lightweight spatial-channel parallel attention module is used to focus on core features within space and channel dynamically and adaptively.The representation ability of the model is further enriched by re-parameterized convolutional unit without increasing computational cost at inference.The proposed model can restore UHD 4K resolution images within about 1s on a CPU(e.g.,Intel i7-7700K) and run at 24 fps on a GPU(e.g.,NVIDIA GTX 1080Ti),which is almost four times faster than existing advanced methods(e.g.,UNet) while still maintaining competitive restoration quality.

Key words: Extreme low-light denoising, Multi-scale fusion, Re-parameterization convolutional unit, Spatial-channel parallel attention module

中图分类号: 

  • TP391
[1]DABOV K,FOI A,KATKOVNIK V,et al.Image denoising by sparse 3-d transform-domain collaborative filtering[J].IEEE Transactions on Image Processing,2007,16 (8):2080-2095.
[2]BUADES A,COLL B,MOREL J M.A non-local algorithm for image denoising [C]//IEEE Conference on Computer Vision and Pattern Recognition.2005:60-65.
[3]GU S,LEI Z,ZUO W,et al.Weighted nuclear norm minimization with application to image denoising [C]//IEEE Conference on Computer Vision and Pattern Recognition.2014:2862-2869.
[4]SCHMIDT U,ROTH S.Shrinkage fields for effective image restoration [C]//IEEE Conference on Computer Vision and Pattern Recognition.2014:2774-2781.
[5]CHEN Y,YU W,POCK T.On learning optimized reaction diffusion processes for effective image restoration [C]//IEEE Conference on Computer Vision and Pattern Recognition.2015:5261-5269.
[6]ZHANG K,ZUO W,CHEN Y,et al.Beyond a gaussian denoi-ser:residual learning of deep cnn for image denoising [J].IEEE Transactions on Image Processing,2017,26 (7):3142-3155.
[7]TAI Y,YANG J,LIU X,et al.Memnet:a persistent memorynetwork for image restoration [C]//IEEE International Confe-rence on Computer Vision.2017:4549-4557.
[8]GUO S,YAN Z,ZHANG K,et al.Toward convolutional blind denoising of real photographs [C]//IEEE Conference on Computer Vision and Pattern Recognition.2019:1712-1722.
[9]CHEN C,CHEN Q,XU J,et al.Learning to see in the dark[C]//IEEE Conference on Computer Vision and Pattern Recognition.2018:3291-3300.
[10]RONNEBERGER O,FISCHER P,BROX T.U-net:convolu-tional networks for biomedical image segmen-tation [C]//International Conference on Medical image computing and computer-assisted intervention.2015:234-241.
[11]GU S,LI Y,GOOL L V,et al.Self-guided network for fastimage denoising [C]//IEEE International Conference on Computer Vision.2019:2511-2520.
[12]OSHER S,BURGER M,GOLDFARB D,et al.An iterative regu-larization method for total variation-based image restoration [J].Multiscale Modeling and Simulation,2005,4(2):460-489.
[13]ELAD M,AHARON M.Image denoising via sparse and redundant representations over learned dictio-naries[J].IEEETransa-ctions on Image Processing,2006,15 (12):3736-3745.
[14]MALLAT S G.A theory for multiresolution signal decomposition:the wavelet representation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1989,11(7):674-693.
[15]ZHANG Y,TIAN Y,KONG Y,et al.Residual dense network for image restoration [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2020,43 (7):2480-2495.
[16]PENG Y,ZHANG L,LIU S,et al..Dilated residual networks with symmetric skip connection for image denoising [J].Neurocomputing,2019,345:67-76.
[17]BURGER H C,SCHULER C J,HARMELING S.Image denoi-sing:Can plain neural networks compete with BM3D?[C]//IEEE Conference on Computer Vision and Pattern Recognition.2012:2392-2399.
[18]ZHANG Y,LI K,LI K,et al.Residual non-local attention networks for image restoration[J].arXiv:1903.10082,2019.
[19]WEI K,FU Y,YANG J,et al.A physics-based noise formation model for extreme low-light raw denoising[C]//IEEE Confe-rence on Computer Vision and Pattern Recognition.2020:2758-2767.
[20]WEI K,FU Y,ZHENG Y,et al.Physics-based Noise Modeling for Extreme Low-light Photography[J].arXiv:2108.02158,2021.
[21]XU K,YANG X,YIN B,et al.Learning to restore low-lightimages via decomposition-and-enhancement[C]//IEEE Confe-rence on Computer Vision and Pattern Recognition.2020:2281-2290.
[22]LAMBA M,MITRA K.Restoring Extremely Dark Images in Real Time[C]//IEEE Conference on Computer Vision and Pattern Recognition.2021:3487-3497.
[23]JIANG H,ZHENG Y.Learning to see moving objects in the dark[C]//IEEE International Conference on Computer Vision.2019:7324-7333.
[24]ZHENG Y,ZHANG M,LU F.Optical flow in the dark[C]//IEEE International Conference on Computer Vision.2020:6749-6757.
[25]CHEN C,CHEN Q,DO M N,et al.Seeing motion in the dark[C]//IEEE International Conference on Computer Vision.2019:3185-3194.
[26]ARORA S,COHEN N,HAZAN E.On the optimization of deep networks:Implicit acceleration by overparameterization[C]//International Conference on Machine Learning.PMLR,2018:244-253.
[27]ZAGORUYKO S,KOMODAKIS N.Diracnets:Training verydeep neural networks without skip-connections[J].arXiv:1706.00388,2017.
[28]DING X,GUO Y,DING G,et al.Acnet:Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks[C]//IEEE International Conference on Computer Vision.2019:1911-1920.
[29]DING X,ZHANG X,MA N,et al.Repvgg:Making vgg-style convnets great again[C]//IEEE Conference on Computer Vision and Pattern Recognition.2021:13733-13742.
[30]HE K,ZHANG X,REN S,et al.Deep residual learning forimage recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[31]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[32]DING X,ZHANG X,HAN J,et al.Diverse branch block:Buil-ding a convolution as an inception-like unit[C]//IEEE Confe-rence on Computer Vision and Pattern Recognition.2021:10886-10895.
[33]LIANG J,CAO J,SUN G,et al.Swinir:Image restoration using swin transformer[C]//IEEE International Conference on Computer Vision.2021:1833-1844.
[34]JIANG Y,WRONSKI B,MILDENHALL B,et al.Fast and High-Quality Image Denoising via Malleable Convolutions[J].arXiv:2201.00392,2022.
[35]LIANG J,ZENG H,ZHANG L.High-resolution photorealistic image translation in real-time:A Laplacian pyramid translation network[C]//IEEE Conference on Computer Vision and Pattern Recognition.2021:9392-9400.
[36]ZHANG K,ZUO W,ZHANG L.FFDNet:Toward a fast andflexible solution for CNN-based image denoising[J].IEEE Transactions on Image Processing,2018,27(9):4608-4622.
[37]ZAMIR S W,ARORA A,KHAN S,et al.Multi-stage progressive image restoration[C]//IEEE Conference on Computer Vision and Pattern Recognition.2021:14821-14831.
[38]KANOPOULOS N,VASANTHAVADA N,BAKER R L.Design of an image edge detection filter using the Sobel operator[J].IEEE Journal of Solid-state Circuits,1988,23(2):358-367.
[39]WANG X.Laplacian operator-based edge detectors[J].IEEETransactions on Pattern Analysis and Machine Intelligence,2007,29(5):886-890.
[40]ZHAO H,GALLO O,FROSIO I,et al.Loss functions for image restoration with neural networks[J].IEEE Transactions on Computational Imaging,2016,3(1):47-57.
[41]KINGMA D P,BA J.Adam:A method for stochastic optimization[J].arXiv:1412.6980,2014.
[42]GOYAL P,DOLLÁR P,GIRSHICK R,et al.Accurate,large minibatch sgd:Training imagenet in 1 hour[J].arXiv:1706.02677,2017.
[43]WANG Z,SIMONCELLI E P,BOVIK A C.Multiscale structu-ral similarity for image quality assessment[C]//The Thrity-Se-venth Asilomar Conference on Signals,Systems & Computers.IEEE,2003:1398-1402.
[44]WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[45]ABDELHAMED A,LIN S,BROWN M S.A high-quality denoising dataset for smartphone cameras[C]//IEEE Conference on Computer Vision and Pattern Recognition.2018:1692-1700.
[1] 冼广铭 王知衍 冼广淋.
小波融合图像效果的因子分析评价方法

计算机科学, 2006, 33(8): 218-220.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发