Computer Science

Computer Science ›› 2022, Vol. 49 ›› Issue (8): 120-126.doi: 10.11896/jsjkx.220200179

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Re-parameterized Multi-scale Fusion Network for Efficient Extreme Low-light Raw Denoising

WEI Kai-xuan, FU Ying   

  1. School of Computer Science and Technology,Beijing Institute of Technology,Beijing 100081,China
  • Received:2022-02-27 Revised:2022-03-27 Published:2022-08-02
  • About author:WEI Kai-xuan,born in 1996,postgra-duate.His main research interests include computer vision,computational photography and computational imaging.
    FU Ying,born in 1987,Ph.D,professor.Her main research interests include physics-based vision,image and video processing,and computational photo-graphy.
  • Supported by:
    National Natural Science Foundation of China(62171038,61827901,62088101).

PDF (PC)

773

Abstract: Practical low-light denoising/enhancement solutions often require fast computation,high memory efficiency,and can achieve visually high-quality restoration results.Most existing methods aim to restore quality but compromise on speed and memory requirements,which limits their usefulness to a large extent.This paper proposes a new deep denoising architecture,a re-parameterized multi-scale fusion network for extreme low-light raw denoising,which greatly improves the inference speed without losing high-quality denoising performance.Specifically,image features are extracted in multi-scale space,and a lightweight spatial-channel parallel attention module is used to focus on core features within space and channel dynamically and adaptively.The representation ability of the model is further enriched by re-parameterized convolutional unit without increasing computational cost at inference.The proposed model can restore UHD 4K resolution images within about 1s on a CPU(e.g.,Intel i7-7700K) and run at 24 fps on a GPU(e.g.,NVIDIA GTX 1080Ti),which is almost four times faster than existing advanced methods(e.g.,UNet) while still maintaining competitive restoration quality.

Key words: Extreme low-light denoising, Multi-scale fusion, Re-parameterization convolutional unit, Spatial-channel parallel attention module

CLC Number: 

  • TP391
[1]DABOV K,FOI A,KATKOVNIK V,et al.Image denoising by sparse 3-d transform-domain collaborative filtering[J].IEEE Transactions on Image Processing,2007,16 (8):2080-2095.
[2]BUADES A,COLL B,MOREL J M.A non-local algorithm for image denoising [C]//IEEE Conference on Computer Vision and Pattern Recognition.2005:60-65.
[3]GU S,LEI Z,ZUO W,et al.Weighted nuclear norm minimization with application to image denoising [C]//IEEE Conference on Computer Vision and Pattern Recognition.2014:2862-2869.
[4]SCHMIDT U,ROTH S.Shrinkage fields for effective image restoration [C]//IEEE Conference on Computer Vision and Pattern Recognition.2014:2774-2781.
[5]CHEN Y,YU W,POCK T.On learning optimized reaction diffusion processes for effective image restoration [C]//IEEE Conference on Computer Vision and Pattern Recognition.2015:5261-5269.
[6]ZHANG K,ZUO W,CHEN Y,et al.Beyond a gaussian denoi-ser:residual learning of deep cnn for image denoising [J].IEEE Transactions on Image Processing,2017,26 (7):3142-3155.
[7]TAI Y,YANG J,LIU X,et al.Memnet:a persistent memorynetwork for image restoration [C]//IEEE International Confe-rence on Computer Vision.2017:4549-4557.
[8]GUO S,YAN Z,ZHANG K,et al.Toward convolutional blind denoising of real photographs [C]//IEEE Conference on Computer Vision and Pattern Recognition.2019:1712-1722.
[9]CHEN C,CHEN Q,XU J,et al.Learning to see in the dark[C]//IEEE Conference on Computer Vision and Pattern Recognition.2018:3291-3300.
[10]RONNEBERGER O,FISCHER P,BROX T.U-net:convolu-tional networks for biomedical image segmen-tation [C]//International Conference on Medical image computing and computer-assisted intervention.2015:234-241.
[11]GU S,LI Y,GOOL L V,et al.Self-guided network for fastimage denoising [C]//IEEE International Conference on Computer Vision.2019:2511-2520.
[12]OSHER S,BURGER M,GOLDFARB D,et al.An iterative regu-larization method for total variation-based image restoration [J].Multiscale Modeling and Simulation,2005,4(2):460-489.
[13]ELAD M,AHARON M.Image denoising via sparse and redundant representations over learned dictio-naries[J].IEEETransa-ctions on Image Processing,2006,15 (12):3736-3745.
[14]MALLAT S G.A theory for multiresolution signal decomposition:the wavelet representation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1989,11(7):674-693.
[15]ZHANG Y,TIAN Y,KONG Y,et al.Residual dense network for image restoration [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2020,43 (7):2480-2495.
[16]PENG Y,ZHANG L,LIU S,et al..Dilated residual networks with symmetric skip connection for image denoising [J].Neurocomputing,2019,345:67-76.
[17]BURGER H C,SCHULER C J,HARMELING S.Image denoi-sing:Can plain neural networks compete with BM3D?[C]//IEEE Conference on Computer Vision and Pattern Recognition.2012:2392-2399.
[18]ZHANG Y,LI K,LI K,et al.Residual non-local attention networks for image restoration[J].arXiv:1903.10082,2019.
[19]WEI K,FU Y,YANG J,et al.A physics-based noise formation model for extreme low-light raw denoising[C]//IEEE Confe-rence on Computer Vision and Pattern Recognition.2020:2758-2767.
[20]WEI K,FU Y,ZHENG Y,et al.Physics-based Noise Modeling for Extreme Low-light Photography[J].arXiv:2108.02158,2021.
[21]XU K,YANG X,YIN B,et al.Learning to restore low-lightimages via decomposition-and-enhancement[C]//IEEE Confe-rence on Computer Vision and Pattern Recognition.2020:2281-2290.
[22]LAMBA M,MITRA K.Restoring Extremely Dark Images in Real Time[C]//IEEE Conference on Computer Vision and Pattern Recognition.2021:3487-3497.
[23]JIANG H,ZHENG Y.Learning to see moving objects in the dark[C]//IEEE International Conference on Computer Vision.2019:7324-7333.
[24]ZHENG Y,ZHANG M,LU F.Optical flow in the dark[C]//IEEE International Conference on Computer Vision.2020:6749-6757.
[25]CHEN C,CHEN Q,DO M N,et al.Seeing motion in the dark[C]//IEEE International Conference on Computer Vision.2019:3185-3194.
[26]ARORA S,COHEN N,HAZAN E.On the optimization of deep networks:Implicit acceleration by overparameterization[C]//International Conference on Machine Learning.PMLR,2018:244-253.
[27]ZAGORUYKO S,KOMODAKIS N.Diracnets:Training verydeep neural networks without skip-connections[J].arXiv:1706.00388,2017.
[28]DING X,GUO Y,DING G,et al.Acnet:Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks[C]//IEEE International Conference on Computer Vision.2019:1911-1920.
[29]DING X,ZHANG X,MA N,et al.Repvgg:Making vgg-style convnets great again[C]//IEEE Conference on Computer Vision and Pattern Recognition.2021:13733-13742.
[30]HE K,ZHANG X,REN S,et al.Deep residual learning forimage recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[31]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[32]DING X,ZHANG X,HAN J,et al.Diverse branch block:Buil-ding a convolution as an inception-like unit[C]//IEEE Confe-rence on Computer Vision and Pattern Recognition.2021:10886-10895.
[33]LIANG J,CAO J,SUN G,et al.Swinir:Image restoration using swin transformer[C]//IEEE International Conference on Computer Vision.2021:1833-1844.
[34]JIANG Y,WRONSKI B,MILDENHALL B,et al.Fast and High-Quality Image Denoising via Malleable Convolutions[J].arXiv:2201.00392,2022.
[35]LIANG J,ZENG H,ZHANG L.High-resolution photorealistic image translation in real-time:A Laplacian pyramid translation network[C]//IEEE Conference on Computer Vision and Pattern Recognition.2021:9392-9400.
[36]ZHANG K,ZUO W,ZHANG L.FFDNet:Toward a fast andflexible solution for CNN-based image denoising[J].IEEE Transactions on Image Processing,2018,27(9):4608-4622.
[37]ZAMIR S W,ARORA A,KHAN S,et al.Multi-stage progressive image restoration[C]//IEEE Conference on Computer Vision and Pattern Recognition.2021:14821-14831.
[38]KANOPOULOS N,VASANTHAVADA N,BAKER R L.Design of an image edge detection filter using the Sobel operator[J].IEEE Journal of Solid-state Circuits,1988,23(2):358-367.
[39]WANG X.Laplacian operator-based edge detectors[J].IEEETransactions on Pattern Analysis and Machine Intelligence,2007,29(5):886-890.
[40]ZHAO H,GALLO O,FROSIO I,et al.Loss functions for image restoration with neural networks[J].IEEE Transactions on Computational Imaging,2016,3(1):47-57.
[41]KINGMA D P,BA J.Adam:A method for stochastic optimization[J].arXiv:1412.6980,2014.
[42]GOYAL P,DOLLÁR P,GIRSHICK R,et al.Accurate,large minibatch sgd:Training imagenet in 1 hour[J].arXiv:1706.02677,2017.
[43]WANG Z,SIMONCELLI E P,BOVIK A C.Multiscale structu-ral similarity for image quality assessment[C]//The Thrity-Se-venth Asilomar Conference on Signals,Systems & Computers.IEEE,2003:1398-1402.
[44]WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[45]ABDELHAMED A,LIN S,BROWN M S.A high-quality denoising dataset for smartphone cameras[C]//IEEE Conference on Computer Vision and Pattern Recognition.2018:1692-1700.
[1] CHEN Zhi-qiang, HAN Meng, LI Mu-hang, WU Hong-xin, ZHANG Xi-long. Survey of Concept Drift Handling Methods in Data Streams [J]. Computer Science, 2022, 49(9): 14-32.
[2] WANG Ming, WU Wen-fang, WANG Da-ling, FENG Shi, ZHANG Yi-fei. Generative Link Tree:A Counterfactual Explanation Generation Approach with High Data Fidelity [J]. Computer Science, 2022, 49(9): 33-40.
[3] ZHANG Jia, DONG Shou-bin. Cross-domain Recommendation Based on Review Aspect-level User Preference Transfer [J]. Computer Science, 2022, 49(9): 41-47.
[4] ZHOU Fang-quan, CHENG Wei-qing. Sequence Recommendation Based on Global Enhanced Graph Neural Network [J]. Computer Science, 2022, 49(9): 55-63.
[5] SONG Jie, LIANG Mei-yu, XUE Zhe, DU Jun-ping, KOU Fei-fei. Scientific Paper Heterogeneous Graph Node Representation Learning Method Based onUnsupervised Clustering Level [J]. Computer Science, 2022, 49(9): 64-69.
[6] CHAI Hui-min, ZHANG Yong, FANG Min. Aerial Target Grouping Method Based on Feature Similarity Clustering [J]. Computer Science, 2022, 49(9): 70-75.
[7] ZHENG Wen-ping, LIU Mei-lin, YANG Gui. Community Detection Algorithm Based on Node Stability and Neighbor Similarity [J]. Computer Science, 2022, 49(9): 83-91.
[8] LYU Xiao-feng, ZHAO Shu-liang, GAO Heng-da, WU Yong-liang, ZHANG Bao-qi. Short Texts Feautre Enrichment Method Based on Heterogeneous Information Network [J]. Computer Science, 2022, 49(9): 92-100.
[9] XU Tian-hui, GUO Qiang, ZHANG Cai-ming. Time Series Data Anomaly Detection Based on Total Variation Ratio Separation Distance [J]. Computer Science, 2022, 49(9): 101-110.
[10] NIE Xiu-shan, PAN Jia-nan, TAN Zhi-fang, LIU Xin-fang, GUO Jie, YIN Yi-long. Overview of Natural Language Video Localization [J]. Computer Science, 2022, 49(9): 111-122.
[11] CAO Xiao-wen, LIANG Mei-yu, LU Kang-kang. Fine-grained Semantic Reasoning Based Cross-media Dual-way Adversarial Hashing Learning Model [J]. Computer Science, 2022, 49(9): 123-131.
[12] ZHOU Xu, QIAN Sheng-sheng, LI Zhang-ming, FANG Quan, XU Chang-sheng. Dual Variational Multi-modal Attention Network for Incomplete Social Event Classification [J]. Computer Science, 2022, 49(9): 132-138.
[13] DAI Yu, XU Lin-feng. Cross-image Text Reading Method Based on Text Line Matching [J]. Computer Science, 2022, 49(9): 139-145.
[14] QU Qian-wen, CHE Xiao-ping, QU Chen-xin, LI Jin-ru. Study on Information Perception Based User Presence in Virtual Reality [J]. Computer Science, 2022, 49(9): 146-154.
[15] ZHOU Le-yuan, ZHANG Jian-hua, YUAN Tian-tian, CHEN Sheng-yong. Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion [J]. Computer Science, 2022, 49(9): 155-161.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.