Computer Science ›› 2023, Vol. 50 ›› Issue (9): 168-175.doi: 10.11896/jsjkx.221000100

• Database & Big Data & Data Science • Previous Articles     Next Articles

Image Relighting Network Based on Context-gated Residuals and Multi-scale Attention

WANG Wei, DU Xiangcheng, JIN Cheng   

  1. School of Computer Science,Fudan University,Shanghai 200438,China
  • Received:2022-10-13 Revised:2023-04-06 Online:2023-09-15 Published:2023-09-01
  • About author:WANG Wei,born in 1995,postgra-duate,is a member of China Computer Federation.His main research interests include image processing and visual positioning.
    JIN Cheng,born in 1978,Ph.D,professor,Ph.D supervisor,is a member of China Computer Federation.His main research interests include computer vision and multimedia information retrieval.
  • Supported by:
    National Key R & D Program of China (2019YFB2102800).

Abstract: Image relighting is commonly used in image editing and data augmentation tasks.Existing image relighting methods suffer from estimating accurate shadows and obtaining consistent structures and clear texture when removing and rendering sha-dows in complex scenes.To address these issues,this paper proposes an image relighting network based on context-gated resi-duals and multiscale attention.Contextual gating residuals capture the long-range dependencies of pixels by aggregating local and global spatial context information,which maintains the consistency of shadow and lighting direction.Besides,gating mechanisms can effectively improve the network's ability to recover textures and structures.Multiscale attention increases the receptive field without losing resolution by iteratively extracting and aggregating features of different scales.It activates important features by concatenating channel attention and spatial attention,and suppresses the responses of irrelevant features.In this paper,lighting gradient loss is also proposed to obtain satisfactory visual images through efficiently learning the lighting gradients in all directions.Experimental results show that,compared with the current state-of-the-art methods,the proposed method improves PSNR and SSIM by 7.47% and 12.37%,respectively.

Key words: Image relighting, Contextual information, Gating mechanism, Lighting gradient, Attention

CLC Number: 

  • TP391
[1]HOU Q,ZHOU D,FENG J.Coordinate attention for efficient mobile network design[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:13713-13722.
[2]ZHANG E,COHEN M F,CURLESS B.Emptying,refurni-shing,and relighting indoor spaces[J].ACM Transactions on Graphics(TOG),2016,35(6):1-14.
[3]DUCHENE S,RIANT C,CHAURASIA G,et al.Multi-view intrinsic images of outdoors scenes with an application to religh-ting[J].ACM Transactions on Graphics,2015,34(5):1-16.
[4]KARSCH K,HEDAU V,FORSYTH D,et al.Rendering synthetic objects into legacy photographs[J].ACM Transactions on Graphics(TOG),2011,30(6):1-12.
[5]PEERS P,MAHAJAN D K,LAMOND B,et al.Compressivelight transport sensing[J].ACM Transactions on Graphics(TOG),2009,28(1):1-18.
[6]REDDY D,RAMAMOORTHI R,CURLESS B.Frequency-space decomposition and acquisition of light transport under spatially varying illumination[C]//European Conference on Computer Vision.Berlin:Springer,2012:596-610.
[7]RONNEBERGER O,FISCHER P,BROX T.U-net:Convolu-tional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Compu-ter-assisted Intervention.Cham:Springer,2015:234-241.
[8]WANG L W,SIU W C,LIU Z S,et al.Deep relighting networks for image light source manipulation[C]//European Conference on Computer Vision.Cham:Springer,2020:550-567.
[9]HU Z,HUANG X,LI Y,et al.SA-AE for any-to-any relighting[C]//European Conference on Computer Vision.Cham:Sprin-ger,2020:535-549.
[10]ISOLA P,ZHU J Y,ZHOU T,et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:1125-1134.
[11]WANG Y,LU T,ZHANG Y,et al.Multi-scale self-calibratednetwork for image light source transfer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:252-259.
[12]KUBIAK N,MUSTAFA A,PHILLIPSON G,et al.SILT:Self-supervised Lighting Transfer Using Implicit Image Decomposition[J].arXiv:2110.12914,2021.
[13]YAZDANI A,GUO T,MONGA V.Physically inspired densefusion networks for relighting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:497-506.
[14]YANG H H,CHEN W T,LUO H L,et al.Multi-modal bifurcated network for depth guided image relighting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:260-267.
[15]El HELOU M,ZHOU R,SUSSTRUNK S,et al.NTIRE 2021 depth guided image relighting challenge[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:566-577.
[16]JIANG Y,GONG X,LIU D,et al.Enlightengan:Deep light enhancement without paired supervision[J].IEEE Transactions on Image Processing,2021,30:2340-2349.
[17]LIU J J,HOU Q,CHENG M M,et al.Improving convolutional networks with self-calibrated convolutions[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:10096-10105.
[18]HENDRYCKS D,GIMPEL K.Gaussian error linear units(gelus)[J].arXiv:1606.08415,2016.
[19]HU J,SHEN L,SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:7132-7141.
[20]LAI W S,HUANG J B,AHUJA N,et al.Fast and accurateimage super-resolution with deep laplacian pyramid networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2018,41(11):2599-2613.
[21]HELOU M E,ZHOU R,BARTHAS J,et al.VIDIT:Virtualimage dataset for illumination transfer[J].arXiv:2005.05460,2020.
[22]KINGMA D P,BA J.Adam:A method for stochastic optimization[J].arXiv:1412.6980,2014.
[23]PUTHUSSERY D,PANIKKASSERIL S H,KURIAKOSE M,et al.Wdrn:A wavelet decomposed relightnet for image religh-ting[C]//European Conference on Computer Vision.Cham:Springer,2020:519-534.
[24]LIU X,MA Y,SHI Z,et al.Griddehazenet:Attention-basedmulti-scale network for image dehazing[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2019:7314-7323.
[25]WEI C,WANG W,YANG W,et al.Deep retinex decomposition for low-light enhancement[J].arXiv:1808.04560,2018.
[26]WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[27]ZHANG R,ISOLA P,EFROS A A,et al.The unreasonable effectiveness of deep features as a perceptual metric[C]//Procee-dings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:586-595.
[28]EI HELOU M,ZHOU R,SUSSTRUNK S,et al.AIM 2020:Scene relighting and illumination estimation challenge[C]//European Conference on Computer Vision.Cham:Springer,2020:499-518.
[29]TAO X,GAO H,SHEN X,et al.Scale-recurrent network for deep image deblurring[C]//Proceedings of the IEEE Confe-rence on Computer Vision and Pattern Recognition.2018:8174-8182.
[1] CHEN Guojun, YUE Xueyan, ZHU Yanning, FU Yunpeng. Study on Building Extraction Algorithm of Remote Sensing Image Based on Multi-scale Feature Fusion [J]. Computer Science, 2023, 50(9): 202-209.
[2] BAI Zhengyao, XU Zhu, ZHANG Yihan. Deep Artificial Correspondence Generation for 3D Point Cloud Registration [J]. Computer Science, 2023, 50(9): 210-219.
[3] LI Xiang, FAN Zhiguang, LIN Nan, CAO Yangjie, LI Xuexiang. Self-supervised Learning for 3D Real-scenes Question Answering [J]. Computer Science, 2023, 50(9): 220-226.
[4] YI Liu, GENG Xinyu, BAI Jing. Hierarchical Multi-label Text Classification Algorithm Based on Parallel Convolutional Network Information Fusion [J]. Computer Science, 2023, 50(9): 278-286.
[5] LUO Yuanyuan, YANG Chunming, LI Bo, ZHANG Hui, ZHAO Xujian. Chinese Medical Named Entity Recognition Method Incorporating Machine ReadingComprehension [J]. Computer Science, 2023, 50(9): 287-294.
[6] LI Ke, YANG Ling, ZHAO Yanbo, CHEN Yonglong, LUO Shouxi. EGCN-CeDML:A Distributed Machine Learning Framework for Vehicle Driving Behavior Prediction [J]. Computer Science, 2023, 50(9): 318-330.
[7] WANG Huaiqin, LUO Jian, WANG Haiyan. Feature Weight Perception-based Prediction of Virtual Network Function Resource Demands [J]. Computer Science, 2023, 50(9): 331-336.
[8] YANG Zhizhuo, XU Lingling, Zhang Hu, LI Ru. Answer Extraction Method for Reading Comprehension Based on Frame Semantics and GraphStructure [J]. Computer Science, 2023, 50(8): 170-176.
[9] WANG Jiahao, ZHONG Xin, LI Wenxiong, ZHAO Dexin. Human Activity Recognition with Meta-learning and Attention [J]. Computer Science, 2023, 50(8): 193-201.
[10] WANG Yu, WANG Zuchao, PAN Rui. Survey of DGA Domain Name Detection Based on Character Feature [J]. Computer Science, 2023, 50(8): 251-259.
[11] ZHANG Yian, YANG Ying, REN Gang, WANG Gang. Study on Multimodal Online Reviews Helpfulness Prediction Based on Attention Mechanism [J]. Computer Science, 2023, 50(8): 37-44.
[12] TENG Sihang, WANG Lie, LI Ya. Non-autoregressive Transformer Chinese Speech Recognition Incorporating Pronunciation- Character Representation Conversion [J]. Computer Science, 2023, 50(8): 111-117.
[13] YAN Mingqiang, YU Pengfei, LI Haiyan, LI Hongsong. Arbitrary Image Style Transfer with Consistent Semantic Style [J]. Computer Science, 2023, 50(7): 129-136.
[14] DAI Xuesong, LI Xiaohong, ZHANG Jingjing, QI Meibin, LIU Yimin. Unsupervised Domain Adaptive Pedestrian Re-identification Based on Counterfactual AttentionLearning [J]. Computer Science, 2023, 50(7): 160-166.
[15] LIANG Meiyan, ZHANG Yu, LIANG Jianan, CHEN Qinghui, WANG Ru, WANG Lin. Pathological Image Super-resolution Reconstruction Based on Sparse Coding Non-local AttentionDual Network [J]. Computer Science, 2023, 50(6A): 220700016-8.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!