计算机科学

计算机科学 ›› 2024, Vol. 51 ›› Issue (5): 117-124.doi: 10.11896/jsjkx.230300049

• 计算机图形学&多媒体 • 上一篇    下一篇

基于渐进式多尺度Transformer的图像去雾算法

周宇1, 陈志华1, 盛斌2, 梁磊1   

  1. 1 华东理工大学信息科学与工程学院 上海 200237
    2 上海交通大学计算机科学与技术系 上海 200240
  • 收稿日期:2023-03-06 修回日期:2023-06-28 出版日期:2024-05-15 发布日期:2024-05-08
  • 通讯作者: 陈志华(czh@ecust.edu.cn)
  • 作者简介:(1178910860@qq.com)
  • 基金资助:
    国家自然科学基金(62272164);空间智能控制技术实验室开放基金(HTKJ2022KL502010)

Multi Scale Progressive Transformer for Image Dehazing

ZHOU Yu1, CHEN Zhihua1, SHENG Bin2, LIANG Lei1   

  1. 1 College of Information Science and Engineering,East China University of Science and Technology,Shanghai 200237,China
    2 Department of Computer Science and Technology,Shanghai Jiao Tong University,Shanghai 200240,China
  • Received:2023-03-06 Revised:2023-06-28 Online:2024-05-15 Published:2024-05-08
  • About author:ZHOU Yu,born in 1994,Ph.D,is a member of CCF(No.D2304G).Her main research interests include compu-ter vision and image processing.
    CHEN Zhihua,born in 1969,Ph.D,professor,Ph.D supervisor,is a member of CCF(No.12441D).His main research interests include computer vision,machine learning,object detection and image video processing.
  • Supported by:
    National Natural Science Foundation of China(62272164) and Science and Technology on Space Intelligent Control Laboratory(HTKJ2022KL502010).

PDF (PC)

70

摘要： 现有的去雾方法难以在复原图像细节的同时保持全局信息。为了解决此问题,文中提出了一种基于渐进式多尺度Transformer(Multi Scale Progressive Transformer,MSP-Transformer)的图像去雾算法。该模型能够有效提取和利用不同尺度的雾相关特征,实现了特征和图像的多尺度学习和融合,渐进式地从有雾图像中复原清晰图像。所提出的MSP-Transformer分为编码、解码和复原3个阶段。在编码阶段,利用基于Transformer模块的编码器将输入图像分解为不同尺度的雾图像特征,以全面表征真实有雾图像的信息损失。在解码阶段,考虑到有雾图像的不同区域存在不同尺度的信息丢失,设计了一个包含多尺度注意力机制的特征聚合模块,利用通道注意力和多尺度空间注意力来融合不同尺度的特征信息。复原阶段包含了复原模块和融合模块,首先基于多尺度特征融合的复原模块聚合不同尺度的雾相关特征以增加不同尺度特征的联系,并在每个尺度复原出清晰的无雾图像,然后将每个尺度的复原图像送入融合模块以获得最终的去雾结果。定性和定量的实验结果表明,所提出的MSP-Transformer在真实图像和合成数据集上能够实现雾的有效去除,具有良好的鲁棒性。在公开的RESIDE数据集上与11种去雾方法进行定量和定性比较,MSP-Transformer取得了最高的PSNR(39.53db)和SSIM(0.9954),并获得了良好的视觉效果。此外,消融实验也证明了MSP-Transformer中所提出的模块的有效性。

关键词: 图像去雾, 多尺度, Transformer, 注意力机制, 特征融合

Abstract: In order to simultaneously recover image details and maintain global information in the dehazed image,a multi scale progressive transformer(MSP-Transformer) is proposed for image dehazing.The MSP-Transformer can effectively extract haze-related features from different scales,and restore clear image in a progressive way,achieving multi-scale learning and fusion of features and images.The proposed MSP-Transformer is divided into an encoding stage,a decoding stage,and a restoration stage.In the encoding stage,a Transformer block-based encoder is used to decompose the input image into different scales.The extracted haze-relevant features from different scales can fully characterize the information loss of the haze image.In the decoding stage,considering that different regions of the haze image have different information loss,this paper designs a feature aggregation module containing a multi-scale attention mechanism in decoder.The multi-scale attention contains channel attention and multi-scale spatial attention,and can fuse the feature information from different scales.The restoration stage contains restoration block and fusion block,firstly,the multi-scale feature fusion restoration block aggregates the haze relevant features from different scales to increase the association between these features,then the aggregated features are used to restore a haze-free image at each scale.Besides,the restored images from each scale are fused by fusion block to obtain the final dehazed result.Qualitative and quantitative experiments on both real and synthetic datasets show that the proposed MSP-Transformer has good dehazing performance.Compared with 11 state-of-the-art methods,MSP-Transformer obtains the best PSNR(39.53db) and SSIM(0.9954) on the RESIDE dataset,and achieves good visual effect.In addition,the ablation experiments also demonstrate the effectiveness of the proposed dehazing method.

Key words: Image dehazing, Multi scale, Transformer, Attention mechanism, Feature fusion

中图分类号: 

  • TP391
[1]HE K M,SUN J,TANG X.Single image haze removal usingdark channel prior[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2011,33(12):2341-2353.
[2]ZHU Q S,MAI J M,SHAO L.A Fast Single Image Haze Removal Algorithm Using Color Attenuation Prior[J].IEEE Transactions on Image Processing,2015,24(11):3522-3533.
[3]BERMAN D,TREIBITZ T,AVIDAN S.2020.Single Image Dehazing Using Haze-Lines[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2020,42(3):720-734.
[4]ZHANG J L,SHI D Y,JIA B.Insulator image defogging algorithm based on dark channel prior theory[J].Journal of Chongqing University of Technology(Natural Science),2022,36(7):208-215.
[5]QIN X,WANG Z L,BAI Y C,et al.FFA-Net:Feature Fusion Attention Network for Single Image Dehazing[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2020,34:11908-11915.
[6]ZHANG J L,SHI D Y,JIA B.Insulator image defogging algorithm based on dark channel prior theory[J].Journal of Chongqing University of Technology(Natural Science),2022,36(7):208-215.
[7]LIU X H,MA Y R,SHI Z H,et al.GridDehazeNet:Attention-Based Multi-Scale Network for Image Dehazing[C]//Procee-dings of the IEEE/CVF International Conference on Computer Vision.2019:7313-7322.
[8]VASWANI A,SHAZEER N,PARMER N,et al.Attention is all you need[C]//Neural Information Processing Systems.2017:5998-6008.
[9]ZAMIR S W,ARORA A,KHAN S,et al.Restormer:Efficient Transformer for High-Resolution Image Restoration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:5718-5729.
[10]WANG Z D,CUN X D,BAO J M,et al.Uformer:A GeneralU-Shaped Transformer for Image Restoration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:17662-17672.
[11]SONG Y,HE Z Q,QIAN H,et al.2022.Vision Transformers for Single Image Dehazing[J].arXiv:2204.03883,2022.
[12]LIU Z,LIN Y T,CAO Y,et al.Swin transformer:Hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2021:10012-10022.
[13]CAI B L,XU X M,JIA K,et al.Dehazenet:An end-to-end system for single image haze removal[J].IEEE Transactions on Image Processing,2016,25(11):5187-5198.
[14]REN W Q,LIU S,ZHANG H,et al.Single image dehazing via multi-scale convolutional neural networks[C]//Proceedings of the European Conference on Computer Vision.Springer,2016:154-169.
[15]WU H Y,QU Y Y,LIN S H,et al.Contrastive Learning for Compact Single Image Dehazing[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:10551-10560.
[16]YU H,ZHENG N S,ZHOU M,et al.Frequency and SpatialDual Guidance for Image Dehazing[C]//Proceedings of the European Conference on Computer Vision.Springer,2022:181-198.
[17]SHAO Y J,LI L R H,REN W Q,et al.Domain Adaptation for Image Dehazing[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:2805-2814.
[18]LI B Y,GOU Y B,GU S H,et al.You Only Look Yourself:Unsupervised and Untrained Single Image Dehazing Neural Network [J].International Journal of Computer Vision,2021,129(5):1754-1767.
[19]CHEN H T,WANG Y H,GUO T Y,et al.Pre-Trained Image Processing Transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:12299-12310.
[20]LIANG J Y,CAO J Z,SUN G L,et al.SwinIR:Image Restoration Using Swin Transformer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop.2021:1833-1844.
[21]LI X,JIN X,YU T,et al.Learning Omni-Frequency Region-adaptive Representations for Real Image Super-Resolution[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2021:1975-1983.
[22]LI X,WANG W H,HU X L,et al.Selective Kernel Networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:510-519.
[23]WANG X T,KELVIN C K C,YU K,et al.EDVR:Video Restoration With Enhanced Deformable Convolutional Networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.2019:1954-1963.
[24]LIU Y,PAN J S,REN J,et al.Learning Deep Priors for Image Dehazing[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2019:2492-2500.
[25]DONG H,PAN J S,XIANG L,et al.Multi Scale Boosted Deha-zing Network With Dense Feature Fusion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:2154-2164.
[26]HONG M,XIE Y,LI C H,et al.Distilling Image Dehazing With Heterogeneous Task Imitation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:3459-3468.
[27]CHEN Z Y,WANG Y C,YANG Y,et al.PSD:Principled Synthetic-to-Real Dehazing Guided by Physical Priors[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:7180-7189.
[28]ZHANG R,ISOLA P,EFROS A A,et al.The Unreasonable Effectiveness of Deep Features as a Perceptual Metric[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2018:586-595.
[29]MITTAL A,SOUNDARARAJAN R,BOVIK A C.Making a“Completely Blind” Image Quality Analyzer[J].IEEE Signal Processing Letters,2013,20(3):209-212.
[30]LI B Y,REN W Q,FU D P,et al.Benchmarking Single-Image Dehazing and Beyond [J].IEEE Transactions on Image Proces-sing,2010,28(1):492-505.
[31]YIN W,ZHANG J M,WANG O,et al.Learning To Recover 3D Scene Shape From a Single Image[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:204-213.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发