深度学习驱动的水下图像处理研究进展

doi:10.11896/jsjkx.230400107

摘要/Abstract

摘要： 随着人工智能和水下设备的发展,水下机器人能够便捷地获取水下图像。水下图像对于海洋的探索和开发等活动至关重要,但由于水下成像环境十分复杂,导致获取到的水下图像成像质量较低,具有低对比度、模糊和颜色失真等退化问题,难以满足水下生产活动的要求。近年来,基于深度学习的水下图像处理方法和质量评价指标发展迅速,受到诸多学者的关注。目前已有基于深度学习的水下图像处理方法的综述,但仍存在总结不够全面及缺少最新研究成果等问题。因此,文中首先分析水下图像退化成因并提出所需处理的问题,根据各类算法的原理特点将水下图像处理方法进行分类;其次,详细分析和归纳了基于深度学习的水下图像处理最新研究成果,总结出各类算法的主要特征;然后,详细整理了现有的公开水下图像数据集和当前主流以及最新的基于学习的水下图像质量评测指标,并通过设计实验将传统算法和基于深度学习的水下图像处理方法进行对比分析;最后,分析总结了一些水下图像处理领域尚未解决的问题,并对未来的发展方向进行展望。

关键词: 深度学习, 水下机器人, 水下图像, 图像处理, 图像质量评测

Abstract: With the development of artificial intelligence and underwater equipments,autonomous underwater vehicles can conveniently obtain underwater images.Underwater images are essential for exploring and developing the ocean.However,due to the complex underwater imaging environment,the acquired underwater images have low image quality,such as low contrast,blurring,and color distortion,making it difficult to meet the requirements of underwater production activities.In recent years,the development of deep learning-based underwater image processing methods and quality evaluation metrics has received much attention from scholars.Although there have been some reviews on deep learning-based underwater image processing methods,there are still issues such as incomplete summarization and a lack of the latest research results.Therefore,this paper first analyzes the causes of underwater image degradation and proposes the necessary processing issues,and classifies underwater image processing methods based on the principles and characteristics of various algorithms.Secondly,the latest research results on deep learning-based underwater image processing are analyzed and summarized,and the main features of various algorithms are summarized.Then,existing publicly available underwater image datasets and current mainstream and latest learning-based underwater image quality evaluation metrics are detailed,and traditional algorithms and deep learning-based underwater image processing methods are compared and analyzed through experimental design.Finally,some unresolved issues in the field of underwater image proces-sing are analyzed and summarized,and future development directions are discussed.

Key words: Deep learning, Autonomous underwater vehicle, Underwater image, Image processing, Image quality evaluation

中图分类号:

TP391

张天驰, 刘宇轩. 深度学习驱动的水下图像处理研究进展[J]. 计算机科学, 2024, 51(6A): 230400107-12. https://doi.org/10.11896/jsjkx.230400107

ZHANG Tianchi, LIU Yuxuan. Research Progress of Underwater Image Processing Based on Deep Learning[J]. Computer Science, 2024, 51(6A): 230400107-12. https://doi.org/10.11896/jsjkx.230400107

参考文献

[1]YANG B,LIU Y Y,LIAO J W.Manned Submersible:A “National Treasure” for Deep-Sea Scientific Exploration and Ocean Resource Development[J].Bulletin of Chinese Academy of Sciences,2021,36(5):622-631.
[2]LI S,WU Y T,LI C,et al.Applications and Prospects of Underwater Robots[J].Bulletin of Chinese Academy of Sciences,2022,37(7):910-920.
[3]FENG X S,LI Y P,XU H X,et al.Development of Deep-Sea Autonomous Underwater Robots and Their Applications in Resource Investigation[J].The Chinese Journal of Nonferrous Metals,2021,31(10):2746-2756.
[4]XU Y L,DU J H,LEI Z Y,et al.Current status and key technologies review of underwater robots in fishery applications[J].Robot,2023,45(1),110-128.
[5]HU K,WENG C,ZHANG Y,et al.An overview of underwater vision enhancement:from traditional methods to recent deep learning[J].Journal of Marine Science and Engineering,2022,10(2):241.
[6]LUO H L,AO Y,YUAN P.A Method of Image RestorationBased on Generative Adversarial Network[J].Acta Electronica Sinica,2020,48(10):1891-1898.
[7]FU X,CAO X.Underwater image enhancement with global-local networks and compressed-histogram equalization[J].Signal Processing:Image Communication,2020,86:115892.
[8]HE K,ZHANG X,REN S,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[9]JAFFE J S.Computer modeling and the design of optimal underwater imaging systems[J].IEEE Journal of Oceanic Engineering,1990,15(2):101-111.
[10]SCHECHNER Y Y,KARPEL N.Recovery of underwater visibility and structure by polarization analysis[J].IEEE Journal of Oceanic Cngineering,2005,30(3):570-587.
[11]CONG R M,ZHANG Y M,ZHANG C,et al.Research Progress on Deep Learning-Driven Underwater Image Enhancement and Restoration[J].Signal Processing,2020,36(9):1377-1389.
[12]ZHOU J,YANG T,ZHANG W.Underwater vision enhance-ment technologies:A comprehensive review,challenges,and recent trends[J].Applied Intelligence,2023,53(3):3594-3621.
[13]FENG F,WU G J,WU Y F,et al.Underwater Polarization Imaging Algorithm Based on Global Estimation[J].Acta Optica Sinica,2020,40(21):75-83.
[14]SHPILMAN B,ABOOKASIS D.Experimental results of imaging objects in turbid liquid integrating multiview circularly polarized speckle images and deconvolution method[J].Optics & Laser Technology,2020,121:105774.
[15]XIANG Y,YANG X,REN Q,et al.Underwater PolarizationImaging Recovery Based on Polarimetric Residual Dense Network[J].IEEE Photonics Journal,2022,14(6):1-6.
[16]HE K,SUN J,TANG X.Single image haze removal using dark channel prior[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2010,33(12):2341-2353.
[17]DREWS P,NASCIMENTO E,MORAES F,et al.Transmission estimation in underwater single images[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops.2013:825-830.
[18]BERMAN D,LEVY D,AVIDAN S,et al.Underwater singleimage color restoration using haze-lines and a new quantitative dataset[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2020,43(8):2822-2837.
[19]ZHOU Y,GU X T,LI Q W.Underwater Image Restoration Based on Background Light Correction Imaging Model[J].Journal of Electronics & Information Technology,2022,44(10):3363-3371.
[20]QI P,LI X,HAN Y,et al.U2R-pGAN:Unpaired underwater-image recovery with polarimetric generative adversarial network[J].Optics and Lasers in Engineering,2022,157:107112.
[21]HUMMEL R.Image enhancement by histogram transformation[J].Computer Graphics and Image Processing,1977,6(2):184-195.
[22]PIZER S M,AMBURN E P,AUSTIN J D,et al.Adaptive histogram equalization and its variations[J].Computer Vision,Graphics,and Image Processing,1987,39(3):355-368.
[23]REZA A M.Realization of the contrast limited adaptive histogram equalization(CLAHE) for real-time image enhancement[J].Journal of VLSI Signal Processing Systems for Signal,Image and Video Technology,2004,38:35-44.
[24]LI X,HOU G,TAN L,et al.A hybrid framework for underwater image enhancement[J].IEEE Access,2020,8:197448-197462.
[25]LEI X Y,ZHANG W D,PAN X P.Underwater Image Color Correction Method Based on Histogram Distribution Characteristics[J].Computer Engineering and Design,2022,43(8):2284-2293.
[26]LAND E H,MCCANN J J.Lightness and retinex theory[J].Josa,1971,61(1):1-11.
[27]FU X,ZHUANG P,HUANG Y,et al.A retinex-based enhancing approach for single underwater image[C]//2014 IEEE International Conference on Image Processing(ICIP).IEEE,2014:4572-4576.
[28]SHI L,XI M L,SUN J.Underwater Image Enhancement Algorithm Based on Controllable Kernel Bilateral Filtering Retinex[J].Chinese Journal of Quantum Electronics,2018,35(1):7-12.
[29]ZOU L,LU J Y,HU Y,et al.Underwater Image Enhancement Algorithm under Non-uniform Illumination Conditions[J].Journal of Shandong University of Science and Technology(Natural Science Edition),2020,39(2):118-125.
[30]TANG Z,JIANG L,LUO Z.A new underwater image enhancement algorithm based on adaptive feedback and Retinex algorithm[J].Multimedia Tools and Applications,2021,80(18):28487-28499.
[31]ANCUTI C,ANCUTI C O,HABER T,et al.Enhancing underwater images and videos by fusion[C]//2012 IEEE conference on computer vision and pattern recognition.IEEE,2012:81-88.
[32]GAO F,WANG K,YANG Z,et al.Underwater image enhancement based on local contrast correction and multi-scale fusion[J].Journal of Marine Science and Engineering,2021,9(2):225.
[33]SONG H,WANG R.Underwater image enhancement based on multi-scale fusion and global stretching of dual-model[J].Mathematics,2021,9(6):595.
[34]FU Z,LIN H,YANG Y,et al.Unsupervised Underwater Image Restoration:From a Homology Perspective[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2022:643-651.
[35]WU S,LUO T,JIANG G,et al.A two-stage underwater en-hancement network based on structure decomposition and characteristics of underwater imaging[J].IEEE Journal of Oceanic Engineering,2021,46(4):1213-1227.
[36]SHIN Y-S,CHO Y,PANDEY G,et al.Estimation of ambient light and transmission map with common convolutional architecture[C]//OCEANS 2016 MTS/IEEE Monterey.IEEE,2016:1-7.
[37]CAO K,PENG Y-T,COSMAN P C.Underwater image restoration using deep networks to estimate background light and scene depth[C]//2018 IEEE Southwest Symposium on Image Analysis and Interpretation(SSIAI).IEEE,2018:1-4.
[38]WANG K,HU Y,CHEN J,et al.Underwater image restoration based on a parallel convolutional neural network[J].Remote sensing,2019,11(13):1591.
[39]WANG Y,SONG W,FORTINO G,et al.An experimental-based review of image enhancement and image restorationme-thods for underwater imaging[J].IEEE Access,2019,7:140233-140251.
[40]KAR A,DHARA S K,SEN D,et al.Zero-shot single image restoration through controlled perturbation of koschmieder’s mo-del[C]//Proceedings of the IEEE/CVF Conference on Compu-ter Vision and Pattern Recognition.2021:16205-16215.
[41]CHAI S,FU Z,HUANG Y,et al.Unsupervised and Untrained Underwater Image Restoration Based on Physical Image Formation Model[C]//ICASSP 2022-2022 IEEE International Confe-rence on Acoustics,Speech and Signal Processing(ICASSP).IEEE,2022:2774-2778.
[42]LI C,ANWAR S,HOU J,et al.Underwater image enhancement via medium transmission-guided multi-color space embedding[J].IEEE Transactions on Image Processing,2021,30:4985-5000.
[43]WANG Y,ZHANG J,CAO Y,et al.A deep CNN method for underwater image enhancement[C]//2017 IEEE International Conference on Image Processing(ICIP).IEEE,2017:1382-1386.
[44]SUN X,LIU L,LI Q,et al.Deep pixel-to-pixel network for underwater image enhancement and restoration[J].IET Image Processing,2019,13(3):469-474.
[45]LI C,GUO C,REN W,et al.An underwater image enhancement benchmark dataset and beyond[J].IEEE Transactions on Image Processing,2019,29:4376-4389.
[46]DOU Z,WANG N,LI B,et al.Dual color space guided sketch colorization[J].IEEE Transactions on Image Processing,2021,30:7292-7304.
[47]WANG Y,GUO J,GAO H,et al.UIEC^∧ 2-Net:CNN-based underwater image enhancement using two color space[J].Signal Processing:Image Communication,2021,96:116250.
[48]CHEN Q J,XIE Y L.Underwater Image Enhancement Based on Dense Cascaded Convolutional Neural Network[J].Laser & Optoelectronics Progress,2022,59(22):235-244.
[49]ZHANG B,GU S,ZHANG B,et al.Styleswin:Transformer-based gan for high-resolution image generation[C]//Procee-dings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:11304-11314.
[50]MARATHE A,JAIN P,WALAMBE R,et al.RestoreX-AI:A Contrastive Approach towards Guiding Image Restoration via Explainable AI Systems[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:3030-3039.
[51]TORBUNOV D,HUANG Y,YU H,et al.Uvcgan:Unet vision transformer cycle-consistent gan for unpaired image-to-image translation[C]//Proceedings of the IEEE/CVF Winter Confe-rence on Applications of Computer Vision.2023:702-712.
[52]GUI J,SUN Z,WEN Y,et al.A review on generative adversarial networks:Algorithms,theory,and applications[J].IEEE Transactions on Knowledge and Data Engineering,2021,35(4):3313-3332.
[53]MIRZA M,OSINDERO S.Conditional generative adversarialnets[J].arXiv:1411.1784,2014.
[54]ISLAM M J,XIA Y,SATTAR J.Fast underwater image enhancement for improved visual perception[J].IEEE Robotics and Automation Letters,2020,5(2):3227-3234.
[55]YANG M,HU K,DU Y,et al.Underwater image enhancement based on conditional generative adversarial network[J].Signal Processing:Image Communication,2020,81:115723.
[56]LI Y,YANG D Y,LIU L Y,et al.Underwater Image Enhancement Using Generative Adversarial Networks[J].Journal of Shanghai Jiao Tong University,2022,56(2):134-142.
[57]PENG Y F,LI J,GU L R,et al.Underwater Image Enhancement Method Based on Improved Conditional Generative Adversarial Network[J].Chinese Journal of Liquid Crystals and Displays,2022,37(6):768-776.
[58]TANG P,LI L,XUE Y,et al.Real-World Underwater ImageEnhancement Based on Attention U-Net[J].Journal of Marine Science and Engineering,2023,11(3):662.
[59]LIU X,LIN S,TAO Z.Learning multiscale pipeline gated fusion for underwater image enhancement[J].Multimedia Tools and Applications,2023,82(21):32281-32304.
[60]ZHU J Y,PARK T,ISOLA P,et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision.2017:2223-2232.
[61]FABBRI C,ISLAM M J,SATTAR J.Enhancing underwater imagery using generative adversarial networks[C]//2018 IEEE International Conference on Robotics and Automation(ICRA).IEEE,2018:7159-7165.
[62]LU J,LI N,ZHANG S,et al.Multi-scale adversarial network for underwater image restoration[J].Optics & Laser Technology,2019,110:105-113.
[63]ZONG X,CHEN Z,WANG D.Local-CycleGAN:a general end-to-end network for visual enhancement in complex deep-water environment[J].Applied Intelligence,2021,51:1947-1958.
[64]HU K,ZHANG Y,WENG C,et al.An underwater image enhancement algorithm based on generative adversarial network and natural image quality evaluation index[J].Journal of Marine Science and Engineering,2021,9(7):691.
[65]MITTAL A,SOUNDARARAJAN R,BOVIK A C.Making a“completely blind” image quality analyzer[J].IEEE Signal Processing Letters,2012,20(3):209-212.
[66]ZHANG H,SUN L,WU L,et al.DuGAN:An effective framework for underwater image enhancement[J].IET Image Processing,2021,15(9):2010-2019.
[67]LIU C,WANG H R.An Improved CycleGAN-Based Method for Underwater Color Image Enhancement[J].Mechanical Science and Technology for Aerospace Engineering,2023,42(12):2093-2099.
[68]JIAN M,QI Q,DONG J,et al.The OUC-vision large-scale underwater image database[C]//2017 IEEE International Confe-rence on Multimedia and Expo(ICME).IEEE,2017:1297-1302.
[69]AKKAYNAK D,TREIBITZ T.Sea-thru:A method for remo-ving water from underwater images[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:1682-1691.
[70]LIU R,FAN X,ZHU M,et al.Real-world underwater enhancement:Challenges,benchmarks,and solutions under natural light[J].IEEE Transactions on Circuits and Systems for Video Technology,2020,30(12):4861-4875.
[71]ISLAM M J,LUO P,SATTAR J.Simultaneous enhancement and super-resolution of underwater imagery for improved visual perception[J].arXiv:2002.01155,2020.
[72]SONG W,WANG Y,HUANG D,et al.Enhancement of underwater images with statistical model of background light and optimization of transmission map[J].IEEE Transactions on Broadcasting,2020,66(1):153-169.
[73]PENG L,ZHU C,BIAN L.U-shape transformer for underwater image enhancement[J].IEEE Transactions on Image Proces-sing,2023,32:3066-3079.
[74]HAN J,SHOEIBY M,MALTHUS T,et al.Underwater image restoration via contrastive learning and a real-world dataset[J].Remote Sensing,2022,14(17):4297.
[75]CHANG L,SONG H,LI M,et al.UIDEF:A real-world underwater image dataset and a color-contrast complementary image enhancement framework[J].ISPRS Journal of Photogrammetry and Remote Sensing,2023,196:415-428.
[76]ZHAI G,MIN X.Perceptual image quality assessment:a survey[J].Science China Information Sciences,2020,63:1-52.
[77]MÜNSSON L.Methods for the evaluation of image quality:a review[J].Radiation Protection Dosimetry,2000,90(1/2):89-99.
[78]BOSSE S,MANIRY D,MÜLLER K R,et al.Deep neural networks for no-reference and full-reference image quality assessment[J].IEEE Transactions on Image Processing,2017,27(1):206-219.
[79]WANG Z,BOVIK A C,SHEIKH H R,et al.Image qualityassessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[80]YANG M,SOWMYA A.An underwater color image quality evaluation metric[J].IEEE Transactions on Image Processing,2015,24(12):6062-6071.
[81]PANETTA K,GAO C,AGAIAN S.Human-visual-system-inspired underwater image quality measures[J].IEEE Journal of Oceanic Engineering,2015,41(3):541-551.
[82]ZHENG Y,CHEN W,LIN R,et al.UIF:An objective qualityassessment for underwater image enhancement[J].IEEE Transactions on Image Processing,2022,31:5456-5468.
[83]KHAN Z A,DARDOURI T,KAANICHE M,et al.NNCD-IQA:A new neural networks based compressed database for image quality assessment[J].Multimedia Tools and Applications,2023,82(9):13951-13971.
[84]GAO R,HUANG Z,LIU S.QL-IQA:Learning distance distribution from quality levels for blind image quality assessment[J].Signal Processing:Image Communication,2022,101:116576.
[85]CAO Y D,LIU H Y,JIA X,et al.A Survey on Image QualityEvaluation Methods Based on Deep Learning[J].Computer Engineering and Applications,2021,57(23):27-36.
[86]FU Z,FU X,HUANG Y,et al.Twice mixing:a rank learning based quality assessment approach for underwater image enhancement[J].Signal Processing:Image Communication,2022,102:116622.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed