Computer Science ›› 2024, Vol. 51 ›› Issue (8): 176-182.doi: 10.11896/jsjkx.230700088

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Color Transfer Method for Unpaired Medical Images Based on Color Flow Model

WANG Xiaojie, LIU Jinhua, LU Shuyi, ZHOU Yuanfeng   

  1. School of Software,Shandong University,Jinan 250101,China
  • Received:2023-07-12 Revised:2023-11-15 Online:2024-08-15 Published:2024-08-13
  • About author:WANG Xiaojie,born in 1990,doctoral student.Her main research interests include medical image processing and so on.
    ZHOU Yuanfeng,born in 1980,Ph.D,professor.His main research interests include geometric modeling,information visualization,and image processing.
  • Supported by:
    National Key R&D Plan on Strategic International Scientific and Technological Innovation Cooperation Special Project(2021YFE0203800),NSFC-Zhejiang Joint Fund of the Integration of Informatization and Industrialization(U1909210) and National Natural Science Foundation of China(62172257).

Abstract: In clinical applications,CT image is a kind of image data that is relatively easy to obtain,but there is a large gap between them and the real human body color.The tomographic color image of the human body is the color response of the real human body,but it is a rare data.Combining the two,so that each case can get its own color CT data,which will have a effect on the doctor’s surgery and the patient’s understanding to the disease.Therefore,this paper proposes a medical image colorization framework based on a color flow model.It first inputs the CT and human color data into the color flow model and extracts the content and color features.Then,the color and texture transfer work is performed at the feature level.Finally,the processed feature information is re-input into the reversible color flow model for image reconstruction.After each flow module,we add a texture constraint loss to make the shaded image more textured.At the same time,we add edge constraints to ensure that the characteristics of small blood vessels and other tissues on the medical image are not lost.Qualitative and quantitative experiments prove that our method is more robust than other colorization methods,and the experimental results are more realistic.And we conduct extensive experiments on different data domains,proving that our method is not affected by domain shift and can obtain stable experimental results.At the same time,the proposed method can display a clear organizational structure without adjusting the window width/level.

Key words: Flow module, Colorization, Texture constraint, Stability, Edge constraint

CLC Number: 

  • TP391.41
[1]WELSH T,ASHIKHMIN M,MUELLER K.Transferring color to greyscale images[C]//Proceedings of the 29th Annual Confe-rence on Computer Graphics and Interactive Techniques.2002:277-280.
[2]KHAN T H,MOHAMMED S K,IMTIAZ M S,et al.Efficient color reproduction algorithm for endoscopic images based on dynamic color map[J].Journal of Medical and Biological Enginee-ring,2016,36:226-235.
[3]MATHUR A N,KHATTAR A,SHARMA O.2D to 3D medical image colorization[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.2021:2847-2856.
[4]AN J,HUANG S,SONGY,et al.Artflow:Unbiased image style transfer via reversible neural flows[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:862-871.
[5]GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Ge-nerative adversarial networks[J].Communications of the ACM,2020,63(11):139-144.
[6]DINH L,KRUEGER D,BENGIO Y.Nice:Non-linear indepen-dent components estimation[J].arXiv:1410.8516,2014.
[7]DINH L,SOHL-DICKSTEIN J,BENGIO S.Density estimation using real nvp[J].arXiv:1605.08803,2016.
[8]KINGMA D P,DHARIWAL P.Glow:Generative flow with invertible 1x1 convolutions[J].Advances in Neural Information Processing Systems,2018,31:10215-10224.
[9]KLOKOV R,BOYER E,VERBEEK J.Discrete point flow networks for efficient point cloud generation[C]//European Conference on Computer Vision.Cham:Springer International Publishing,2020:694-710.
[10]PUMAROLA A,POPOV S,MORENO-NOGUERF,et al.C-flow:Conditional generative flow models for images and 3d point clouds[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:7949-7958.
[11]LIU R,LIU Y,GONG X,et al.Conditional adversarial generative flow for controllable image synthesis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:7992-8001.
[12]LIU R,LIU Y,GONG X,et al.Conditional adversarial generative flow for controllable image synthesis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:7992-8001.
[13]ARDIZZONE L,LÜTH C,KRUSE J,et al.Guided image ge-neration with conditional invertible neural networks[J].arXiv:1907.02392,2019.
[14]WINKLER C,WORRALL D,HOOGEBOOME,et al.Learning likelihoods with conditional normalizing flows[J].arXiv:1912.00042,2019.
[15]LUGMAYR A,DANELLJAN M,VAN GOOL L,et al.Srflow:Learning the super-resolution space with normalizing flow[C]//Computer Vision-ECCV 2020:16th European Conference,Glasgow,UK,Part V 16.Springer International Publishing,2020:715-732.
[16]GATYS L A,ECKER A S,BETHGE M.Image style transferusing convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2414-2423.
[17]LUAN F,PARIS S,SHECHTMAN E,et al.Deep photo style transfer[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:4990-4998.
[18]ISOLA P,ZHU J Y,ZHOU T,et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:1125-1134.
[19]ZHU J Y,PARK T,ISOLAP,et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision.2017:2223-2232.
[20]DHARIWAL P,NICHOL A.Diffusion models beat gans onimage synthesis[J].Advances in Neural Information Processing Systems,2021,34:8780-8794.
[21]HO J,JAIN A,ABBEEL P.Denoising diffusion probabilisticmodels[J].Advances in Neural Information Processing Systems,2020,33:6840-6851.
[22]KINGMA D P,WELLING M.Auto-encoding variational bayes[J].arXiv:1312.6114,2013.
[23]IOFFE S,SZEGEDY C.Batch normalization:Accelerating deep network training by reducing internal covariate shift[C]//International Conference on Machine Learning.PMLR,2015:448-456.
[24]LI Y,FANG C,YANG J,et al.Diversified texture synthesiswith feed-forward networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:3920-3928.
[25]GATYS L,ECKER A S,BETHGE M.Texture synthesis using convolutional neural networks[J].Advances in Neural Information Processing Systems,2015,28:262-270.
[26]ULYANOV D,LEBEDEV V,VEDALDI A,et al.Texture networks:Feed-forward synthesis of textures and stylized images[J].arXiv:1603.03417,2016.
[27]ACKERMAN M J.The visible human project[J].Proceedings of the IEEE,1998,86(3):504-511.
[28]ARMATO III S G,MCLENNAN G,BIDAUT L,et al.The lung image database consortium(LIDC) and image database resource initiative(IDRI):a completed reference database of lung nodules on CT scans[J].Medical Physics,2011,38(2):915-931.
[29]WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality as-sessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[30]HUYNH-THU Q,GHANBARI M.Scope of validity of PSNR in image/video quality assessment[J].Electronics Letters,2008,44(13):800-801.
[31]CHOI E,LEE C.Feature extraction based on the Bhattacharyya distance[J].Pattern Recognition,2003,36(8):1703-1709.
[32]ROFER T.Using histogram correlation to create consistent laser scan maps[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems.IEEE,2002,1:625-630.
[33]TORBUNOV D,HUANG Y,YU H,et al.Uvcgan:Unet vision transformer cycle-consistent gan for unpaired image-to-image translation[C]//Proceedings of the IEEE/CVF Winter Confe-rence on Applications of Computer Vision.2023:702-712.
[1] SONG Jianfeng, ZHANG Wenying, HAN Lu, HU Guozheng, MIAO Qiguang. Multi-stage Intelligent Color Restoration Algorithm for Black-and-White Movies [J]. Computer Science, 2024, 51(5): 92-99.
[2] CHANG Sha, WU Yahui, DENG Su, MA Wubin, ZHOU Haohao. Online Task Allocation Strategy Based on Lyapunov Optimization in Mobile Crowdsensing [J]. Computer Science, 2023, 50(2): 50-56.
[3] ZHENG Wen-ping, LIU Mei-lin, YANG Gui. Community Detection Algorithm Based on Node Stability and Neighbor Similarity [J]. Computer Science, 2022, 49(9): 83-91.
[4] DOU Zhi, WANG Ning, WANG Shi-jie, WANG Zhi-hui, LI Hao-jie. Sketch Colorization Method with Drawing Prior [J]. Computer Science, 2022, 49(4): 195-202.
[5] ZHANG Jiu-jie, CHEN Chao, NIE Hong-xuan, XIA Yu-qin, ZHANG Li-ping, MA Zhan-fei. Empirical Study on Stability of Clone Code Sets Based on Class Granularity [J]. Computer Science, 2021, 48(5): 75-85.
[6] WANG Meng, DING Zhi-jun. New Device Fingerprint Feature Selection and Model Construction Method [J]. Computer Science, 2020, 47(7): 257-262.
[7] ZHU Lin-li, HUA Gang, GAO Wei. Stability Analysis of Ontology Learning Algorithm in Decision Graph Setting [J]. Computer Science, 2020, 47(5): 43-50.
[8] YANG Jia-ning, HUANG Xiang-sheng, LI Zong-han, RONG Can, LIU Dao-wei. Spatio-temporal Trajectory Prediction of Power Grid Based on Double Layers Stacked Long Short-term Memory [J]. Computer Science, 2019, 46(11A): 23-27.
[9] CAI Yu-xin, GONG Si-liang, YANG Ming, TANG Zhi-wei, ZHAO Bo. Personnel Identification System Based on Mobile Police [J]. Computer Science, 2019, 46(11A): 446-449.
[10] ZHU Jiang, LEI Yun, WANG Yan. Stability Based Energy-efficient Routing Protocol in Cognitive Wireless Sensor Networks [J]. Computer Science, 2018, 45(11): 97-102.
[11] MA Yuan-yuan, SHI Yong-yi, ZHANG Hong, LIN Qi and LI Qian-mu. Feature Processing Approach Based on MA-LSSVM in Safety Data [J]. Computer Science, 2017, 44(3): 237-241.
[12] XUAN Han-yu, LIU Hong-zhe, YUAN Jia-zheng, LI Qing and NIU Xiao-ning. Robust Multi-lane Detection Algorithm [J]. Computer Science, 2017, 44(11): 305-313.
[13] WANG Ying-bo and CHAI Jia-jia. Employing AS-FOA for Optimization of GRNN Network with Application to Financial Warning Research [J]. Computer Science, 2016, 43(7): 275-280.
[14] HU Yong-dong. Stability Analysis for Predicting LWDF Scheduling Algorithm in M-WiMAX [J]. Computer Science, 2016, 43(7): 120-124.
[15] CUI Jian-qun, CHEN Ai-ling, XIA Zhen-chang and WU Li-bing. High Stability Low Delay Spanning Tree Algorithm for Application Layer Multicast [J]. Computer Science, 2016, 43(6): 77-81.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!