计算机科学

计算机科学 ›› 2025, Vol. 52 ›› Issue (10): 123-133.doi: 10.11896/jsjkx.240800013

• 计算机图形学&多媒体 • 上一篇    下一篇

SAM-Retina:基于SAM的双模态视网膜图像动静脉分割

许恒宇1, 陈坤2, 徐琳1,3, 孙明斋4,5, 陆洲1   

  1. 1 安徽师范大学物理与电子信息学院光电信息材料功能调控与应用安徽省重点实验室 安徽 芜湖 241002
    2 中国科学技术大学精密机械与精密仪器系 合肥 230000
    3 安徽师范大学计算机与信息学院工业智能数据安全安徽省重点实验室 安徽 芜湖 241002
    4 中国科学技术大学生物医学工程学院 合肥 230026
    5 中国科学技术大学苏州高等研究院 江苏 苏州 215000
  • 收稿日期:2024-08-02 修回日期:2024-11-25 出版日期:2025-10-15 发布日期:2025-10-14
  • 通讯作者: 徐琳(linxu@ahnu.edu.cn)
  • 作者简介:(2121021186@ahnu.edu.cn)
  • 基金资助:
    国家自然科学基金面上项目(22073001);网络与信息安全安徽省重点实验室课题(AHNIS2021001);芜湖市重点研发与成果转化项目(2023yf110);安徽师范大学校级科研项目培育项目(2022xjxm052)

SAM-Retina:Arteriovenous Segmentation in Dual-modal Retinal Image Based on SAM

XU Hengyu1, CHEN Kun2, XU Lin1,3, SUN Mingzhai4,5, LU Zhou1   

  1. 1 Anhui Key Laboratory for Control, Applications of Optoelectronic Information Materials, School of Physics, Electronic Information, Anhui Normal University, Wuhu, Anhui 241002,China
    2 Department of Precision Machinery and Precision Instrumentation,University of Science and Technology of China,Hefei 230000,China
    3 Anhui Provincial Key Laboratory of Industrial Intelligence Data Security,School of Computer and Information,Anhui Normal University,Wuhu,Anhui 241002,China
    4 School of Biomedical Engineering,University of Science and Technology of China,Hefei 230026,China
    5 Suzhou Institute for Advanced Research,University of Science and Technology of China,Suzhou,Jiangsu 215000,China
  • Received:2024-08-02 Revised:2024-11-25 Online:2025-10-15 Published:2025-10-14
  • About author:XU Hengyu,born in 1999,postgraduate,is a member of CCF(No.X8654G).His main research interest is deep learning.
    XU Lin,born in 1981,master,lecturer,master supervisor.Her main research interests include smart health and computer vision.
  • Supported by:
    National Natural Science Foundation of China(22073001),Anhui Provincial Key Laboratory of Network and Information Security Project(AHNIS2021001),Key R&D and Achievement Transformation Projects in Wuhu(2023yf110) and Anhui Normal University School-level Research Project Incubation Project(2022xjxm052).

PDF (PC)

2

摘要： 动脉与静脉在RGB视网膜成像中形态高度相似,且其本身结构兼具细微性和复杂性,导致现阶段多数视网膜图像处理所使用的动静脉分割模型难以取得理想效果。为提高动静脉分割的准确性,同时降低训练成本,提出了一种基于SAM(Segment Anything Model)的视网膜分割模型——SAM-Retina。SAM-Retina采用特征融合器-适配型图像编码器-掩码解码器架构,使用同时包含RGB图像以及570 nm和610 nm单波长图像的结构-功能双模态视网膜图像代替原有的单模态(RGB)图像作为输入,利用特征融合器融合这3种图像的特征;通过在视觉转换器中插入Adapter模块并对其加以更新,保留图像编码器在大规模自然图像数据集上的预训练参数;使用静态提示嵌入代替提示编码器,去除原有SAM分割流程中的提示输入过程和提示编码过程。实验阶段将模型在DualModal2019和HRF数据集上进行训练和评估,并与U-Net,CRU-Net和TW-GAN进行对比。结果表明,相较于对比模型,SAM-Retina在各项评估指标上效果更好,尤其是双模态图像的引入,使得在无需扩大模型规模的前提下,有效提升了分割性能。

关键词: 双模态视网膜图像, 动静脉分割, 图像编码器, 视觉转换器, 静态提示嵌入

Abstract: The shapes of arteries and veins are highly similar in RGB retinal imaging,and their inherent structures are both subtle and complex,making it difficult for most retinal image processing models to achieve ideal results.To improve the accuracy of arteriovenous segmentation and reduce training costs,a retinal segmentation model based on segment anything model is proposed—SAM-Retina.SAM-Retina adopts a feature fusion,adaptive image encoder,and mask decoder architecture,using structure-and-function dual-modal retinal images simultaneously containing RGB as well as 570 nm and 610 nm single wavelength images instead of a single mode(RGB) image as input.The features of these three images are fused through a feature fusion.While retaining the pre-trained parameters of the image encoder on a large-scale natural image dataset,the model's feature extraction capability on retinal medical images is enhanced by inserting an adapter module and updating it within the vision transformer(ViT) block.The static prompt embedding instead of prompt encoder is adopted to remove the input and encoding process of prompts in the original SAM segmentation process.During the experimental phase,the model is trained and evaluated on the DualModal2019 and HRF datasets,and compared with U-Net,CRU-Net,and TW-GAN.The experimental results show that SAM-Retina is more advanced than other models in various evaluation indicators and the employment of dual-modal image also improves segmentation perfor-mance without increasing the model size.

Key words: Dual-modal retinal image,Arteriovenous segmentation,Image encoder,Vision transformer,Static prompt embedding

中图分类号: 

  • TP391
[1]CHEN Y X,WEN L,PEI C W,et al.Changes of microvascular diameter in non-proliferative diabetic retinopathy[J].International Eye Science,2021,9(21):1632-1636.
[2]MURSCH-EDLMAYR A S,BOLZ M,STROHMAIER C.Vascular aspects in glaucoma:from pathogenesis to therapeutic approaches[J].International Journal of Molecular Sciences,2021,22(9):4662.
[3]CHEUNG C Y,MOK V,FOSTER P J,et al.Retinal imaging in Alzheimer's disease[J].Journal of Neurology,Neurosurgery & Psychiatry,2021,92(9):983-994.
[4]RIM T H,TEO A W J,YANG H H S,et al.Retinal vascular signs and cerebrovascular diseases[J].Journal of Neuro-Ophthalmology,2020,40(1):44-59.
[5]IKRAM M K,DE JONG F J,VINGERLING J R,et al.Are retinal arteriolar or venular diameters associated with markers for cardiovascular disorders? The Rotterdam Study[J].Investigative Ophthalmology & Visual Science,2004,45(7):2129-2134.
[6]DASHTBOZORG B,MENDONÇA A M,CAMPILHO A.Anautomatic method for the estimation of arteriolar-to-venular ratio in retinal images[C]//Proceedings of the 26th IEEE International Symposium on Computer-based Medical Systems.IEEE,2013:512-513.
[7]CIURICĂ S,LOPEZ-SUBLET M,LOEYS B L,et al.Arterialtortuosity:novel implications for an old phenotype[J].Hypertension,2019,73(5):951-960.
[8]TELISCHAK N A,YEDAVALLI V,MASSOUD T F.Tortuosi-ty of superior cerebral veins:comparative magnetic resonance imaging morphometrics in normal subjects and arteriovenous malformation patients[J].Clinical Anatomy,2021,34(3):326-332.
[9]YU S,LAKSHMINARAYANAN V.Fractal dimension and reti-nal pathology:a meta-analysis[J].Applied Sciences,2021,11(5):2376.
[10]MILANI P,MONTESANO G,ROSSETTI L,et al.Vessel density,retinal thickness,and choriocapillaris vascular flow in myopic eyes on OCT angiography[J].Graefe's Archive for Clinical and Experimental Ophthalmology,2018,256:1419-1427.
[11]DASHTBOZORG B,MENDONCA A M,CAMPILHO A.AnAutomatic Graph-Based Approach for Artery/Vein Classification in Retinal Images[J].IEEE Transactions on Image Proces-sing,2014,23(3):1073-1083.
[12]ZHAO Y,XIE J,ZHANG H,et al.Retinal vascular network topology reconstruction and artery/vein classification via dominant set clustering[J].IEEE Transactions on Medical Imaging,2019,39(2):341-356.
[13]WELIKALA R A,FOSTER P J,WHINCUP P H,et al.Automated arteriole and venule classification using deep learning for retinal images from the UK Biobank cohort[J].Computers in Biology and Medicine,2017,90:23-32.
[14]REMESEIRO B,MENDONÇA A M,CAMPILHO A.Automa-tic classification of retinal blood vessels based on multilevel thresholding and graph propagation[J].The Visual Computer,2021,37:1247-1261.
[16]RONNEBERGER O,FISCHER P,BROX T.U-net:Convolu-tional networks for biomedical image segmentation[C]//Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015:18th International Conference,Munich,Germany,October 5-9,2015,Proceedings,Part III 18.Springer,2015:234-241.
[17]HEMELINGS R,ELEN B,STALMANS I,et al.Artery-veinsegmentation in fundus images using a fully convolutional network[J].Computerized Medical Imaging and Graphics,2019,76:101636.
[18]KARLSSON R A,HARDARSON S H.Artery vein classification in fundus images using serially connected U-Nets[J].Computer Methods and Programs in Biomedicine,2022,216:106650.
[19]CHEN W,YU S,MA K,et al.TW-GAN:Topology and width aware GAN for retinal artery/vein classification[J].Medical Image Analysis,2022,77:102340.
[20]ZHANG J,YANG K,SHEN Z,et al.End-to-End AutomaticClassification of Retinal Vessel Based on Generative Adversarial Networks with Improved U-Net[J].Diagnostics,2023,13(6):1148.
[21]WALD T,ROY S,KOEHLER G,et al.SAM.MD:Zero-shotmedical image segmentation capabilities of the Segment Anything Model[C]//Medical Imaging with Deep Learning.2023.
[22]MA J,HE Y,LI F,et al.Segment Anything in Medical Images[J].arXiv:2304.12306,2023.
[23]MAZUROWSKI M A,DONG H,GU H,et al.Segment any-thing model for medical image analysis:an experimental study[J].Medical Image Analysis,2023,89:102918.
[24]DENG R,CUI C,LIU Q,et al.Segment anything model(sam) for digital pathology:Assess zero-shot segmentation on whole slide imaging[J].arXiv:2304.04155,2023.
[25]KIRILLOV A,MINTUN E,RAVI N,et al.Segment Anything[C]//2023 IEEE/CVF International Conference on Computer Vision.2023.
[26]CHEN H Y.Calibration of Retinal Oximetry Devices by Fun-dus-simulating Phantoms[D].Hefei:University of Science and Technology of China,2019.
[27]TEAM G,ANIL R,BORGEAUD S,et al.Gemini:a family ofhighly capable multimodal models[J].arXiv:2312.11805,2023.
[28]ROMBACH R,BLATTMANN A,LORENZ D,et al.High-reso-lution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:10684-10695.
[29]RAMESH A,PAVLOV M,GOH G,et al.Zero-shot text-to-image generation[C]//International Conference on Machine Learning.PMLR,2021:8821-8831.
[30]WU J,ZHANG Y,FU R,et al.Medical SAM Adapter:Adapting Segment Anything Model for Medical Image Segmentation[J].arXiv:2306.12620,2023.
[31]DOSOVITSKIY A,BEYER L,KOLESNIKOV A,et al.Animage is worth 16x16 words:Transformers for image recognition at scale[J].arXiv:2010.11929,2020.
[32]HU E J,SHEN Y,WALLIS P,et al.LORA:Low-rank adaptation of large language models[J].arXiv:2106.09685,2021.
[33]ZHANG S,ZHENG R,LUO Y,et al.Simultaneous Arterioleand Venule Segmentation of Dual-Modal Fundus Images Using a Multi-Task Cascade Network[J].IEEE Access,2019,7:57561-57573.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发