计算机科学 ›› 2026, Vol. 53 ›› Issue (4): 284-290.doi: 10.11896/jsjkx.250600188

• 计算机图形学&多媒体 • 上一篇    下一篇

基于张量的多模态融合诊断微血管侵犯

汪少东1, 李柳军2, 李蕊1, 苏中振2, 陆遥1   

  1. 1 中山大学计算机学院 广州 510006
    2 中山大学附属第五医院超声科 广东 珠海 519000
  • 收稿日期:2025-06-26 修回日期:2025-09-08 出版日期:2026-04-15 发布日期:2026-04-08
  • 通讯作者: 陆遥(luyao23@mail.sysu.edu.cn)
  • 作者简介:(wangshd6@mail2.sysu.edu.cn)
  • 基金资助:
    国家重点研发计划(2023YFE0204300);琶洲实验室(黄埔区)研发项目(2023K0606);国家自然科学基金(82441027,62371476);广州市科学技术局项目(2023B03J1237);湖南省卫生健康委卫生健康科研重大项目(W20241010);广东省计算科学重点实验室(中山大学)(2020B1212060032)

Tensor-based Multimodal Fusion Technique to Diagnose Microvascular Invasion

WANG Shaodong1, LI Liujun2, LI Rui1, SU Zhongzhen2, LU Yao1   

  1. 1 School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006, China
    2 Department of Ultrasound, the Fifth Affiliated Hospital of Sun Yat-Sen University, Zhuhai, Guangdong 519000, China
  • Received:2025-06-26 Revised:2025-09-08 Published:2026-04-15 Online:2026-04-08
  • About author:WANG Shaodong,born in 2000,postgraduate.His main research interests include deep learning and medical imaging.
    LU Yao,born in 1979,Ph.D,professor.His main research interests include medical artificial intelligence,computer-aided diagnosis,medical image analysis,machine learning,radiomics,and medical big data analysis.
  • Supported by:
    National Key R & D Program of China(2023YFE0204300),the R&D Project of Pazhou Lab(HuangPu)(2023K0606),National Natural Science Foundation of China(82441027,62371476),Guangzhou Science and Technology Bureau(2023B03J1237),Health Research Major Projects of Hunan Health Commission(W20241010) and Guangdong Province Key Laboratory of Computational Science at the Sun Yat-sen University(2020B1212060032).

摘要: 微血管侵犯(MVI)作为肝细胞癌(HCC)术后复发和生存率低的关键预后因素,其术前精准定位对治疗决策至关重要。针对现有放射组学方法特征泛化弱、可解释性差且忽略瘤周MVI空间分布的问题,提出通过病理全切片(WSI)与三维超声(3D US)的空间融合实现MVI三维定位,并设计特征张量融合深度学习模型(融合多尺度特征、特征张量及正交化损失函数)提取瘤周MVI分布语义特征。在收集的数据集上开展了详细的对比分析和消融实验研究,使用受试者工作特征曲线下的面积(AUC)、准确度(Accuracy)和F1分数等指标证明了该模型的有效性。实验验证了该模型性能优异(AUC:0.910,ACC:0.930,F1 score:0.852),证实了其在术前MVI精确诊断中的临床潜力。

关键词: 微血管侵犯, 多模态, 张量融合, 多尺度融合, 肝细胞癌

Abstract: Microvascular invasion(MVI) is a critical prognostic factor for postoperative recurrence and reduced survival in hepatocellular carcinoma(HCC),making its precise preoperative localization essential for treatment planning.To address limitations of existing radiomics approaches—including poor feature generalizability,weak interpretability,and neglect of spatial peritumoral MVI distribution crucial for surgical strategies—this study proposes:1) multimodal fusion-based 3D localization of MVI via spatial alignment of whole-slide pathology images(WSIs) with 3D ultrasound(3D US);2) a feature tensor fusion deep learning model integrating multiscale features,tensor fusion,and orthogonal loss functions to extract semantic features of peritumoral MVI distribution.Model performance is evaluated on the curated dataset using metrics such as the area under the receiver operating cha-racteristic curve(AUC),Accuracy and F1 score,demonstrating its effectiveness.Experimental validation demonstrates exceptional performance(AUC:0.910,Accuracy:0.930,F1-score:0.852),confirming the proposed model’s clinical potential for precise preoperative MVI diagnosis.

Key words: Microvascular invasion, Multimodal, Tensor fusion, Multi-scale fusion, Hepatocellular carcinoma

中图分类号: 

  • TP183
[1]SUNG H,FERLAY J,SIEGEL R L,et al.Global cancer statistics 2020:GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries[J].CA:A Cancer Journal for Clinicians,2021,71(3):209-249.
[2]VILLANUEVA A.Hepatocellular carcinoma.reply[J].TheNew England journal of medicine,2019,381(1):e2.
[3]VITALE A,PECK-RADOSAVLJEVIC M,GIANNINI E G,et al.Personalized treatment of patients with very early hepatocellular carcinoma[J].Journal of Hepatology,2017,66(2):412-423.
[4]ZHOU J,SUN H,WANG Z,et al.Guidelines for the diagnosis and treatment of hepatocellularcarcinoma(2019 edition)[J].Liver Cancer,2020,9(6):682-720.
[5]NJEI B,ROTMAN Y,DITAH I,et al.Emerging trends in hepatocellular carcinoma incidence and mortality[J].Hepatology,2015,61(1):191-199.
[6]MARENGO A,ROSSO C,BUGIANESI E.Liver cancer:con-nections with obesity,fatty liver,and cirrhosis[J].Annual Review of Medicine,2016,67(1):103-117.
[7]LAI C L,RATZIU V,YUEN M F,et al.Viral hepatitis B[J].The Lancet,2003,362(9401):2089-2094.
[8]YANG L,GU D,WEI J,et al.A radiomics nomogram for preoperative prediction of microvascular invasion in hepatocellular carcinoma[J].Liver Cancer,2019,8(5):373-386.
[9]CHAN A W H,ZHONG J,BERHANE S,et al.Development ofpre and post-operative models to predict early recurrence of hepatocellular carcinoma after surgical resection[J].Journal of Hepatology,2018,69(6):1284-1293.
[10]CHEN Z H,ZHANG X P,FENG J K,et al.Actual long-term survival in hepatocellular carcinoma patients with microvascular invasion:a multicenter study from China[J].Hepatology International,2021,15:642-650.
[11]DONG Y,WANG Q M,LI Q,et al.Preoperative prediction of microvascular invasion of hepatocellular carcinoma:radiomics algorithm based on ultrasound original radio frequency signals[J].Frontiers in Oncology,2019,9:1203.
[12]XU X,ZHANG H L,LIU Q P,et al.Radiomic analysis of contrast-enhanced CT predicts microvascular invasion and outcome in hepatocellular carcinoma[J].Journal of Hepatology,2019,70(6):1133-1144.
[13]NITTA H,ALLARD M A,SEBAGH M,et al.Prognostic value and prediction of extratumoral microvascular invasion for hepatocellular carcinoma[J].Annals of Surgical Oncology,2019,26:2568-2576.
[14]LI S,ZENG Q,LIANG R,et al.Using systemic inflammatorymarkers to predict microvascular invasion before surgery in patients with hepatocellular carcinoma[J].Frontiers in Surgery,2022,9:833779.
[15]MAO S,YU X,YANG Y,et al.Preoperative nomogram for microvascular invasion prediction based on clinical database in hepatocellular carcinoma[J].Scientific Reports,2021,11(1):13999.
[16]SHI L,QU Z,YANG Y,et al.Analysis of Related Risk Factors of Microvascular Invasion in Hepatocellular Carcinoma[J].Computational and Mathematical Methods in Medicine,2022,2022(1):8195512.
[17]ZHOU Q,ZHOU C,YIN Y,et al.Development and validation of a nomogram combining hematological and imaging features for preoperative prediction of microvascular invasion in hepatocellular carcinoma patients[J].Annals of Translational Medicine,2021,9(5):402.
[18]COMEN E A,BOWMAN R L,KLEPPE M.Underlying causes and therapeutic targeting of the inflammatory tumor microenvironment[J].Frontiers in Cell and Developmental Biology,2018,6:56.
[19]WANG H,LU Y,LIU R,et al.A non-invasive nomogram for preoperative prediction of microvascular invasion risk in hepatocellular carcinoma[J].Frontiers in Oncology,2021,11:745085.
[20]ZHAN J,WANG J,ZHANG Z,et al.Noninvasive diagnosis of significant liver inflammation in patients with chronic hepatitis B in the indeterminate phase[J].Virulence,2023,14(1):2268497.
[21]CHONG H H,YANG L,SHENG R F,et al.Multi-scale and multi-parametric radiomics of gadoxetate disodium-enhanced MRI predicts microvascular invasion and outcome in patients with solitary hepatocellular carcinoma≤ 5 cm[J].European Radiology,2021,31:4824-4838.
[22]ABI-JAOUDEH N,KRUECKER J,KADOURY S,et al.Multimodality image fusion-guided procedures:technique,accuracy,and applications[J].Cardiovascular and Interventional Radiology,2012,35:986-998.
[23]CARCINOMA VILLANUEVA A.Hepatocellular[J].The New England Journal of Medicine,2019,380:1450-1462.
[24]ZHANG R,XU L,WEN X,et al.A nomogram based on bi-regional radiomics features from multimodal magnetic resonance imaging for preoperative prediction of microvascular invasion in hepatocellular carcinoma[J].Quantitative Imaging in Medicine and Surgery,2019,9(9):1503.
[25]LIU B,ZENG Q,HUANG J,et al.IVIM using convolutional neural networks predicts microvascular invasion in HCC[J].European Radiology,2022,32(10):7185-7195.
[26]WANG L,WU M,LI R,et al.MVI-Mind:a novel deep-learning strategy using computed tomography(CT)-based radiomics for end-to-end high efficiency prediction of microvascular invasion in hepatocellular carcinoma[J].Cancers,2022,14(12):2956.
[27]PERCIE DU SERT N,HURST V,AHLUWALIA A,et al.The ARRIVE guidelines 2.0:Updated guidelines for reporting animal research[J].Journal of Cerebral Blood Flow & Metabolism,2020,40(9):1769-1777.
[28]BEUTEL J.Handbook of medical imaging[M].Spie Press,2000.
[29]VASWANI A,SHAZEER N,PARMAR N,et al.Attention is all you need[C]//Proceedings of the 31st International Confe-rence on Neural Information Processing Systems.2017:6000-6010.
[1] 尹创, 刘建毅, 张茹.
跨模态融合的少样本勒索软件分类器:基于预训练模型的多模态编码
Cross-modal Fusion Few-sample Ransomware Classifier:Multimodal Encoding Based on Pre-trained Models
计算机科学, 2026, 53(4): 435-444. https://doi.org/10.11896/jsjkx.250500078
[2] 徐成, 刘宇轩, 王欣, 张铖, 姚登峰, 袁家政.
大语言模型驱动的言语障碍评估方法综述
Review of Speech Disorder Assessment Methods Driven by Large Language Models
计算机科学, 2026, 53(3): 307-320. https://doi.org/10.11896/jsjkx.250300125
[3] 杜剑彤, 管泽礼, 薛哲.
基于多任务学习的眼科视频特征融合与多维画像
Multi-task Learning-based Ophthalmic Video Feature Fusion and Multi-dimensional Profiling
计算机科学, 2026, 53(3): 383-391. https://doi.org/10.11896/jsjkx.260200058
[4] 卓铁农, 英迪, 赵晖.
融合跨模态注意力与角色交互的学生课堂专注度研究
Research on Student Classroom Concentration Integrating Cross-modal Attention and Role
Interaction
计算机科学, 2026, 53(2): 67-77. https://doi.org/10.11896/jsjkx.250300026
[5] 陈海涛, 梁俊威, 陈晨, 王宇帆, 周宇.
基于多模态体育教育数据的图空间融合动作识别方法
Multimodal Physical Education Data Fusion via Graph Alignment for Action Recognition
计算机科学, 2026, 53(2): 89-98. https://doi.org/10.11896/jsjkx.250800007
[6] 黄靖, 王腾, 刘健, 胡凯, 彭鑫, 黄亚敏, 文元桥.
多模态水声图像目标视觉检测
Multimodal Visual Detection for Underwater Sonar Target Images
计算机科学, 2026, 53(2): 227-235. https://doi.org/10.11896/jsjkx.241200082
[7] 万盛华, 徐兴业, 甘乐, 詹德川.
基于多模态大模型辅助视频动作生成的预训练世界模型
Pre-training World Models from Videos with Generated Actions by Multi-modal Large Models
计算机科学, 2026, 53(1): 51-57. https://doi.org/10.11896/jsjkx.250800033
[8] 赵文豪, 梅萌, 王小平, 罗航宇.
PKHOI:利用先验知识增强人-物交互检测算法
PKHOI:Enhancing Human-Object Interaction Detection Algorithms with Prior Knowledge
计算机科学, 2026, 53(1): 141-152. https://doi.org/10.11896/jsjkx.250100086
[9] 卜韵阳, 齐彬廷, 卜凡亮.
跨模态不一致感知下双视角交互融合的多模态情感分析
Multimodal Sentiment Analysis for Interactive Fusion of Dual Perspectives Under Cross-modalInconsistent Perception
计算机科学, 2026, 53(1): 187-194. https://doi.org/10.11896/jsjkx.241100029
[10] 范家斌, 王宝会, 陈继轩.
基于文本-图像多模态融合的变电所布局图纸图符检测方法
Method for Symbol Detection in Substation Layout Diagrams Based on Text-Image MultimodalFusion
计算机科学, 2026, 53(1): 206-215. https://doi.org/10.11896/jsjkx.250200090
[11] 宋志超, 张建平, 张其阳, 方玺, 谢良, 宋少莉, 胡战利.
面向肿瘤早期诊断的延迟PET图像重建:多模态PET/CT核矩阵约束延迟成像算法
Delayed PET Reconstruction for Early Tumor Diagnosis:Multimodal PET/CT Nuclear Matrix- constrained Delayed Imaging Algorithm
计算机科学, 2025, 52(9): 119-127. https://doi.org/10.11896/jsjkx.250400037
[12] 刘威, 许勇, 方娟, 李城, 祝玉军, 方群, 何昕.
基于雷达和视觉融合的多模态空中手写体识别
Multimodal Air-writing Gesture Recognition Based on Radar-Vision Fusion
计算机科学, 2025, 52(9): 259-268. https://doi.org/10.11896/jsjkx.240400143
[13] 高龙, 李旸, 王素格.
基于分步协作融合表示的情感分类方法
Sentiment Classification Method Based on Stepwise Cooperative Fusion Representation
计算机科学, 2025, 52(9): 313-319. https://doi.org/10.11896/jsjkx.240700161
[14] 刘健, 姚任远, 高楠, 梁荣华, 陈朋.
VSRI:基于视觉语义关系交互的图像字幕生成方法
VSRI:Visual Semantic Relational Interactor for Image Caption
计算机科学, 2025, 52(8): 222-231. https://doi.org/10.11896/jsjkx.240600082
[15] 岳彩通, 叶文豪, 张颖洁, 梁静, 林泓宇.
基于局部中心解聚类的多模态多目标优化算法
Multimodal Multiobjective Optimization Algorithm Based on Local Center Clustering
计算机科学, 2025, 52(8): 288-299. https://doi.org/10.11896/jsjkx.240700094
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!