计算机科学 ›› 2022, Vol. 49 ›› Issue (2): 69-82.doi: 10.11896/jsjkx.210900140
颜锐1,2, 梁智勇3, 李锦涛1, 任菲1
YAN Rui1,2, LIANG Zhi-yong3, LI Jin-tao1, REN Fei1
摘要: 肿瘤的精确诊断对患者的治疗方案选择和预后预测都非常重要。病理学诊断被认为是肿瘤诊断的 “金标准”,但是,病理学发展目前仍然面临着巨大的挑战,如病理医生的缺乏,特别是在欠发达地区和小医院,这将导致病理医生长期超负荷工作,同时,病理诊断严重依赖于病理医生的专业知识和诊断经验,病理医生的主观性导致了诊断不一致性的激增。全切片扫描图像 (Whole Slide Images,WSI)技术和深度学习方法的突破性进展为计算机辅助诊断和预后预测提供了新的发展机遇。苏木精-伊红( Hematoxylin-Eosin,H&E) 染色的组织病理切片可以很好地显示细胞形态和组织结构,而且制作简单、成本便宜、使用广泛。仅仅从H&E染色的病理图像可以预测什么?在将深度学习方法应用到病理图像领域之后,这个问题得到了新的答案。文中首先总结了基于深度学习和病理图像的肿瘤相关指标预测的整体研究框架,按照整体研究框架发展的顺序将其总结为3个逐渐推进的阶段:基于人工选取感兴趣的单张图片小块进行WSI预测研究、基于多数投票的WSI预测研究以及具有普遍适用性的WSI预测研究。其次简单介绍了4种在WSI预测中经常用到的监督学习或弱监督学习方法:卷积神经网络、循环神经网络、图神经网络和多示例学习。然后综述了可以通过病理图像预测的肿瘤相关指标以及其最新研究进展,文中主要从两个方面进行文献的综述:预测专家可以阅片识别的肿瘤相关指标(肿瘤分类、肿瘤分级、肿瘤区域识别)和预测专家无法阅片识别的肿瘤相关指标(基因变异预测、分子亚型预测、治疗效果评估、生存期预测)。最后展望了该领域面临的挑战和机遇。
中图分类号:
[1]ZHOU S K,GREENSPAN H,DAVATZIKOS C,et al.A re-view of deep learning in medical imaging:Image traits,technology trends,case studies with progress highlights,and future promises[J].arXiv:2008.09104,2020. [2]DENG S,ZHANG X,YAN W,et al.Deep learning in digital pathology image analysis:A survey[J].Frontiers of Medicine,2020,14(4):18. [3]ECHLE A,RINDTORFF N T,BRINKER T J,et al.Deeplearning in cancer pathology:A new generation of clinical biomarkers[J].British Journal of Cancer,2020,124:1-11. [4]YAN R,CHEN L M,LI J T,et al.Research progress of cancer classification based on deep learning and histopathological images[J].Medical Journal of Peking Union Medical College Hospital,2021,12:742-748. [5]LITJENS G,KOOI T,BEJNORDI B E,et al.A survey on deep learning in medical image analysis[J].Med. Image. Anal.,2017,42:60-88. [6]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014. [7]HE K,ZHANG X,REN S,et al.Deep residual learning forimage recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778. [8]SZEGEDY C,VANHOUCKE V,IOFFE S,et al.Rethinking the inception architecture for computer vision[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2818-2826. [9]HUANG G,LIU Z,MAATEN L V D,et al.Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:4700-4708. [10]RONNEBERGER O,FISCHER P,BROX T.U-net:Convolu-tional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Compu-ter-assisted Intervention.Springer,2015:234-241. [11]CHEN L C,PAPANDREOU G,KOKKINOS I,et al.Deeplab:Semantic image segmentation with deep convolutional nets,atrous convolution,and fully connected crfs[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,40:834-848. [12]CHEN L C,PAPANDREOU G,KOKKINOS I,et al.Semanticimage segmentation with deep convolutional nets and fully connected crfs[J].arXiv:1412.7062,2014. [13]DOSOVITSKIY A,BEYER L,KOLESNIKOV A,et al.Animage is worth 16x16 words:Transformers for Image Recognition at Scale[J].arXiv:2010.11929,2020. [14]WANG X,YANG S,ZHANG J,et al.Transpath:Transformer-based self-supervised learning for histopathological image classification[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Springer,2021:186-195. [15]GAO Z,HONG B,ZHANG X,et al.Instance-based vision trans-former for subtyping of papillary renal cell carcinoma in histopathological image[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Springer,2021:299-308. [16]LI H,YANG F,ZHAO Y,et al.Dt-mil:Deformable transformer for multi-instance learning on histopathological image[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Springer,2021:206-216. [17]WU Z,PAN S,CHEN F,et al.A comprehensive survey ongraph neural networks[J].IEEE Transactions on Neural Networks and Learning Systems,2020,32:4-24. [18]ILSE M,TOMCZAK J,WELLING M.Attention-based deepmultiple instance learning[C]//International Conference on Machine Learning.PMLR,2018:2127-2136. [19]PRADHAN A,PAUDYAL P,SINHA A,et al.Grading,staging and nottingham prognostic index scoring of breast carcinoma[J].Journal of Pathology of Nepal,2017,7:1078-1083. [20]LOEFFLER C M L,BRUECHLE N O,JUNG M,et al.Artificial intelligence-based detection of fgfr3 mutational status directly from routine histology in bladder cancer:A possible preselection for molecular testing?[J].European Urology Focus,2021. [21]KATHER J N,HEIJ L R,GRABSCH H I,et al.Pan-cancerimage-based detection of clinically actionable genetic alterations[J].Nature Cancer,2020,1:789-799. [22]SPANHOL F A,OLIVEIRA L S,PETITJEAN C,et al.Breast cancer histopathological image classification using convolutional neural networks[C]//International Joint Conference on Neural Networks.2016:717-726. [23]BAYRAMOGLU N,KANNALA J,HEIKKIL J.Deep learningfor magnification independent breast cancer histopathology image classification[C]//International Conference on Pattern Recognition.2017:2440-2445. [24]ARAÚJO T,ARESTA G,CASTRO E,et al.Classification ofbreast cancer histology images using convolutional neural networks[J].Plos One,2017,12(6):e0177544. [25]JIANG Y,CHEN L,ZHANG H,et al.Breast cancer histopathological image classification using convolutional neural networks with small se-resnet module[J].PloS one,2019,14:e0214587. [26]VESAL S,RAVIKUMAR N,DAVARI A A,et al.Classification of breast cancer histology images using transfer learning[C]//International Conference Image Analysis and Recognition.2018:812-819. [27]VANG Y S,CHEN Z,XIE X.Deep learning framework formulti-class breast cancer histology image classification[C]//International Conference Image Analysis and Recognition.2018:914-922. [28]RAKHLIN A,SHVETS A,IGLOVIKOV V,et al.Deep convolutional neural networks for breast cancer histology image analysis[C]//International Conference Image Analysis and Recognition.2018:737-744. [29]SHABAN M,AWAN R,FRAZ M M,et al.Context-aware convolutional neural network for grading of colorectal cancer histo-logy images[J].IEEE Transactions on Medical Imaging,2020,39(7):2395-2405. [30]YAN R,REN F,WANG Z,et al.Breast cancer histopathological image classification using a hybrid deep neural network[J].Methods,2020,173:52-60. [31]GAO Z,SHI J,WANG J.Gq-gcn:Group quadratic graph convolutional network for classification of histopathological images[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Springer,2021:121-131. [32]COUDRAY N,OCAMPO P S,SAKELLAROPOULOS T,et al.Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning[J].Nature Medicine,2018,24:1559-1567. [33]TAKAHAMA S,KUROSE Y,MUKUTA Y,et al.Multi-stage pathological image classification using semantic segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2019:10702-10711. [34]ADNAN M,KALRA S,TIZHOOSH H R.Representationlearning of histopathology images using graph neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.2020:988-989. [35]TELLEZ D,LITJENS G,LAAK J V D,et al.Neural imagecompression for gigapixel histopathology image analysis[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2019,43:567-578. [36]CAMPANELLA G,HANNA M G,GENESLAW L,et al.Clinical-grade computational pathology using weakly supervised deep learning on whole slide images[J].Nature Medicine,2019,25:1301-1309. [37]WANG X,CHEN H,GAN C,et al.Weakly supervised deeplearning for whole slide lung cancer image analysis[J].IEEE Transactions on Cybernetics,2019,50:3950-3962. [38]WANG S,ZHU Y,YU L,et al.Rmdl:Recalibrated multi-instance deep learning for whole slide gastric image classification[J].Medical Image Analysis,2019,58:101549. [39]LI B,LI Y,ELICEIRI K W.Dual-stream multiple instancelearning network for whole slide image classification with self-supervised contrastive learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:14318-14328. [40]KÄLLÉN H,MOLIN J,HEYDEN A,et al.Towards gradinggleason score using generically trained deep convolutional neural networks[C]//2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).IEEE,2016:1163-1167. [41]WAN T,CAO J,CHEN J,et al.Automated grading of breastcancer histopathology using cascaded ensemble with combination of multi-level image features[J].Neurocomputing,2017,229:34-44. [42]YAN R,LI J,RAO X,et al.Nanet:Nuclei-aware network for grading of breast cancer in he stained pathological images[C]//2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).IEEE,2020:865-870. [43]AWAN R,SIRINUKUNWATTANA K,EPSTEIN D,et al.Glandular morphometrics for objective grading of colorectal adenocarcinoma histology images[J].Scientific Reports,2017,7:1-12. [44]ZHOU Y,GRAHAM S,KOOHBANANI N A,et al.Cgcnet:Cell graph convolutional network for grading of colorectal cancerhistology images[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops.2019. [45]STRÖM P,KARTASALO K,OLSSON H,et al.Artificial intelligence for diagnosis and grading of prostate cancer in biopsies:A population-based,diagnostic study[J].The Lancet Oncology,2020,21:222-232. [46]BULTEN W,PINCKAERS H,BOVEN H V,et al.Automated deep-learning system for gleason grading of prostate cancer using biopsies:A diagnostic study[J].The Lancet Oncology,2020,21:233-241. [47]BULTEN W,BALKENHOL M,BELINGA J J A,et al.Artificial intelligence assistance significantly improves gleason grading of prostate biopsies by pathologists[J].Modern Pathology,2021,34:660-671. [48]NAGPAL K,FOOTE D,TAN F,et al.Development and validation of a deep learning algorithm for gleason grading of prostate cancer from biopsy specimens[J].JAMA Oncology,2020,6:1372-1380. [49]RAJU A,YAO J,HAQ M M,et al.Graph attention multi-in-stance learning for accurate colorectal cancer staging[C]//International Conference on Medical Image Compu-ting and Compu-ter-Assisted Intervention.Springer,2020:529-539. [50]WANG D,KHOSLA A,GARGEYA R,et al.Deep learning for identifying metastatic breast cancer[J].arXiv:1606.05718,2016. [51]LIU Y,GADEPALLI K,NOROUZI M,et al.Detecting cancer metastases on gigapixel pathology images[J].arXiv:1703.02442,2017. [52]LI Y,PING W.Cancer metastasis detection with neural condi-tional random field[J].arXiv:1806.07064,2018. [53]GUO Z,LIU H,NI H,et al.A fast and refined cancer regions segmentation framework in whole-slide breast pathological images[J].Scientific Reports,2019,9:1-10. [54]LE H,SAMARAS D,KURC T,et al.Pancreatic cancer detection in whole slide images using noisy label annotations[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Springer,2019:541-549. [55]TOLKACH Y,RGEN T D,TOMA M,et al.High-accuracyprostate cancer pathology using deep learning[J].Nature Machine Intelligence,2020,2:411-418. [56]SCHAUMBERG A J,RUBIN M A,FUCHS T J.H&E-stained whole slide image deep learning predicts spop mutation state in prostate cancer[J].BioRxiv,2018:064279. [57]DING K,LIU Q,LEE E,et al.Feature-enhanced graph net-works for genetic mutational prediction using histopathological images in colon cancer[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Sprin-ger,2020:294-304. [58]KATHER J N,PEARSON A T,HALAMA N,et al.Deeplearning can predict microsatellite instability directly from histology in gastrointestinal cancer[J].Nature Medicine,2019,25:1054-1056. [59]YAMASHITA R,LONG J,LONGACRE T,et al.Deep learning model for the prediction of microsatellite instability in colorectal cancer:A diagnostic study[J].The Lancet Oncology,2021,22:132-141. [60]XU H,PARK S,LEE S H,et al.Using transfer learning on whole slide images to predict tumor mutational burden in bladder cancer patients[J].BioRxiv,2019:554527. [61]WOERL A C,ECKSTEIN M,GEIGER J,et al.Deep learning predicts molecular subtype of muscle-invasive bladder cancer from conventional histopathological slides[J].European Urology,2020,78:256-264. [62]RAWAT R R,ORTEGA I,ROY P,et al.Deep learned tissue“fingerprints” classify breast cancers by er/pr/her2 status from H&E images[J].Scientific Reports,2020,10:1-13. [63]HU J,CUI C,YANG W,et al.Using deep learning to predictanti-pd-1 response in melanoma and lung cancer patients from histopathology images[J].Translational Oncology,2021,14:100921. [64]JOHANNET P,COUDRAY N,DONNELLY D M,et al.Usingmachine learning algorithms to predict immunotherapy response in patients with advanced melanoma[J].Clinical Cancer Research,2021,27:131-140. [65]BARRERA C,VELU P,BERA K,et al.Computer-extractedfeatures relating to spatial arrangement of tumor infiltrating lymphocytes to predict response to nivolumab in non-small cell lung cancer (nsclc)[C]//American Society of Clinical Oncology.2018:12115. [66]JOHANNET P,COUDRAY N,JOUR G,et al.Using machine learning algorithms to predict response and toxicity to immune checkpoint inhibitors (icis) in melanoma patients[C]//American Society of Clinical Oncology.2019:2581. [67]LIU K,XIA W,QIANG M,et al.Deep learning pathological microscopic features in endemic nasopharyngeal cancer:Prognostic value and protentional role for individual induction chemotherapy[J].Cancer Medicine,2020,9:1298-1306. [68]LI X,TANG Q,YU J,et al.Microvascularity detection andquantification in glioma:A novel deep-learning-based framework[J].Laboratory Investigation,2019,99:1515-1526. [69]ZHU X,YAO J,HUANG J.Deep convolutional neural network for survival analysis with pathological images[C]//2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).IEEE,2016:544-547. [70]ZHU X,YAO J,ZHU F,et al.Wsisa:Making survival prediction from whole slide histopathological images[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:7234-7242. [71]LI R,YAO J,ZHU X,et al.Graph cnn for survival analysis on whole slide pathological images[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Springer,2018:174-182. [72]YAO J,ZHU X,JONNAGADDALA J,et al.Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks[J].Medical Image Analysis,2020,65:101789. [73]KOOHBANANI N A,UNNIKRISHNAN B,KHURRAM S A,et al.Self-path:Self-supervision for classification of pathology images with limited annotations[J].IEEE Transactions on Me-dical Imaging,2021,40:2845-2856. [74]SELVARAJU R R,COGSWELL M,DAS A,et al.Grad-cam:Visual explanations from deep networks via gradient-based localization[C]//Proceedings of the IEEE International Confe-rence on Computer Vision.2017:618-626. |
[1] | 徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺. 时序知识图谱表示学习 Temporal Knowledge Graph Representation Learning 计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204 |
[2] | 饶志双, 贾真, 张凡, 李天瑞. 基于Key-Value关联记忆网络的知识图谱问答方法 Key-Value Relational Memory Networks for Question Answering over Knowledge Graph 计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277 |
[3] | 汤凌韬, 王迪, 张鲁飞, 刘盛云. 基于安全多方计算和差分隐私的联邦学习方案 Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy 计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108 |
[4] | 王剑, 彭雨琦, 赵宇斐, 杨健. 基于深度学习的社交网络舆情信息抽取方法综述 Survey of Social Network Public Opinion Information Extraction Based on Deep Learning 计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099 |
[5] | 郝志荣, 陈龙, 黄嘉成. 面向文本分类的类别区分式通用对抗攻击方法 Class Discriminative Universal Adversarial Attack for Text Classification 计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077 |
[6] | 姜梦函, 李邵梅, 郑洪浩, 张建朋. 基于改进位置编码的谣言检测模型 Rumor Detection Model Based on Improved Position Embedding 计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046 |
[7] | 孙奇, 吉根林, 张杰. 基于非局部注意力生成对抗网络的视频异常事件检测方法 Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection 计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061 |
[8] | 胡艳羽, 赵龙, 董祥军. 一种用于癌症分类的两阶段深度特征选择提取算法 Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification 计算机科学, 2022, 49(7): 73-78. https://doi.org/10.11896/jsjkx.210500092 |
[9] | 程成, 降爱莲. 基于多路径特征提取的实时语义分割方法 Real-time Semantic Segmentation Method Based on Multi-path Feature Extraction 计算机科学, 2022, 49(7): 120-126. https://doi.org/10.11896/jsjkx.210500157 |
[10] | 侯钰涛, 阿布都克力木·阿布力孜, 哈里旦木·阿布都克里木. 中文预训练模型研究进展 Advances in Chinese Pre-training Models 计算机科学, 2022, 49(7): 148-163. https://doi.org/10.11896/jsjkx.211200018 |
[11] | 周慧, 施皓晨, 屠要峰, 黄圣君. 基于主动采样的深度鲁棒神经网络学习 Robust Deep Neural Network Learning Based on Active Sampling 计算机科学, 2022, 49(7): 164-169. https://doi.org/10.11896/jsjkx.210600044 |
[12] | 苏丹宁, 曹桂涛, 王燕楠, 王宏, 任赫. 小样本雷达辐射源识别的深度学习方法综述 Survey of Deep Learning for Radar Emitter Identification Based on Small Sample 计算机科学, 2022, 49(7): 226-235. https://doi.org/10.11896/jsjkx.210600138 |
[13] | 祝文韬, 兰先超, 罗唤霖, 岳彬, 汪洋. 改进Faster R-CNN的光学遥感飞机目标检测 Remote Sensing Aircraft Target Detection Based on Improved Faster R-CNN 计算机科学, 2022, 49(6A): 378-383. https://doi.org/10.11896/jsjkx.210300121 |
[14] | 王建明, 陈响育, 杨自忠, 史晨阳, 张宇航, 钱正坤. 不同数据增强方法对模型识别精度的影响 Influence of Different Data Augmentation Methods on Model Recognition Accuracy 计算机科学, 2022, 49(6A): 418-423. https://doi.org/10.11896/jsjkx.210700210 |
[15] | 毛典辉, 黄晖煜, 赵爽. 符合监管合规性的自动合成新闻检测方法研究 Study on Automatic Synthetic News Detection Method Complying with Regulatory Compliance 计算机科学, 2022, 49(6A): 523-530. https://doi.org/10.11896/jsjkx.210300083 |
|