Computer Science ›› 2020, Vol. 47 ›› Issue (6A): 172-175.doi: 10.11896/JsJkx.190500154

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Semi-supervised Surgical Video Workflow Recognition Based on Convolution Neural Network

QI Bao-lian1, 3, ZHONG Kun-hua1, 2, 3 and CHEN Yu-wen1, 2, 3   

  1. 1 Chengdu Computing Institute of the Chinese Academy of Sciences,Chengdu 610041,China
    2 Chongqing Institute of Green and Intelligent Technology,Chongqing 400714,China
    3 University of Chinese Academy of Sciences,BeiJing 100049,China
  • Published:2020-07-07
  • About author:I Bao-lian, postgraduate.Her main research interests include video analysis, surgical workflow recognition, and artificial intelligence for healthcare.
    CHEN Yu-wen, doctorial student.His main research interests include automated reasoning and programming, computer vision and artificial intelligent for healthcare.
  • Supported by:
    This work was supported by the National Key Research & Development Plan of China (2018YFC0116704) and Chongqing Technology Innovation and Application Development ProJect(cstc2019Jscx-msxmX0237).

Abstract: The real-time and robust open surgery workflow automatic detection will be the core component of the future artificial intelligent medical operation room.The key technology combined with other artificial intelligence technologies can help medical staff to automatically and intelligently complete a number of routine activities in the operation.However,the use of artificial intelligence and computer vision for surgical workflow recognition requires a large amount of data to be learned.In order to train this method,a large amount of labeled surgical video data is required.However,in the medical field,the labeling of surgical video data requires expert knowledge,and collecting enough numbers of marked surgical video data is difficult and time-consuming.Therefore,in this paper,the video data of laparoscopic cholecystectomy data is taken as the research obJect,the video spatial feature extraction is carried out by convolution self-encoder with semi-supervised learning method,and combined with a pair of video frames in the context of the same video for sequential feature extraction.The unstructured surgical video data is structured to build a bridge between the video characteristics of low-level surgery and the semantics of high-level surgical procedures,trying to realize the intelligent recognition of the surgical workflow at a low cost,and effectively determining the progress of the surgical workflow.Finally,the Jaccard coefficient of the proposed algorithm in this paper on a public dataset is 71.3% and the accuracy is 86.6%,achieving good experimental results.

Key words: CNN, Semi-supervised, Surgical workflow

CLC Number: 

  • TP181
[1] 徐大华.高科技引领微创外科发展.科技导报,2017,35(11):69-70.
[2] 曹晖.人工智能医疗给外科医生带来的挑战、机遇与思考.中国实用外科杂志,2017(12):387-388.
[3] JIN Y,DOU Q,CHEN H,et al.SV-RCNet:Workflow recognition from surgical videos using recurrent convolutional network.IEEE Trans.Med.Imaging,2018,37(5):1114-1126.
[4] TWINANDA A,YENGERA G,MUTTER D.RSDNet:Lear-ning to Predict Remaining Surgery Duration from Laparoscopic Videos Without Manual Annotations.arXiv:1802.03243v2,2018.
[5] LOUKAS C.Video content analysis of surgical procedures.Surgical Endoscopy,2018,32(2):553-568.
[6] LI X,URICCHIO T,BALLAN L,et al.Socializing the Semantic Gap:A Comparative Survey on Image Tag Assignment,Refinement,and Retrieval.Acm Computing Surveys,2016,49(1):14.
[7] KLANK U,PADOY N,FEUSSNER H.Navab N (2008) Automatic feature generation in endoscopic images.Int J Comput Assist Radiol Surg 3:331-339.
[8] BLUM T,FEUSSNER H,NAVAB N.Modeling and segmentation of surgical workflow from laparoscopic video.Lect Notes Comput Sci,2010,6363:400-407.
[9] DERGACHYOVA O,BOUGET D,HUAULM A,et al.Automatic data-driven real-time segmentation and recognition of surgical workflow.Int J Comput Assist Radiol Surg,2016,11:1081-1089.
[10] TWINANDA A P,SHEHATA S,MUTTER D,et al.EndoNet:a deep architecture for recognition tasks on laparoscopic videos.IEEE Trans Med Imaging,2017,36:86-97.
[11] LOUKAS C.Surgical Phase Recognition of Short Video Shots Based on Temporal Modeling of Deep Features.arXiv:1807.07853,2018.
[12] LECUN Y,BENGIO Y,HINTON G.Deep learning.Nature,2015,521(7553):436.
[13] GENG C,SONG J X.Human Action Recognition based on Convolutional Neural Networks with a Convolutional Auto-Encoder//International Conference on Computer Sciences and Automation Engineering.Atlantis Press,2016.
[14] RONNEBERGER O,FISCHER P,BROX T.U-Net:Convolutional Networks for Biomedical Image Segmentation//Medi-cal Image Computing and Computer-Assisted Intervention-MICCAI 2015.Springer International Publishing,2015:234-241.
[1] WU Hong-xin, HAN Meng, CHEN Zhi-qiang, ZHANG Xi-long, LI Mu-hang. Survey of Multi-label Classification Based on Supervised and Semi-supervised Learning [J]. Computer Science, 2022, 49(8): 12-25.
[2] JIN Fang-yan, WANG Xiu-li. Implicit Causality Extraction of Financial Events Integrating RACNN and BiLSTM [J]. Computer Science, 2022, 49(7): 179-186.
[3] YUE Qing, YIN Jian-yu, WANG Sheng-sheng. Automatic Detection of Pulmonary Nodules in Low-dose CT Images Based on Improved CNN [J]. Computer Science, 2022, 49(6A): 54-59.
[4] HOU Xia-ye, CHEN Hai-yan, ZHANG Bing, YUAN Li-gang, JIA Yi-zhen. Active Metric Learning Based on Support Vector Machines [J]. Computer Science, 2022, 49(6A): 113-118.
[5] YU Ben-gong, ZHANG Zi-wei, WANG Hui-ling. TS-AC-EWM Online Product Ranking Method Based on Multi-level Emotion and Topic Information [J]. Computer Science, 2022, 49(6A): 165-171.
[6] WANG Shan, XU Chu-yi, SHI Chun-xiang, ZHANG Ying. Study on Cloud Classification Method of Satellite Cloud Images Based on CNN-LSTM [J]. Computer Science, 2022, 49(6A): 675-679.
[7] ZHU Wen-tao, LAN Xian-chao, LUO Huan-lin, YUE Bing, WANG Yang. Remote Sensing Aircraft Target Detection Based on Improved Faster R-CNN [J]. Computer Science, 2022, 49(6A): 378-383.
[8] WANG Yu-fei, CHEN Wen. Tri-training Algorithm Based on DECORATE Ensemble Learning and Credibility Assessment [J]. Computer Science, 2022, 49(6): 127-133.
[9] ZHAO Zheng-peng, LI Jun-gang, PU Yuan-yuan. Low-light Image Enhancement Based on Retinex Theory by Convolutional Neural Network [J]. Computer Science, 2022, 49(6): 199-209.
[10] XU Hua-jie, CHEN Yu, YANG Yang, QIN Yuan-zhuo. Semi-supervised Learning Method Based on Automated Mixed Sample Data Augmentation Techniques [J]. Computer Science, 2022, 49(3): 288-293.
[11] HOU Hong-xu, SUN Shuo, WU Nier. Survey of Mongolian-Chinese Neural Machine Translation [J]. Computer Science, 2022, 49(1): 31-40.
[12] HUANG Xiao-sheng, XU Jing. Multi-focus Image Fusion Method Based on PCANet in NSST Domain [J]. Computer Science, 2021, 48(9): 181-186.
[13] XING Hao, LI Ming. Deepfake Video Detection Based on 3D Convolutional Neural Networks [J]. Computer Science, 2021, 48(7): 86-92.
[14] ZHAO Min, LIU Jing-lei. Semi-supervised Clustering Based on Gaussian Fields and Adaptive Graph Regularization [J]. Computer Science, 2021, 48(7): 137-144.
[15] CUI Wen-hao, JIANG Mu-rong, YANG Lei, FU Peng-ming, ZHU Ling-xiao. Combining MCycleGAN and RFCNN to Realize High Resolution Reconstruction of Solar Speckle Image [J]. Computer Science, 2021, 48(6A): 38-42.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!