计算机科学 ›› 2022, Vol. 49 ›› Issue (3): 86-91.doi: 10.11896/jsjkx.210700199
所属专题: 大数据&数据科学 虚拟专题
苗旭鹏1, 周跃1, 邵蓥侠2, 崔斌1
MIAO Xu-peng1, ZHOU Yue1, SHAO Ying-xia2, CUI Bin1
摘要: 深度学习在各种实际应用中取得了巨大成功,如何有效提高各种复杂的深度学习模型在硬件设备上的执行效率是该领域重要的研究内容之一。深度学习框架通常将深度学习模型表达为由基础算子构成的计算图,为了提高计算图的执行效率,传统的深度学习系统通常基于一些专家设计的子图替换规则,采用启发式搜索算法来优化计算图。它们的不足主要有:1)搜索空间大,效率低下;2)缺乏可拓展性;3)难以利用历史优化结果。为了解决上述问题,文中提出了GSO,即一个基于图神经网络的深度学习计算图子图替换优化框架。该框架将计算图的子图优化建模成经典的子图匹配问题,基于计算图中算子的特征信息和计算图的拓扑结构信息,通过图神经网络模型来估计每种子图替换规则的匹配可行性和位置。基于与主流深度学习系统兼容的Python接口实现了GSO,实验结果表明:1)相比全量的子图替换规则,基于图神经网络的子图匹配预测可以最多减少92%的搜索空间;2)相比现有的启发式搜索算法,GSO可以更快地完成计算图子图替换优化(2倍以上),并使优化后的子图最多得到34%的加速。
中图分类号:
[1]CANZIANI A,PASZKE A,CULURCIELLO E.An analysis ofdeep neural network models for practical applications[J].arXiv:1605.07678v4,2016. [2]MARTÍN A,PAUL B,JIANMIN C,et al.TensorFlow:a sys-tem for large-scale machine learning[C]//12th USENIX Symposium on Operating Systems Design and Implementation.2016:265-283. [3]JIA Z H,JAMES T,TODD W,et al.Optimizing DNN Computation with Relaxed Graph Substitutions[C]//Proceedings of the 2nd SysML Conference.2019. [4]JIA Z H,ODED P,JAMES T,et al.TASO:optimizing deeplearning computation with automatic generation of graph substitutions[C]//Proceedings of the 27th ACM Symposium on Operating Systems Principles.2019:47-62. [5]FANG J Z,SHEN Y Y,WANG Y,et al.Optimizing DNN computation graph using graph substitutions[C]//Proceedings of the VLDB Endowment.2020:2734-2746. [6]CHEN T Q,THIERRY M,JIANG Z H,et al.TVM:an automated end-to-end optimizing compiler for deep learning[C]//13th USENIX Symposium on Operating Systems Design and Implementation.2018:578-594. [7]PASZKE A,GROSS S,MASSA F,et al.Pytorch:An imperative style,high-performance deep learning library[C]//Advances in Neural Information Processing Systems.2019,32:8026-8037. [8]Nvidia tensorrt:Programmable inference accelerator[Z/OL].https://developer.nvidia.com/tensorrt. [9]THOMAS N,MAX W.Semi-supervised classification withgraph convolutional networks[J].arXiv:1609.02907,2016. [10]MIAO X P,ZHANG W T,SHAO Y X,et al.Lasagne:A multi-layer graph convolutional network framework via node-aware deep architecture[J/OL].IEEE Transactions on Knowledge and Data Engineering,2021.https://ieeexplore.ieee.org/document/9513581. [11]MIAO X P,GÜREL N M,ZHANG W T,et al.Degnn:Improving graph neural networks with graph decomposition[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining.2021:1223-1233. [12]ZHANG W T,MIAO X P,SHAO Y X,et al.Reliable Data Distillation on Graph Convolutional Network [C]//Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data.2020:1399-1414. [13]MINAR M R,NAHER J.Recent advances in deep learning:An overview[J].arXiv:1807.08169,2018. [14]MIAO X P,MA L X,YANG Z,et al.Cuwide:towards efficient flow-based training for sparse wide models on gpus[J/OL].IEEE Transactions on Knowledge and Data Engineering,2020.https://ieeexplore.ieee.org/document/9261124. [15]HE K,ZHANG X Y,REN S Q,et al.Deep Residual Learning for Image Recognition [C]//Proceedings of the IEEE Confe-rence on Computer Vision and Pattern Recognition.2016:770-778. [16]ALEX K,ILYA S,GEOFFREY E.ImageNet classification with deep convolutional neural networks [C]//Advances in Neural Information Processing Systems.2012,25:1097-1105. [17]KAREN S,ANDREW Z.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556v4,2014. [18]ZAREMBA W,ILYA S,ORIOL V.Recurrent neural network regularization[J].arXiv:1409.2329,2014. |
[1] | 饶志双, 贾真, 张凡, 李天瑞. 基于Key-Value关联记忆网络的知识图谱问答方法 Key-Value Relational Memory Networks for Question Answering over Knowledge Graph 计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277 |
[2] | 汤凌韬, 王迪, 张鲁飞, 刘盛云. 基于安全多方计算和差分隐私的联邦学习方案 Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy 计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108 |
[3] | 周芳泉, 成卫青. 基于全局增强图神经网络的序列推荐 Sequence Recommendation Based on Global Enhanced Graph Neural Network 计算机科学, 2022, 49(9): 55-63. https://doi.org/10.11896/jsjkx.210700085 |
[4] | 徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺. 时序知识图谱表示学习 Temporal Knowledge Graph Representation Learning 计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204 |
[5] | 王剑, 彭雨琦, 赵宇斐, 杨健. 基于深度学习的社交网络舆情信息抽取方法综述 Survey of Social Network Public Opinion Information Extraction Based on Deep Learning 计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099 |
[6] | 郝志荣, 陈龙, 黄嘉成. 面向文本分类的类别区分式通用对抗攻击方法 Class Discriminative Universal Adversarial Attack for Text Classification 计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077 |
[7] | 姜梦函, 李邵梅, 郑洪浩, 张建朋. 基于改进位置编码的谣言检测模型 Rumor Detection Model Based on Improved Position Embedding 计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046 |
[8] | 孙奇, 吉根林, 张杰. 基于非局部注意力生成对抗网络的视频异常事件检测方法 Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection 计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061 |
[9] | 闫佳丹, 贾彩燕. 基于双图神经网络信息融合的文本分类方法 Text Classification Method Based on Information Fusion of Dual-graph Neural Network 计算机科学, 2022, 49(8): 230-236. https://doi.org/10.11896/jsjkx.210600042 |
[10] | 侯钰涛, 阿布都克力木·阿布力孜, 哈里旦木·阿布都克里木. 中文预训练模型研究进展 Advances in Chinese Pre-training Models 计算机科学, 2022, 49(7): 148-163. https://doi.org/10.11896/jsjkx.211200018 |
[11] | 周慧, 施皓晨, 屠要峰, 黄圣君. 基于主动采样的深度鲁棒神经网络学习 Robust Deep Neural Network Learning Based on Active Sampling 计算机科学, 2022, 49(7): 164-169. https://doi.org/10.11896/jsjkx.210600044 |
[12] | 苏丹宁, 曹桂涛, 王燕楠, 王宏, 任赫. 小样本雷达辐射源识别的深度学习方法综述 Survey of Deep Learning for Radar Emitter Identification Based on Small Sample 计算机科学, 2022, 49(7): 226-235. https://doi.org/10.11896/jsjkx.210600138 |
[13] | 齐秀秀, 王佳昊, 李文雄, 周帆. 基于概率元学习的矩阵补全预测融合算法 Fusion Algorithm for Matrix Completion Prediction Based on Probabilistic Meta-learning 计算机科学, 2022, 49(7): 18-24. https://doi.org/10.11896/jsjkx.210600126 |
[14] | 杨炳新, 郭艳蓉, 郝世杰, 洪日昌. 基于数据增广和模型集成策略的图神经网络在抑郁症识别上的应用 Application of Graph Neural Network Based on Data Augmentation and Model Ensemble in Depression Recognition 计算机科学, 2022, 49(7): 57-63. https://doi.org/10.11896/jsjkx.210800070 |
[15] | 胡艳羽, 赵龙, 董祥军. 一种用于癌症分类的两阶段深度特征选择提取算法 Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification 计算机科学, 2022, 49(7): 73-78. https://doi.org/10.11896/jsjkx.210500092 |
|