计算机科学 ›› 2022, Vol. 49 ›› Issue (3): 86-91.doi: 10.11896/jsjkx.210700199

所属专题: 大数据&数据科学 虚拟专题

• 数据库&大数据&数据科学 • 上一篇    下一篇

GSO:基于图神经网络的深度学习计算图子图替换优化框架

苗旭鹏1, 周跃1, 邵蓥侠2, 崔斌1   

  1. 1 北京大学信息科学技术学院 北京100871
    2 北京邮电大学计算机学院 北京100871
  • 收稿日期:2021-07-19 修回日期:2021-08-16 出版日期:2022-03-15 发布日期:2022-03-15
  • 通讯作者: 崔斌(bin.cui@pku.edu.cn)
  • 作者简介:(xupeng.miao@pku.edu.cn)
  • 基金资助:
    国家重点研发计划(2018YFB1004403);国家自然科学基金(61832001);北京大学-腾讯协同创新实验室项目

GSO:A GNN-based Deep Learning Computation Graph Substitutions Optimization Framework

MIAO Xu-peng1, ZHOU Yue1, SHAO Ying-xia2, CUI Bin1   

  1. 1 School of Electronics Engineering and Computer Science,Peking University,Beijing 100871,China
    2 School of Computer Science,Beijing University of Posts and Telecommunications,Beijing 100871,China
  • Received:2021-07-19 Revised:2021-08-16 Online:2022-03-15 Published:2022-03-15
  • About author:MIAO Xu-peng,born in 1995,Ph.D candidate,is a student member of China Computer Federation.His main research interests include deep learning system and distributed optimization.
    CUI Bin,born in 1975,Ph.D,professor,Ph.D supervisor,is a distinguished member of China Computer Federation.His main research interests include database system architectures,query and index techniques,big data management and mining.
  • Supported by:
    National Key Research and Development Program of China(2018YFB1004403),National Natural Science Foundation of China(61832001) and PKU-Tencent Joint Research Lab.

摘要: 深度学习在各种实际应用中取得了巨大成功,如何有效提高各种复杂的深度学习模型在硬件设备上的执行效率是该领域重要的研究内容之一。深度学习框架通常将深度学习模型表达为由基础算子构成的计算图,为了提高计算图的执行效率,传统的深度学习系统通常基于一些专家设计的子图替换规则,采用启发式搜索算法来优化计算图。它们的不足主要有:1)搜索空间大,效率低下;2)缺乏可拓展性;3)难以利用历史优化结果。为了解决上述问题,文中提出了GSO,即一个基于图神经网络的深度学习计算图子图替换优化框架。该框架将计算图的子图优化建模成经典的子图匹配问题,基于计算图中算子的特征信息和计算图的拓扑结构信息,通过图神经网络模型来估计每种子图替换规则的匹配可行性和位置。基于与主流深度学习系统兼容的Python接口实现了GSO,实验结果表明:1)相比全量的子图替换规则,基于图神经网络的子图匹配预测可以最多减少92%的搜索空间;2)相比现有的启发式搜索算法,GSO可以更快地完成计算图子图替换优化(2倍以上),并使优化后的子图最多得到34%的加速。

关键词: 计算图优化, 深度学习, 图神经网络, 子图替换

Abstract: Deep learning has achieved great success in various practical applications.How to effectively improve the model execution efficiency is one of the important research issues in this field.The existing deep learning frameworks usually model deep learning in the form of computational graphs,try to optimize computational graphs through subgraph substitution rules designed by experts and mainly use heuristic algorithms to search substitution sequences.Their shortcomings mainly include:1)the exis-ting subgraph substitution rules result in a large search space and the heuristic algorithms are not efficient;2)these algorithms are not scalable for large computation graphs;3)cannot utilize the history optimization results.In order to solve the above problem,we propose GSO,a graph neural network-based deep learning computation graph optimization framework.We transfer the graph substitution optimization problem as the subgraph matching problem.Based on the feature information from the operators and the computation graph topology,we utilize the graph neural network to predict the subgraph matching feasibility and positions.We implement the framework using Python,which is compatible with the mainstream deep learning systems.The experimental results show that:1)compared to the total graph substitution rules,the proposed rule can reduce the search space by up to 92%;2)compared to the existing heuristic algorithms,GSO can complete the subgraph replacement process of the computational graph 2 times faster.The optimized computation graph is up to 34% faster the original graph.

Key words: Deep learning, Graph neural network, Graph substitutions, Optimizing DNN computation

中图分类号: 

  • TP311
[1]CANZIANI A,PASZKE A,CULURCIELLO E.An analysis ofdeep neural network models for practical applications[J].arXiv:1605.07678v4,2016.
[2]MARTÍN A,PAUL B,JIANMIN C,et al.TensorFlow:a sys-tem for large-scale machine learning[C]//12th USENIX Symposium on Operating Systems Design and Implementation.2016:265-283.
[3]JIA Z H,JAMES T,TODD W,et al.Optimizing DNN Computation with Relaxed Graph Substitutions[C]//Proceedings of the 2nd SysML Conference.2019.
[4]JIA Z H,ODED P,JAMES T,et al.TASO:optimizing deeplearning computation with automatic generation of graph substitutions[C]//Proceedings of the 27th ACM Symposium on Operating Systems Principles.2019:47-62.
[5]FANG J Z,SHEN Y Y,WANG Y,et al.Optimizing DNN computation graph using graph substitutions[C]//Proceedings of the VLDB Endowment.2020:2734-2746.
[6]CHEN T Q,THIERRY M,JIANG Z H,et al.TVM:an automated end-to-end optimizing compiler for deep learning[C]//13th USENIX Symposium on Operating Systems Design and Implementation.2018:578-594.
[7]PASZKE A,GROSS S,MASSA F,et al.Pytorch:An imperative style,high-performance deep learning library[C]//Advances in Neural Information Processing Systems.2019,32:8026-8037.
[8]Nvidia tensorrt:Programmable inference accelerator[Z/OL].https://developer.nvidia.com/tensorrt.
[9]THOMAS N,MAX W.Semi-supervised classification withgraph convolutional networks[J].arXiv:1609.02907,2016.
[10]MIAO X P,ZHANG W T,SHAO Y X,et al.Lasagne:A multi-layer graph convolutional network framework via node-aware deep architecture[J/OL].IEEE Transactions on Knowledge and Data Engineering,2021.https://ieeexplore.ieee.org/document/9513581.
[11]MIAO X P,GÜREL N M,ZHANG W T,et al.Degnn:Improving graph neural networks with graph decomposition[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining.2021:1223-1233.
[12]ZHANG W T,MIAO X P,SHAO Y X,et al.Reliable Data Distillation on Graph Convolutional Network [C]//Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data.2020:1399-1414.
[13]MINAR M R,NAHER J.Recent advances in deep learning:An overview[J].arXiv:1807.08169,2018.
[14]MIAO X P,MA L X,YANG Z,et al.Cuwide:towards efficient flow-based training for sparse wide models on gpus[J/OL].IEEE Transactions on Knowledge and Data Engineering,2020.https://ieeexplore.ieee.org/document/9261124.
[15]HE K,ZHANG X Y,REN S Q,et al.Deep Residual Learning for Image Recognition [C]//Proceedings of the IEEE Confe-rence on Computer Vision and Pattern Recognition.2016:770-778.
[16]ALEX K,ILYA S,GEOFFREY E.ImageNet classification with deep convolutional neural networks [C]//Advances in Neural Information Processing Systems.2012,25:1097-1105.
[17]KAREN S,ANDREW Z.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556v4,2014.
[18]ZAREMBA W,ILYA S,ORIOL V.Recurrent neural network regularization[J].arXiv:1409.2329,2014.
[1] 饶志双, 贾真, 张凡, 李天瑞.
基于Key-Value关联记忆网络的知识图谱问答方法
Key-Value Relational Memory Networks for Question Answering over Knowledge Graph
计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277
[2] 汤凌韬, 王迪, 张鲁飞, 刘盛云.
基于安全多方计算和差分隐私的联邦学习方案
Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy
计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108
[3] 周芳泉, 成卫青.
基于全局增强图神经网络的序列推荐
Sequence Recommendation Based on Global Enhanced Graph Neural Network
计算机科学, 2022, 49(9): 55-63. https://doi.org/10.11896/jsjkx.210700085
[4] 徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺.
时序知识图谱表示学习
Temporal Knowledge Graph Representation Learning
计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204
[5] 王剑, 彭雨琦, 赵宇斐, 杨健.
基于深度学习的社交网络舆情信息抽取方法综述
Survey of Social Network Public Opinion Information Extraction Based on Deep Learning
计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099
[6] 郝志荣, 陈龙, 黄嘉成.
面向文本分类的类别区分式通用对抗攻击方法
Class Discriminative Universal Adversarial Attack for Text Classification
计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077
[7] 姜梦函, 李邵梅, 郑洪浩, 张建朋.
基于改进位置编码的谣言检测模型
Rumor Detection Model Based on Improved Position Embedding
计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046
[8] 孙奇, 吉根林, 张杰.
基于非局部注意力生成对抗网络的视频异常事件检测方法
Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection
计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[9] 闫佳丹, 贾彩燕.
基于双图神经网络信息融合的文本分类方法
Text Classification Method Based on Information Fusion of Dual-graph Neural Network
计算机科学, 2022, 49(8): 230-236. https://doi.org/10.11896/jsjkx.210600042
[10] 侯钰涛, 阿布都克力木·阿布力孜, 哈里旦木·阿布都克里木.
中文预训练模型研究进展
Advances in Chinese Pre-training Models
计算机科学, 2022, 49(7): 148-163. https://doi.org/10.11896/jsjkx.211200018
[11] 周慧, 施皓晨, 屠要峰, 黄圣君.
基于主动采样的深度鲁棒神经网络学习
Robust Deep Neural Network Learning Based on Active Sampling
计算机科学, 2022, 49(7): 164-169. https://doi.org/10.11896/jsjkx.210600044
[12] 苏丹宁, 曹桂涛, 王燕楠, 王宏, 任赫.
小样本雷达辐射源识别的深度学习方法综述
Survey of Deep Learning for Radar Emitter Identification Based on Small Sample
计算机科学, 2022, 49(7): 226-235. https://doi.org/10.11896/jsjkx.210600138
[13] 齐秀秀, 王佳昊, 李文雄, 周帆.
基于概率元学习的矩阵补全预测融合算法
Fusion Algorithm for Matrix Completion Prediction Based on Probabilistic Meta-learning
计算机科学, 2022, 49(7): 18-24. https://doi.org/10.11896/jsjkx.210600126
[14] 杨炳新, 郭艳蓉, 郝世杰, 洪日昌.
基于数据增广和模型集成策略的图神经网络在抑郁症识别上的应用
Application of Graph Neural Network Based on Data Augmentation and Model Ensemble in Depression Recognition
计算机科学, 2022, 49(7): 57-63. https://doi.org/10.11896/jsjkx.210800070
[15] 胡艳羽, 赵龙, 董祥军.
一种用于癌症分类的两阶段深度特征选择提取算法
Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification
计算机科学, 2022, 49(7): 73-78. https://doi.org/10.11896/jsjkx.210500092
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!