计算机科学 ›› 2019, Vol. 46 ›› Issue (9): 1-14.doi: 10.11896/j.issn.1002-137X.2019.09.001

• 综述 •    下一篇

深度神经网络压缩综述

李青华1,2, 李翠平1,2, 张静1,2, 陈红1,2, 王绍卿1,2,3   

  1. (中国人民大学数据工程与知识工程教育部重点实验室 北京100872)1;
    (中国人民大学信息学院 北京100872)2;
    (山东理工大学计算机科学与技术学院 山东 淄博255091)3
  • 收稿日期:2018-12-11 出版日期:2019-09-15 发布日期:2019-09-02
  • 通讯作者: 李翠平(1971-),女,博士,教授,主要研究方向为推荐系统,E-mail:licuiping@ruc.edu.cn
  • 作者简介:李青华(1991-),男,博士生,主要研究方向为深度学习、模型优化,E-mail:qinghuali@ruc.edu.cn;张 静(1984-),女,博士,讲师,主要研究方向为社交网络分析;陈 红(1965-),女,博士,教授,主要研究方向为高性能数据库;王绍卿(1981-),男,博士,主要研究方向为推荐系统。

Survey of Compressed Deep Neural Network

LI Qing-hua1,2, LI Cui-ping1,2, ZHANG Jing1,2, CHEN Hong1,2, WANG Shao-qing1,2,3   

  1. (Key Laboratory of Data Engineering and Knowledge Engineering (Renmin University of China),Ministry of Education,Beijing 100872,China)1;
    (School of Information,Renmin University of China,Beijing 100872,China)2;
    (School of Computer Science and Technology,Shandong University of Technology,Zibo,Shandong 255091,China)3
  • Received:2018-12-11 Online:2019-09-15 Published:2019-09-02

摘要: 近年来深度神经网络在目标识别、图像分类等领域取得了重大突破,然而训练和测试这些大型深度神经网络存在几点限制:1)训练和测试这些深度神经网络需要进行大量的计算(训练和测试将消耗大量的时间),需要高性能的计算设备(例如GPU)来加快训练和测试速度;2)深度神经网络模型通常包含大量的参数,需要大容量的高速内存来存储模型。上述限制阻碍了神经网络等技术的广泛应用(现阶段神经网络的训练和测试通常是在高性能服务器或者集群下面运行,在一些对实时性要求较高的移动设备(如手机)上的应用受到限制)。文中对近年来的压缩神经网络算法进行了综述,系统地介绍了深度神经网络压缩的主要方法,如裁剪方法、稀疏正则化方法、分解方法、共享参数方法、掩码加速方法、离散余弦变换方法,最后对未来深度神经网络压缩的研究方向进行了展望。

关键词: 深度学习, 神经网络, 模型压缩

Abstract: In recent years,deep neural networks have achieved significant breakthroughs in target recognition,image classification,etc.However,training and testing for these deep neural network have several limitations.Firstly,training and testing for these deep neural networks require a lot of computation (training and testing consume a lot of time),which requires high-performance computing devices (such as GPUs) to improve the training and testing speed,and shorten training and testing time.Secondly,the deep neural network model usually contains a large number of parameters that require high-capacity,high-speed memory to store.These limitations hinder the widespread use of deep neural networks.At present,training and testing of deep neural networks usually run under high-performance servers or clusters.In some mobile devices with high real-time requirements,such as mobile phones,applications are limited.This paper reviewed the progress of compression deep neural network algorithm in recent years,and introduced the main me-thods of compressing neural network,such as cropping method,sparse regularization method,decomposition method,shared parameter method,mask acceleration method and discrete cosine transform method.Finally,the future research direction of compressed deep neural network was prospected.

Key words: Deep learning, Neural network, Model compression

中图分类号: 

  • TP39
[1]YANIV T M,YANG M,RANZATO M A,et al.Deepface:Clo-sing the gap to human-level performance in face verification[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2014.
[2]SUN Y,WANG X G,TANG X O,et al.Deep learning face representation from predicting10000 classes[C]//IEEE Confe-rence on Computer Vision and Pattern Recognition (CVPR).2014.
[3]GIRSHICK R B.Fast R-CNN[C]//International Conference on Computer Vision(ICCV).2015:1440-1448.
[4]GIRSHICK R B,DONAHUE J,DARRELL T,et al.Rich feature hierarchies for accurate object detection and semantic segmentation[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2014:580-587.
[5]REN S,HE K,GIRSHICK R B,et al.Faster R-CNN:towards real-time object detection with region proposal networks[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2015,39(1):1137-1149.
[6]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenet classification with deep convolutional neural networks[J].Neural Information Processing Systems (NIPS),2012,25(2):1106-1114.
[7]ZEILER M D,FERGUSR.Visualizing and understanding convo-lutional networks[C]//European Conference on Computer Vision (ECCV).2014:818-833.
[8]SZEGEDY C,LIU W,JIA Y,et al.Going deeper with convolutions[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2015:1-9.
[9]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[C]//International Conference on Learning Representations (ICLR).2015.
[10]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[C]//International Conference on Learning Representations (ICLR).2015.
[11]CHEN L C,PAPANDREOU G,KOKKINOS I,et al.Semantic image segmentation with deep convolutional nets and fully connected crfs[C]//International Conference on Learning Representations (ICLR).2015.
[12]GONG Y,WANG L,GUO R,et al.Multiscale orderless pooling of deep convolutional activation features[C]//European Conference on Computer Vision (ECCV).2014.
[13]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556.
[14]DENIL M,SHAKIBI B,DINH L,et al.Predicting parameters in deep learning[C]//Neural Information Processing Systems(NIPS).2013.
[15]HORNIK K,STINCHCOMBE M,WHITE H.Multilayer feedforward networks are universal approximators [J].Neural Networks,1989,2(5):359-366.
[16]GARDNER M W,DORLING S R.Artificial neural networks (the multilayer perceptron )-a review of applications in the atmospheric sciences[J].Atmospheric Environment,1998,32(14/15):2627-2636.
[17]ZHOU F Y,JIN L P,DONG J.Review of Convolutional Neural Network[J].Chinese Journal of Computers,2017,40(7):1229-1251.(in Chinese)周飞燕,金林鹏,董军.卷积神经网络研究综述[J].计算机学报,2017,40(7):1229-1251.
[18]RUMELHART D E,HINTON G,WILLIAMS R J.Learning representations by back-propagating errors[J].Nature,1986,323(6088):533-536.
[19]LECUN Y,BOTTOU L,BENGIO Y S,et al.Gradient-Based Learning Applied to Document Recognition[J].Proceedings of the IEEE,1998,86(11):2278-2324.
[20]ZHANG Q H,WAN C X.Review of Convolutional Neural Network[J].Journal of Zhongyuan University of Technology,2017,28(3):1671-6906.(in Chinese)张庆辉,万晨霞.卷积神经网络综述[J].中原工学院学报,2017,28(3):1671-6906.
[21]SONG H,JEFF P,JOHN T,et al.Learning both weights and connections for efficient neural network[C]//Neural Information Processing Systems(NIPS).2015:1135-1143.
[22]GUO Y W,YAO A B,CHEN Y R.Dynamic Network Surgery for Efficient DNNs[C]//Neural Information Processing Systems(NIPS).2016.
[23]LI H,KADAV A,DURDANOVIC I,et al.Pruning Filters for Efficient Convents[C]//International Conference on Learning Representations (ICLR).2017.
[24]PAVIO M,STEPHEN T,TERO K,et al.Pruning Convolutional Neural Networks for Resource Efficient Inference[C]//International Conference on Learning Representations (ICLR).2017.
[25]HARVEY L,ARNOLD B,ZIPURSKY L S,et al.Molecular Cell Biology:Neurotransmitters,Synapses,and Impulse Transmission[M].New York:W.H.Freeman,2000.
[26]JOSE M A,MATHIEU S.Learning the Number of Neurons in Deep Networks[C]//Neural Information Processing Systems(NIPS).2016.
[27]WEI W,WU C P,WANG Y D,et al.Learning Structured Sparsity in Deep Neural Networks[C]//Neural Information Proces-sing Systems(NIPS).2016.
[28]WANG S J,CAI H R,JEFF B,et al.Training Compressed Fully-Connected Networks with a Density-diversity Penalty[C]//International Conference on Learning Representations (ICLR).2017.
[29]YUAN M,LIN Y.Model selection and estimation in regression with grouped variables[J].Journal of The Royal Statistical So-ciety Series B-statistical Methodology,2006,68(1):49-67.
[30]KIM S,XING E P.Tree-guided group lasso for multi-task regression with structured sparsity[C]//Proceedings of the 27th International Conference on Machine Learning.2010.
[31]YUAN M,LIN Y.Model selection and estimation in regression with grouped variables[J].Journal of the Royal Statistical Society,Series B,2006,68(1):49-67.
[32]PETER L B.For valid generalization the size of the weights is more important than the size of the network[C]//Neural Information Processing Systems(NIPS).1996.
[33]KROGH A,JOHN A H.A simple weight decay can improve generalization[C]//Neural Information Processing Systems(NIPS).1992.
[34]THEODORIDIS S.Machine Learning A Bayesian and Optimization Perspective[M].Academic Press,2015.
[35]COLLINS M D,KOHLI P.Memory Bounded Deep Convolutional Networks[J].arXiv.1412.1442v1.
[36]SIMON N,FRIEDMAN J,HASTIE T,et al.A sparse-group lasso[J].Journal of Computational and Graphical Statistics,2013,22(2):231-245.
[37]PARIKH N,BOYD S.Proximal algorithms[J].Foundations and Trends in Optimization,2014,1(3):123-231.
[38]MAX J,ANDREA V,ANDREW Z.Speeding up Convolutional Neural Networks with Low Rank Expansions[C]//British Machine Vision Conference.2014.
[39]VADIM L,YAROSLAV G,MAKSIM R,et al.Speeding-up Convolutional Neural Networks Using Fine-tuned Cp-decomposition[C]//International Conference on Learning Representations (ICLR).2015.
[40]JONGHOON J,AYSEGUL D,EUGENIO C.Flattened Convolutional Neural Networks for Feedforward Acceleration[C]//International Conference on Learning Representations (ICLR).2015.
[41]KIM Y D,EUNHYEOK P,YOO S J,et al.Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications[C]//International Conference on Learning Representations (ICLR).2016.
[42]RIGAMONTI R,SIRONI A,LEPETIT V,et al.Learning separable filters[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2013:2754-2761.
[43]KOLDA T G,BADER B W.Tensor decompositions and applications[J].Siam Review,2009,51(3):455-500.
[44]EMILY D,WOJCIECH Z,JOAN B,et al.Exploiting linear structure within convolutional networks for efficient evaluation[J].arXiv:1404.0736.
[45]SORBER L,VAN B M,DELATHAUWER L.Tensorlab v2.0[EB/OL].http://tensorlab.net.
[46]TOMASI G,BRO R.A comparison of algorithms for fitting the parafac model[J].Computational Statistics & Data Analysis,2006,50(7):1700-1734.
[47]TUCKER L R.Some mathematical notes on three-mode factor analysis[J].Psychometrika,1966,31(3):279-311.
[48]GONG Y C,LIU L,YANG M,et al.Compressing Deep Convolutional Networks Using Vector Quantization[C]//InternationalConference on Learning Representations (ICLR).2015.
[49]CHEN W L,JAMES T W,STEPHEN T,et al.Compressing Neural Networks with the Hashing Trick[C]//International Conference on Machine Learning.2015:2285-2294.
[50]SONG H,MAO H Z,WILLIAM J.Deep Compression:Com-pressing Deep Neural Networks with Pruning,Trained Quantization and Huffman Coding[C]//International Conference on Learning Representations (ICLR).2016.
[51]CHOI Y J,MOSTAFA E K,JUNWON L.Towards the Limit of Network Quantization[C]//International Conference onLear-ning Representations (ICLR).2017.
[52]ZHOU A J,YAO A B,GUO Y W,et al.Incremental Network Quantization: Towards Lossless Cnns with Low-precision Weights[C]//International Conference on Learning Representations (ICLR).2017.
[53]JAN V L.On the construction of huffman trees[C]//International colloquium on automata,languages and programming(ICALP).1976:382-410.
[54]MICHAEL F,AIJAN L,DMITRY V,et al.PerforatedCNNs:Acceleration through Elimination of Redundant Convolutions[C]//Neural Information Processing Systems(NIPS).2016.
[55]LIN S H,JI R R,CHEN C,et al.ESPACE:Accelerating Convolutional Neural Networks via Eliminating Spatial & Channel Redundancy[C]//AAAI Conference on Artificial Intelligence(AAAI).2017.
[56]STELIOS S D,SASA M,HENRY C H,et al.Managing performance vs.accuracy trade-offs with loop perforation[C]//ESEC.2011:124-134.
[57]MISAILOVIC S,SIDIROGLOU S,HOFFMANN H,et al.Quality of service profiling[C]//International conference on software engineering(ICSE).2010:25-34.
[58]MISAILOVIC S,ROY D M,RINARD M C.Probabilistically ac-curate program transformations[C]//Static Analysis Sympo-sium.2011:316-333.
[59]CHEN W L,JAMES T,WILSON S T.Compressing Convolu-tional Neural Networks[J].arXiv:1506.04449v1.
[60]WANG Y H,XU C,YOU S,et al.CNNpack:Packing Convolutional Neural Networks in the Frequency Domain[C]//Neural Information Processing Systems(NIPS).2016.
[61]RAO K R,YIP P.Discrete cosine transform:algorithms,advantages,applications[M].Academic Press Professional Inc,2014.
[1] 余雪勇, 陈涛. 边缘计算场景中基于虚拟映射的隐私保护卸载算法[J]. 计算机科学, 2021, 48(1): 65-71.
[2] 单美静, 秦龙飞, 张会兵. L-YOLO:适用于车载边缘计算的实时交通标识检测模型[J]. 计算机科学, 2021, 48(1): 89-95.
[3] 何彦辉, 吴桂兴, 吴志强. 基于域适应的X光图像的目标检测[J]. 计算机科学, 2021, 48(1): 175-181.
[4] 李亚男, 胡宇佳, 甘伟, 朱敏. 基于深度学习的miRNA靶位点预测研究综述[J]. 计算机科学, 2021, 48(1): 209-216.
[5] 王瑞平, 贾真, 刘畅, 陈泽威, 李天瑞. 基于DeepFM的深度兴趣因子分解机网络[J]. 计算机科学, 2021, 48(1): 226-232.
[6] 于文家, 丁世飞. 基于自注意力机制的条件生成对抗网络[J]. 计算机科学, 2021, 48(1): 241-246.
[7] 仝鑫, 王斌君, 王润正, 潘孝勤. 面向自然语言处理的深度学习对抗样本综述[J]. 计算机科学, 2021, 48(1): 258-267.
[8] 张艳梅, 楼胤成. 基于深度神经网络的庞氏骗局合约检测方法[J]. 计算机科学, 2021, 48(1): 273-279.
[9] 丁钰, 魏浩, 潘志松, 刘鑫. 网络表示学习算法综述[J]. 计算机科学, 2020, 47(9): 52-59.
[10] 庄世杰, 於志勇, 郭文忠, 黄昉菀. 基于Zoneout的跨尺度循环神经网络及其在短期电力负荷预测中的应用[J]. 计算机科学, 2020, 47(9): 105-109.
[11] 何鑫, 许娟, 金莹莹. 行为关联网络:完整的变化行为建模[J]. 计算机科学, 2020, 47(9): 123-128.
[12] 张佳嘉, 张小洪. 多分支卷积神经网络肺结节分类方法及其可解释性[J]. 计算机科学, 2020, 47(9): 129-134.
[13] 叶亚男, 迟静, 于志平, 战玉丽, 张彩明. 基于改进CycleGan模型和区域分割的表情动画合成[J]. 计算机科学, 2020, 47(9): 142-149.
[14] 朱玲莹, 桑庆兵, 顾婷婷. 基于视差信息的无参考立体图像质量评价[J]. 计算机科学, 2020, 47(9): 150-156.
[15] 赵钦炎, 李宗民, 刘玉杰, 李华. 基于信息熵的级联Siamese网络目标跟踪[J]. 计算机科学, 2020, 47(9): 157-162.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] 雷丽晖,王静. 可能性测度下的LTL模型检测并行化研究[J]. 计算机科学, 2018, 45(4): 71 -75 .
[2] 孙启,金燕,何琨,徐凌轩. 用于求解混合车辆路径问题的混合进化算法[J]. 计算机科学, 2018, 45(4): 76 -82 .
[3] 张佳男,肖鸣宇. 带权混合支配问题的近似算法研究[J]. 计算机科学, 2018, 45(4): 83 -88 .
[4] 伍建辉,黄中祥,李武,吴健辉,彭鑫,张生. 城市道路建设时序决策的鲁棒优化[J]. 计算机科学, 2018, 45(4): 89 -93 .
[5] 史雯隽,武继刚,罗裕春. 针对移动云计算任务迁移的快速高效调度算法[J]. 计算机科学, 2018, 45(4): 94 -99 .
[6] 周燕萍,业巧林. 基于L1-范数距离的最小二乘对支持向量机[J]. 计算机科学, 2018, 45(4): 100 -105 .
[7] 刘博艺,唐湘滟,程杰仁. 基于多生长时期模板匹配的玉米螟识别方法[J]. 计算机科学, 2018, 45(4): 106 -111 .
[8] 耿海军,施新刚,王之梁,尹霞,尹少平. 基于有向无环图的互联网域内节能路由算法[J]. 计算机科学, 2018, 45(4): 112 -116 .
[9] 崔琼,李建华,王宏,南明莉. 基于节点修复的网络化指挥信息系统弹性分析模型[J]. 计算机科学, 2018, 45(4): 117 -121 .
[10] 王振朝,侯欢欢,连蕊. 抑制CMT中乱序程度的路径优化方案[J]. 计算机科学, 2018, 45(4): 122 -125 .