计算机科学 ›› 2019, Vol. 46 ›› Issue (9): 1-14.doi: 10.11896/j.issn.1002-137X.2019.09.001
• 综述 • 下一篇
李青华1,2, 李翠平1,2, 张静1,2, 陈红1,2, 王绍卿1,2,3
LI Qing-hua1,2, LI Cui-ping1,2, ZHANG Jing1,2, CHEN Hong1,2, WANG Shao-qing1,2,3
摘要: 近年来深度神经网络在目标识别、图像分类等领域取得了重大突破,然而训练和测试这些大型深度神经网络存在几点限制:1)训练和测试这些深度神经网络需要进行大量的计算(训练和测试将消耗大量的时间),需要高性能的计算设备(例如GPU)来加快训练和测试速度;2)深度神经网络模型通常包含大量的参数,需要大容量的高速内存来存储模型。上述限制阻碍了神经网络等技术的广泛应用(现阶段神经网络的训练和测试通常是在高性能服务器或者集群下面运行,在一些对实时性要求较高的移动设备(如手机)上的应用受到限制)。文中对近年来的压缩神经网络算法进行了综述,系统地介绍了深度神经网络压缩的主要方法,如裁剪方法、稀疏正则化方法、分解方法、共享参数方法、掩码加速方法、离散余弦变换方法,最后对未来深度神经网络压缩的研究方向进行了展望。
中图分类号:
[1]YANIV T M,YANG M,RANZATO M A,et al.Deepface:Clo-sing the gap to human-level performance in face verification[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2014. [2]SUN Y,WANG X G,TANG X O,et al.Deep learning face representation from predicting10000 classes[C]//IEEE Confe-rence on Computer Vision and Pattern Recognition (CVPR).2014. [3]GIRSHICK R B.Fast R-CNN[C]//International Conference on Computer Vision(ICCV).2015:1440-1448. [4]GIRSHICK R B,DONAHUE J,DARRELL T,et al.Rich feature hierarchies for accurate object detection and semantic segmentation[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2014:580-587. [5]REN S,HE K,GIRSHICK R B,et al.Faster R-CNN:towards real-time object detection with region proposal networks[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2015,39(1):1137-1149. [6]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenet classification with deep convolutional neural networks[J].Neural Information Processing Systems (NIPS),2012,25(2):1106-1114. [7]ZEILER M D,FERGUSR.Visualizing and understanding convo-lutional networks[C]//European Conference on Computer Vision (ECCV).2014:818-833. [8]SZEGEDY C,LIU W,JIA Y,et al.Going deeper with convolutions[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2015:1-9. [9]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[C]//International Conference on Learning Representations (ICLR).2015. [10]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[C]//International Conference on Learning Representations (ICLR).2015. [11]CHEN L C,PAPANDREOU G,KOKKINOS I,et al.Semantic image segmentation with deep convolutional nets and fully connected crfs[C]//International Conference on Learning Representations (ICLR).2015. [12]GONG Y,WANG L,GUO R,et al.Multiscale orderless pooling of deep convolutional activation features[C]//European Conference on Computer Vision (ECCV).2014. [13]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556. [14]DENIL M,SHAKIBI B,DINH L,et al.Predicting parameters in deep learning[C]//Neural Information Processing Systems(NIPS).2013. [15]HORNIK K,STINCHCOMBE M,WHITE H.Multilayer feedforward networks are universal approximators [J].Neural Networks,1989,2(5):359-366. [16]GARDNER M W,DORLING S R.Artificial neural networks (the multilayer perceptron )-a review of applications in the atmospheric sciences[J].Atmospheric Environment,1998,32(14/15):2627-2636. [17]ZHOU F Y,JIN L P,DONG J.Review of Convolutional Neural Network[J].Chinese Journal of Computers,2017,40(7):1229-1251.(in Chinese)周飞燕,金林鹏,董军.卷积神经网络研究综述[J].计算机学报,2017,40(7):1229-1251. [18]RUMELHART D E,HINTON G,WILLIAMS R J.Learning representations by back-propagating errors[J].Nature,1986,323(6088):533-536. [19]LECUN Y,BOTTOU L,BENGIO Y S,et al.Gradient-Based Learning Applied to Document Recognition[J].Proceedings of the IEEE,1998,86(11):2278-2324. [20]ZHANG Q H,WAN C X.Review of Convolutional Neural Network[J].Journal of Zhongyuan University of Technology,2017,28(3):1671-6906.(in Chinese)张庆辉,万晨霞.卷积神经网络综述[J].中原工学院学报,2017,28(3):1671-6906. [21]SONG H,JEFF P,JOHN T,et al.Learning both weights and connections for efficient neural network[C]//Neural Information Processing Systems(NIPS).2015:1135-1143. [22]GUO Y W,YAO A B,CHEN Y R.Dynamic Network Surgery for Efficient DNNs[C]//Neural Information Processing Systems(NIPS).2016. [23]LI H,KADAV A,DURDANOVIC I,et al.Pruning Filters for Efficient Convents[C]//International Conference on Learning Representations (ICLR).2017. [24]PAVIO M,STEPHEN T,TERO K,et al.Pruning Convolutional Neural Networks for Resource Efficient Inference[C]//International Conference on Learning Representations (ICLR).2017. [25]HARVEY L,ARNOLD B,ZIPURSKY L S,et al.Molecular Cell Biology:Neurotransmitters,Synapses,and Impulse Transmission[M].New York:W.H.Freeman,2000. [26]JOSE M A,MATHIEU S.Learning the Number of Neurons in Deep Networks[C]//Neural Information Processing Systems(NIPS).2016. [27]WEI W,WU C P,WANG Y D,et al.Learning Structured Sparsity in Deep Neural Networks[C]//Neural Information Proces-sing Systems(NIPS).2016. [28]WANG S J,CAI H R,JEFF B,et al.Training Compressed Fully-Connected Networks with a Density-diversity Penalty[C]//International Conference on Learning Representations (ICLR).2017. [29]YUAN M,LIN Y.Model selection and estimation in regression with grouped variables[J].Journal of The Royal Statistical So-ciety Series B-statistical Methodology,2006,68(1):49-67. [30]KIM S,XING E P.Tree-guided group lasso for multi-task regression with structured sparsity[C]//Proceedings of the 27th International Conference on Machine Learning.2010. [31]YUAN M,LIN Y.Model selection and estimation in regression with grouped variables[J].Journal of the Royal Statistical Society,Series B,2006,68(1):49-67. [32]PETER L B.For valid generalization the size of the weights is more important than the size of the network[C]//Neural Information Processing Systems(NIPS).1996. [33]KROGH A,JOHN A H.A simple weight decay can improve generalization[C]//Neural Information Processing Systems(NIPS).1992. [34]THEODORIDIS S.Machine Learning A Bayesian and Optimization Perspective[M].Academic Press,2015. [35]COLLINS M D,KOHLI P.Memory Bounded Deep Convolutional Networks[J].arXiv.1412.1442v1. [36]SIMON N,FRIEDMAN J,HASTIE T,et al.A sparse-group lasso[J].Journal of Computational and Graphical Statistics,2013,22(2):231-245. [37]PARIKH N,BOYD S.Proximal algorithms[J].Foundations and Trends in Optimization,2014,1(3):123-231. [38]MAX J,ANDREA V,ANDREW Z.Speeding up Convolutional Neural Networks with Low Rank Expansions[C]//British Machine Vision Conference.2014. [39]VADIM L,YAROSLAV G,MAKSIM R,et al.Speeding-up Convolutional Neural Networks Using Fine-tuned Cp-decomposition[C]//International Conference on Learning Representations (ICLR).2015. [40]JONGHOON J,AYSEGUL D,EUGENIO C.Flattened Convolutional Neural Networks for Feedforward Acceleration[C]//International Conference on Learning Representations (ICLR).2015. [41]KIM Y D,EUNHYEOK P,YOO S J,et al.Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications[C]//International Conference on Learning Representations (ICLR).2016. [42]RIGAMONTI R,SIRONI A,LEPETIT V,et al.Learning separable filters[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2013:2754-2761. [43]KOLDA T G,BADER B W.Tensor decompositions and applications[J].Siam Review,2009,51(3):455-500. [44]EMILY D,WOJCIECH Z,JOAN B,et al.Exploiting linear structure within convolutional networks for efficient evaluation[J].arXiv:1404.0736. [45]SORBER L,VAN B M,DELATHAUWER L.Tensorlab v2.0[EB/OL].http://tensorlab.net. [46]TOMASI G,BRO R.A comparison of algorithms for fitting the parafac model[J].Computational Statistics & Data Analysis,2006,50(7):1700-1734. [47]TUCKER L R.Some mathematical notes on three-mode factor analysis[J].Psychometrika,1966,31(3):279-311. [48]GONG Y C,LIU L,YANG M,et al.Compressing Deep Convolutional Networks Using Vector Quantization[C]//InternationalConference on Learning Representations (ICLR).2015. [49]CHEN W L,JAMES T W,STEPHEN T,et al.Compressing Neural Networks with the Hashing Trick[C]//International Conference on Machine Learning.2015:2285-2294. [50]SONG H,MAO H Z,WILLIAM J.Deep Compression:Com-pressing Deep Neural Networks with Pruning,Trained Quantization and Huffman Coding[C]//International Conference on Learning Representations (ICLR).2016. [51]CHOI Y J,MOSTAFA E K,JUNWON L.Towards the Limit of Network Quantization[C]//International Conference onLear-ning Representations (ICLR).2017. [52]ZHOU A J,YAO A B,GUO Y W,et al.Incremental Network Quantization: Towards Lossless Cnns with Low-precision Weights[C]//International Conference on Learning Representations (ICLR).2017. [53]JAN V L.On the construction of huffman trees[C]//International colloquium on automata,languages and programming(ICALP).1976:382-410. [54]MICHAEL F,AIJAN L,DMITRY V,et al.PerforatedCNNs:Acceleration through Elimination of Redundant Convolutions[C]//Neural Information Processing Systems(NIPS).2016. [55]LIN S H,JI R R,CHEN C,et al.ESPACE:Accelerating Convolutional Neural Networks via Eliminating Spatial & Channel Redundancy[C]//AAAI Conference on Artificial Intelligence(AAAI).2017. [56]STELIOS S D,SASA M,HENRY C H,et al.Managing performance vs.accuracy trade-offs with loop perforation[C]//ESEC.2011:124-134. [57]MISAILOVIC S,SIDIROGLOU S,HOFFMANN H,et al.Quality of service profiling[C]//International conference on software engineering(ICSE).2010:25-34. [58]MISAILOVIC S,ROY D M,RINARD M C.Probabilistically ac-curate program transformations[C]//Static Analysis Sympo-sium.2011:316-333. [59]CHEN W L,JAMES T,WILSON S T.Compressing Convolu-tional Neural Networks[J].arXiv:1506.04449v1. [60]WANG Y H,XU C,YOU S,et al.CNNpack:Packing Convolutional Neural Networks in the Frequency Domain[C]//Neural Information Processing Systems(NIPS).2016. [61]RAO K R,YIP P.Discrete cosine transform:algorithms,advantages,applications[M].Academic Press Professional Inc,2014. |
[1] | 余雪勇, 陈涛. 边缘计算场景中基于虚拟映射的隐私保护卸载算法[J]. 计算机科学, 2021, 48(1): 65-71. |
[2] | 单美静, 秦龙飞, 张会兵. L-YOLO:适用于车载边缘计算的实时交通标识检测模型[J]. 计算机科学, 2021, 48(1): 89-95. |
[3] | 何彦辉, 吴桂兴, 吴志强. 基于域适应的X光图像的目标检测[J]. 计算机科学, 2021, 48(1): 175-181. |
[4] | 李亚男, 胡宇佳, 甘伟, 朱敏. 基于深度学习的miRNA靶位点预测研究综述[J]. 计算机科学, 2021, 48(1): 209-216. |
[5] | 王瑞平, 贾真, 刘畅, 陈泽威, 李天瑞. 基于DeepFM的深度兴趣因子分解机网络[J]. 计算机科学, 2021, 48(1): 226-232. |
[6] | 于文家, 丁世飞. 基于自注意力机制的条件生成对抗网络[J]. 计算机科学, 2021, 48(1): 241-246. |
[7] | 仝鑫, 王斌君, 王润正, 潘孝勤. 面向自然语言处理的深度学习对抗样本综述[J]. 计算机科学, 2021, 48(1): 258-267. |
[8] | 张艳梅, 楼胤成. 基于深度神经网络的庞氏骗局合约检测方法[J]. 计算机科学, 2021, 48(1): 273-279. |
[9] | 丁钰, 魏浩, 潘志松, 刘鑫. 网络表示学习算法综述[J]. 计算机科学, 2020, 47(9): 52-59. |
[10] | 庄世杰, 於志勇, 郭文忠, 黄昉菀. 基于Zoneout的跨尺度循环神经网络及其在短期电力负荷预测中的应用[J]. 计算机科学, 2020, 47(9): 105-109. |
[11] | 何鑫, 许娟, 金莹莹. 行为关联网络:完整的变化行为建模[J]. 计算机科学, 2020, 47(9): 123-128. |
[12] | 张佳嘉, 张小洪. 多分支卷积神经网络肺结节分类方法及其可解释性[J]. 计算机科学, 2020, 47(9): 129-134. |
[13] | 叶亚男, 迟静, 于志平, 战玉丽, 张彩明. 基于改进CycleGan模型和区域分割的表情动画合成[J]. 计算机科学, 2020, 47(9): 142-149. |
[14] | 朱玲莹, 桑庆兵, 顾婷婷. 基于视差信息的无参考立体图像质量评价[J]. 计算机科学, 2020, 47(9): 150-156. |
[15] | 赵钦炎, 李宗民, 刘玉杰, 李华. 基于信息熵的级联Siamese网络目标跟踪[J]. 计算机科学, 2020, 47(9): 157-162. |
|