深度学习研究进展

doi:10.11896/j.issn.1002-137X.2015.05.006

摘要/Abstract

摘要： 深度学习(Deep Learning) 是一个近几年备受关注的研究领域,在机器学习中起着重要的作用。如果说浅层学习是机器学习的一次浪潮,那么深度学习作为机器学习的一个新领域,将掀起机器学习的又一次浪潮。深度学习通过建立、模拟人脑的分层结构来实现对外部输入的数据进行从低级到高级的特征提取,从而能够解释外部数据。首先介绍了深度学习的由来,分析了浅层学习存在的弊端；其次列举了深度学习的经典方法,主要以监督学习和无监督学习来展开介绍；然后对深度学习的最新研究进展及其应用进行了综述；最后总结了深度学习发展所面临的问题。

关键词: 机器学习,浅层学习,深度学习,卷积神经网络,深度置信网

Abstract: Deep learning plays an important role in machine learning.If shallow learning is a wave of machine learning, as a new field of machine learning,the deep learning will set off another wave of machine learning.Deep learning establishes and simulates the human brain’s hierarchical structure to extract the external input data’s features from lower to higher,which can explain the external data.Firstly,this paper discussed the origin of deep learning.Secondly,it described the common methods of deep learning illustrated by the example of supervised lear-ning and unsupervised learning.Then it generalized deep learning’s recent research and applications.Finally,it concluded the problems of development.

Key words: Machine learning,Shallow learning,Deep learning,CNNs,DBNs

郭丽丽,丁世飞. 深度学习研究进展[J]. 计算机科学, 2015, 42(5): 28-33. https://doi.org/10.11896/j.issn.1002-137X.2015.05.006

GUO Li-li and DING Shi-fei. Research Progress on Deep Learning[J]. Computer Science, 2015, 42(5): 28-33. https://doi.org/10.11896/j.issn.1002-137X.2015.05.006

参考文献

[1] 丁世飞.人工智能[M].北京:清华大学出版社,2011
[2] 史忠值.神经网络[M].北京:高等教育出版社,2009
[3] Rumelhart D,Hinton G,Williams R.Learning representationsby back-propagating errors[J].Nature,1986,323(6088):533-536
[4] 余凯,贾磊,陈雨强.深度学习的昨天、今天和明天[J].计算机研究与发展,2013,50(9):1799-1804
[5] Hinton G,Salakhutdinov R.Reducing the dimensionality of data with neural networks[J].Science,2006,313(5786):504-507
[6] Ding Shi-fei,Zhang Yan-an,Chen Jin-rong,et al.Research onUsing Genetic Algorithms to Optimize Elman Neural Networks[J].Neural Computing and Applications,2013,23(2):293-297
[7] Ding Shi-fei,Jia Wei-kuan,Su Chun-yang,et al.Research ofNeural Network Algorithm Based on Factor Analysis and Cluster Analysis[J].Neural Computing and Applications,2011,20(2):297-302
[8] Lee T S,Mumford D.Hierarchical Bayesian inference in the vi-sual cortex[J].Optical Society of America,2003,20(7):1434-1448
[9] Serre T,Wolf L,Bileschi S,et al.Robust object recognition with cortex-like mechanisms[J].IEEE Trans on Pattern Analysis and Machine Intelligence,2007,29(3):411-426
[10] Lee T S,Mumford D,Romero R,et al.The role of the primary visual cortex in higher level vision[J].Vision Research,1998,38 (15):2429-2454
[11] Bengio Y.Learning deep architectures for AI[J].Foundations and Trends in Machine Learning,2009,2(1):1-127
[12] Bengio Y,LeCun Y.Scaling learning algorithms towards AI[M]∥Bottou L,Chapelle O,Decoste D,et al.Large-Scale Kernel Machines.Cambridge:MIT Press,2007:321-358
[13] 李海峰,李纯果.深度学习结构和算法比较分析[J].河北大学学报:自然科学版,2012,32(5):538-544
[14] Hinton G E.Learning distributed representations of concepts[C]∥Proc.of the 8th Annual Conference of the Cognitive ScienceSociety.1986:1-12
[15] 孙志军,薛磊,许阳明.深度学习研究综述[J].计算机应用研究,2012,29(8):2806-2810
[16] Bengio Y,Delalleau O.On the expressive power of deep architectures[C]∥Proceedings of the 22nd International Conference on Algorithmic Learning Theory.Berlin Heidelberg,2011:18-36
[17] Vincent P,Larochelle H,Lajoie I,et al.Stacked denoising autoencoders:learning useful representations in a deep network with a local denoising criterion[J].Journal of Machine Learning Research,2010,11(12):3371-3408
[18] Hubel D H,Wiesel T N.Receptive Fields,Binocular Interaction and Functional Architecture in the Cat’s Visual Cortex [J].Journal of Physiology,1962,160:106-154
[19] Fukushima K.Neocognition:A Self-Organizing Neural Network Model for a mechanism of Pattern Recognition Unaffected by Shift in Postion[J].Biological Cybermetics,1980,36:193-202
[20] LeCun Y,Bottou L,Bengio Y,et al.Granient-based learning applied to document recognition[J].Proceedings of IEEE,1988,6(11):2278-2324
[21] LeCun Y,Boser B,Denker J S,et al.Backpropagation Applied to Handwritten Zip Code Recognition[J].Neural Computation,1989,1(4):541-551
[22] Serre T,Keriman G,Kouch M.A Quantitative Theory of Immediate Visual Recognition” Progress in Brain Research,Computational Neuroscience[J].Theoretical Insights into Brain Function,2007,165:33-56
[23] LeCun Y,Kavukcuogl U K,Farabe C.Convolutional networksand applications in vision[Z].International Symposium on Circuits and Systems,Paris,2010
[24] Kwolek B.Face Detection Using Convolutional Neural Net-works And Gobor Filters[J].Artificial Neural Networks:Biological Inspirations,2005,3699:551-556
[25] Rosenblatt F.The Perceptron:A Probabilistic Model For Information Storage and Organization in the Brain[J].Psychological Review,1958,65:386-408
[26] Neubauer C.Shape,position and size invariant visual patternrecognition based on principles of neocognitron and perception in artificial neural networks[M].Netherlands:North Holland,1992:833-837
[27] Laserson J.From neural networks to deep learning:zeroing in on the human brain[J].ACM Crossroads Student Magazine,2011,18(1):29-34
[28] LeCun Y,Botton L,Bengio Y,et al.Gradient-based learning applied to document recognition[J].Proceedings of IEEE,1998,86(11):2278-2324
[29] Vincent P,Larochelle H,Bengio Y,et al.Extracting and composing robust features with denoising autoencoders[C]∥Proceedings of the 25th International Conference on Machine Learning (ICML’2008).New York:ACM Press,2008:1096-1103
[30] Huang Fu-jie,LeCun Y.Large-scale learning with SVM andconvolutional for generic object categorization [C]∥Proc.of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Washington DC:IEEE Computer Society,2006:284-291
[31] 聂仁灿,姚绍文,周冬明.基于简化脉冲耦合神经网络的人脸识别[J].计算机科学,2014,41(2):297-301
[32] Simard P Y,Steinkraus D,Platt J C.Best Practice for Convolutional Neural Networks Applied to Visual Document Analysis[C]∥Seventh International Conference on Document Analysis and Recognition.2003:963-985
[33] Sukittanon S,Surendran A C,Burges C J C,et al.Convolutional Networks for speech Detection.http://131.107.65.14/pubs/68033/convnet_speechdetect.pdf
[34] Chen Y,Han C,Wang C,et al.The application of a convolution neural network on face and license plate detection[C]∥18th International Conference on Pattern Recognition(ICPR).Hong Kong,China:IEEE Computer Society,2006:552-555
[35] 赵元庆,吴华.多尺度特征和神经网络相融合的手写体数字识别[J].计算机科学,2013,40(8):316-318
[36] Ji S,Xu W,Yang M,et al.3D Convolutional Neural Networks for Human Action Recognition[J].IEEE Transaction on Pattern Analysis and Machine Intelligence,2013,35(1):221-231
[37] Hinton G E.Distributed representations[R].Tech.Report,Uni-versity of Toronto,1984
[38] Hinton G E,Osindero S.A fast learning algorithm for deep belief nets[J].Neural Computation,2006,18:1527-1554
[39] Lawrence C E,Altschul S F,Boguski M S,et al.Detecting Subtle Sequence Signals:A Gibbs Sampling Strategy for Multiple Alignment[J].Science,1993,262:208-214
[40] Bishop C M.Pattern recognition and machine learning[M].New York:Springer,2006
[41] Hinton G E,Dayanp,Frey B,et al.The wake-sleep algorithm for unsupervised neural network[J].Science,1995,268:1158-1161
[42] Liu Yan,Zhou Shu-sen,Chen Qing-cai.Discriminative deep belief networks for visual data classification[J].Pattern Recognition,2010,44(10):2287-2296
[43] Yu Dong,Deng Li.Deep convex net:a scalable architecture for speech pattern classification[C]∥Proc.of the 12th Annual Conference of International Speech Communication Association.2011:2296-2299
[44] Zhou Shu-sen,Chen Qing-cai,Wang Xiao-long.Convolutional Deep Networks for Visual Data Classification[J].Neural Process Lett,2013,38:17-27
[45] Wong W K,Sun M M.Deep learning regularized fisher mappings[J].IEEE Transactions on Neural Networks,2011,22(10):1668-1675
[46] 孙志军,薛磊,许阳明.基于深度学的边际Fisher分析特征提取算法[J].电子与信息学报,2013,35(4):805-811
[47] Zhou Shu-sen,Chen Qing-cai,Wang Xiao-long.Active deeplearning method for semi-supervised sentiment classification[J].Neurocomputing,2013,120:536-546
[48] Tom A.Stanford algorithm analyzes sentence sentiment,ad-vances machine learning [N].Stanford University,2013
[49] Li Fei-fei,Fergus R,Perona P.Learning generative visual mo-dels from few training examples:an incremental Bayesian approach tested on 101 object categories[J].Computer Vision and Image Understanding,2004,106(1):59-70
[50] Hinton G E,Li Deng,Dong Yu,et al.Deep neural networks for acoustic modeling in speech recognition [J].IEEE Signal Processing Magazine,2012,29(6):82-97
[51] Markoff J.Scientists see promise in deep-learning programs[N].The New York Times,2012-11-23
[52] Bach F,Jenatton R,Obozinski G.Structured sparsity throughconvex optimization.http:arxiv.org/pdf.1109.2397/pdf
[53] Bengio Y,Courville A,Vincent P.Representation Learning:AReview and New Perspectives[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(8):1798-1828
[54] Markoff J.How many computers to identify a cat?[N].The new York Times,2012-06-25
[55] 李彦宏.2012百度年会主题报告:相信技术的力量[R].北京:百度,2013
[56] 10 Breakthrough Technologies 2013 [N].MIT Technology Review,2013-04-2

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed