Computer Science ›› 2016, Vol. 43 ›› Issue (8): 249-253.doi: 10.11896/j.issn.1002-137X.2016.08.050

Previous Articles     Next Articles

Deep Belief Networks Research Based on Maximum Information Coefficient

ZENG An and ZHENG Qi-mi   

  • Online:2018-12-01 Published:2018-12-01

Abstract: The traditional deep belief networks use reconstruction error as the evaluation criteria of restricted boltzmann machine(RBM) networks in the training process,which can reflect the likelihood between RBM network and training samples to some extent.However,it is not reliable.Maximum information coefficient (MIC),based on the estimations of Shannon entropy and conditional entropy,identifies interesting relationships between pairs of variables in large data sets and captures a subset of highly related features.The MIC can be used as a criterion for evaluating a network since it is robust to outliers.In order to construct models that fit data well and reduce classification error,a deep belief networks based on MIC method was proposed.MIC is used not only in dimensionality reduction,but also in improving the unreliability of the reconstruction error.Classification experiments were performed on handwriting data sets of MNIST and USPS by several traditional methods and deep belief networks based on MIC method.The results show that the latter can improve the recognition rate effectively.

Key words: DBNs,MIC,Reconstruction error,Dimensionality reduction

[1] Catanzaro B,Sundaram N,Keutzer K.Fast support vector machine training and classification on graphics processors[C]∥Helsinki Finland.ACM,2008:104-111
[2] Pradhan B,Lee S.Regional landslide susceptibility analysis using back-propagation neural network model at Cameron High-land,Malaysia [J].Landslides,2010,7(1):13-30
[3] Yoshua B.Learning Deep Architectures for AI [J].Foundations and Trends in Machine Learning,2009,2(1):1-127
[4] George D,Marc’Aurelio R,Hinton G.Phone Recognition with the Mean-Covariance Restricted Boltzmann Machine[C]∥24th Annual Conference on Neural Information Processing Systems.2010:1-9
[5] Ragheb W,Ali L.Handwritten Digit Recognition using SparseDeep Architectures[C]∥2014 9th International Conference on Intelligent Systems:Theories and Applications.2014:1-6
[6] Peng Bo,Zang Di.Vehicle Logo Recognition Based on Deep Learning[J].Computer Science,2015,42(4):268-273(in Chinese) 彭博,臧笛.基于深度学习的车标识别方法研究[J].计算机科学,2015,42(4):268-273
[7] Sarikaya R,Hinton G,Deoras A.Application of Deep Belief Networks for Natural Language Understanding[J].ACM Transactions on Audio,Speech and Language Processing,2014,22(4):778-784
[8] Guo Li-li,Ding Shi-fei.Research Progress on Deep Learning[J].Computer Science,2015,42(5):28-33(in Chinese) 郭丽丽,丁世飞.深度学习研究进展[J].计算机科学,2015,42(5):28-33
[9] Abdel O,Mohamed A,Jiang Hui,et al.Convolutional Neural Networks for Speech Recognition[J].Audio,Speech,and Language Processing,2014,22(10):1533-1545
[10] Vidya R,Nasira G M,Priyankka R.Sparse Coding:A DeepLearning Using Unlabeled Data for High-Level Representation[C]∥Computing and Communication Technologies.2014:124-127
[11] Jiang Xiao-juan,Zhang Ying-hua,Zhang Wen-sheng,et al.A novel sparse auto-encoder for deep unsupervised learning[C]∥2013 Sixth International Conference on Advanced Computatio-nal Intelligence.2013:256-261
[12] Liu Jian-wei,Liu Yuan,Luo Xiong-lin.Research and Development on Boltzmann Machine[J].Journal of Computer Research and Development,2014,1(1):1-16(in Chinese) 刘建伟,刘媛,罗雄麟.玻尔兹曼机研究进展[J].计算机研究与发展,2014,1(1):1-16
[13] Courville A,Bergstra J,Bengio Y.A Spike and Slab Restricted Boltzmann Machine[C]∥Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics.2011:233-241
[14] Bengio Y,Lamblin P,Popovici D,et al.Greedy layer-wise trai-ning of deep networks[J].Advances in Neural Information Processing System,2007,9:153-160
[15] Hinton G,Osindero S,Teh Y.A fast learning algorithm for deep belief nets[J].Neural Comput,2006,8(7):1527-1554
[16] Hinton G,Salakhutdinov R.Reducing the Dimensionality of Data with Neural Networks[J].Science,2006,313:504-507
[17] Liu Yan,Zhou Shu-sen,Chen Qing-cai.Discriminative deep belief networks for visual data classification[J].Pattern Recognition,2011,4(10/11):2287-2296
[18] Zhu Ming,Wu Yan.A novel deep model for image recognition[C]∥2014 5th IEEE International Conference on Software Engineering and Service Science.2014:373-376
[19] Hinton G.Training products of experts by minimizing contrastive divergence[J].Neural Comput,2002,14(8):1771-1800
[20] Hinton G.A practical guide to training Restricted BoltzmannMachines[J]∥Momentum,2010,9(1):599-619
[21] David N,et al.Detecting Novel Associations in Large Data Sets[J].Science,2011,334(6062):1518-1524
[22] Zhao Xi,Deng Wei,Shi Yong.Feature Selection with Attributes Clustering by Maximal Information Coefficient[J].Procedia Computer Science,2013,7:70-79
[23] LeCun Y,Chopra S,Hadsell R,et al.A tutorial on energy-based learning[C]∥Conference on Predicting Structured Data.2006:191-246
[24] Dahl G,Dong D,Li D,et al.Context-Dependent Pre-trainedDeep Neural Networks for Large Vocabulary Speech Recognition[J].IEEE Transactions on Audio,Speech,and Language Processing,2011(20):30-42
[25] Larochelle H,Mande M,Pascanu R,et al.Learning Algorithms for the Classification Restricted Boltzmann Machine[J].Journal of Machine Learning Research,2012(13):643-669
[26] Roux N,Bengio Y.Representational power of Restricted Boltzmann Machines and deep belief networks[J].Neural Comput,2008(6):1631-1649
[27] Robert M.Entropy and Information Theory [M].Springer-Verlag,New York,2011
[28] David N,Yakir A,Michael M,et al.Equitability Analysis of the Maximal Information Coefficient with Comparisons[J].Compu-ter Science,2013:1-22
[29] Justin B,Gurinder S.Equitability,mutual information,and the maximal information coefficient[C]∥Proceedings of the National Academy of Sciences of the United States of America.2014:3354-3359
[30] Speed T.A Correlation for the 21st Century [J].Science,2011,334(6062):1502-1503
[31] Tan Qiu-heng,Jiang Hang-jin,Ding Yi-ming.Model SelectionMethod Based on Maximal Information Coefficient of Residual[J].Acta Mathematica Scientia,2014,4(2):579-592
[32] Liu Jian-wei,Chi Guang-hui,Luo Xiong-lin.Contrastive Diver-gence Learning for the Restricted Boltzmann Machine[C]∥2013 Ninth International Conference on Natural Computation.2013:18-22

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!