Computer Science ›› 2019, Vol. 46 ›› Issue (5): 1-12.doi: 10.11896/j.issn.1002-137X.2019.05.001

    Next Articles

Survey on Cost-sensitive Deep Learning Methods

WU Yu-xi, WANG Jun-li, YANG Li, YU Miao-miao   

  1. (College of Electronics and Information Engineering,Tongji University, Shanghai 201804,China)
  • Received:2018-06-17 Revised:2018-09-29 Published:2019-05-15

Abstract: Cost-sensitive learning method can effectively alleviate the problem of data imbalance in classification tasks and has been successfully applied to various traditional machine learning techniques.With the continuous development of deep learning technology,cost-sensitive method has become a research hotspot again.The combination of deep learning with cost-sensitive methods can not only breaks through the limitations of traditional machine learning technology,but also improve the data sensitivity and classification accuracy of the model,especially when there is a certain imbalance in the data.However,how to effectively combine theabove two factors has become the focus and difficulty of the research.From the aspects of network structure,loss function and training method,researchers have improved the performance of the deep learning model combined with cost-sensitive method.In this paper,the development of the combination of deep learning and cost-sensitive method was described in detail,several innovative models were analyzed and the classification performance of these model was compared.Finally,the development trend of combination of deep learning and cost-sensitive method was discussed.

Key words: Cost-sensitive, Deep learning, Imbalanced data, Neural network

CLC Number: 

  • TP18
[1]SUN Y,WONG A K C,KAMEL M S.Classification of Imba-lanced Data:A REVIEW[J].International Journal of Pattern Recognition & Artificial Intelligence,2009,23(4):687-719.
[2]KHALILIA M,CHAKRABORTY S,POPESCU M.Predicting disease risks from highly imbalanced data using random forest[J].Bmc Medical Informatics & Decision Making,2011,11(1):51.
[3]ZHANG D,XU W.A Data-Distribution-Based Imbalanced Data Classification Method for Credit Scoring Using Neural Networks[C]∥Sixth International Conference on Business Intelligence and Financial Engineering.IEEE,2014:557-561.
[4]XIONG B Y,WANG G Y,DENG W B,et al.Unsaturated data undersampling method based on sample weight[J].Journal of Computer Research and Development,2016,53(11):2613-2622.(in Chinese)熊冰妍,王国胤,邓维斌,等.基于样本权重的不平衡数据欠抽样方法[J].计算机研究与发展,2016,53(11):2613-2622.
[5]FERNÁNDEZ A,LÓPEZ V,GALAR M,et al.Analysing the classification of imbalanced data-sets with multiple classes:Binarization techniques and ad-hoc approaches[J].Knowledge-Based Systems,2013,42(2):97-110.
[6]CHAWLA N V,BOWYER K W,HALL L O,et al.SMOTE:synthetic minority over-sampling technique[J].Journal of Artificial Intelligence Research,2002,16(1):321-357.
[7]CHAWLA N V.Data Mining for Imbalanced Datasets:An Overview[M]∥Data Mining and Knowledge Discovery Handbook.US:Springer,2005:853-867.
[8]YE Z F,WEN Y M,LU B L,et al.A Review of Researches on Unbalanced Classification[J].Journal of Intelligent Systems,2009,4(2):148-156.(in Chinese)叶志飞,文益民,吕宝粮,等.不平衡分类问题研究综述[J].智能系统学报,2009,4(2):148-156.
[9]KUBAT M,MATWIN S.Addressing the Curse of Imbalanced Training Sets:One-Sided Selection[C]∥International Confe-rence on Machine Learning.1997:179-186.
[10]SARDARI S,EFTEKHARI M.A Fuzzy Decision tree approach for imbalanced data classification[C]∥International Conference on Computer and Knowledge Engineering.IEEE,2017:292-297.
[11]WANG B,PINEAU J.Online Bagging and Boosting for Imba-lanced Data Streams[J].IEEE Transactions on Knowledge and Data Engineering,2016,28(12):3353-3366.
[12]CHAWLA N V,LAZAREVIC A,HALL L O,et al.SMOTEBoost:Improving Prediction of the Minority Class in Boosting[C]∥Knowledge Discovery in Databases:Pkdd 2003,European Conference on Principles and Practice of Knowledge Discovery in Databases.Cavtat-Dubrovnik,Croatia,DBLP,2003:107-119.
[13]ABDI L,HASHEMI S.To combat multi-class imbalanced problems by means of over-sampling and boosting techniques[M]∥Soft Computing.Springer-Verlag,2015:3369-3385.
[14]TAO X M,HAO S Y,ZHANG D X,et al.An Overview of Unbalanced Data Classification Algorithms[J].Journal of Chongqing University of Posts and Telecommunications(Natural Science),2013,25(1):101-110.(in Chinese)陶新民,郝思媛,张冬雪,等.不均衡数据分类算法的综述[J].重庆邮电大学学报(自然科学版),2013,25(1):101-110.
[15]YANG J M,GAO C,QU C Y,et al.Cost-sensitive Random Fo-rest Unbalanced Data Classification Algorithm[J].Science Technology and Engineering,2018(6):285-290.(in Chinese)杨杰明,高聪,曲朝阳,等.基于代价敏感的随机森林不平衡数据分类算法[J].科学技术与工程,2018(6):285-290.
[16]ZANGENEH V,SHAJARI M.A cost-sensitive move selection strategy for moving target defense[J].Computers & Security,2018,75:S0167404817302821.
[17]XIE J,LI M.Software defect location based on cost-sensitive interval distribution optimization[J].Acta Journal of Software,2017,28(11):3072-3079.(in Chinese)解铮,黎铭.基于代价敏感间隔分布优化的软件缺陷定位[J].软件学报,2017,28(11):3072-3079.
[18]SHI Y W,WANG H J.A cost-sensitive random forest classifier based on a new measure of impureness[J].Journal of Computer Science,2017,44(S2):98-101.(in Chinese)师彦文,王宏杰.基于新型不纯度度量的代价敏感随机森林分类器[J].计算机科学,2017,44(S2):98-101.
[19]ZHAO P,ZHANG Y,WU M,et al.Adaptive Cost-Sensitive Online Classification[J].IEEE Transactions on Knowledge & Data Engineering,2018,PP(99):1-1.
[20]VERMA A,HANAWAL M K.Unsupervised cost sensitive predictions with side information[C]∥the ACM India Joint International Conference.ACM,2018:364-367.
[21]BACH M,WERNER A.Cost-Sensitive Feature Selection forClass Imbalance Problem[M]∥Information Systems Architecture and Technology:Proceedings of 38th International Confe-rence on Information Systems Architecture and Technology- ISAT 2017.2018:182-194.
[22]LI Q J,ZHAO Y Q,GU Z,et al.Loss function design in cost-sensitive learning[J].Control Theory & Applications,2015,32(5):689-694.(in Chinese)李秋洁,赵亚琴,顾洲,等.代价敏感学习中的损失函数设计[J].控制理论与应用,2015,32(5):689-694.
[23]CHAN P,STOLFO S.Toward scalable learning with non-uniform class and cost distributions[C]∥International Conference on Knowledge Discovery & Data Mining.1998:164-168.
[24]ELKAN C.The Foundations of Cost-Sensitive Learning[C]∥Seventeenth International Joint Conference on Artificial Intelligence.2001:973-978.
[25]DOMINGOS P.MetaCost:a general method for making classi-fiers cost-sensitive[C]∥ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.ACM,1999:155-164.
[26]LI J,FONG S,MOHAMMED S,et al.Improving the classification performance of biological imbalanced datasets by swarm optimization algorithms[J].Journal of Supercomputing,2016,72(10):1-21.
[27]RYU D,JANG J I,BAIK J.A transfer cost-sensitive boosting approach for cross-project defect prediction[J].Software Quality Journal,2017,25(1):1-38.
[28]RYU D,JANG J I,BAIK J.A transfer cost-sensitive boosting approach for cross-project defect prediction.Software Quality Journal,2017,25(1):235-272.
[29]DRUMMOND C,HOLTE R C.Exploiting the Cost(In)sensiti-vity of Decision Tree Splitting Criteria[C]∥Seventeenth International Conference on Machine Learning.Morgan Kaufmann Publishers Inc.2000:239-246.
[30]LOMAX S,VADERA S.A survey of cost-sensitive decision tree induction algorithms[J].Acm Computing Surveys,2013,45(2):1-35.
[31]KNOLL U,NAKHAEIZADEH G,TAUSEND B.Cost-sensitive pruning of decision trees[C]∥Machine Learning:ECML-94,European Conference on Machine Learning.Catania,Italy,DBLP,1994:383-386.
[32]BRADFORD J P,KUNZ C,KOHAVI R,et al.Pruning Decision Trees with Misclassification Costs[M]∥Machine Learning:ECML-98.Springer Berlin Heidelberg,1998:131-136.
[33]ZHENG E H,LI P,SONG Z H,et al.Cost-Sensitive Support Vector Machines[J].Control and Decision,2006,21(4):473-476.(in Chinese)郑恩辉,李平,宋执环,等.代价敏感支持向量机[J].控制与决策,2006,21(4):473-476.
[34]ADACHI Y,ONIMURA N,YAMASHITA T,et al.Classification of Imbalanced Documents by Feature Selection[C]∥International Conference on Compute and Data Analysis.ACM,2017:228-232.
[35]ZHOU Y H,ZHOU Z H.Cost-sensitive large interval distributed learning machine[J].Journal of Computer Research and Development,2016,53(9):1964-1970.(in Chinese)周宇航,周志华.代价敏感大间隔分布学习机[J].计算机研究与发展,2016,53(9):1964-1970.
[36]KUKAR M Z,KONONENKO I.Cost-Sensitive Learning with Neural Networks[C]∥Proceedings of the 13th European Conference on Artificial Intelligence,Wiley,New York,1998:445-449.
[37]ZHOU Z H,LIU X Y.Training cost-sensitive neural networks with methods addressing the class imbalance problem[J].IEEE Transactions on Knowledge & Data Engineering,2006,18(1):63-77.
[38]HUANG L W,JIANG B T,LU S Y,et al.A review of recommendation systems based on deep learning [J].arxiv:1707.07435,2018.(in Chinese)黄立威,江碧涛,吕守业,等.基于深度学习的推荐系统研究综述[J].arxiv:1707.07435,2018.
[39]HE Y X,SUN S T,NIU F F,et al.An in-depth learning model for emotional semantics enhancement in Weibo emotion analysis [J].Chinese Journal of Computers,2017,40(4):773-790.(in Chinese)何炎祥,孙松涛,牛菲菲,等.用于微博情感分析的一种情感语义增强的深度学习模型[J].计算机学报,2017,40(4):773-790.
[40]CHENG X Q,QI X L,WANG Y Z,et al.A review of big data systems and analysis techniques[J].Journal of Software,2014,25(9):1889-1908.(in Chinese)程学旗,靳小龙,王元卓,等.大数据系统和分析技术综述[J].软件学报,2014,25(9):1889-1908.
[41]YAN X F,ZHOU G D.Research on Deep Learning for Natural Language Processing[J].Journal of Automation,2016,42(10):1445-1465.(in Chinese)奚雪峰,周国栋.面向自然语言处理的深度学习研究[J].自动化学报,2016,42(10):1445-1465.
[42]KAMILARIS A,PRENAFETA-BOLDú F X.Deep learning in agriculture:A survey[J].Computers & Electronics in Agriculture,2018,147(1):70-90.
[43]HATCHER W G,YU W.A Survey of Deep Learning:Plat-forms,Applications and Emerging Research Trends[J].IEEE Access,2018,PP(99):1-1.
[44]MA S L,WUNIRI Q Q G,LI X P.Summary of big data and deep learning [J].Journal of Intelligent Systems,2016,11(6):728-742.(in Chinese)马世龙,乌尼日其其格,李小平.大数据与深度学习综述[J].智能系统学报,2016,11(6):728-742.
[45]HINTON G,OSINDERO S,TEH Y.A fast learning algorithm for deep belief nets[J].Neural Computation,2006,18(7):1527-1554.
[46]BENGIO Y,LAMBLIN P,POPOVICI D,et al.Greedy layer-wise training of deep networks[C]∥Proc of the 12th Annual Conference on Neural Information Processing System.2006:153-160.
[47]BENGIO Y.Learning deep architectures for AI [J].Foundations and Trends in Machine Learning,2009,2(1):1-127.
[48]BENGIO Y,LAMBLIN P,POPOVICI D,et al.Greedy Layer-wise training of deep networks [C]∥Advances in Neural Information Processing Systems.Cambridge,MA:MIT Press,2007:153-160.
[49]YIN B C,WANG W T,WANG L C.A summary of deep lear-ning research [J].Journal of Beijing University of Technology,2015,41(1):48-59.(in Chinese)尹宝才,王文通,王立春.深度学习研究综述[J].北京工业大学学报,2015,41(1):48-59.
[50]CHEN W H,AN J Y,LI R F,et al.Summary of deep learning cognitive computing [J].Automation Journal,2017,43(11):1886-1897.(in Chinese)陈伟宏,安吉尧,李仁发,等.深度学习认知计算综述[J].自动化学报,2017,43(11):1886-1897.
[51]REN S,HE K,GIRSHICK R,et al.Faster R-CNN:Towards Real-Time Object Detection with Region Proposal Networks[J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2015,39(6):1137-1149.
[52]CHENG B,WEI Y,SHI H,et al.Revisiting RCNN:On Awa-kening the Classification Power of Faster RCNN[M]∥Compu-ter Vision(ECCV 2018).2018.
[53]MAHAPATRA D,ROY P K,SEDAI S,et al.Retinal Image Quality Classification Using Saliency Maps and CNNs[J].Inte-racting with Computers,2016,10(4):385-399.
[54]ZHONG Y,FEI F,LIU Y,et al.SatCNN:satellite image dataset classification using agile convolutional neural networks[J].Remote Sensing Letters,2017,8(2):136-145.
[55]YUE S.Imbalanced Malware Images Classification:a CNNbased Approach[J].arXiv:1708.08042,2017.
[56]WINDRIM L,RAMAKRISHNAN R,MELKUMYAN A,et al.Hyperspectral CNN Classification with Limited Training Samples[J].arXiv:1611.09007v1,2016.
[57]HOCHREITER S,SCHMIDHUBER J.Long short-term memory[J].Neural Computation,1997,9(8):1735-1780.
[58]MNIH V,HEESS N,GRAVES A,et al.Recurrent Models of Visual Attention[J].Advances in Neural Information Processing Systems,2014:2204-2212.
[59]SHEN L,ZHANG J.Empirical Evaluation of RNN Architec-tures on Sentence Classification Task[J].arXiv:1609.09171,2016.
[60]KANG S I,LEE S M.Improvement of Speech/Music Classification Based on RNN in EVS Codec for Hearing Aids[J].Journal of Rehabilitation Welfare Engineering & Assistive Technology,2017,11.
[61]MUN S,PARK D,KO H.A Novel Discriminative Feature Extraction for Emotion Classification using RNN based Source Separation[J].Ieice Transactions on Information & Systems,2017,100(12):3041-3044.
[62]CHENG J,LI P,DING Z,et al.Sentiment Classification of Chinese Microblogging Texts with Global RNN[C]∥IEEE International Conference on Data Science in Cyberspace.IEEE,2017:653-657.
[63]WANG J,YANG Y,MAO J,et al.CNN-RNN:A UnifiedFramework for Multi-label Image Classification[J].arXiv:1604.04573,2016.
[64]LIU F,LI S,ZHANG L,et al.3DCNN-DQN-RNN:A Deep Reinforcement Learning Framework for Semantic Parsing of Large-Scale 3D Point Clouds[C]∥IEEE International Confe-rence on Computer Vision.IEEE Computer Society,2017:5679-5688.
[65]HORI T,WATANABE S,ZHANG Y,et al.Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM[J].arXiv:1706.02737,2017.
[66]XIONG H K,GAO X,LI Y H,et al.Explicit,structured andmultimodal deep neural networks[J].Pattern Recognition & Artificial Intelligence,2018,31(1):1-11.(in Chinese)熊红凯,高星,李劭辉,等.可解释化、结构化、多模态化的深度神经网络[J].模式识别与人工智能,2018,31(1):1-11.
[67]GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Ge-nerative adversarial nets[C]∥Proceedings of the 2014 Conference on Advances in Neural Information Processing Systems 27.Montreal,Canada:Curran Associates,Inc.,2014:2672-2680.
[68]WANG K F,GOU C,DUAN Y J,et al.Generative Adversarial Networks:The State of the Art and Beyond[J].ACTA AUTOMATICA SINICA,2017,43(3):321-332.(in Chinese)王坤峰,苟超,段艳杰,等.生成式对抗网络GAN的研究进展与展望[J].自动化学报,2017,43(3):321-332.
[69]ARJOVSKY M,CHINTALA S,BOTTOU L.Wasserstein GAN[J].arXiv:1701.07875,2017.
[70]CHEN X,DUAN Y,HOUTHOOFT R,et al.InfoGAN:inter-pretable representation learning by information maximizing ge-nerative adversarial nets[C]∥Proceedings of the 2016 Neural Information Processing Systems.2016.
[71]LI Y B,ZHOU S M.A New Loss Function[J].Journal of Wuhan University of Technology,1999,21(4):86-87.(in Chinese)李跃波,周树民.一种新的损失函数[J].武汉理工大学学报,1999,21(4):86-87.
[72]WANG S,LIU W,WU J,et al.Training deep neural networks on imbalanced data sets[C]∥International Joint Conference on Neural Networks.IEEE,2016:4368-4374.
[73]CHENG D,GONG Y,ZHOU S,et al.Person Re-identification by Multi-Channel Parts-Based CNN with Improved Triplet Loss Function[C]∥IEEE Conference on Computer Vision and Pattern Recognition.IEEE Computer Society,2016:1335-1344.
[74]ZHANG X L.Speech separation by cost-sensitive deep learning[C]∥APSIPA.2017:159-162.
[75]JIANG J,LIU X,ZHANG K,et al.Automatic diagnosis of imbalanced ophthalmic images using a cost-sensitive deep convolutional neural network[J].Biomedical Engineering Online,2017,16(1):132.
[76]HUANG C,LI Y,CHEN C L,et al.Learning Deep Representation for Imbalanced Classification[C]∥Computer Vision and Pattern Recognition.IEEE,2016:5375-5384.
[77]KHAN S H,HAYAT M,BENNAMOUN M,et al.Cost-Sensitive Learning of Deep Feature Representations From Imbalanced Data[J].IEEE Transactions on Neural Networks & Learning Systems,2015,PP(99):1-15.
[78]YUAN C,LIN H T,YANG S W.Cost-aware Pre-training for Multiclass Cost-sensitive Deep Learning[J].arXiv:1511.09337,2015.
[79]YUAN C,LIN H T.Cost-Sensitive Deep Learning with Layer-Wise Cost Estimation[J].arXiv:1611.05134,2016.
[80]TU H H,LIN H T.One-sided support vector regression for multiclass cost-sensitive classification[C]∥International Conference on International Conference on Machine Learning.Omnipress,2010:1095-1102.
[81]LI H,ZHOU X,HUANG B,et al.Cost-Sensitive Three-Way Decision:A Sequential Strategy[M]∥Rough Sets andKnow-ledge Technology.Springer Berlin Heidelberg,2013:325-337.
[82]PANG J B,LIN H H,DUAN L J,et al.Accelerate convolutional neural networks for binary classification:a cascading cost-sensitive feature approach[C]∥Proceedings of the 13th European Conference on Artificial Intelligence(ECAI-98).2016.
[83]LEE Y J,MANGASARIAN O L.SSVM:A Smooth Support Vector Machine for Classification[J].Computational Optimization & Applications,2001,20(1):5-22.
[84]HINTON G E,OSINDERO S,TEH Y.A fast learning algorithm for deep belief nets.Neural Computation,2006,18(7):1527-1554.
[85]VINCENT P.A connection between score matching and denoi-sing autoencoders[M].MIT Press,2011.
[86]RAJ V,MAGG S,WERMTER S.Towards Effective Classification of Imbalanced Data with Convolutional Neural Networks[M]∥Artificial Neural Networks in Pattern Recognition.Springer International Publishing,2016:150-162.
[87]SIMONYAN K,ZISSERMAN A.Very Deep Convolutional Networks for Large-Scale Image Recognition[J].arXiv:1409.1556,2014.
[88]FIORE U,SANTIS A D,PERLA F,et al.Using generative adversarial networks for improving classification effectiveness in credit card fraud detection[J].Information Sciences,2017:S0020025517311519.
[89]GENG Y,LUO X.Cost-Sensitive Convolution based NeuralNetworks for Imbalanced Time-Series Classification[J].arXiv:1801.04396 2018.
[1] RAO Zhi-shuang, JIA Zhen, ZHANG Fan, LI Tian-rui. Key-Value Relational Memory Networks for Question Answering over Knowledge Graph [J]. Computer Science, 2022, 49(9): 202-207.
[2] NING Han-yang, MA Miao, YANG Bo, LIU Shi-chang. Research Progress and Analysis on Intelligent Cryptology [J]. Computer Science, 2022, 49(9): 288-296.
[3] TANG Ling-tao, WANG Di, ZHANG Lu-fei, LIU Sheng-yun. Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy [J]. Computer Science, 2022, 49(9): 297-305.
[4] ZHOU Fang-quan, CHENG Wei-qing. Sequence Recommendation Based on Global Enhanced Graph Neural Network [J]. Computer Science, 2022, 49(9): 55-63.
[5] ZHOU Le-yuan, ZHANG Jian-hua, YUAN Tian-tian, CHEN Sheng-yong. Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion [J]. Computer Science, 2022, 49(9): 155-161.
[6] XU Yong-xin, ZHAO Jun-feng, WANG Ya-sha, XIE Bing, YANG Kai. Temporal Knowledge Graph Representation Learning [J]. Computer Science, 2022, 49(9): 162-171.
[7] WANG Jian, PENG Yu-qi, ZHAO Yu-fei, YANG Jian. Survey of Social Network Public Opinion Information Extraction Based on Deep Learning [J]. Computer Science, 2022, 49(8): 279-293.
[8] HAO Zhi-rong, CHEN Long, HUANG Jia-cheng. Class Discriminative Universal Adversarial Attack for Text Classification [J]. Computer Science, 2022, 49(8): 323-329.
[9] JIANG Meng-han, LI Shao-mei, ZHENG Hong-hao, ZHANG Jian-peng. Rumor Detection Model Based on Improved Position Embedding [J]. Computer Science, 2022, 49(8): 330-335.
[10] WANG Run-an, ZOU Zhao-nian. Query Performance Prediction Based on Physical Operation-level Models [J]. Computer Science, 2022, 49(8): 49-55.
[11] CHEN Yong-quan, JIANG Ying. Analysis Method of APP User Behavior Based on Convolutional Neural Network [J]. Computer Science, 2022, 49(8): 78-85.
[12] ZHU Cheng-zhang, HUANG Jia-er, XIAO Ya-long, WANG Han, ZOU Bei-ji. Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism [J]. Computer Science, 2022, 49(8): 113-119.
[13] SUN Qi, JI Gen-lin, ZHANG Jie. Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection [J]. Computer Science, 2022, 49(8): 172-177.
[14] YAN Jia-dan, JIA Cai-yan. Text Classification Method Based on Information Fusion of Dual-graph Neural Network [J]. Computer Science, 2022, 49(8): 230-236.
[15] HOU Yu-tao, ABULIZI Abudukelimu, ABUDUKELIMU Halidanmu. Advances in Chinese Pre-training Models [J]. Computer Science, 2022, 49(7): 148-163.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!