计算机科学 ›› 2018, Vol. 45 ›› Issue (6A): 487-492.

• 大数据与数据挖掘 • 上一篇    下一篇

基于自适应随机梯度下降方法的非平衡数据分类

陶秉墨1,鲁淑霞1,2   

  1. 河北大学数学与信息科学学院 河北 保定0710021
    河北省机器学习与计算智能重点实验室 河北 保定 0710022
  • 出版日期:2018-06-20 发布日期:2018-08-03
  • 作者简介:陶秉墨(1991-),男,硕士生,主要研究方向为机器学习,E-mail:281035569@qq.com;鲁淑霞(1966-),女,博士,教授,CCF会员,主要研究方向为机器学习,E-mail:cmclusx@126.com(通信作者)。
  • 基金资助:
    河北省自然科学基金(F2015201185)资助

Adaptive Stochastic Gradient Descent for Imbalanced Data Classification

TAO Bing-mo1,LU Shu-xia1,2   

  1. College of Mathematics and Information Science,Hebei University,Baoding,Hebei 071002,China1
    Hebei Province Key Laboratory of Machine Learning and Computational Intelligence,Baoding,Hebei 071002,China2
  • Online:2018-06-20 Published:2018-08-03

摘要: 对于不平衡数据分类问题,传统的随机梯度下降方法在求解一般的支持向量机问题时会产生一定的偏差,导致效果较差。自适应随机梯度下降算法定义了一个分布p,在选择样例进行迭代更新时,其依据分布p而非依据均匀分布来选择样例,并且在优化问题中使用光滑绞链损失函数。对于不平衡的训练集,依据均匀分布选择样例时,数据的不平衡比率越大,多数类中的样例被选择的次数就越多,从而导致结果偏向少数类。分布p在很大程度上解决了这个问题。普通的随机梯度下降算法没有明确的停机准则,这导致何时停机成为一个很重要的问题,尤其是在大型数据集上进行训练时。以训练集或训练集的子集中的分类准确率为标准来设定停机准则,如果参数设定恰当,算法几乎可以在迭代的早期就停止,这种现象在大中型数据集上表现得尤为突出。在一些不平衡数据集上的实验证明了所提算法的有效性。

关键词: 非均匀分布, 随机梯度下降, 损失函数, 停机准则, 支持向量机

Abstract: For imbalanced data classification,the performance of using traditional stochastic gradient descent for solving SVM problems is not very well.Adaptive stochastic gradient descent algorithm defines a distribution pinstead of using uniform distribution to choose examples,and the smoothing hinge loss function is used in the optimization problem.Because of the training sets are imbalanced,using uniform distribution will cause the algorithm choose more majority class based on the imbalanced ratio.That would result the classifier bias towards the minority class.The distribution p largely overcomes this issue.When to stop the programs becomes an important problem,because the normal stochastic gradient descent algorithm does not have a stop criterion especially for large data sets.The stop criterion was setted according to the classification accuracy on the training sets or its subsets.This stop criterion could stop the programs very early especially for large data sets if the parameters are chosen properly.Some experiments on imbalanced data sets show that the proposed algorithm is effective.

Key words: Loss function, Nonuniform distribution, Stochastic gradient descent, Stop criterion, Support vector machine

中图分类号: 

  • TP181
[1]CORTES C,VAPNIK V.Support-vector networks [J].Machine Learning,1995,20(3):273-297.
[2]PLATT J C.Sequential Minimal Optimization:A Fast Algo- rithm for Training Support Vector Machines [J].Technical Report,1998,208(1):212-223.
[3]WRIGHT S J.Coordinate Descent Algorithms [J].Mathematical Programming,2015,151(1):3-34.
[4]NESTEROV Y,STICH S U.Efficiency of the Accelerated Coordinate Descent Method on Structured Optimization Problems [J].Core Discussion Papers,2016,27(1):110-123.
[5]SHALEV-SHWARTZ S,ZHANG T.Accelerated Proximal Sto- chastic Dual Coordinate Ascent for regularized Loss Minimization [J].Mathematical Programming,2016,155(1/2):105-145.
[6]SHALEV-SHWATRZ S,ZHANG T.Stochastic Dual Coordi- nate Ascent Methods for Regularized Loss Minimization [J].Journal of Machine learning Research,2012,14(1):2013.
[7]CSIBA Q,ZHENG Q,RICHTARIK Q.Stochastic Dual Coordinate Ascent with Adaptive Probabilities [C]∥International Conference on Machine Learning.2015:674-683.
[8]WANG X,ZHANG W,YAN J,et al.On the Flexibility of Block Coordinate Descent for Large-Scale Optimization [J].Neurocomputing,2018,272(10):471-480.
[9]JOHNSON R,ZHANG T.Accelerating Stochastic Gradient Using Predictive Variance Reduction [C]∥International Confe-rence on Neural Information Processing Systems.2013:315-232.
[10]SHALEV-SHWARTZ S,SINGER Y.Primal Estimated Sub- gradient Solver for SVM [J].Mathematical Programming,2011,127(1):3-30.
[11]LIN C F,WANG S D.Fuzzy support vector machines [J].IEEE Trans.Neural Network,2002,13(2):464-471.
[12]FAN Q,WANG Z,LI D,et al.Entropy-based fuzzy support vector machine for imbalanced data-sets [J].Knowledge-Based Systems,2017,115(1):87-89.
[13]ZHANG T,ZHOU Z H.Large margin distribution machine [C]∥ Acm Sigkdd International Conference on Knowledge Discovery &Data Mining.2014:313-322.
[14]CHENG F,ZHANG J,WEN C,et al.Large Cost-Sensitive Margin Distribution Machine for Imbalanced Data Classification [J].Neruocomputing,2016,224(8):45-57.
[15]CHAWLA N V,BOWYER K W,HALL L O,et al.SMOTE:synthetic minority over-sampling technique [J].Journal of Artificial Intelligence Research,2002,16(1):321-357.
[16]GALAR M,BARRENECHEA E,HERRERA F.EUSBoost: Enhancing ensembles for highly imbalanced data-sets by evolutionary under-sampling [J].Pattern Recognition,2013,46(12):3460-3471.
[17]LIU X Y,WU J,ZHOU Z H.Exploratory Under-Sampling for Class-Imbalanced Learning [J].IEEE Transactions on Systems Man & Cybernetics Part B Cybernetics A publication of the IEEE Systems Man & Cybernetics Society,2009,39(2):539-550.
[18]KUBAT M,MATWIN S.Addressing the Curse of Imbalanced Training Sets:One-Sided Selection [C]∥International Confe-rence on Machine Learning.2012:179-186.
[1] 孟月波, 穆思蓉, 刘光辉, 徐胜军, 韩九强.
基于向量注意力机制GoogLeNet-GMP的行人重识别方法
Person Re-identification Method Based on GoogLeNet-GMP Based on Vector Attention Mechanism
计算机科学, 2022, 49(7): 142-147. https://doi.org/10.11896/jsjkx.210600198
[2] 单晓英, 任迎春.
基于改进麻雀搜索优化支持向量机的渔船捕捞方式识别
Fishing Type Identification of Marine Fishing Vessels Based on Support Vector Machine Optimized by Improved Sparrow Search Algorithm
计算机科学, 2022, 49(6A): 211-216. https://doi.org/10.11896/jsjkx.220300216
[3] 陈景年.
一种适于多分类问题的支持向量机加速方法
Acceleration of SVM for Multi-class Classification
计算机科学, 2022, 49(6A): 297-300. https://doi.org/10.11896/jsjkx.210400149
[4] 侯夏晔, 陈海燕, 张兵, 袁立罡, 贾亦真.
一种基于支持向量机的主动度量学习算法
Active Metric Learning Based on Support Vector Machines
计算机科学, 2022, 49(6A): 113-118. https://doi.org/10.11896/jsjkx.210500034
[5] 高荣华, 白强, 王荣, 吴华瑞, 孙想.
改进注意力机制的多叉树网络多作物早期病害识别方法
Multi-tree Network Multi-crop Early Disease Recognition Method Based on Improved Attention Mechanism
计算机科学, 2022, 49(6A): 363-369. https://doi.org/10.11896/jsjkx.210500044
[6] 邢云冰, 龙广玉, 胡春雨, 忽丽莎.
基于SVM的类别增量人体活动识别方法
Human Activity Recognition Method Based on Class Increment SVM
计算机科学, 2022, 49(5): 78-83. https://doi.org/10.11896/jsjkx.210400024
[7] 武玉坤, 李伟, 倪敏雅, 许志骋.
单类支持向量机融合深度自编码器的异常检测模型
Anomaly Detection Model Based on One-class Support Vector Machine Fused Deep Auto-encoder
计算机科学, 2022, 49(3): 144-151. https://doi.org/10.11896/jsjkx.210100142
[8] 张晓宇, 王彬, 安卫超, 阎婷, 相洁.
基于融合损失函数的3D U-Net++脑胶质瘤分割网络
Glioma Segmentation Network Based on 3D U-Net++ with Fusion Loss Function
计算机科学, 2021, 48(9): 187-193. https://doi.org/10.11896/jsjkx.200800099
[9] 黄颖琦, 陈红梅.
基于代价敏感卷积神经网络的非平衡问题混合方法
Cost-sensitive Convolutional Neural Network Based Hybrid Method for Imbalanced Data Classification
计算机科学, 2021, 48(9): 77-85. https://doi.org/10.11896/jsjkx.200900013
[10] 侯春萍, 赵春月, 王致芃.
基于自反馈最优子类挖掘的视频异常检测算法
Video Abnormal Event Detection Algorithm Based on Self-feedback Optimal Subclass Mining
计算机科学, 2021, 48(7): 199-205. https://doi.org/10.11896/jsjkx.200800146
[11] 冯姣, 陆昶谕.
基于残差注意力网络的跨媒体检索方法
Cross Media Retrieval Method Based on Residual Attention Network
计算机科学, 2021, 48(6A): 122-126. https://doi.org/10.11896/jsjkx.201100026
[12] 郭福民, 张华, 胡瑢华, 宋岩.
一种基于表面肌电信号的腕部肌力估计方法研究
Study on Method for Estimating Wrist Muscle Force Based on Surface EMG Signals
计算机科学, 2021, 48(6A): 317-320. https://doi.org/10.11896/jsjkx.200600021
[13] 卓雅倩, 欧博.
噪声环境下的人脸防伪识别算法研究
Face Anti-spoofing Algorithm for Noisy Environment
计算机科学, 2021, 48(6A): 443-447. https://doi.org/10.11896/jsjkx.200900207
[14] 段菲, 王慧敏, 张超.
面向数据表示的Cauchy非负矩阵分解
Cauchy Non-negative Matrix Factorization for Data Representation
计算机科学, 2021, 48(6): 96-102. https://doi.org/10.11896/jsjkx.200700195
[15] 雷剑梅, 曾令秋, 牟洁, 陈立东, 王淙, 柴勇.
基于整车EMC标准测试和机器学习的反向诊断方法
Reverse Diagnostic Method Based on Vehicle EMC Standard Test and Machine Learning
计算机科学, 2021, 48(6): 190-195. https://doi.org/10.11896/jsjkx.200700204
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!