基于零阶减小方差方法的鲁棒支持向量机

doi:10.11896/jsjkx.181001840

摘要/Abstract

摘要： 采用传统的支持向量机方法对含有噪声的数据进行分类时会产生较大的损失,使得分类超平面严重偏离最优超平面,从而导致分类性能较差。为了解决此问题,文中提出了一种鲁棒的支持向量机(Robust Support Vector Machine,RSVM )方法,该方法给出了一种正弦平方形式的损失函数,根据正弦函数的特点,即使对于噪声数据,其损失函数的值也会被限制在[0,1]区间,从而提高了支持向量机的抗噪性。另外,在求解支持向量机时,传统的随机梯度下降方法在每次迭代中利用单个样本梯度近似代替全梯度,这样必然会产生方差,而随着迭代次数的增加,方差也不断累积,从而严重影响算法的分类性能。为了减小方差的影响,引入零阶减小方差的随机梯度下降(Zeroth Order-Stochastic Variance Reduced Gradient,ZO-SVRG )算法。该算法使用坐标梯度估计方法近似代替梯度,通过在每轮迭代中引入梯度修正项来减小方差的影响;同时,采取加权平均的输出形式进行内外循环的输出,加快了优化问题的收敛速度。实验结果表明,提出的基于零阶减小方差方法的鲁棒支持向量机算法对噪声数据具有更好的鲁棒性,且有效降低了方差的影响;为了进一步提高算法的性能,对实验中主要参数λ,k对算法精度的影响进行了分析。对于线性和非线性两种情况,当其参数对(λ,k)分别满足(λ＝1,k=5)和(λ＝10,k=3)时,可以达到各自的最高精度。

关键词: 方差约简, 零阶优化, 损失函数, 噪声, 支持向量机

Abstract: Great losses will be produced when traditional SVM methods are used to deal with the classification problem with noisy data,which makes the classification hyperplane seriously deviates from the optimal hyperplane,resulting in poor classification performance.In order to solve this problem,this paper proposed a robust support vector machine (RSVM) and gave a loss function in sinusoidal square form.According to the characteristics of sinusoidal function,the value of loss function is limited to the range of [0,1],even for noise data,which improves the anti-noise ability of SVM.In addition,when the traditional stochastic gradient descent method is used to solve the SVM,a single sample gradient is used to approximately replace the full gradient in each iteration,which will inevitably produce variance.As the number of iterations increases,the variance also accumulates,which seriously affects the classification performance of the algorithm.In order to reduce the influence of variance,this paper introduced a zeroth order-stochastic variance reduced gradient (ZO-SVRG) algorithm.This algorithm uses coordinate gradient estimation method to replace gradient approximately,and reduces the influence of variance by introducing the gradient correction term in each iteration.Besides,in the output of the internal and external loop,the weighted average output form is adopted,and then the convergence speed of the optimization problem is accelerated.The experimental results show that the robust support vector machine based on zeroth-order variance reduction algorithm has better robustness to noise data and effectively reduces the influence of variance.In order to further improve the performance of the algorithm,the influence of the main parameters λ and k on the accuracy of algorithm were analyzed.For both linear and nonlinear cases,when its parameter pairs (λ,k) are satisfied (λ＝1,k=5) and (λ＝10,k=3),respectively,the highest accuracy of each can be achieved.

Key words: Loss function, Noise, Support vector machine, Variance reduction, Zeroth order optimization

中图分类号:

TP181

鲁淑霞, 蔡莲香, 张罗幻. 基于零阶减小方差方法的鲁棒支持向量机[J]. 计算机科学, 2019, 46(11): 193-201. https://doi.org/10.11896/jsjkx.181001840

LU Shu-xia, CAI Lian-xiang, ZHANG Luo-huan. Robust SVM Based on Zeroth Order Variance Reduction[J]. Computer Science, 2019, 46(11): 193-201. https://doi.org/10.11896/jsjkx.181001840

参考文献

[1]FRENAY B,VERLEYSEN M.Classification in the Presence of Label Noise:A Survey ∥IEEE Transactions on Neural Networks and Learning Systems.2014:845-869.
[2]LIU Y F,ZHANG H H.Support Vector Machines with Adaptive Lq Penalty .Computational Statistic & Data Analysis,2007,51(12):6380-6394.
[3]LIN C F,WANG S D.Fuzzy support vector machines .IEEE Transactions on Neural Networks,2002,13(2):464-471.
[4]WU Y,LIU Y.Adaptively Weighted Large Margin Classifiers.Journal of Computational and Graphical Statistics,2013,22(2):416-432.
[5]SUN S C,HUANG D.A Novel Robust Smooth Support Vector Machine.Applied Mechanics and Materials,2012,1603(297):1438-1441.
[6]HUANG X L,SHI L.Ramp Loss Linear Programming Support Vector Machine .Machine Learning,2014,15(1):2185-2211.
[7]XU G B,CAO Z,HU B G.Robust Support Vector Machines Based on the Rescaled Hinge loss function.Pattern Recognition,2017,63:139-148.
[8]SHALEV-SHWARTZ S,SINGER Y.Pegasos:Primal Estimated Sub-Gradient Solver for SVM.Mathematical Programming,2011,127(1):3-30.
[9]ZHANG J R.Accelerating Stochastic Gradient Descent usingPredictive Variance Reduction ∥Advances in Neural Information Processing Systems.2013:315-323.
[10]NESTEROV Y.Random gradient-free minimization of convexfunctions.Foundations of Computational Mathematics,2015,2(17):527-566.
[11]LIU S,KAILKHURA B.Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimization .arXiv:1805.10367v2.
[12]LIU L,CHENG M.Stochastic Zeroth-order Optimization viaVariance Reduction Method .arXiv:1805.11811v1.
[13]NEMIROVSKI A,JUDITSKY A.Robust Stochastic Approximation Approach to Stochastic Programming .SIAM Journal on Optimization :A Publication of the Society for Industrial and Applied Mathematics,2009,19(4):1574-1609.
[14]DUCHII J C,JORDAN M I. Optimal rates for zeroth-order convex optimization:The power of two function evaluations .IEEE Transactions on Information Theory,2015,61(5):2788-2806.
[15]GU B,HUO Z. Zeroth-Order Asynchronous Doubly Stochastic Algorithm with Variance Reduction .arXiv:1612.01425v1.
[16]CHENG F,ZHANG J. Large Cost-Sensitive Margin Distribution Machine for Imbalanced Data Classification .Neurocomputing,2017,224(8):45-57.
[17]LACOSTE-JULIEN S,SCHMIDT M.A Simpler Approach to Obtaining an O(1/t) Convergence Rate for Projected Stochastic sub-gradient Method .arXiv:1212.2002v2.

相关文章 15

[1]	周慧, 施皓晨, 屠要峰, 黄圣君. 基于主动采样的深度鲁棒神经网络学习 Robust Deep Neural Network Learning Based on Active Sampling 计算机科学, 2022, 49(7): 164-169. https://doi.org/10.11896/jsjkx.210600044
[2]	徐鸣珂, 张帆. Head Fusion:一种提高语音情绪识别的准确性和鲁棒性的方法 Head Fusion:A Method to Improve Accuracy and Robustness of Speech Emotion Recognition 计算机科学, 2022, 49(7): 132-141. https://doi.org/10.11896/jsjkx.210100085
[3]	孟月波, 穆思蓉, 刘光辉, 徐胜军, 韩九强. 基于向量注意力机制GoogLeNet-GMP的行人重识别方法 Person Re-identification Method Based on GoogLeNet-GMP Based on Vector Attention Mechanism 计算机科学, 2022, 49(7): 142-147. https://doi.org/10.11896/jsjkx.210600198
[4]	侯夏晔, 陈海燕, 张兵, 袁立罡, 贾亦真. 一种基于支持向量机的主动度量学习算法 Active Metric Learning Based on Support Vector Machines 计算机科学, 2022, 49(6A): 113-118. https://doi.org/10.11896/jsjkx.210500034
[5]	单晓英, 任迎春. 基于改进麻雀搜索优化支持向量机的渔船捕捞方式识别 Fishing Type Identification of Marine Fishing Vessels Based on Support Vector Machine Optimized by Improved Sparrow Search Algorithm 计算机科学, 2022, 49(6A): 211-216. https://doi.org/10.11896/jsjkx.220300216
[6]	陈景年. 一种适于多分类问题的支持向量机加速方法 Acceleration of SVM for Multi-class Classification 计算机科学, 2022, 49(6A): 297-300. https://doi.org/10.11896/jsjkx.210400149
[7]	高荣华, 白强, 王荣, 吴华瑞, 孙想. 改进注意力机制的多叉树网络多作物早期病害识别方法 Multi-tree Network Multi-crop Early Disease Recognition Method Based on Improved Attention Mechanism 计算机科学, 2022, 49(6A): 363-369. https://doi.org/10.11896/jsjkx.210500044
[8]	邢云冰, 龙广玉, 胡春雨, 忽丽莎. 基于SVM的类别增量人体活动识别方法 Human Activity Recognition Method Based on Class Increment SVM 计算机科学, 2022, 49(5): 78-83. https://doi.org/10.11896/jsjkx.210400024
[9]	唐超尘, 仇洪冰, 刘鑫, 唐清华. 非均匀白噪声条件下的相干MIMO雷达角度估计 Angle Estimation of Coherent MIMO Radar Under the Condition of Non-uniform Noise 计算机科学, 2022, 49(5): 262-265. https://doi.org/10.11896/jsjkx.210300162
[10]	赵亮, 张洁, 陈志奎. 基于双图正则化的自适应多模态鲁棒特征学习 Adaptive Multimodal Robust Feature Learning Based on Dual Graph-regularization 计算机科学, 2022, 49(4): 124-133. https://doi.org/10.11896/jsjkx.210300078
[11]	武玉坤, 李伟, 倪敏雅, 许志骋. 单类支持向量机融合深度自编码器的异常检测模型 Anomaly Detection Model Based on One-class Support Vector Machine Fused Deep Auto-encoder 计算机科学, 2022, 49(3): 144-151. https://doi.org/10.11896/jsjkx.210100142
[12]	郑建炜, 黄娟娟, 秦梦洁, 徐宏辉, 刘志. 基于非局部相似及加权截断核范数的高光谱图像去噪 Hyperspectral Image Denoising Based on Non-local Similarity and Weighted-truncated NuclearNorm 计算机科学, 2021, 48(9): 160-167. https://doi.org/10.11896/jsjkx.200600135
[13]	张晓宇, 王彬, 安卫超, 阎婷, 相洁. 基于融合损失函数的3D U-Net++脑胶质瘤分割网络 Glioma Segmentation Network Based on 3D U-Net＋+ with Fusion Loss Function 计算机科学, 2021, 48(9): 187-193. https://doi.org/10.11896/jsjkx.200800099
[14]	黄颖琦, 陈红梅. 基于代价敏感卷积神经网络的非平衡问题混合方法 Cost-sensitive Convolutional Neural Network Based Hybrid Method for Imbalanced Data Classification 计算机科学, 2021, 48(9): 77-85. https://doi.org/10.11896/jsjkx.200900013
[15]	陶星朋, 徐宏辉, 郑建炜, 陈婉君. 基于非凸低秩矩阵逼近和全变分正则化的高光谱图像去噪 Hyperspectral Image Denoising Based on Nonconvex Low Rank Matrix Approximation and TotalVariation Regularization 计算机科学, 2021, 48(8): 125-133. https://doi.org/10.11896/jsjkx.200400143

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed