计算机科学 ›› 2022, Vol. 49 ›› Issue (11): 309-315.doi: 10.11896/jsjkx.211200006
王冬霞, 雷咏梅, 张泽宇
WANG Dong-xia, LEI Yong-mei, ZHANG Ze-yu
摘要: 分布式交替方向乘子法(Alternating Direction Method of Multipliers,ADMM)是求解大规模机器学习问题使用最广泛的方法之一。现有大多数分布式ADMM算法都基于完整的模型更新。随着系统规模及数据量的不断增长,节点间的通信开销逐渐成为限制分布式ADMM算法发展的瓶颈。为了减少节点间通信开销,提出了一种通信高效的通用一致性异步分布式ADMM算法(General Form Consensus Asynchronous Distributed ADMM,GFC-ADADMM ),该算法通过分析高维稀疏数据集的特性,节点间利用关联模型参数代替完整模型参数进行通信,并对模型参数进行过滤以进一步减少节点间传输负载。同时结合过时同步并行(Stale Synchronous Parallel,SSP)计算模型、allreude通信模型及混合编程模型的优势,利用异步allreduce框架并基于MPI/OpenMP混合编程模型实现GFC-ADADMM算法,提高算法计算与通信效率。文中利用GFC-ADADMM算法求解稀疏logistic回归问题,实验测试表明,与现有分布式ADMM算法相比,GFC-ADADMM算法可减少15%~63%的总运行时间,且算法收敛时可达到更高的准确率。
中图分类号:
[1]LI Y,WANG X,FANG W,et al.A Distributed ADMM Approach for Collaborative Regression Learning in Edge Computing[J].Computers,Materials and Continua,2019,58(2):493-508. [2]BALAMURUGAN P,POSINASETTY A,SHEVADE S.ADMM for Training Sparse Structural SVMs with Augmented l1 Regularizers[C]//Siam International Conference on Data Mining.2016. [3]DHAR S,YI C,RAMAKRISHNAN N,et al.ADMM BasedScalable Machine Learning on Spark[C]//2015 IEEE International Conference on Big Data.Santa Clara:IEEE,2015:1174-1182. [4]LI W,LIU Y,TIAN Z,et al.Communication-Censored Linea-rized ADMM for Decentralized Consensus Optimization[J].IEEE Transactions on Signal Processing,2020,6:18-34. [5]LIU Y,YUAN K,WU G,et al.Decentralized Dynamic ADMM with Quantized and Censored Communications[C]//53rd Asilomar Conference on Signals,Systems,and Computers.2019. [6]ISSAID C B,ELGABLI A,PARK J,et al.Communication Efficient Distributed Learning with Censored,Quantized,and Ge-neralized Group ADMM[J].arXiv:2009.06459,2020. [7]ZHANG R,KWOK J.Asynchronous distributed ADMM forconsensus optimization[C]//International Conference on Machine Learning.2014:1701-1709. [8]CTANG T H,HONG M,LIAO W C,et al.Asynchronous Distributed ADMM for Large-Scale Optimization—Part I:Algorithm and Convergence Analysis[J].IEEE Transactions on Signal Processing,2016,64(12):3118-3130. [9]BOYD S,PARIKH N,CHU E,et al.Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers[J].Foundations & Trends in Machine Learning,2010,3(1):1-122. [10]ZHU R,NIU D,LI Z.A Block-wise,Asynchronous and Distri-buted ADMM Algorithm for General Form Consensus Optimization[J].arXiv:1802.08882,2018. [11]JIN H,JESPERSEN D,MEHROTRA P,et al.High Perfor-mance Computing Using MPI and OpenMP on Multi-core Parallel Systems[J].Parallel Computing,2011,37(9):562-575. [12]CHORLEY M J,WALKER D W.Performance Analysis of aHybrid MPI/OpenMP Application on Multi-core Clusters[J].Journal of Computational Science,2010,1(3):168-174. [13]GENKIN A,MADIGAN L D.Large-scale Bayesian Logistic Regression for Text Categorization[J].Technometrics,2007,49(3):291-304. [14]ALGAMAL Z Y,LEE M H.A Two-stage Sparse Logistic Regression for Optimal Gene Selection in High-dimensional Microarray Data Classification[J].Advances in Data Analysis and Classification,2019,13(3):753-771. [15]SHEVADE S K,KEERTHI S S.A Simple and Efficient Algorithm for Gene Selection Using Sparse Logistic Regression[J].Bioinformatics,2003,19(17):2246-2253. [16]ELGABLI A,PARK J,BEDI A S,et al.GADMM:Fast andCommunication Efficient Framework for Distributed Machine Learning[J].Journal of Machine Learning Research,2020,21(76):1-39. [17]WANG S,LEI Y.Fast Communication Structure for Asyn-chronous Distributed ADMM under Unbalance Process Arrival Pattern[C]//27th International Conference on Artificial Neural Networks.2018:362-371. [18]XIE J,LEI Y.ADMMLIB:A Library of Communication-Effi-cient AD-ADMM for Distributed Machine Learning[C]//IFIP International Conference on Network and Parallel Computing.Cham:Springer,2019:322-326. [19]WANG D,LEI Y,XIE J,et al.HSAC-ALADMM:An Asynchronous Lazy ADMM Algorithm Based on Hierarchical Sparse Allreduce Communication[J].The Journal of Supercomputing.2021,77(8):8111-8134. [20]WANG D,LEI Y,ZHOU J.Hybrid MPI/OpenMP ParallelAsynchronous Distributed Alternating Direction Method of Multipliers[J].Computing,2021,103(12):2737-2762. [21]HSIA C Y,ZHU Y,LIN C J.A Study on Trust Region Update Rules in Newton Methods for Large-scale Linear Classification[C]//Proceedings of the Ninth Asian Conference on Machine Learning.2017:33-48. |
[1] | 李京泰, 王晓丹. 基于代价敏感激活函数XGBoost的不平衡数据分类方法 XGBoost for Imbalanced Data Based on Cost-sensitive Activation Function 计算机科学, 2022, 49(5): 135-143. https://doi.org/10.11896/jsjkx.210400064 |
[2] | 张晓风,张德平. 基于拟似然估计方法的软件失效预测模型 Software Failure Prediction Model Based on Quasi-likelihood Method 计算机科学, 2016, 43(Z11): 486-489. https://doi.org/10.11896/j.issn.1002-137X.2016.11A.109 |
[3] | . 基于Logistic回归的中文垃圾邮件过滤方法 计算机科学, 2008, 35(10): 197-199. |
|