计算机科学

计算机科学 ›› 2026, Vol. 53 ›› Issue (6A): 250800062-11.doi: 10.11896/jsjkx.250800062

• 大数据&数据科学 • 上一篇    下一篇

Torlink:面向动态流速数据的高性能流式机器学习框架

梁哲恒1,3, 于然2,4, 崔磊1,3, 秦政2,4, 张金波1,3, 张子扬1,3, 吴铭钞2,4   

  1. 1 广东电网信息中心 广州 510000
    2 中国科学院软件研究所基础软件与系统重点实验室 北京 100190
    3 南方电网网络空间安全联合实验室 广州 510000
    4 中国科学院大学 北京 100049
  • 出版日期:2026-06-16 发布日期:2026-06-12
  • 通讯作者: 秦政(qinzheng19@otcaix.iscas.ac.cn)
  • 作者简介:(liangzheheng@qq.com)
  • 基金资助:
    广东电网有限责任公司项目(037800KC23090006);中国科学院软件研究所重大项目(ISCAS-ZD-202302)

Torlink:High-performance Streaming ML Framework for Dynamic Flow-rate Data

LIANG Zheheng1,3, YU Ran2,4, CUI Lei1,3, QIN Zheng2,4, ZHANG Jinbo1,3, ZHANG Ziyang1,3, WU Mingchao2,4   

  1. 1 Information Center,Guangdong Power Grid Limited Liability Company,Guangzhou 510000,China
    2 Key Laboratory of System Software,Chinese Academy of Sciences,Beijing 100190,China
    3 Joint Laboratory on Cyberspace Security,China Southern Power Grid,Guangzhou 510000,China
    4 University of Chinese Academy of Sciences,Beijing 100049,China
  • Published:2026-06-16 Online:2026-06-12
  • About author:LIANG Zheheng,born in 1986,postgraduate.His main research interests include big data processing,and so on.
    QIN Zheng,born in 1997,Ph.D.His main research interests include streamingprocessing and machine learning system.
  • Supported by:
    Guangdong Power Grid Limited Liability Company(037800KC23090006) and Major Project of ISCAS(ISCAS-ZD-202302).

PDF (PC)

54

摘要： 随着大数据时代的到来,流式机器学习理论和方法被广泛关注并应用。其核心在于能够实时处理连续到达的数据流,并迅速响应数据的动态变化。已有典型流式机器学习框架缺乏通用的流式学习算法支持,同时在面对动态流速数据时缺乏有效的性能优化机制。为了解决上述问题,首先对流式机器学习应用与计算特征进行分析与总结,设计出一个较为通用的流式机器学习数据流。针对现有框架,分析其潜在的性能瓶颈,进一步提出两种性能优化方法,即基于距离的动态采样机制和基于梯度的窗口预聚合机制。最后,基于Flink实现了原型系统Torlink,并在4个典型数据集上进行了实验与评价。结果表明,Torlink在4节点集群上,总体吞吐率约为现有框架的4.1倍,水平加速比可达3.3。

关键词: 流式机器学习, 流式数据, 流式处理, 性能优化

Abstract: With the advent of the big data era,streaming machine learning theories and methods have gained widespread attention and application.Their core lies in the ability to process continuously arriving data streams in real time and respond quickly to dynamic changes in data.The typical streaming machine learning frameworks lack support for general streaming learning algorithms and effective performance optimization mechanisms when handling dynamic data flow rates.To address these issues,this paper first analyzes and summarizes the application and computational characteristics of streaming machine learning,designing a relatively general streaming machine learning data flow.For existing frameworks,it analyzes their potential performance bottlenecks and proposes two performance optimization methods:a distance-based dynamic sampling mechanism and a gradient-based window pre-aggregation mechanism.Finally,a prototype system,Torlink,is implemented based on Flink,and experiments are conducted on four typical datasets.Results show that Torlink achieves an overall throughput approximately 4.1 times higher than existing frameworks on a 4-node cluster,with a horizontal speedup ratio of up to 3.3.

Key words: Streaming learning, Stream data, Streaming processing, Performance optimization

中图分类号: 

  • TP311
[1] SUN D W,ZHANG G Y,ZHENG W M.Big Data Stream Computing:Technologies and Instances [J].Journal of Software,2014,25(4):839-862.
[2] MONTIEL J,HALFORD M,MASTELINI S M,et al.River:machine learning for streaming data in python[J].Journal of Machine Learning Research,2021,22(110):1-8.
[3] scikit-multiflow[EB/OL].[2023-06-18] .https://scikit-multiflow.github.io/.
[4] Apache Flink Machine Learning Library[EB/OL].[2023-06-18] .https://nightlies.apache.org/flink/flink-ml-docs-release-2.2/.
[5] Alink[CP].Alibaba,2023.
[6] MORALES G D F,BIFET A.SAMOA:scalable advanced massive online analysis[J].Journal of Machine Learning Research,2015,16(1):149-153.
[7] MENG X,BRADLEY J,YAVUZ B,et al.Mllib:Machine lear-ning in apache spark[J].Journal of Machine Learning Research,2016,17(34):1-7.
[8] CARBONE P,KATSIFODIMOS A,EWEN S,et al.Apacheflink:Stream and batch processing in a single engine[J].The Bulletin of the Technical Committee on Data Engineering,2015,38(4).
[9] SCHLIMMER J,FISHERD.A Case Study of Incremental Concept Induction[C]//Proceedings of the Fifth National Confe-rence on Artificial Intelligence.1986:496-501.
[10] BORS A G,PITAS I.Introduction of the Radial Basis Function(RBF) Networks:Vol.1[M].Physica Verlag Rudolf Liebing KG,2001:1-7.
[11] POLIKAR R,UPDA L,UPDA S S,et al.Learn++:An incremental learning algorithm for supervised neural networks[J].IEEE Transactions on Systems,Man,and Cybernetics,Part C:Applications and Reviews,2001,31:497-508.
[12] TSCHEREPANOWM.TopoART:A Topology Learning Hierarchical ART Network[C]//Artificial Neural Networks－ICANN 2010.Berlin,Heidelberg:Springer,2010:157-167.
[13] LAMIREL J C,BOULILA Z,GHRIBI M,et al.A New Incremental Growing Neural Gas Algorithm Based on Clusters Labeling Maximization:Application to Clustering of Heterogeneous Textual Data[C]//GARCÍA-PEDRAJAS N,HERRERA F,FYFE C,et al.Trends in Applied Intelligent Systems.Berlin,Heidelberg:Springer,2010:139-148.
[14] AILON N,JAISWAL R,MONTELEONI C.Streaming k-means approximation[C]//International Conference on Neural Information Processing Systems.2009.
[15] DIEHL C P,CAUWENBERGHS G.Svm incremental learning,adaptation and optimization[C]//Proceedings of the International Joint Conference on Neural Networks.IEEE,2003:2685-2690.
[16] VAHIDI A,STEFANOPOULOU A,PENG H.Recursive least squares with forgetting for online estimation of vehicle mass and road grade:theory and experiments[J].Vehicle System Dynami-cs,2005,43(1):31-55.
[17] CHO Y,SAUL L.Kernel Methods for Deep Learning[C]//Advances in Neural Information Processing Systems.Curran Associates,Inc.,2009.
[18] SAHOO D,PHAM Q,LU J,et al.Online Deep Learning:Learning Deep Neural Networks on the Fly[C]//IJCAI-18.2017.
[19] LARA-BENÍTEZ P,CARRANZA-GARCÍA M,GARCÍA-GU-TIÉRREZ J,et al.Asynchronous dual-pipeline deep learning framework for online data stream classification[J].ICA,2020,27(2):101-119.
[20] WAHAB O A.Intrusion detection in the iot under data and concept drifts:Online deep learning approach[J].IEEE Internet of Things Journal,2022,9(20):19706-19716.
[21] MAO K,ZHU J,SU L,et al.FinalMLP:An Enhanced Two-Stream MLP Model for CTR Prediction[J].Proceedings of the AAAI Conference on Artificial Intelligence,2023,37(4):4552-4560.
[22] REN H,ANICIC D,RUNKLER T A.TinyOL:TinyML withOnline-Learning on Microcontrollers[C]//2021 International Joint Conference on Neural Networks(IJCNN).2021:1-8.
[23] ZHANG M J,HE Y L,LI X,et al.Distributed Two-stage Clustering Method Based on Node Sampling[J].Computer Science,2025,52(2):134-144.
[24] WU Z M,CAO J J,TANG Q.Online Parallel SDN Routing Optimization Algorithm Based on Deep Reinforcement Learning [J].Computer Science,2025,52(S1):240900018-9.
[25] RÖCKER S.sroecker/creme[CP/OL].(2023-03-16)[2023-06-18] .https://github.com/sroecker/creme.
[26] VowpalWabbit[EB/OL].[2023-06-18] .https://vowpalwabbit.org/.
[27] BIFET A,HOLMES G,PFAHRINGER B,et al.MOA:Massive Online Analysis,a Framework for Stream Classification and Clustering[C]//Proceedings of the First Workshop on Applications of Pattern Analysis.PMLR,2010:44-50.
[28] ZAHARIA M,CHOWDHURY M,DAS T,et al.Resilient dis-tributed datasets:A Fault-Tolerant abstraction for In-Memory cluster computing[C]//9th USENIX Symposium on Networked Systems Design and Implementation(NSDI 12).2012:15-28.
[29] Spark Streaming－Spark 3.5.0 Documentation[EB/OL].[2023-12-12] .https://spark.apache.org/docs/latest/streaming-programming-guide.html.
[30] RisingWave:Open-Source Streaming Database[EB/OL].[2023-12-12] .https://risingwave.com/.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市两江新区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发