计算机科学 ›› 2019, Vol. 46 ›› Issue (6A): 1-5.

• 综述研究 •    下一篇

分布式消息系统研究综述

吴璨1,2, 王小宁1, 肖海力1, 曹荣强1, 赵一宁1, 迟学斌1,2   

  1. 中国科学院计算机网络信息中心 北京1001901;
    中国科学院大学 北京1000492
  • 出版日期:2019-06-14 发布日期:2019-07-02
  • 通讯作者: 王小宁(1981-),女,博士,副研究员,主要研究方向为网格计算、云计算和分布式系统,E-mail:wxn@sccas.cn
  • 作者简介:吴 璨(1992-),女,博士生,主要研究方向为高性能计算、可视化与网格技术,E-mail:wucan@sccas.cn;肖海力(1978-),男,博士,研究员,主要研究方向为网格技术和分布式系统;曹荣强(1982-),男,博士,副研究员,主要研究方向为网格计算和云计算;赵一宁(1983-),男,博士,助理研究员,主要研究方向为分布式系统与大数据分析;迟学斌(1963-),男,博士,研究员,博士生导师,主要研究方向为高性能计算、可视化与网格技术。
  • 基金资助:
    本文受国家重点研发计划课题(2018YFB0204001),中国科学院青年创新促进会(2017216)资助。

Survey on Distributed Message System

WU Can1,2, WANG Xiao-ning1, XIAO Hai-li1, CAO Rong-qiang1, ZHAO Yi-ning1, CHI Xue-bin1,2   

  1. Computer Network Information Center,Chinese Academy of Sciences,Beijing 100190,China1;
    University of Chinese Academy of Sciences,Beijing 100049,China2
  • Online:2019-06-14 Published:2019-07-02

摘要: 随着大数据时代的到来,各类软硬件系统的高并发访问、海量数据处理等需求越来越多,系统的高可用、易伸缩、可扩展成为系统研发的首要目标,分布式系统应运而生,提供了满足高性能需求的解决方案。然而,系统分布式地部署在不同的计算机上,使得系统间的消息通信成为重要问题。文章综述了4种流行的开源分布式消息系统,对比分析了RabbitMQ,Kafka,ActiveMQ和RocketMQ的架构及性能,为科研人员和系统开发者选择分布式消息系统提供了参考意见。

关键词: ActiveMQ, Kafka, RabbitMQ, RocketMQ, 分布式消息系统

Abstract: With the advent of the Big Data Era,there have been increasing demands for the high concurrent access and mass data processing of all kinds of hardware and software systems.High availability,extensibility and scalability are the main driving forces for system development.Distributed systems emerge as new age comes,providing solutions for higher performance requirements.However,as distributed systems are deployed on different computers,message communication between the systems has become an important problem.This paper provided an overview of the research progress of four popular open source distributed message systems:RabbitMQ,Kafka,ActiveMQ and RocketMQ.The architecture and performance of them are compared and analyzed,providing information and references for researchers and developers when choosing distributed message systems as an option.

Key words: ActiveMQ, Distributed message system, Kafka, RabbitMQ, RocketMQ

中图分类号: 

  • TP311
[1]Distributed Systems Concepts and Design.George Coulouris [M].America:Addison-Wesley,2012.
[2]WOOD I.Distributed Message Transmission System and Me-thod:WO,EP1477034[P].2004.
[3]GE Y,LIANG X X,PAN Z,et al.MESSAGE PARSING IN A DISTRIBUTED STREAM PROCESSING SYSTEM:U.S.Patent Application 15/258,629[P].2018-3-8.
[4]CONSULTANT M P C T.Chapter 3.Point-to-Point(PTP) and Point-to-Multipoint(PMP) Wireless Systems and Antennas[M]∥Wireless Access Networks:Fixed Wireless Access and WLL Networks-Design and Operation.John Wiley & Sons,Ltd,2001:41-56.
[5]EUGSTER P T.The many faces of publish/subscribe[J].ACM Computing Surveys(CSUR),2003,35(2):114-131.
[6]GUPTA A,SAHIN O D,AGRAWAL D,et al.Meghdoot:Content-Based Publish/Subscribe over P2P Networks[M]∥Middleware 2004.Springer Berlin Heidelberg,2004:254-273.
[7]SURHONE L M,TIMPLEDON M T,MARSEKEN S F,et al.RabbitMQ[M].Betascript Publishing,2010.
[8]VIDELA A,WILLIAMS J J W.RabbitMQ in action:distributed messaging for everyone.Manning Publications Co.,2012.
[9]ROSTANSKI M,GROCHLA K,SEMAN A.Evaluation of highly available and fault-tolerant middleware clustered architectures using RabbitMQ[C]∥Computer Science and Information Systems.IEEE,2014:879-884.
[10]AYANOGLU E,AYTAS Y,NAHUM D.Mastering RabbitMQ[M].Packt Publishing,2016,1:76-77.
[11]TKRAMER J.Advanced message queuing protocol(AMQP).Belltown Media,2009.
[12]VIDELA A,VIDELA A.RabbitMQ in Action[M].Manning Publications,2012.
[13]GILLES.Kafka:toward a minor literature[M].University of Minnesota Press,1986.
[14]ESMAILI K S,ESMAILI K S.Kafka versus RabbitMQ:A comparative study of two industry reference publish/subscribe implementations:Industry Paper[C]∥ACM International Confe-rence on Distributed and Event-Based Systems.ACM,2017:227-238.
[15]PROULX T,HEINE S J.Connections from Kafka:exposure to meaning threats improves implicit learning of an artificial grammar[J].Psychological Science,2009,20(9):1125.
[16]COMPUTING C C,WANG M H,LI Y,et al.Redesigning Kafka Message Queue System:Toward a Decentralized Stateful Broker System[J].2017.
[17]GARG N.Apache Kafka[M].Packt Publishing Ltd,2013.
[18]HENJES R,SCHLOSSER D,MENTH M,et al.Throughput Performance of the ActiveMQ JMS Server[M]∥Kommunikation in VerteiltenSystemen(KiVS).DBLP,2007:113-124.
[19]IONESCU V M.The analysis of the performance of RabbitMQ and ActiveMQ[C]∥Roedunet International Conference-Networking in Education and Research.IEEE,2015:132-137.
[20]KLEIN A F,STEFANESCU M,SAIED A,et al.An experimental comparison of ActiveMQ and OpenMQ brokers in asynchronous cloud environment[C]∥Fifth International Confe-rence on Digital Information Processing and Communications.IEEE,2015:24-30.
[21]HENJES R,MENTH M,HIMMLER V.Impact of Complex Filters on the Message Throughput of the ActiveMQ JMS Ser-ver[M]∥Managing Traffic Performance in Converged Networks.Springer Berlin Heidelberg,2007:192-203.
[22]CHEN G,DU Y,QIN P,et al.Design and Realization of ActiveMQ Based High Performance Message Cluster[C]∥International Conference on Chemical Science and Engineering.2013.
[23]ZHANG Y,XU L X.Feature and Configuration Research of Ac-tiveMQ[J].Computer Programming Skills & Maintenance,2011,12:6-13.
[24]SURHONE L M,TENNOE M T,HENSSONOW S F,et al.OpenWire (binary protocol).Betascript Publishing,2010.
[25]SURHONE L M,TENNOE M T,HENSSONOW S F,et al.WS-Notification[M].Betascript Publishing,2010.
[26]MILLER J.A Feature Set for the Extensible Messaging and Presence Protocol (XMPP).China Educational Technology,2015,17:2025-2032.
[27]SNYDER B,BOSANAC D,DAVIES R.ActiveMQ in Action [M].Manning Publications Co,2011.
[28]LIAO J,ZHUANG X,FAN R,et al.Toward a General Distributed Messaging Framework for Online Transaction Processing Applications[J].IEEE Access,2017,PP(99):1.
[29]WANG X R,WANG X R,FENG J.From Alibaba to Apache:RocketMQ’s Past,Present,and Future[EB/OL].https://www.infoq.com/articles/alibaba-apache-rocketmq.
[30]MA Y,YAN R,SUN J,et al.A MQTT Protocol Message Push Server Based on RocketMQ[C]∥International Conference on Intelligent Computation Technology and Automation.2017:295-298.
[31]The Apache Software Foundation.RocketMQArchitecture[EB/OL].http://rocketmq.apache.org/docs/rmq-arc.
[32]HINTJENS P.ZeroMQ:messaging for many applications[M].O’Reilly Media,Inc.,2013.
[33]FENGPING P,JIANZHENG C.Distributed system based on ZeroMQ[J].Electronic Test,2012,7(7):24-29.
[34]HINTJENS P.ZeroMQ:云时代极速消息通信库.北京:电子工业出版社,2015.
[35]CARLSON J L.Redis in action[M].Manning Publications Co.,2013.
[36]CARLSON J L.Redis in Action.Manning Publications Co.,2012.
[1] 谢文康, 樊卫北, 张玉杰, 徐鹤, 李鹏.
ENLHS:一种基于抽样的Kafka自适应调优方法
ENLHS:Sampling Approach to Auto Tuning Kafka Configurations
计算机科学, 2020, 47(8): 119-126. https://doi.org/10.11896/jsjkx.200300010
[2] 高子妍, 王勇.
面向云服务的分布式消息系统负载均衡策略
Load Balancing Strategy of Distributed Messaging System for Cloud Services
计算机科学, 2020, 47(6A): 318-324. https://doi.org/10.11896/JsJkx.191100012
[3] 王绪亮, 聂铁铮, 唐欣然, 黄菊, 李迪, 闫铭森, 刘畅.
流式数据处理的动态自适应缓存策略研究
Study on Dynamic Adaptive Caching Strategy for Streaming Data Processing
计算机科学, 2020, 47(11): 122-127. https://doi.org/10.11896/jsjkx.190800093
[4] 易佳,薛晨,王树鹏.
分布式流数据加载和查询技术优化
Optimization on Distributed Stream Data Loading and Querying
计算机科学, 2017, 44(5): 172-177. https://doi.org/10.11896/j.issn.1002-137X.2017.05.031
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!