HBase架构中RPC客户端的通信性能优化

doi:10.11896/j.issn.1002-137X.2016.04.019

计算机科学 ›› 2016, Vol. 43 ›› Issue (4): 97-101.doi: 10.11896/j.issn.1002-137X.2016.04.019

HBase架构中RPC客户端的通信性能优化

胡波,谭良

四川师范大学计算机学院成都610101,四川师范大学计算机学院成都610101;中国科学院计算技术研究所北京100190

出版日期:2018-12-01 发布日期:2018-12-01
基金资助:
本文受国家自然科学基金(61373162),四川省科技厅科技支撑项目(2014GZ007),四川省可视化计算与虚拟现实项目(KJ201402)资助

Optimization of Communication Performance of RPC Client in HBase Architecture

HU Bo and TAN Liang

Online:2018-12-01 Published:2018-12-01

摘要/Abstract

摘要： HBase已成为大数据存储、分析和处理的关键部件,对其进行性能优化是当前产业界和学术界的一个研究热点。HBase架构包括多个子系统,子系统之间的通信采用远程过程调用(Remote Procedure Call,RPC)通信机制,但这些子系统的RPC客户端采用的是阻塞通信模式,这种模式在客户端数据请求密集的情况下会引起线程的阻塞,影响了子系统之间的通信效率,降低了HBase的性能。首先分析了HBaseRPC客户端与服务端的通信机制,然后提出了一种HBaseRPC客户端非阻塞的通信模型,并通过Java NIO技术实现。实验结果表明,该模型有效降低了阻塞模式对通信性能的影响,提高了HBaseRPC客户端的通信性能。

关键词: HBase,大数据,远程过程调用,非阻塞

Abstract: HBase has become a key component of big data storage,analysis and processing.Its performance optimization is a hot research topic in nowaday industry and academia.HBase architecture consists of multiple subsystems,among which communication is realized via remote procedure call (RPC) communication mechanism.But the RPC client of subsystem adopts the blocking communication patterns,which will cause threads to block when there are intensive client data requests,thus affecting communication efficiency among subsystems,and reducing HBase performance.We first ana-lyzed communication mechanism of HBase RPC client side and its server side.Then we proposed a nonblocking communication pattern of HBase RPC client side via Java NIO.Experiment results show that this communication pattern can minimize the effect of blocking patterns on communication performance,and improve communication performance of HBase RPC client.

Key words: HBase,Big data,RPC,Nonblocking

胡波,谭良. HBase架构中RPC客户端的通信性能优化[J]. 计算机科学, 2016, 43(4): 97-101. https://doi.org/10.11896/j.issn.1002-137X.2016.04.019

HU Bo and TAN Liang. Optimization of Communication Performance of RPC Client in HBase Architecture[J]. Computer Science, 2016, 43(4): 97-101. https://doi.org/10.11896/j.issn.1002-137X.2016.04.019

参考文献

[1] NoSQL Database[EB/OL].2014.8.http://nosql-database.org
[2] Apache HBase[EB/OL].2014.8.http://hbase.apache.org
[3] Sun Wei-qin.Java Network Programming[M].Beijing:Publi-shing House of Electronics Industry,2007:82-86(in Chinese) 孙卫琴.Java网络编程精解[M].北京:电子工业出版社,2007:82-86
[4] Chang F,Dean J,Ghemawat S,et al.Bigtable:A distributed sto-rage system for structured data[C]∥Proceedings of the Seventh Symposium on Operating System Desgin and Implementation.2006
[5] Huang Jian,Ouyang Xiang-yong,Jose J,et al.High-Performance Design of HBase with RDMA over InfiniBand[C]∥IEEE International Parallel and Distributed Processing Symposium.IEEE,2012:774-785
[6] Lu Xiao-yi,Islam N S,Wasi-ur-Rahman M,et al.High-Perfor-mance Design of Hadoop RPC with RDMA over InfiniBand[C]∥International Conference on Parallel Processing.IEEE,2013:641-650
[7] Bao Xian-qiang,Liu Ling,Xiao Nong,et al.HConfig:Resourceadaptive fast bulk loading in HBase[C]∥ International Confe-rence on Collaborative Computing:Networking,Applications and Worksharing.IEEE,2014:215-224
[8] Tian Sheng-li,Xu Xi-shan,Yang Shu-qiang,et al.Optimizationfor the Access Interface of MapReduce in HBase[C]∥Collections of Ninth Annual Academic Conference of China Institute of Communications.China Institute of Communications,2012(in Chinese) 田胜利,徐锡山,杨树强,等.针对HBase的MapReduce访问接口的优化[C]∥第九届中国通信学会学术年会论文集.中国通信学会,2012
[9] Luo Yan-xin.Research and Implementation on HBASE BasedColumn-Oriented Compression Algorithms[D].Guangzhou:South China University of Technology,2011(in Chinese) 罗燕新.基于HBASE的列存储压缩算法的研究与实现[D].广州 :华南理工大学,2011
[10] Cheng Peng-sen,An Jun-xiu.The key as dictionary compression method of inverted index table under the HBase database[J].Journal of Software,2013,8(5):1086-1093
[11] Kang Yi.The Design and Implementation of HBase Large Object Storage[D].Nanjing:Nanjing University,2013(in Chinese) 康毅.HBase大对象存储方案的设计与实现[D].南京:南京大学,2013
[12] Harter T,Borthankur D,Dong Si-ying,et al.Analysis of HDFS Under HBase:A Facebook Messages Case Study[C]∥Procee-dings of the 12th USENIX Conference on File and Storage Technologies.USENIX,2014:199-212
[13] A Summary of Application and Optimization of HBase in Tao-bao[EB/OL].2014.8.http://blog.nosqlfan.com/html/3694.html(in Chinese) HBase在淘宝的应用和优化小结[EB/OL].2014.8.http://blog.nosqlfan.com/html/3694.html
[14] Perfection and Innovation of HBase in Jingdong[EB/OL].2014.8.http://www.sootoo.com/content/455783.shtml(in Chinese) HBase在京东的完善与创新[EB/OL].2014.8.http://www.sootoo.com/content/455783.shtml
[15] Liu Shao-hui.HBase Used in Xiaomi[R].China Hadoop Sum-mit.2013(in Chinese) 刘绍辉.小米Hbase实践[R].中国Hadoop技术峰会,2013
[16] Geoge L.HBase权威指南[M].代志远,刘佳,蒋杰,译.北京:人民邮电出版社,2013:302-304
[17] Hbase Hmaster Architecture[EB/OL].2014-8.http://blog.zahoor.in/2012/08/hbase-hmaster-architecture/

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

HBase架构中RPC客户端的通信性能优化

Optimization of Communication Performance of RPC Client in HBase Architecture

PDF (PC)

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 0

Metrics

本文评价

推荐阅读 0