计算机科学 ›› 2016, Vol. 43 ›› Issue (6): 68-71.doi: 10.11896/j.issn.1002-137X.2016.06.014

• 目次 • 上一篇    下一篇

基于融合数据库的海量传感器信息存储架构

类兴邦,房俊   

  1. 山东科技大学信息科学与工程学院 青岛266590,北方工业大学云计算研究中心 北京100144;大规模流数据集成与分析技术北京市重点实验室 北京100144
  • 出版日期:2018-12-01 发布日期:2018-12-01
  • 基金资助:
    本文受北京市属高等学校创新团队建设与教师职业发展计划基金资助

Mass Sensor Information Storage Infrastructure Based on Fusion Database

LEI Xing-bang and FANG Jun   

  • Online:2018-12-01 Published:2018-12-01

摘要: 在物联网、工业监控等系统中,庞大规模的传感器每时每刻都在产生大量的数据。实时数据库在处理高时效性数据方面具有较强的优势,但是在处理大规模传感器数据方面存在着存储量低、不便于扩展的弊端。而HBase在处理海量数据方面具有高读写性能、高扩展性、高可靠性和高存储量的优势。通过将实时数据库与HBase相结合,设计并实现了基于融合数据库的传感器信息存储架构。该架构采用多租户机制,对HBase写入进行了优化,将原来分散的传感器数据集中式存储,并把传感器元数据与历史数据分离存储,同时维持了实时数据库原有的查询、数据组织结构的特点。经过实验验证,该架构具有较高的读写性能以及良好的可扩展性,有效避免了Region写入热点,实现了集群负载均衡。

关键词: 实时数据库,HBase,传感器数据,集中式存储,写入优化

Abstract: In the internet of things,industrial control and other systems,large-scale sensor generates a large amount of data all the time.The Real-time database has an advantage in terms of time-sensive data processing,while it has a problem in storage capacity and scalability.On the contrary,the HBase has the advantage of high read and write perfor-mance,high scalability and high reliablity.Through the combination of real-time database and HBase,we designed and implemented the sensor information storage architecture based on fusion database.The architecture uses a multi-tenant mechanism to optimize HBase writes,the original sensor data are centrally stored,and the sensor metadata and historical data are stored separately,while maintaining the original real-time database queries,data structure characteristics.Our experiments verify that the system has high read and write performance and good scalability.And it effectively avoids the region write hot and achieves the objective of the cluster load balancing.

Key words: Real-time database,HBase,Sensor data,Centralized storage,Write-optimized

[1] Ding Zhi-ming,Gao Xu.A Database Cluster System Framework for Managing Massive Sensor Sampling Data in the Internet of Things[J].Chinese Journal of Computers,2012,5(6):1175-1191(in Chinese) 丁治明,高需.面向物联网海量传感器采样数据管理的数据库集群系统框架[J].计算机学报,2012,35(6):1175-1191
[2] Chen Qing-kui,Zhou Li-zhen.HBase-based storage system for large-scale data in wireless sensor network[J].Journal of Computer Application,2012,32(7):1920-1923,7(in Chinese) 陈庆奎,周利珍.基于HBase的大规模无线传感网络数据存储系统[J].计算机应用,2012,32(7):1920-1923,7
[3] Lu Ting,Fang Jun,Qiao Yan-ke.HBase-based Real-time Sto-rage System for Traffic Stream Data[J].Journal of Computer Application,2015,35(1):103-107,135(in Chinese) 陆婷,房俊,乔彦克.基于HBase的交通流数据实时存储系统[J].计算机应用,2015,35(1):103-107,135
[4] Lu Hui-ming,Zhou Zhao,Liao Chang-bin.Historical Data Processing Based On Real-time Database System [J].Electric PowerAutomation Equipment,2012,29(3):127-131(in Chinese) 陆会明,周钊,廖常斌.基于实时数据库系统的历史数据处理[J].电力自动化设备,2012,29(3):127-131
[5] George L.HBase:The Definitive Guide[M].2.Inc,USA:O’Reilly Media,2013:339-350
[6] Ku W Y,Chou T Y,Chung L K.The CloudBased Sensor Data Warehouse[C]∥International Symposium on Grids and Clouds and the Open Grid Forum.Taipei,Taiwan,2011:21-24
[7] Carstoiu D,Cernian A,Olteanu A.Hadoop Hbase-0.20.2 performance evaluation[C]∥2010 4th International Conference on New Trends in Information Science and Service Science (NISS).IEEE,2010:84-87
[8] The Apache Software Foundation.http://hadoop.apache.org
[9] Rabl T,Gómez-Villamor S,Sadoghi M,et al.Solving Big Data Challenges for Enterprise Application Performance Management[J].Proceedings of the Vldb Endowment,2012,5(12):1724-1735
[10] Kallman R,Kimura H,Natkins J,et al.H-store:a high-perfor-mance,distributed main memorytransaction processing system[J].PVLDB,2008,1(2):1496-1499
[11] Lakshman A,Malik P.Cassandra:a decentralized structured storage system[J].SIGOPS Operating Systems Review,2010,44(2):35-40
[12] http://planetcassandra.org/nosql-performance-benchmarks

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!