计算机科学 ›› 2014, Vol. 41 ›› Issue (Z11): 393-395.

• 软件工程与数据库技术 • 上一篇    下一篇

一种分布式大数据管理系统的设计与实现

陈海燕   

  1. 华东政法大学计算机科学与技术系 上海201620
  • 出版日期:2018-11-14 发布日期:2018-11-14
  • 基金资助:
    本文受国家社会科学基金项目(06BFX051),上海高校选拔培养优秀青年教师科研专项基金(hzf05046),华东政法大学校级科研项目(09HZK014)资助

Design and Realization of Distributed Big Data Management System

CHEN Hai-yan   

  • Online:2018-11-14 Published:2018-11-14

摘要: 随着云计算、物联网、移动互联网等技术的飞速发展,海量数据在这些崭新的领域迅猛地生长着,大数据作为一项颠覆性技术,为处理海量数据提供了无限可能。而传统的关系型数据库的不再适用,导致了分布式数据库NoSQL的应运而生。针对大数据领域面临的种种现实难题,设计并实现了一种基于Hadoop和NoSQL的新型分布式大数据管理系统(DBDMS),其提供大数据的实时采集、检索以及永久存储的功能。实验表明,DBDMS可以显著提高大数据处理能力,适用于海量日志备份和检索、海量网络报文抓取和分析等领域。

关键词: 大数据,分布式,数据存储,数据检索,Hadoop,NoSQL

Abstract: With the pretty rapid development of cloud computing,internet of things,mobile internet,and other technologies,mass data grows in those areas in violent speed.Big Data provides a possibility of handling mass data,which acts as a subversive technique.By the way,traditional relation database is no more effective of mass data that causes distributed database NoSQL to appear and evolve.Facing with actual and various difficulties,we designed and realized a new distributed big data management system (DBDMS),which is based on Hadoop and NoSQL techniques,and it provides big data real-time collection,search and permanent storage.Proved by some experiment,DBDMS can enhance the processing capacity of mass data,and very fits for mass log backup and retrieval,mass network packet grab and analysis,and other applied areas.

Key words: Big data,Distributed,Data storage,Data query,Hadoop,NoSQL

[1] Bari N,Mani G,Berkovich S.Internet of Things as a Methodological Concept[C]∥Fourth International Conference on Computing for Geospatial Research and Application.2013:48-50
[2] Dikaiakos M D,Pallis G,Katsaros D.Cloud Computing:Distri-buted Internet Computing for IT and Scientific Research[C]∥IEEE Internet Computing.2009:10-13
[3] Song Juan,Tang Shou-lian.Operator’s Mobile Internet Strategy in the process of Converged Network[C]∥2010 International Conference Management and Service Science (MASS).2010:1-4
[4] Wu Yu-lin,Gong Guang-hong.A Fully Distributed CollectionTechnology for Mass Simulation Data[C]∥2013 Fifth International Conference Computational and Information Sciences (ICCIS).2013:1679-1683
[5] Ringel D M,Skiera B.Understanding Competition using BigConsumer Search Data [C]∥2014 47th Hawaii International Conferences,System Sciences (HICSS).2014:3129-3138
[6] Membrey P,Chan K C C,Demchenko Y.A Disk Based Stream Oriented Approach For Storing Big Data[C]∥2013 International Conference Collaboration Technologies and Systems (CTS).2013:56-64
[7] Han J,Ishii M,Makino H.A Hadoop Performance Model forMulti-Rack Clusters[C]∥2013 5th International Conference Computer Science and Information Technology (CSIT).2013:265-274
[8] He Chen,Weitzel D,Swanson D,et al.HOG:Distributed Ha-doop MapReduce on the Grid[C]∥2012 SC Companion High Performance Computing,Networking,Storage and Analysis (SCC).2012:1276-1283
[9] von der Weth C,Datta A.Multiterm Keyword Search in NoSQL Systems[C]∥Digital Object Identifier.2012:34-42
[10] Kaur K,Rani R.Modeling and Querying Data in NoSQL Databases[C]∥2013 IEEE International Conference Big Data.2013:1-7

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!