计算机科学 ›› 2018, Vol. 45 ›› Issue (6A): 601-605.

• 综合、交叉与应用 • 上一篇    

基于SWIFT的海量数据存储平台设计

李朋远1,张志勇2   

  1. 北京跟踪与通信技术研究所 北京1000891
    太原卫星发射中心 太原0300002
  • 出版日期:2018-06-20 发布日期:2018-08-03
  • 作者简介:李朋远(1989-),男,硕士,主要研究方向为计算机系统总体设计、高性能计算,E-mail:pengyuan_li@163.com;张志勇(1988-),男,主要研究方向为计算机系统总体设计。

Design of Storage Platform for Large Scale Data Based on SWIFT System

LI Peng-yuan1,ZHANG Zhi-yong2   

  1. Beijing Institute of Tracking and Telecommunications Technology,Beijing 100089,China1
    Taiyuan Satellite Launch Center,Taiyuan 030000,China2
  • Online:2018-06-20 Published:2018-08-03

摘要: 随着我国航天事业的迅速发展,利用海量数据存储平台存储产生的海量科研数据变得越来越重要。文中提出了一种基于SWIFT分布式存储系统的云存储方案,并构建了存储平台的基础架构。存储平台中SWIFT系统的设计主要包含数据存储的哈希过程、Ring优化、Partition优化和Replica副本决策4部分,通过数据模拟的方式验证了关键设计的有效性。

关键词: Partition, Replica, Ring, SWIFT, 分布式存储

Abstract: With the rapid development of China’s space activities,storing the massive data based on the huge data stora-ge platform becomes increasingly important.This paper presented a cloud storage solution based on one distributed storage system which is named SWIFT,and built the infrastructure architecture for this storage platform.The design of SWIFT mainly includes four parts,the hash process of data storage,Ring,Partition,and Replica policy.And this paper verified the validity of the key design of SWIFT through the way of data simulation.

Key words: Distributed storage, Partition, Replica, Ring, SWIFT

中图分类号: 

  • TP393.07
[1]刘鹏.云计算——将计算变成水和电.中国计算机学会通讯,2009,5(10):49-54.
[2]WEN X,GU G,LI Q,et al.Comparison of open-source cloud management platforms:OpenStack and OpenNebula∥2012 9th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD).IEEE,2012:2457-2461.
[3]GUPTA R.Above the Clouds:A view of cloud computing[J].Eecs Department University of California Berkeley,2009,53(4):50-58.
[4]陆平,赵培,左奇.OpenStack系统架构设计实战[M].北京:机械工业出版社,2016.
[5]ARNOLD J.OpenStack Swift:Using,Administering,and Deve- loping for Swift Object Storage[M].O’Reilly Media,Inc.,2014.
[6]TOOR S,TOEBBICKE R,RESINES M Z,et al.Investigating an open source cloud storage infrastructure for CERN-specific data analysis∥2012 IEEE 7th International Conference on Networking,Architecture and Storage (NAS).IEEE,2012:84-88.
[7]李俊,李勇.联邦式异构数据库应用系统的集成框架和实现技术研究[J].计算机应用研究,2001,18(4):19-22.
[8]钟茂生.软件设计模式及其使用[J].计算机应用,2002,22(8):32-35.
[9]LEWIN D M.Consistent hashing and random trees:algorithms for caching in distributed networks[D].Massachusetts Institute of Technology,1998.
[10]武志学,赵阳,马超英.云存储系统——Swift的原理、架构及实践[M].北京:人民邮电出版社,2015.
[11]GODFREY B,LAKSHMINARAYANAN K,SURANA S,et al.Load balancing in dynamic structured P2P systems∥Twenty-third Annual Joint Conference of the IEEE Computer and Communications Societies(INFOCOM 2004).IEEE,2004:2253-2262.
[1] Ran WANG, Jiang-tian NIE, Yang ZHANG, Kun ZHU.
Clustering-based Demand Response for Intelligent Energy Management in 6G-enabled Smart Grids
Clustering-based Demand Response for Intelligent Energy Management in 6G-enabled Smart Grids
计算机科学, 2022, 49(6): 44-54. https://doi.org/10.11896/jsjkx.220400002
[2] 张航, 唐聃, 蔡红亮.
分布式存储系统中的预测式纠删码研究
Study on Predictive Erasure Codes in Distributed Storage System
计算机科学, 2021, 48(5): 130-139. https://doi.org/10.11896/jsjkx.200300124
[3] 张晓, 张思蒙, 石佳, 董聪, 李战怀.
Ceph分布式存储系统性能优化技术研究综述
Review on Performance Optimization of Ceph Distributed Storage System
计算机科学, 2021, 48(2): 1-12. https://doi.org/10.11896/jsjkx.201000149
[4] 钟凤艳, 王艳, 李念爽.
异构分布式存储系统再生码数据修复的节点选择方案
Node Selection Scheme for Data Repair in Heterogeneous Distributed Storage Systems
计算机科学, 2019, 46(8): 35-41. https://doi.org/10.11896/j.issn.1002-137X.2019.08.006
[5] 王俊生, 李丽丽, 颜拥, 赵微, 徐彧.
区块链技术应用的安全与监管问题
Security Incidents and Solutions of Blockchain Technology Application
计算机科学, 2018, 45(6A): 352-355.
[6] 张传国,王伶俐.
基于MEAN+SpringMVC的服务管理平台的设计与实现
Design and Implementation of Service Management Platform Based on MEAN and SpringMVC
计算机科学, 2017, 44(Z11): 538-541. https://doi.org/10.11896/j.issn.1002-137X.2017.11A.114
[7] 张栗粽,崔园,罗光春,陈爱国,卢国明,王晓雪.
面向大数据分布式存储的动态负载均衡算法
Dynamic Load Balance Algorithm for Big-data Distributed Storage
计算机科学, 2017, 44(5): 178-183. https://doi.org/10.11896/j.issn.1002-137X.2017.05.032
[8] 王青芸,程春玲.
基于位置信息的移动SNS数据动态划分复制算法
Mobile SNS Data Dynamic Partitioning and Replication Algorithm Based on Location Information
计算机科学, 2017, 44(3): 220-225. https://doi.org/10.11896/j.issn.1002-137X.2017.03.046
[9] 朱康林.
分布式虚拟化存储在公安专科类学校中的应用
Application of Distributed Virtualized Storage in Public Security College
计算机科学, 2016, 43(Z6): 571-576. https://doi.org/10.11896/j.issn.1002-137X.2016.6A.137
[10] 王静,罗威,欧阳明生,姜灿,王新梅.
基于简单再生码的分段编码方案
Segmentation Coding Scheme Based on Simple Regenerating Codes
计算机科学, 2016, 43(8): 148-153. https://doi.org/10.11896/j.issn.1002-137X.2016.08.031
[11] 董书暕,汪璟玢,陈远.
HMSST+:基于分布式内存数据库的HMSST算法优化
HMSST+:HMSST Algorithm Optimization Based on Distributed Memory Database
计算机科学, 2016, 43(3): 220-224. https://doi.org/10.11896/j.issn.1002-137X.2016.03.040
[12] 李 璋,杜慧敏,张丽果.
基于分布式存储的正则表达式匹配算法设计与实现
Fine-grained Parallel Multi-pattern Matching for Backbone Network NIDS
计算机科学, 2013, 40(3): 74-76.
[13] 陈达智,赵荣彩,姚远,韩林.
MPI自动并行化编译系统中消息传递代码生成算法
Message-passing Code Generation Algorithm in the MPI Automatic Parallelizing Compilation System
计算机科学, 2012, 39(6): 301-304.
[14] 龚燕,张浩.
基于子群极值和Sharing重分布的粒子群优化算法
Particle Swarm Optimization Algorithm Based on Extreme Value of Sub-swarm and Sharing Redistribution
计算机科学, 2010, 37(12): 224-226.
[15] 费定舟.
心灵即计算:哲学,逻辑和实践

计算机科学, 2009, 36(4): 56-59.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!