计算机科学 ›› 2013, Vol. 40 ›› Issue (3): 100-103.

• 2012多值逻辑专栏 • 上一篇    下一篇

基于Hadoop的高性能海量数据处理平台研究

翟岩龙,罗 壮,杨 凯,徐晟晨   

  1. (北京理工大学计算机学院 北京100081);(北京仿真中心 北京100854)
  • 出版日期:2018-11-16 发布日期:2018-11-16

High Performance Massive Data Computing Framework Based on Hadoop Cluster

  • Online:2018-11-16 Published:2018-11-16

摘要: 海量数据高性能计算蕴藏着巨大的应用价值,但是目前云计算体系只具有海量数据处理能力,而不具有足够的高性能计算能力。将具有超强并行计算能力的CPU与云计算相融合,提出了基于CPU/GPU协同的异构高性能云计算体系结构。以开源Hadoop为基础,采用注释码的形式对MapReduce函数中需要并行的部分进行标记。通过 定制GPU类加载器,将被标记代码转换为CUDA代码并动态编译运行。该平台将GPU的计算能力融合到MapReduce框架中,可高效处理海量数据。

关键词: CPU/UPU协同计算,Hadoop,海量数据处理,高性能计算

Abstract: HPC of massive data presents tremendous value. However, cloud systems still lack HPC computing power.This study improved the HPC ability of cloud computing technology by adding GPU to the cloud system. The proposed platform is based on Hadoop MapReduce programming model, and it defines some OpenMP like directives to annotate MapReduce program. The annotated code will try to be executed in parallel. A GPUClassloader was designed to convert annotated java code regions to CUDA code. With JNI,generated CUDA code and run on the GPUs. The computing resups of GPUs can be transferred back to the map function, in the end, the map function finishes the rest computing. The platform can support the user to complete CPU, GPU collaborative large-scale data parallel processing programming conveniently.

Key words: CPU/GPU collaborative computing, Hadoop, Massive data processing, HPC

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!