计算机科学 ›› 2013, Vol. 40 ›› Issue (3): 77-78.

• 2012多值逻辑专栏 • 上一篇    下一篇

支持多集群数据并行的On-demand文件传输算法

魏晓辉,周芹芹,李洪亮   

  1. (吉林大学计算机科学与技术学院 长春130012)
  • 出版日期:2018-11-16 发布日期:2018-11-16

On-demand File Transfer Algorithm Supporting Data-parallel over Multi-cluster

  • Online:2018-11-16 Published:2018-11-16

摘要: 数据密集型应用通常需要在广域网分布式共享计算环境中高效地传输海量数据。并行处理中,大量的数据需要在生成集群、存储集群、处理集群间进行传输。针对该传输问题提出了一个支持多集群数据并行传输的按需文件传输算法(On-demand File Transfer),该算法以批量传输请求的整体完成时间最小为目的,根据集群内部快速传输的特点,实现目的端并行,分散单个节点的传输负载;在传输路径上,采用多重路径和多跳路径分割方法实现并行传输。对于批量传输请求,依据每个请求的传输负载,全局按需分配带宽,以解决传输路径的带宽冲突,从而充分利用当前网络带宽,快速传输批量传输请求。

关键词: 多集群,目的端并行,按需,文件传输

Abstract: Data-intensive applications often require the efficient transfer of large amounts of data over wide-area distribirted computing environments. In parallel processing, such data transmissions usually happen between data generation clusters and the data storage or processing clusters from time to time. This paper presented an on-demand file transfer algorithm,which aims at minimizing the makespan of overall file transfer rectuests. Taking advantage of the high transfer performance within a single cluster, the algorithm performs simultaneous transfer of different pieces of files to multiple nodes of a cluster when one node of the cluster requests the files. It employs multi-hop path splitting and multi patking methods in[1]to improve the transfer performance of a single source-destination pair. When several requests compete for the physical links, the algorithm evenly allocates the bandwidth resources upon the transmission load of each request, to make full use of the network bandwidth.

Key words: Multi-clusters,Data intensive,On-demand schedule,File transfer

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!