计算机科学 ›› 2012, Vol. 39 ›› Issue (3): 174-182.

• 数据库与数据挖掘 • 上一篇    下一篇

基于分布式协调系统的并行频繁模式增长算法的优化

王洁,戴清濒,李环   

  1. (首都师范大学管理学院 北京100089);(中国科学院计算技术研究所 北京100190)
  • 出版日期:2018-11-16 发布日期:2018-11-16

Tuning of Parallel Frequent Pattern Growth Algorithm Based on Distributed Coordination System

WANG Jie,DAI Qing-hao,LI Huan   

  • Online:2018-11-16 Published:2018-11-16

摘要: 频繁模式挖掘可以发现数据中频繁出现的模式,是关联规则挖掘的重要步骤。并行频繁模式算法将其应用到并行环境中,以对海量数据进行挖掘。在Apachc软件基金会的Mahout项目实现的基础上,对计数和排序阶段以及算法的执行顺序提出了新的优化策略。优化后的设计将计数信息存储在分布式协调系统上,充分地利用了分布式协调系统的高可用性、适宜存储元数据信息的特点。该设计减小了小文件在分布式文件系统(HDFS)上的开销,同时保留了其优点,还能使计数过程和排序过程并行执行,减小了计算节点的内存开销。对比了文件系统I/O的开销,并分析了实现设计中的难点,为未来的工作打下了基础。

关键词: 频繁模式增长算法,并行数据挖掘,分布式协调系统,性能优化

Abstract: Frequent pattern mining can find frequent pattern in data, and iYs an important step in the association rules mining. Parallel frequent pattern(PFP) algorithms apply it into parallel environment, which is suitable for massive data.Based on the implementation of Apache Mahout, this paper proposed a design for optimizing the counting and sorting parts of PFP using distributed coordination system. This design takes advantage of distributed coordination system and reduces the consumption on HDFS and memory of data node. Another benefit is that the counting procedure and sorting procedure start parallclly. At last this paper analyzed the experimental result and the difficulties for implementation for further study.

Key words: Frequent pattern growth algorithm, Parallel data mining, Distributed coordination system, Performance tuning

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!