Computer Science ›› 2018, Vol. 45 ›› Issue (9): 81-88.doi: 10.11896/j.issn.1002-137X.2018.09.012

• NASAC 2017 • Previous Articles     Next Articles

Framework Assisting Storm Application Development Driven by Data Requirements

ZHOU Wen, SHI Xue-fei, WU Yi-jian, ZHAO Wen-yun   

  1. Software School,Fudan University,Shanghai 201203,China
    Shanghai Key Laboratory of Data Science,Fudan University,Shanghai 201203,China
  • Received:2017-10-05 Online:2018-09-20 Published:2018-10-10

Abstract: Storm,a widely used stream calculation framework,supports high efficient real-time calculation for stream data.In the development of Storm applications,developers have to write modules for various stream data requirements,causing repetitive work and difficulties in adapting to changes in data requirements.How to develop Storm applications and configure corresponding environment rapidly based on data requirements such as stream data format and calculations is an important research question for improving the efficiency of stream-oriented application development.An approach for describing stream data requirements was proposed in this paper.A framework assisting Storm application development was designed and implemented for business people to describe domain-specific data requirements and gene-rate Storm applications automatically.Experiments show that the framework is able to help non-developers configure and deploy common Storm-based stream calculation applications.The framework is adaptive to common requirements in real-time stream data calculations.

Key words: Data requirements, Development framework, Storm, Stream calculation

CLC Number: 

  • TP311.5
[1]SUN D W,ZHANG G Y,ZHENG W M.Stream Computing in Big Data Environment:Key Technologies and System Examples[J].Journal of Software,2014,25(4):839-862.(in Chinese)
孙大为,张广艳,郑纬民.大数据流式计算:关键技术及系统实例[J].软件学报,2014,25(4):839-862.
[2]TOSHNIWAL A,TANEJA S,SHUKLA A,et al.Storm@twitter[C]∥Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data.New York:ACM,2014:147-156.
[3]NEUMEYER L,ROBBINS B,NAIR A,et al.S4:Distributed stream computing platform[C]∥The 10th IEEE International Conference on Data Mining Workshops.Washington:IEEE Computer Society,2010:170-177.
[4]KULKARNI S,BHAGAT N,FU M,et al.Twitter Heron:
Stream Processing at Scale[C]∥Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data.New York:ACM,2015:239-250.
[5]AKIDAU T,BALIKOV A,BEKIROGLU K,et al.MillWheel:Fault-Tolerant Stream Processing at Internet Scale[J].Procee-dings of the Vldb Endowment,2013,6(11):1033-1044.
[6]QIAN Z,HE Y,SU C,et al.TimeStream:reliable stream computation in the cloud[C]∥Proceedings of the 8th ACM EuropeanConference on Computer Systems.New York:ACM,2013:1-14.
[7]ZAHARIA M,DAS T,LI H,et al.Discretized streams:fault-tolerant streaming computation at scale[C]∥ACM SIGOPS 24th Symposium on Operating Systems Principles.New York:ACM,2013:423-438.
[8]PAPAGEORGIOU A,POORMOHAMMADY E,CHENG B.
Edge-Computing-Aware Deployment of Stream Processing Tasks Based on Topology-External Information:Model,Algorithms,and a Storm-Based Prototype[C]∥2016 IEEE International Congress on Big Data.Washington:IEEE,2016:259-266.
[9]ANIELLO L,BALDONI R,QUERZONI L.Adaptive online
scheduling in storm[C]∥The 7th ACM International Confe-rence on Distributed Event-Based Systems.New York:ACM,2013:207-218.
[10]XIN Q,YAO X.Distributed QoS-Aware Scheduling in Cognitive Radio Cellular Networks[C]∥Proceedings of the 2015 International Conference on Network and Information Systems for Computers,Wuhan,China.2015:106-110.
[11]XIONG A P,WANG X W,ZOU Y.Scheduling Algorithm Based on Storm Topology Hot-edge[J].Computer Engineering,2017,43(1):37-42.
[12]LI T,TANG J,XU J.Performance Modeling and Predictive
Scheduling for Distributed Stream Data Processing[J].IEEE Transactions on Big Data,2016:2(4):353-364.
[13]SANTURKAR S,ARORA A,CHANDRASEKARAN K.Stor-mgen-A Domain specific Language to create ad-hoc Storm Topologies[C]∥Proceedings of the 2014 Federated Conference on Computer Science and Information Systems.Washington:IEEE,2014:1621-1628.
[14]SUN C H.The Design and Implementation of Data Analysis
System Based on Storm[D].Beijing:Beijing University of Posts and Telecommunications,2014.(in Chinese)
孙朝华.基于Storm的数据分析系统设计与实现[D].北京:北京邮电大学,2014.
[15]LONG S H.Research and Implementation of Real-time Big Data Analysis System Based on Storm[D].Shanghai:Shanghai JiaoTong University,2015.(in Chinese)
龙少杭.基于Storm的实时大数据分析系统的研究与实现[D].上海:上海交通大学,2015.
[1] JIAN Cheng-feng, PING Jing, ZHANG Mei-yu. Edge Computing-oriented Storm Edge Node Scheduling Optimization Method [J]. Computer Science, 2020, 47(5): 277-283.
[2] ZHAO Xin, MA Zai-chao, LIU Ying-bo, DING Yu-ting, WEI Mu-heng. Incremental FFT Based on Apache Storm and Its Application [J]. Computer Science, 2020, 47(11A): 504-507.
[3] YANG Li-peng, ZHANG Yang-sen, ZHANG Wen, WANG Jian, ZENG Jian-rong. Web Log Analysis Method Based on Storm Real-time Streaming Computing Framework [J]. Computer Science, 2019, 46(9): 176-183.
[4] ZHANG Zhou, HUANG Guo-rui, JIN Pei-quan. Task Scheduling on Storm:Current Situations and Research Prospects [J]. Computer Science, 2019, 46(9): 28-35.
[5] LIU Jing-fa, LI Fan, JIANG Sheng-yi. Focused Annealing Crawler Algorithm for Rainstorm Disasters Based on Comprehensive Priority and Host Information [J]. Computer Science, 2019, 46(2): 215-222.
[6] LIANG Kui-kui. Implementation of ETL Scheme Based on Storm Platform [J]. Computer Science, 2019, 46(11A): 208-211.
[7] WANG Jin-ming and WANG Yuan-fang. Parallel Mining of Densest Subgraph Based on Twitter Storm [J]. Computer Science, 2014, 41(1): 274-278.
[8] . [J]. Computer Science, 2006, 33(8): 236-239.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!