Computer Science ›› 2018, Vol. 45 ›› Issue (9): 81-88.doi: 10.11896/j.issn.1002-137X.2018.09.012

• NASAC 2017 • Previous Articles     Next Articles

Framework Assisting Storm Application Development Driven by Data Requirements

ZHOU Wen, SHI Xue-fei, WU Yi-jian, ZHAO Wen-yun   

  1. Software School,Fudan University,Shanghai 201203,China
    Shanghai Key Laboratory of Data Science,Fudan University,Shanghai 201203,China
  • Received:2017-10-05 Online:2018-09-20 Published:2018-10-10

Abstract: Storm,a widely used stream calculation framework,supports high efficient real-time calculation for stream data.In the development of Storm applications,developers have to write modules for various stream data requirements,causing repetitive work and difficulties in adapting to changes in data requirements.How to develop Storm applications and configure corresponding environment rapidly based on data requirements such as stream data format and calculations is an important research question for improving the efficiency of stream-oriented application development.An approach for describing stream data requirements was proposed in this paper.A framework assisting Storm application development was designed and implemented for business people to describe domain-specific data requirements and gene-rate Storm applications automatically.Experiments show that the framework is able to help non-developers configure and deploy common Storm-based stream calculation applications.The framework is adaptive to common requirements in real-time stream data calculations.

Key words: Stream calculation, Development framework, Data requirements, Storm

CLC Number: 

  • TP311.5
[1] SUN D W,ZHANG G Y,ZHENG W M.Stream Computing in Big Data Environment:Key Technologies and System Examples[J].Journal of Software,2014,25(4):839-862.(in Chinese)孙大为,张广艳,郑纬民.大数据流式计算:关键技术及系统实例[J].软件学报,2014,25(4):839-862.
[2] TOSHNIWAL A,TANEJA S,SHUKLA A,et al.Storm@twitter[C]∥Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data.New York:ACM,2014:147-156.
[3] NEUMEYER L,ROBBINS B,NAIR A,et al.S4:Distributed stream computing platform[C]∥The 10th IEEE International Conference on Data Mining Workshops.Washington:IEEE Computer Society,2010:170-177.
[4] KULKARNI S,BHAGAT N,FU M,et al.Twitter Heron:Stream Processing at Scale[C]∥Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data.New York:ACM,2015:239-250.
[5] AKIDAU T,BALIKOV A,BEKIROGLU K,et al.MillWheel:Fault-Tolerant Stream Processing at Internet Scale[J].Procee-dings of the Vldb Endowment,2013,6(11):1033-1044.
[6] QIAN Z,HE Y,SU C,et al.TimeStream:reliable stream computation in the cloud[C]∥Proceedings of the 8th ACM EuropeanConference on Computer Systems.New York:ACM,2013:1-14.
[7] ZAHARIA M,DAS T,LI H,et al.Discretized streams:fault-tolerant streaming computation at scale[C]∥ACM SIGOPS 24th Symposium on Operating Systems Principles.New York:ACM,2013:423-438.
[8] PAPAGEORGIOU A,POORMOHAMMADY E,CHENG B.Edge-Computing-Aware Deployment of Stream Processing Tasks Based on Topology-External Information:Model,Algorithms,and a Storm-Based Prototype[C]∥2016 IEEE International Congress on Big Data.Washington:IEEE,2016:259-266.
[9] ANIELLO L,BALDONI R,QUERZONI L.Adaptive onlinescheduling in storm[C]∥The 7th ACM International Confe-rence on Distributed Event-Based Systems.New York:ACM,2013:207-218.
[10] XIN Q,YAO X.Distributed QoS-Aware Scheduling in Cognitive Radio Cellular Networks[C]∥Proceedings of the 2015 International Conference on Network and Information Systems for Computers,Wuhan,China.2015:106-110.
[11] XIONG A P,WANG X W,ZOU Y.Scheduling Algorithm Based on Storm Topology Hot-edge[J].Computer Engineering,2017,43(1):37-42.
[12] LI T,TANG J,XU J.Performance Modeling and PredictiveScheduling for Distributed Stream Data Processing[J].IEEE Transactions on Big Data,2016:2(4):353-364.
[13] SANTURKAR S,ARORA A,CHANDRASEKARAN K.Stor-mgen-A Domain specific Language to create ad-hoc Storm Topologies[C]∥Proceedings of the 2014 Federated Conference on Computer Science and Information Systems.Washington:IEEE,2014:1621-1628.
[14] SUN C H.The Design and Implementation of Data AnalysisSystem Based on Storm[D].Beijing:Beijing University of Posts and Telecommunications,2014.(in Chinese)孙朝华.基于Storm的数据分析系统设计与实现[D].北京:北京邮电大学,2014.
[15] LONG S H.Research and Implementation of Real-time Big Data Analysis System Based on Storm[D].Shanghai:Shanghai JiaoTong University,2015.(in Chinese)龙少杭.基于Storm的实时大数据分析系统的研究与实现[D].上海:上海交通大学,2015.
[1] ZHANG Zhou, HUANG Guo-rui, JIN Pei-quan. Task Scheduling on Storm:Current Situations and Research Prospects [J]. Computer Science, 2019, 46(9): 28-35.
[2] YANG Li-peng, ZHANG Yang-sen, ZHANG Wen, WANG Jian, ZENG Jian-rong. Web Log Analysis Method Based on Storm Real-time Streaming Computing Framework [J]. Computer Science, 2019, 46(9): 176-183.
[3] LIU Jing-fa, LI Fan, JIANG Sheng-yi. Focused Annealing Crawler Algorithm for Rainstorm Disasters Based on Comprehensive Priority and Host Information [J]. Computer Science, 2019, 46(2): 215-222.
[4] WANG Jin-ming and WANG Yuan-fang. Parallel Mining of Densest Subgraph Based on Twitter Storm [J]. Computer Science, 2014, 41(1): 274-278.
[5] . [J]. Computer Science, 2006, 33(8): 236-239.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] . [J]. Computer Science, 2018, 1(1): 1 .
[2] LEI Li-hui and WANG Jing. Parallelization of LTL Model Checking Based on Possibility Measure[J]. Computer Science, 2018, 45(4): 71 -75, 88 .
[3] XIA Qing-xun and ZHUANG Yi. Remote Attestation Mechanism Based on Locality Principle[J]. Computer Science, 2018, 45(4): 148 -151, 162 .
[4] LI Bai-shen, LI Ling-zhi, SUN Yong and ZHU Yan-qin. Intranet Defense Algorithm Based on Pseudo Boosting Decision Tree[J]. Computer Science, 2018, 45(4): 157 -162 .
[5] WANG Huan, ZHANG Yun-feng and ZHANG Yan. Rapid Decision Method for Repairing Sequence Based on CFDs[J]. Computer Science, 2018, 45(3): 311 -316 .
[6] SUN Qi, JIN Yan, HE Kun and XU Ling-xuan. Hybrid Evolutionary Algorithm for Solving Mixed Capacitated General Routing Problem[J]. Computer Science, 2018, 45(4): 76 -82 .
[7] ZHANG Jia-nan and XIAO Ming-yu. Approximation Algorithm for Weighted Mixed Domination Problem[J]. Computer Science, 2018, 45(4): 83 -88 .
[8] WU Jian-hui, HUANG Zhong-xiang, LI Wu, WU Jian-hui, PENG Xin and ZHANG Sheng. Robustness Optimization of Sequence Decision in Urban Road Construction[J]. Computer Science, 2018, 45(4): 89 -93 .
[9] LIU Qin. Study on Data Quality Based on Constraint in Computer Forensics[J]. Computer Science, 2018, 45(4): 169 -172 .
[10] ZHONG Fei and YANG Bin. License Plate Detection Based on Principal Component Analysis Network[J]. Computer Science, 2018, 45(3): 268 -273 .