计算机科学 ›› 2020, Vol. 47 ›› Issue (6): 85-91.doi: 10.11896/jsjkx.190500085

• 数据库&大数据&数据科学 • 上一篇    下一篇

基于增量日志的数据组合视图定位更新方法

张元鸣, 李梦妮, 黄浪游, 陆佳炜, 肖刚   

  1. 浙江工业大学计算机科学与技术学院 杭州310023
  • 收稿日期:2019-05-17 出版日期:2020-06-15 发布日期:2020-06-10
  • 通讯作者: 张元鸣(zym@zjut.edu.cn)
  • 基金资助:
    浙江省重大科技专项(2018C01064);浙江省自然科学基金项目(LY19F020034)

Data Composition View Positioning Update Approach with Incremental Logs

ZHANG Yuan-ming, LI Meng-ni, HUANG Lang-you, LU Jia-wei, XIAO Gang   

  1. College of Computer Science and Technology,Zhejiang University of Technology,Hangzhou 310023,China
  • Received:2019-05-17 Online:2020-06-15 Published:2020-06-10
  • About author:ZHANG Yuan-ming,born in 1977,Ph.D, associate professor.His main research interests include service computing,cloud computing and big data processing.
  • Supported by:
    This work was supported by the Major Science and Technology Projects of Zhejiang Province (2018C01064) and Natural Science Foundation of Zhejiang Province(LY19F020034)

摘要: 数据服务作为一种面向跨域异构数据源的统一数据模型,能够将数据资源以服务的形式进行发布,并根据用户的数据需求,通过组合若干个数据服务生成数据组合视图。然而,由于数据源是自治的,当数据发生变化时,如何以最小的代价实时更新数据组合视图是数据服务技术需要解决的关键问题。为此,提出一种基于增量日志的数据组合视图定位更新方法,先根据数据源日志的增量变化获取最新变更数据,然后通过定位属性计算组合视图中差异元组的索引号,并根据变更类型直接对组合视图中的差异元组执行数据更新操作,最终给出了基于日志的更新数据实时获取算法和数据组合视图定位更新算法。在跨域异构电梯数据服务系统中对本更新方法进行了评价,结果表明,当变更元组数量所占比例远小于元组总数或数据组合视图的属性个数较多时,定位更新方法的更新效率比现有方法高。

关键词: 定位更新, 实时性, 数据服务, 数据组合视图, 增量日志

Abstract: Data resources stored in different units and departments in cloud environment are cross-domain,heterogeneous and complex.As a unified data model for cross-origin and heterogeneous data sources,data service can publish data sources in the form of services,and generate data composition view by composing several data services according to users’ data requirements.Since the data sources are autonomous,it becomes a key issue to update data composition view in real time with minimal cost.This paper proposes a data composition view positioning update approach based on incremental logs.The latest data changes of data sources are captured according to incremental logs,and then the attributes and tuples in data composition view are indexed.The index numbers of different tuples can be calculated with positioning attributes.The corresponding tuple update operations can be performed according to data changes’ type.A log-based update data acquisition algorithm and a data composition view positioning update algorithm are presented.The proposed approach has been evaluated in a cross-origin heterogeneous elevator data service system by using datasets from multiple departments.When the proportion of the number of changed tuples is much smaller than the total number of tuples,the update efficiency of positioning update approach is much higher than existing methods.When the number of attributes of the data composition view is larger,the update efficiency of the positioning update approach is much higher than existing methods.

Key words: Data composition view, Data service, Incremental logs, Positioning update, Real time

中图分类号: 

  • TP311
[1]LEMOS A L,DANIEL F,BENATALLAH B.Web service composition[J].ACM Computing Surveys,2016,48(3):1-41.
[2]CAREY M J,ONOSE N,PETROPOULOS M.Data services [J].Communications of the ACM,2012,55(6):86.
[3]KURZADKAR S,BAJPAYEE A.Anatomization of miscellaneous approaches for selection and maintenance of Materialized view[C]//2015 IEEE 9th International Conference on Intelligent Systems and Control (ISCO).IEEE,2015:1-5.
[4]TERZO O,RUIU P,BUCCI E,et al.Data as a service (DaaS) for sharing and processing of large data collections in the cloud[C]//2013 Seventh International Conference on Complex,Intelligent,and Software Intensive Systems.IEEE,2013:475-480.
[5]ZHANG Y M,YE C L,HUANG L Y,et al.Research on data service dependency graph model and automatic composition[J].Journal of Chinese Computer Systems,2018,39(3):450-456.
[6]WEN Y,LIU C,HAN Y B.iViewer:service-based view construction method for just-in-time sharing business data across organizations[J].Journal of Frontiers of Computer Science & Technology,2012,6(3):221-236.
[7]GUPTA A,MUMICK I S.Materialized views[M].The MIT Press,1999.
[8]ZHANG P,HAN Y B,WANG G L.Implementing dynamic nested view update based on data service[J].Chinese Journal of Computers,2013,36(2):226-237.
[9]RAGOTHAMAN P,PANDA B.Analyzing transaction logs for effective damage assessment[M]//Research Directions in Data and Applications Security.Boston,MA:Springer US,2003:89-101.
[10]ZOU X X,JIA W J,PAN J H.Research of log-based change data capture[J].Journal of Chinese Computer Systems,2012,33(3):531-536.
[11]CAREY M,REVELIOTIS P,THATTE S,et al.Data service modeling in the AquaLogic data services platform[C]//2008 IEEE Congress on Services.IEEE,2008:78-80.
[12]ZHANG Y,ZHU L M,XU X W,et al.Data service API design for data analytics[M]//Services Computing-SCC 2018.Cham:Springer International Publishing,2018:87-102.
[13]ALTINEL M,BROWN P,CLINE S,et al.Damia:A Data Mashup Fabric for Intranet Applications[C]//Proceedings of the 33rd International Conference on Very Large Data Bases.Vienna:VLDB Endowment,2007:1370-1373.
[14]ZHANG P,WANG G L,JI G,et al.Optimization update for data composition view based on data service[J].Chinese Journal of Computers,2011,34(12):2344-2354.
[15]CAO B,BADIA A.SQL query optimization through nested relational algebra[J].ACM Transactions on Database Systems,2007,32(3):18.
[16]HAN Y B,WANG G L,JI G,et al.Situational data integration with data services and nested table[J].Service Oriented Computing and Applications,2013,7(2):129-150.
[17]AMDOUNI S,BARHAMGI M,BENSLIMANE D,et al.Handling uncertainty in data services composition[C]//2014 IEEE International Conference on Services Computing.IEEE,2014:653-660.
[18]MALKI A,BARHAMGI M,BENSLIMANE S M,et al.Composing data services with uncertain semantics[J].IEEETran-sactions on Knowledge and Data Engineering,2015,27(4):936-949.
[19]VU Q H,PHAM T V,TRUONG H L,et al.DEMODS:a description model for data-as-a-service[C]//2012 IEEE 26th International Conference on Advanced Information Networking and Applications.IEEE,2012:605-612.
[20]LIU X,HU C,LI Y,et al.The advanced data service architecture for modern enterprise information system[C]//2014 International Conference on Information Science & Applications (ICISA).IEEE,2014:1-4.
[21]ZHANG Z J,ZHANG Y M,LU J W,et al.CMfgIA:a cloud manufacturing application mode for industry alliance[J].The International Journal of Advanced Manufacturing Technology,2018,98(9/10/11/12):2967-2985.
[1] 吴培培, 吴兆贤, 唐文兵.
基于吸收态马尔可夫链的智能无人车系统实时性能分析
Real-time Performance Analysis of Intelligent Unmanned Vehicle System Based on Absorbing Markov Chain
计算机科学, 2021, 48(11A): 147-153. https://doi.org/10.11896/jsjkx.210300050
[2] 庞宇, 刘平, 雷印杰.
基于移动端的“非受控”物体识别算法的实现
Realization of “Uncontrolled” Object Recognition Algorithm Based on Mobile Terminal
计算机科学, 2019, 46(6A): 153-157.
[3] 曲佳博, 秦勃.
基于ST-CNN的交通标志实时检测识别算法
Real-time Detection and Recognition Algorithm of Traffic Signs Based on ST-CNN
计算机科学, 2019, 46(11A): 309-314.
[4] 刘彦,张琳.
位置大数据中一种基于Bloom Filter的匿名保护方法
Improved Location Anonymous Technology for Big Data Based on Bloom Filter
计算机科学, 2017, 44(6): 144-149. https://doi.org/10.11896/j.issn.1002-137X.2017.06.024
[5] 黄中平,白光伟,沈航,承骁,华志翔.
基于MapReduce模型的推测执行优化算法
Speculative Execution Optimization Algorithm with MapReduce
计算机科学, 2017, 44(4): 193-196. https://doi.org/10.11896/j.issn.1002-137X.2017.04.042
[6] 孙磊,杨海燕,吴际.
基于IMA平台的嵌入式软件设计模型仿真及实时性分析方法
Simulation and Real-time Analysis for Embedded Software Design Model with Consideration of Integrated Modular Avionics Platform
计算机科学, 2015, 42(12): 95-97.
[7] 郑远力,胡志坤.
基于滑动扫描框的高速物体的图像实时跟踪算法
Real-time Tracking Algorithm for Fast Target Based on Dynamical Scanning Boxes
计算机科学, 2015, 42(10): 287-291.
[8] 王桂玲,曹波,张赛,耿美珍,张峰.
一个用户主导的情景数据集成应用构造环境
User-steered Application Building Environment for Situational Data Integration
计算机科学, 2014, 41(9): 96-100. https://doi.org/10.11896/j.issn.1002-137X.2014.09.018
[9] 柏骏,夏靖波,吴吉祥,任高明,赵小欢.
实时网络流量分类研究综述
Survey on Real-time Traffic Classification
计算机科学, 2013, 40(9): 8-15.
[10] 余云霞,綦志勇.
基于混沌反馈控制理论的资源选择算法研究
Research on Resource Selection Algorithm Based on Feedback Control of Chaos Theory
计算机科学, 2012, 39(Z6): 452-456.
[11] 李 允,桂盛霖,陈 更,罗 蕾.
嵌入式实时软件模型开发环境研究
Model Development Environment Research of Embedded Real-time Software
计算机科学, 2012, 39(Z11): 226-229.
[12] 唐国明,周广新,谢羿,汤大权,唐九阳.
一种基于双层栅格划分的无线传感器网络目标定位方法
Target Localization Based on Double-level Grid Division in Wireless Sensor Networks
计算机科学, 2012, 39(6): 25-29.
[13] 周伟,安虹,刘谷,李小强,吴石磊.
一种输入感知的雷达回波快速聚类实现
Input-aware Runtime Scheduling Support for Fast Clustering of Radar Reflectivity Data on GPUs
计算机科学, 2012, 39(12): 295-299.
[14] 张慧超,朱红松,孙利民,黄晓伟.
DISP:物联网开放式数据服务系统
DISP:Distributed Information Sharing Platform for IOT
计算机科学, 2012, 39(12): 1-5.
[15] 范荣全,肖红,李琪林.
智能变电站GOOSE通信网实时性分析
Real-time Performance Analysis of GOOSE Communication Network in Smart Substation
计算机科学, 2011, 38(Z10): 444-446.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!