Computer Science ›› 2019, Vol. 46 ›› Issue (11): 145-155.doi: 10.11896/jsjkx.181102210

• Software & Database Technology • Previous Articles     Next Articles

Method of Microservice System Debugging Based on Log Visualization Analysis

LI Wen-hai, PENG Xin, DING DAN, XIANG Qi-lin, GUO Xiao-feng, ZHOU Xiang, ZHAO Wen-yun   

  1. (School of Computer Science,Fudan University,Shanghai 201203,China)
    (Shanghai Key Laboratory of Data Science,Fudan University,Shanghai 201203,China)
  • Received:2018-11-29 Online:2019-11-15 Published:2019-11-14

Abstract: In the era of cloud computing,more and more enterprises are adopting microservice architecture for software development or traditional monolithic application transformation.However,microservice system has high complexity and dynamism.When microservice system fails,there is currently no method or tool that can effectively support the location of the root cause of failure.To this end,the paper first proposed that all business log generated on all of the ser-vices by a single request can be associated by the trace information.And on this basis,this paper studied the method of microservice system debugging based on log visualization analysis.Firstly,the model of microservice log is defined.So the data information required for log visualization analysis can be specified.Then five kinds of visual debug strategies are summarized to support the location of four kinds of typical microservice fault’s root cause.The four kinds of microservice faults are ordinary fault with exceptions,logical fault with no exceptions,fault caused by unexpected service asynchronous invocation sequences and faults caused by service multi-instances.The strategies include single trace with log information,comparison of different traces,service asynchronous invocation analysis,service multi-instances analysis and trace segmentation.Among them,in order to realize service asynchronous invocation analysis and service multi-instances analysis,this paper designed two algorithms.At the same time,a prototype tool named LogVisualization was designed and implemented.LogVisualization can collect log information,trace data,nodes information and service instance information of the cluster,generated by the microservice system runtime.It can associate the business log with trace information by less code intrusion.And it supports users to use five strategies for visual debug.Finally,the prototype tool is applied to the actual micro-service system.Compared with the existing tools (Zipkin+ELK),the usefulness and effectiveness of prototype tool in the root location of four micro-service faults are verified.

Key words: Microservice, Trace, Log, Visualization, Fault, Debugging

CLC Number: 

  • TP311
[1] JAMES L,MARTIN F.“Microservices” [EB/OL].[2018-715].https://martinfowler.com/articles/microservices.html.
[2] ZIMMERMANN O.Microservices tenets[J].Computer Science-Research and Development,2017,32(3/4):301-310.
[3] FRANCESCO P D,MALAVOLTA I,LAGO P.Research onArchitecting Microservices:Trends,Focus,and Potential for Industrial Adoption[C]∥IEEE International Conference on Software Architecture.2017:21-30.
[4] HEORHIADI V,RAJAGOPALAN S,JAMJOOM H,et al.Gremlin:Systematic Resilience Testing of Microservices[C]∥IEEE International Conference on Distributed Computing Systems.2016:57-66.
[5] ZHENG H,LI D,LIANG B,et al.Automated Test Input Generation for Android:Towards Getting There in an Industrial Case[C]∥IEEE/ACM International Conference on Software Engineering:Software Engineering in Practice Track.2017:253-262.
[6] RUSLAN M.Microservices at Netflix Scale[EB/OL].[2018-715].https://gotocon.com/dl/goto-amsterdam-2016/slides/Rus-lanMeshenberg_MicroservicesAtNetflixScaleFirstPrinciplesTrade-offsLessonsLearned.pdf.
[7] JAMSHIDI P,PAHL C,MENDONCA N C,et al.Microservices:The Journey So Far and Challenges Ahead[J].IEEE Software,2018,35(3):24-35.
[8] ZHOU X,PENG X,XIE T,et al.Benchmarking Microservice Systems for Software Engineering Research[C]∥Proceedings of International Conference on Software Engineering:Companion Proceeedings.2018:323-324.
[9] DO N H,DO T V,XUAN T T,et al.A scalable routing mechanism for stateful microservices[C]∥Innovations in Clouds,Internet & Networks.2017:72-78.
[10] SHASHA M.Application Delivery Service Challenges in Microservices-based Applications[EB/OL].[2018-7-21].http://www.thefabricnet.com/application-delivery-service-challenges-in-microservices-based-applications/.
[11] ZHOU X,PENG X,XIE T,et al.Delta debugging microservice systems[C]∥Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering.2018:802-807.
[12] ZELLER A,HILDEBR R.Simplifying and Isolating Failure-Inducing Input[J].IEEE Transactions on Software Engineering,2002,28(2):183-200.
[13] ZIPKIN.Zipkin[EB/OL].[2018-7-20].https://zipkin.io/.
[14] JAEGER.Jaeger[EB/OL].[2018-7-20].https://www.jaegertracing.io/.
[15] ELK.ELK Stack[EB/OL].[2018-7-20].https://www.elastic.co/elk-stack.
[16] SHIVIZ.Shiviz[EB/OL].[2018-7-20].https://bestchai.bitbucket.io/shiviz/.
[17] PAHL C,JAMSHIDI P.Microservices:A Systematic MappingStudy[C]∥International Conference on Cloud Computing & Services Science.2016:137-146.
[18] HASSAN S,BAHSOON R.Microservices and Their DesignTrade-Offs:A Self-Adaptive Roadmap[C]∥IEEE International Conference on Services Computing.2016:813-818.
[19] CAMARGO A D,SALVADORI I,MELLO R D S,et al.An architecture to automate performance tests on microservices[C]∥International Conference on Information Integration & Web-based Applications & Services.2016:422-429.
[20] HEINRICH R,HOORN A V,KNOCHE H,et al.Performance Engineering for Microservices:Research Challenges and Directions[C]∥International Conference on Performance Enginee-ring Companion.2017:223-226.
[21] SCHERMANN G,SCHÖNI D,LEITNER P,et al.Bifrost:Supporting Continuous Deployment with Automated Enactment of Multi-Phase Live Testing Strategies[C]∥International Middleware Conference.2016:12.
[22] HASSELBRING W.Microservices for scalability:keynote talkabstract[C]∥Proceedings of the 7th ACM/SPEC on International Conference on Performance Engineering.2016:133-134.
[23] KLOCK S,VAN DER WERF J M E M,GUELEN J P,et al.Workload-Based Clustering of Coherent Feature Sets in Microservice Architectures[C]∥IEEE International Conference on Software Architecture.2017:11-20.
[24] LEITNERP,JURGEN C,EMANUEL S.Modelling and managing deployment costs of microservice-based cloud applications[C]∥International Conference on Utility & Cloud Computing.2016:165-174.
[25] LIN J,LIN L C,HUANG S.Migrating web applications toclouds with microservice architectures[C]∥International Conference on Applied System Innovation (ICASI).2016:1-4.
[26] 百度百科.可视化[EB/OL].[2018-7-25].https://baike.baidu.com/item/%E5%8F%AF%E8%A7%86%E5%8C%96.
[27] AntV.数据可视化概览[EB/OL].[2018-7-25].https://antv.alipay.com/zh-cn/vis/blog/vis-introduce.html.
[28] YANG H.The Research and Implementation of Visual Log Analysis System[D].Xi’an:Xidian University,2010.(in Chinese)杨华.可视化日志分析系统的研究与实现[D].西安:西安电子科技大学,2010.
[29] ZHANG S,ZHAO J.Research advances on network securitylogs visualization[J].Journal of Frontiers of Computer Science and Technology,2018(5):681-696.(in Chinese)张胜,赵珏.网络安全日志可视化分析研究进展[J].计算机科学与探索,2018(5):681-696.
[30] HU G.Research and Implementation of The Network Security Log Data Analysis System[D].Beijing:Beijing University of Posts and Telecommunications,2012.(in Chinese)胡钢.网络安全日志数据可视分析系统的研究与实现[D].北京:北京邮电大学,2012.
[31] DONG Z W.Design and Implementation of Visual Analysis and Monitoring System for National Domain Name Log[D].Beijing:University of Chinese Academy of Sciences(School of Enginee-ring Science),2014.(in Chinese)董再旺.国家域名日志可视化分析监控系统设计与实现[D].北京:中国科学院大学,2014.
[32] CHEN W W,WU K C.Research and application of massiveDNS log data analysis and visualization[J].Application Research of Computers,2016,33(2):335-338.(in Chinese)陈文文,吴开超.海量域名日志数据分析与可视化研究及应用[J].计算机应用研究,2016,33(2):335-338.
[33] ZHAO G.Visualization Analysis Research of Website Usability and User Behavior Based on Web Server Log[D].Taiyuan:Shanxi University,2007.(in Chinese)赵刚.基于Web日志的网站可用性及用户行为可视化分析方法研究[D].太原:山西大学,2007.
[34] LU D.Implementation and Design of OPAC Search Log Visualize System[J].Modern Computer,2016(12):67-70.(in Chinese)鲁丹.OPAC搜索日志可视化系统的设计与实现[J].现代计算机,2016(12):67-70.
[35] APACHE FLUME.Welcome to Apache Flume[EB/OL].[2018-7-26].https://flume.apache.org/.
[36] GITHUB.facebookarchive/scribe[EB/OL].[2018-7-26].https://github.com/facebookarchive/scribe/wiki.
[37] LIU K.Architecture analysis and application of massive data log system[J].Journal of Changchun University of Technology,2016,37(6):581-586.(in Chinese)刘锴.海量数据日志系统架构分析与应用[J].长春工业大学学报(自然科学版),2016,37(6):581-586.
[38] CHEN J J,LIU H H.Distributed ELK log analysis systembased on Kubernetes[J].Electronic Technology & Software Engineering,2016(15):211-212.(in Chinese)陈建娟,刘行行.基于Kubernetes的分布式ELK日志分析系统[J].电子技术与软件工程,2016(15):211-212.
[39] BESCHASTNIKH I,WANG P,BRUN Y,et al.Debugging Distributed Systems[J].Queue,2016,14(2):91-110.
[40] OPENTRACING.The OpenTracing Semantic Specification[EB/OL].[2018-7-26].https://opentracing.io/specification/.
[41] SIGELMAN B H,BARROSO L A,BURROWS M.Dapper,a Large-Scale Distributed Systems Tracing Infrastructure[EB/OL].[2018-7-26].https://storage.googleapis.com/pub-tools-public-publication-data/pdf/36356.pdf.
[42] JAEGER.Introduction[EB/OL].[2018-7-26].https://www.jaegertracing.io/docs/.
[43] SEMATEXT.Jaeger vs Zipkin-OpenTracing Distributed Tracers[EB/OL].[2018-7-28].https://sematext.com/blog/jaeger-vs-zipkin-opentracing-distributed-tracers/.
[44] INSTANA[EB/OL].[2018-7-28].https://www.instana.com/.
[45] LUXBURG U V.A tutorial on spectral clustering[J].Statistics &Computing,2007,17(4):395-416.
[46] TECHBEACON.3 reasons why you should always run microservices apps in containers[EB/OL].[2018-7-28].https://techbeacon.com/3-reasons-why-you-should-always-run-microserv-ices-apps-containers.
[47] BUOYANT.What’s a service mesh? And why do I need one?[EB/OL].[2018-7-28].https://blog.buoyant.io/2017/04/25/whats-a-service-mesh-and-why-do-i-need-one/.
[48] KUBERNETES[EB/OL].[2018-7-28].https://kubernetes.io/.
[49] ISTIO[EB/OL].[2018-7-28].https://istio.io/.
[50] SUGIYAMA K,TAGAWA S,TODA M.Methods for VisualUnderstanding of Hierarchical System Structures[J].IEEE Transactions on Systems,Man and Cybernetics,1981,11(2):109-125.
[51] 沪江网.以红色为主的色彩搭配,值得推荐[EB/OL].[2018-7-28].https://www.hujiang.com/fyuid_s/p886132/.
[52] 新浪博客.用户体验设计之圆角和直角[EB/OL].[2018-7-28].http://blog.sina.com.cn/s/blog_5d7170af0101dnpk.html.
[53] GITHUB.Dagrejs[EB/OL].[2018-7-29].https://github.com/dagrejs/dagre-d3.
[54] GITHUB.Rogen319/logvisualization_trainticket[EB/OL].[2018-7-28].https://github.com/Rogen319/logvisualization_trainticket.git.
[1] WANG Yang,LI Peng,JI Yi-mu,FAN Wei-bei,ZHANG Yu-jie,WANG Ru-chuan,CHEN Guo-liang. High Performance Computing and Astronomical Data:A Survey [J]. Computer Science, 2020, 47(1): 1-6.
[2] LI Yu,SHANG Zhi-yong,LIU Jing-sen. Improved Cuckoo Search Algorithm for Function Optimization Problems [J]. Computer Science, 2020, 47(1): 219-230.
[3] XU Fei-xiang,YE Xia,LI Lin-lin,CAO Jun-bo,WANG Xin. Comprehensive Calculation of Semantic Similarity of Ontology Concept Based on SA-BP Algorithm [J]. Computer Science, 2020, 47(1): 199-204.
[4] LI Juan,FANG Xian-wen,WANG Li-li,LIU Xiang-wei. Chaotic Activity Filter Method for Business Process Based on Log Automaton [J]. Computer Science, 2020, 47(1): 66-71.
[5] WANG Xin-sheng,MA Shu-zhang. Method of Weibo User Influence Calculation Integrating Users’ Own Factors and Interaction Behavior [J]. Computer Science, 2020, 47(1): 96-101.
[6] YANG Li-peng, ZHANG Yang-sen, ZHANG Wen, WANG Jian, ZENG Jian-rong. Web Log Analysis Method Based on Storm Real-time Streaming Computing Framework [J]. Computer Science, 2019, 46(9): 176-183.
[7] SUN Shu-ya, FANG Huan, FANG Xian-wen. Log-induced Morphological Fragments Process Clustering Method [J]. Computer Science, 2019, 46(8): 71-77.
[8] WANG Yi. Integral Fault Analysis on LED Cryptosystem in Digital Data Forensic [J]. Computer Science, 2019, 46(8): 189-193.
[9] ZHANG Yuan-yuan, QIN Ling. Efficient Access Control Scheme for Internet of Things Search Technology [J]. Computer Science, 2019, 46(8): 194-200.
[10] CHEN Jing, SHU Qiang, XIE Hao-fei. Priority Ranking Method of Test Cases Based on Fault Location [J]. Computer Science, 2019, 46(8): 239-243.
[11] LIU Ting, XU Yang, CHEN Xiu-lan. Unit Clauses and Their Complementary Literals and Redundant Clauses in Propositional Logic [J]. Computer Science, 2019, 46(8): 255-259.
[12] XIAO Zhen-hua, LIANG Yi-wen, TAN Cheng-yu, ZHOU Wen. Fault Detection Method Based on Immune Homeostasis Mechanism [J]. Computer Science, 2019, 46(8): 337-341.
[13] PAN Wei-qiong, TU Juan-juan, GAN Zong-liang, LIU Feng. Low Light Images Enhancement Based on Retinex Adaptive Reflectance Estimation and LIPS Post-processing [J]. Computer Science, 2019, 46(8): 327-331.
[14] WANG Le-le,WANG Bin-qiang,LIU Jian-gang,ZHANG Jian-hui,MIAO Qi-guang. Study on Malicious Program Detection Based on Recurrent Neural Network [J]. Computer Science, 2019, 46(7): 86-90.
[15] WEN Xi-ming,FANG Liang-da,YU Quan,CHANG Liang,WANG Ju. Knowledge Forgetting in Multi-agent Modal Logic System KD45n [J]. Computer Science, 2019, 46(7): 195-205.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] SUN Qi, JIN Yan, HE Kun and XU Ling-xuan. Hybrid Evolutionary Algorithm for Solving Mixed Capacitated General Routing Problem[J]. Computer Science, 2018, 45(4): 76 -82 .
[2] SHI Wen-jun, WU Ji-gang and LUO Yu-chun. Fast and Efficient Scheduling Algorithms for Mobile Cloud Offloading[J]. Computer Science, 2018, 45(4): 94 -99, 116 .
[3] PANG Bo, JIN Qian-kun, HENIGULI·Wu Mai Er and QI Xing-bin. Routing Scheme Based on Network Slicing and ILP Model in SDN[J]. Computer Science, 2018, 45(4): 143 -147 .
[4] LI Bai-shen, LI Ling-zhi, SUN Yong and ZHU Yan-qin. Intranet Defense Algorithm Based on Pseudo Boosting Decision Tree[J]. Computer Science, 2018, 45(4): 157 -162 .
[5] TONG Ze-ping, LI Tao, LI Li-jie and REN Liang. Study on Collaborative Optimization of Supply Chain with Uncertain Demand and Capacity Constraint[J]. Computer Science, 2018, 45(4): 260 -265 .
[6] CAI Li, LIANG Yu, ZHU Yang-yong and HE Jing. History and Development Tendency of Data Quality[J]. Computer Science, 2018, 45(4): 1 -10 .
[7] CUI Jian-jing, LONG Jun, MIN Er-xue, YU Yang and YIN Jian-ping. Survey on Application of Homomorphic Encryption in Encrypted Machine Learning[J]. Computer Science, 2018, 45(4): 46 -52 .
[8] WANG Zheng-li, XIE Tian, HE Kun and JIN Yan. 0-1 Knapsack Variant with Time Scheduling[J]. Computer Science, 2018, 45(4): 53 -59 .
[9] ZHU Hong, LI Qian-mu and LI De-qiang. Facial Multi-landmarks Localization Based on Single Convolution Neural Network[J]. Computer Science, 2018, 45(4): 273 -277, 284 .
[10] HU Qing-cheng, ZHANG Yong, XING Chun-xiao. K-clique Heuristic Algorithm for Influence Maximization in Social Network[J]. Computer Science, 2018, 45(6): 32 -35 .