Computer Science ›› 2022, Vol. 49 ›› Issue (11): 65-75.doi: 10.11896/jsjkx.220200122

• Computer Software • Previous Articles     Next Articles

Web Application Page Element Recognition and Visual Script Generation Based on Machine Vision

LI Zi-dong, YAO Yi-fei, WANG Wei-wei, ZHAO Rui-lian   

  1. School of Information Science and Technology,Beijing University of Chemical Technology,Beijing 100029,China
  • Received:2022-02-21 Revised:2022-05-26 Online:2022-11-15 Published:2022-11-03
  • About author:LI Zi-dong,born in 1997,postgraduate,is a member of China Computer Federation.His main research interests include software testing and so on.
    WANG Wei-wei,born in 1990,Ph.D,lecturer,is a member of China Compu-ter Federation.Her main research in-terests include software testing,Web application testing and search-based test case generation.
  • Supported by:
    National Natural Science Foundation of China(62077003,61872026).

Abstract: In order to provide richer interactive response effect,the visualization elements of the Web application is becoming more complex and diverse.The traditional test based on DOM cannot match the new requirement to test Web application.A new generation test based on computer vision provides an efficient way for the complex elements in Web application.This test for the Web based on computer vision mainly depends on template matching technique to recognize the page elements,so that it can ge-nerate visualization test script.However,the subtle changes of the page elements’ appearance can lead to the failure of template matching technique,so that the Web page elements cannot be recognized and the visualization test script cannot be generated.How to improve the accuracy of Web page element recognition based on machine vision and make it still applicable in complex conditions is a challenging task.Object detection based on deep learning is a research hotspot in the field of today’s computer vision and machine learning.It has been shown from the deep data characteristics gained through the large sample learning that it can recognize the target accurately and has stronger generalization ability.Therefore,this paper targets the Web application,starts from the visual characteristics of the page elements,and proposes a Web page elements recognition method based on deep lear-ning.This method uses YOLOv3 algorithmic structure model based on the target testing to automatically localize the position and boundary of an element,recognize the type of Web page elements as well as describe its function.On the base of the Web page elements recognition,it can automatically generate visualization test script for the Web application.To verify the accuracy of the page elements recognition based on computer vision,experiments are set to test between different versions of the same Web application,and between different Web applications to analyze its accuracy.The results show that the average recall rate of machine vision-based Web page element recognition model is 75.6%.It provides foundation for the Web application’s visualization test script.

Key words: Web application testing, Web application page element recognition, Visual test script, Automatic generation of test script, Machine vision

CLC Number: 

  • TP391
[1]CHANG T H,YEH T,MILLER R C.GUI Testing Using Computer Vision[C]//Proceedings of the 28th International Conference on Human Factors in Computing Systems.2010:1535-1544.
[2]STOCCO A,YANDRAPALLY R,MESBAH A.Vista:Web test repair using computer vision[C]//Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Confe-rence and Symposium on the Foundations of Software Enginee-ring.2018:876-879.
[3]LOWE D G.Distinctive image features from scale-invariant keypoints[J].International Journal of Computer Vision,2004,60(2):91-110.
[4]REDMON J,DIVVALA S,GIRSHICK R,et al.You only look once:Unified,real-time object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:779-788.
[5]Gitee[CP/OL].[2021-12-10].https://gitee.com/explore.
[6]GIRSHICK R,DONAHUE J,DARRELL T,et al.Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2014:580-587.
[7]REDMON J,FARHADI A.Yolov3:An incremental improvement[J]. arXiv:1804.02767,2018.
[8]WANG H J,ZHOU B,PAN Y L.Ship Detection and Classification from High Resolution Optical Remote Sensing Images Based on YOLOv3 Neural Network[J].Bulletin of Science and Technology,2020,36(3):43-48,58.
[9]KOVALENKO A,KOVALENKO A,SMIRNOV A,et al.DOM XSS Testing Technology Vulnerabilities[J].Scientific and Practical Cyber Security Journal,2018,1(1):38-45.
[10]ALTHOMALI I,KAPFHAMMER G M,MCMINN P.Automated visual classification of DOM-based presentation failure reports for responsive web pages[J/OL].Software Testing,Verification and Reliability,2021,31(4).https://doi.org/10.1002/stvr.1756.
[11]LEOTTA M,STOCCO A,RICCA F,etal.Pesto:Automated migration of DOM-based Web tests towards the visual approach[J/OL].Software Testing,Verification And Reliability,2018,28(4).https://doi.org/10.1002/stvr.1665.
[12]WANG J,TANG J,YANG M,et al.Improving OCR-BasedImage Captioning by Incorporating Geometrical Relationship[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:1306-1315.
[13]Mrbs [CP/OL].[2021-12-10].https://github.com/meeting-room-booking-system/mrbs-code.
[14]OrangeHRM [CP/OL].[2021-12-10].https://github.com/orangehrm/orangehrm.
[15]MantisBT [CP/OL].[2021-12-10].https://github.com/man-tisbt/mantisbt.
[16]MyAdmin [CP/OL].[2021-12-10].https://github.com/cdfan/my-admin.
[17]VUE-Manage-System [CP/OL].[2021-12-10].https://github.com/lin-xin/vue-manage-system.
[18]GABS [CP/OL].[2021-12-10].https://github.com/manier13579/GABS-AddressBookSystem.
[19]RAMYA P,SINDHURA V,SAGAR P V.Testing using sele-nium web driver[C]//2017 Second International Conference on Electrical,Computer and Communication Technologies(ICECCT).IEEE,2017:1-7.
[20]XU T,ABBOTT R J,MILNE R I,et al.Phylogeography and allopatric divergence of cypress species(Cupressus L.) in the Qinghai-Tibetan Plateau and adjacent regions[J].BMC Evolutionary Biology,2010,10(1):1-10.
[21]KylinTOP [CP/OL].[2019-04-18].http://www.70testing.com/cloud/help/index.html.
[22]WTF-PageObject-Utility-Chrome-Extension[CP/OL].[2019-04-18].https://github.com/wiredrive/wtframework/wiki/WTF-PageObject-Utility-Chrome-Extension.
[23]AirtestProject [CP/OL].[2019-04-18].https://github.com/AirtestProject.
[24]YU S,FANG C,LI T,et al.Automated Mobile App Test Script Intent Generation via Image and Code Understanding[J].arXiv:2107.05165,2021.
[25]JANG Y,ZHAO J B.An Approach to Automatic Generation of Test Automation Scripts[J].Computer Technology and Deve-lopment,2007,17(7):4-7.
[26]LIU H G,ZHU L L,SONG L X,et al.A Method of Automatically Generating Test Script:CN104484269A[P].2015.
[1] WANG Dong, ZHOU Da-ke, HUANG You-da , YANG Xin. Multi-scale Multi-granularity Feature for Pedestrian Re-identification [J]. Computer Science, 2021, 48(7): 238-244.
[2] HAN Ke-kun, HU Gui-chuan, REN Jing, HE Hong-yu, LIU Jia-yin. Application of Image Processing in Feature Size Detection of Wind Turbine Blade’s Flange Face [J]. Computer Science, 2019, 46(6A): 562-565.
[3] HAN Shao-chao,XU Zun-yi,YIN Zhong-chuan,WANG Jun-xue. Research Review and Development for Automatic Reading Recognition Technology of Pointer Instruments [J]. Computer Science, 2018, 45(6A): 54-57.
[4] ZHANG Wen-yong, CHEN Le-zhu. Tableware Sorting System Based on LabVIEW Machine Vision [J]. Computer Science, 2018, 45(6A): 595-597.
[5] QIAN Ji-de, CHEN Bin, QIAN Ji-ye, ZHAO Heng-jun, CHEN Gang. Machine Vision Based Inspection Method of Mura Defect for LCD [J]. Computer Science, 2018, 45(6): 296-300.
[6] ZHU Hang-jiang, ZHU Fan, PAN Zhen-fu and ZHU Yong-li. Visual Object Tracking Method with Motion Estimation and Scale Estimation [J]. Computer Science, 2017, 44(Z11): 193-198.
[7] XUAN Han-yu, LIU Hong-zhe, YUAN Jia-zheng, LI Qing and NIU Xiao-ning. Robust Multi-lane Detection Algorithm [J]. Computer Science, 2017, 44(11): 305-313.
[8] XU Bang-zhen, TANG Yi-ping and CAI Guo-ning. Machine Vision-based Lightweight Driver Assistance System [J]. Computer Science, 2015, 42(Z11): 520-524.
[9] XU Zhen-chi, JI Lei, LIU Xiao-rong and ZHOU Xiao-jia. Recognition of Impurities Based on their Distinguishing Feature in Mushrooms [J]. Computer Science, 2015, 42(Z11): 203-205.
[10] GUO Qing-hui,LIANG Xiu-xia and ZHANG Rui. Calibration for Single Viewpoint Depth Image Based on Sparse Feature Points [J]. Computer Science, 2014, 41(Z6): 191-195.
[11] TANG Yi-ping,HUANG Lei-lei,YAN Hang-chen and MA Bao-qing. Research on Traffic Holographic State Detection Based on Machine Vision in Lightweight [J]. Computer Science, 2014, 41(5): 308-314.
[12] LIU Yong-po,WU Ji and LIU Shuang-mei. Research of Generic Codec for Web Application Testing [J]. Computer Science, 2013, 40(8): 157-160.
[13] WANG Zhi-liang,ZHANG Qiong,CHI Jian-nan,SHI Xue-fei. Camera Calibration Based on Projective Lines [J]. Computer Science, 2011, 38(8): 272-274.
[14] LU Xiao-li,DONG Yun-wei,ZHAO Hong-bin. Object-oriented Web Application Testing Model [J]. Computer Science, 2010, 37(7): 134-136.
[15] PENG Shu-shen,GU Qing,CHEN Dao-xu. Study of Test Case Generation for Web Applications [J]. Computer Science, 2010, 37(6): 159-163.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!