计算机科学 ›› 2022, Vol. 49 ›› Issue (11): 65-75.doi: 10.11896/jsjkx.220200122
李子东, 姚怡飞, 王微微, 赵瑞莲
LI Zi-dong, YAO Yi-fei, WANG Wei-wei, ZHAO Rui-lian
摘要： 为了给用户提供丰富的交互响应,Web应用的可视化元素越发复杂多样,传统基于DOM的测试已不能满足Web应用的测试新需求。新一代基于机器视觉的测试方法为Web应用复杂元素的测试提供了一种有效途径。目前,此类方法主要依赖于模版匹配技术识别Web页面元素,以生成可视化测试脚本对Web应用进行测试。然而,页面元素外观的细微变化可导致模版匹配技术失效,从而无法识别Web页面元素,更无法生成可视化测试脚本。因此,如何提高基于机器视觉的Web页面元素识别的准确性,使其在复杂条件中仍然适用是一项具有挑战性的工作。基于深度学习的目标检测是当前计算机视觉和机器学习领域的研究热点,可通过大样本学习得到深层的数据特征表示,以准确识别目标,其泛化能力相比模板匹配更强。针对Web应用,从页面元素的视觉特征出发,提出了一种基于深度学习的Web页面元素识别方法,即利用基于深度学习的目标检测算法YOLOv3构建Web页面元素识别模型,自动定位元素的位置和边界,识别Web页面元素类型及功能描述;在此基础上,自动为Web应用生成可视化测试脚本,提升Web应用的测试效率。为了验证基于机器视觉的Web页面元素识别的准确性,针对同一Web应用的不同版本及不同Web应用分别进行实验,结果表明,基于机器视觉的Web页面元素识别模型的平均召回率为75.6%,可有效辅助Web应用可视化测试脚本生成。
中图分类号:
| [1]CHANG T H,YEH T,MILLER R C.GUI Testing Using Computer Vision[C]//Proceedings of the 28th International Conference on Human Factors in Computing Systems.2010:1535-1544. [2]STOCCO A,YANDRAPALLY R,MESBAH A.Vista:Web test repair using computer vision[C]//Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Confe-rence and Symposium on the Foundations of Software Enginee-ring.2018:876-879. [3]LOWE D G.Distinctive image features from scale-invariant keypoints[J].International Journal of Computer Vision,2004,60(2):91-110. [4]REDMON J,DIVVALA S,GIRSHICK R,et al.You only look once:Unified,real-time object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:779-788. [5]Gitee[CP/OL].[2021-12-10].https://gitee.com/explore. [6]GIRSHICK R,DONAHUE J,DARRELL T,et al.Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2014:580-587. [7]REDMON J,FARHADI A.Yolov3:An incremental improvement[J]. arXiv:1804.02767,2018. [8]WANG H J,ZHOU B,PAN Y L.Ship Detection and Classification from High Resolution Optical Remote Sensing Images Based on YOLOv3 Neural Network[J].Bulletin of Science and Technology,2020,36(3):43-48,58. [9]KOVALENKO A,KOVALENKO A,SMIRNOV A,et al.DOM XSS Testing Technology Vulnerabilities[J].Scientific and Practical Cyber Security Journal,2018,1(1):38-45. [10]ALTHOMALI I,KAPFHAMMER G M,MCMINN P.Automated visual classification of DOM-based presentation failure reports for responsive web pages[J/OL].Software Testing,Verification and Reliability,2021,31(4).https://doi.org/10.1002/stvr.1756. [11]LEOTTA M,STOCCO A,RICCA F,etal.Pesto:Automated migration of DOM-based Web tests towards the visual approach[J/OL].Software Testing,Verification And Reliability,2018,28(4).https://doi.org/10.1002/stvr.1665. [12]WANG J,TANG J,YANG M,et al.Improving OCR-BasedImage Captioning by Incorporating Geometrical Relationship[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:1306-1315. [13]Mrbs [CP/OL].[2021-12-10].https://github.com/meeting-room-booking-system/mrbs-code. [14]OrangeHRM [CP/OL].[2021-12-10].https://github.com/orangehrm/orangehrm. [15]MantisBT [CP/OL].[2021-12-10].https://github.com/man-tisbt/mantisbt. [16]MyAdmin [CP/OL].[2021-12-10].https://github.com/cdfan/my-admin. [17]VUE-Manage-System [CP/OL].[2021-12-10].https://github.com/lin-xin/vue-manage-system. [18]GABS [CP/OL].[2021-12-10].https://github.com/manier13579/GABS-AddressBookSystem. [19]RAMYA P,SINDHURA V,SAGAR P V.Testing using sele-nium web driver[C]//2017 Second International Conference on Electrical,Computer and Communication Technologies(ICECCT).IEEE,2017:1-7. [20]XU T,ABBOTT R J,MILNE R I,et al.Phylogeography and allopatric divergence of cypress species(Cupressus L.) in the Qinghai-Tibetan Plateau and adjacent regions[J].BMC Evolutionary Biology,2010,10(1):1-10. [21]KylinTOP [CP/OL].[2019-04-18].http://www.70testing.com/cloud/help/index.html. [22]WTF-PageObject-Utility-Chrome-Extension[CP/OL].[2019-04-18].https://github.com/wiredrive/wtframework/wiki/WTF-PageObject-Utility-Chrome-Extension. [23]AirtestProject [CP/OL].[2019-04-18].https://github.com/AirtestProject. [24]YU S,FANG C,LI T,et al.Automated Mobile App Test Script Intent Generation via Image and Code Understanding[J].arXiv:2107.05165,2021. [25]JANG Y,ZHAO J B.An Approach to Automatic Generation of Test Automation Scripts[J].Computer Technology and Deve-lopment,2007,17(7):4-7. [26]LIU H G,ZHU L L,SONG L X,et al.A Method of Automatically Generating Test Script:CN104484269A[P].2015. |
| [1] | 王栋, 周大可, 黄有达, 杨欣. 基于多尺度多粒度特征的行人重识别 Multi-scale Multi-granularity Feature for Pedestrian Re-identification 计算机科学, 2021, 48(7): 238-244. https://doi.org/10.11896/jsjkx.200600043 |
| [2] | 韩克堃, 胡桂川, 任静, 何鸿宇, 刘佳音. 图像处理在风电叶片法兰端面特征尺寸检测中的应用 Application of Image Processing in Feature Size Detection of Wind Turbine Blade’s Flange Face 计算机科学, 2019, 46(6A): 562-565. |
| [3] | 李堃, 黎向锋. 基于日间行车的灯语识别技术 Lamp Language Recognition Technology Based on Daytime Driving 计算机科学, 2019, 46(11A): 277-282. |
| [4] | 韩绍超,徐遵义,尹中川,王俊雪. 指针式仪表自动读数识别技术的研究现状与发展 Research Review and Development for Automatic Reading Recognition Technology of Pointer Instruments 计算机科学, 2018, 45(6A): 54-57. |
| [5] | 张文勇, 陈乐柱. 基于LabVIEW机器视觉的餐具分拣系统 Tableware Sorting System Based on LabVIEW Machine Vision 计算机科学, 2018, 45(6A): 595-597. |
| [6] | 钱基德, 陈斌, 钱基业, 赵恒军, 陈刚. 基于机器视觉的液晶屏Mura缺陷检测方法 Machine Vision Based Inspection Method of Mura Defect for LCD 计算机科学, 2018, 45(6): 296-300. https://doi.org/10.11896/j.issn.1002-137X.2018.06.052 |
| [7] | 朱航江,朱帆,潘振福,朱永利. 运动状态与尺度估计的核相关目标跟踪方法 Visual Object Tracking Method with Motion Estimation and Scale Estimation 计算机科学, 2017, 44(Z11): 193-198. https://doi.org/10.11896/j.issn.1002-137X.2017.11A.040 |
| [8] | 宣寒宇,刘宏哲,袁家政,李青,牛小宁. 一种鲁棒性的多车道线检测算法 Robust Multi-lane Detection Algorithm 计算机科学, 2017, 44(11): 305-313. https://doi.org/10.11896/j.issn.1002-137X.2017.11.047 |
| [9] | 徐邦振,汤一平,蔡国宁. 基于机器视觉的轻量级驾驶辅助系统 Machine Vision-based Lightweight Driver Assistance System 计算机科学, 2015, 42(Z11): 520-524. |
| [10] | 徐振驰,纪磊,刘晓荣,周晓佳. 基于显著性特征的食用菌中杂质检测 Recognition of Impurities Based on their Distinguishing Feature in Mushrooms 计算机科学, 2015, 42(Z11): 203-205. |
| [11] | 郭庆慧,梁秀霞,张锐. 基于稀疏特征点的单视点深度图像校准 Calibration for Single Viewpoint Depth Image Based on Sparse Feature Points 计算机科学, 2014, 41(Z6): 191-195. |
| [12] | 汤一平,黄磊磊,严杭晨,马宝庆. 轻量级的全息道路交通状态视觉检测的研究 Research on Traffic Holographic State Detection Based on Machine Vision in Lightweight 计算机科学, 2014, 41(5): 308-314. https://doi.org/10.11896/j.issn.1002-137X.2014.05.066 |
| [13] | 柳永坡,吴际,刘霜梅. Web应用测试的通用编解码器研究 Research of Generic Codec for Web Application Testing 计算机科学, 2013, 40(8): 157-160. |
| [14] | 王志良，张琼，迟健男，史雪飞. 基于直线射影特征的摄像机参数标定方法 Camera Calibration Based on Projective Lines 计算机科学, 2011, 38(8): 272-274. |
| [15] | 王燕清，陈德运，石朝侠，刘泊，房国志. 基于一种新的目标识别的边缘爬行算法 Object Recognition Based on a New Method of Edge Crawling 计算机科学, 2010, 37(8): 266-269272. |
|
||
首页