基于录制回放的移动应用可访问性增强方法

doi:10.11896/jsjkx.230300164

摘要/Abstract

摘要： 移动应用的可访问性指在不受身体和认知障碍的影响下便捷地使用移动应用的能力,其对老年人与残疾人群体具有重要意义。缩短一个应用使用过程中的交互路径(减少操作步骤数)是增强移动应用可访问性的一种重要方式。录制回放技术基于所录制的脚本自动执行交互过程中的固定操作,实现交互操作的缩减。然而,现有的录制回放工具仍存在局限性,包括依赖ROOT权限或使用侵入式手段才能进行录制并实现脚本可迁移。另外,已有工具录制的脚本不支持参数化的操作。针对这些问题,提出了基于录制回放的移动应用可访问性增强方法。在该方法的录制过程中,以可访问性服务为媒介,避免申请ROOT权限或使用侵入式手段,设计路径索引算法保证脚本的可迁移性,设计脚本参数化算法记录参数化操作,从而生成具有终端迁移性和操作数据泛化性的应用执行脚本。基于该方法开发了录制回放原型工具RRA并构建了面向10个流行应用的50个常见执行脚本。使用这些脚本在同一设备上的回放成功率为80%,与对比方法SARA相当。对RRA成功回放的40个脚本中的5个脚本进行参数化录制,在同一设备上的回放成功率达100%。对两种方法均能录制成功的29个脚本以及5个参数化脚本进行迁移执行,RRA的回放成功率为94%,高于SARA。

关键词: 安卓, 移动应用, 可访问性, 录制回放, 可迁移

Abstract: Accessibility of mobile applications refers to the ability to use mobile applications conveniently without being affected by physical and cognitive impairments,which is of great significance to the elderly and disabled groups.Shortening the interaction path(reducing the number of steps) during the use of an application is an important way to enhance the accessibility of mobile applications.The recording and playback technology automatically executes fixed operations in the interactive process based on the recorded scripts to reduce the interactive operations.However,existing recording and playback tools still have limitations,including relying on ROOT permissions or using intrusive means to record and realize script migration.In addition,scripts recorded by existing tools do not support parameterized operations.In response to these problems,this paper proposes a mobile application accessibility enhancement method based on recording and playback.In the recording process of this method,the accessibility service is used as the medium to avoid applying for ROOT permission or using intrusive means.The path index algorithm is designed to ensure the portability of scripts,and the script parameterization algorithm is designed to record parameterized operations,so as to generate application execution scripts with terminal mobility and operation data generalization.Based on the proposed method,a prototype tool RRA for recording and playback is developed and 50 common execution scripts for 10 popular applications are constructed.The playback success rate on the same device using these scripts is 80%,which is comparable to the comparison method SARA.Parameterized recording of 5 scripts out of 40 scripts successfully played back by RRA with a 100% playback success rate on the same device.The 29 scripts and 5 parameterized scripts that can be successfully recorded by both methods are migrated and executed,and the playback success rate of RRA is 94%,which is higher than that of SARA.

Key words: Android, Mobile application, Accessibility, Recording and playback, Transferable

中图分类号:

TP311

李向民, 沈立炜, 董震. 基于录制回放的移动应用可访问性增强方法[J]. 计算机科学, 2023, 50(12): 32-48. https://doi.org/10.11896/jsjkx.230300164

LI Xiangmin, SHEN Liwei, DONG Zhen. Mobile Application Accessibility Enhancement Method Based on Recording and Playback[J]. Computer Science, 2023, 50(12): 32-48. https://doi.org/10.11896/jsjkx.230300164

参考文献

[1]IWARSSON S,STÅHL A.Accessibility,usability and universal design－positioning and definition of concepts describing person-environment relationships[J].Disability and rehabilitation,2003,25(2):57-66.
[2]BIGHAM J P,LIN I,SAVAGE S.The Effects of “Not Knowing What You Don't Know” on Web Accessibility for Blind Web Users[C]//Proceedings of the 19thInternational ACM SIGACCESS Conference on Computers and Accessibility(ASSETS).New York,NY:ACM,2017:101-109.
[3]ONEY S,LUNDGARD A,KROSNICK R,et al.Arboretum and arbility:Improving web accessibility through a shared browsing architecture[C]//Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology(UIST).New York,NY:ACM,2018:937-949.
[4]LAZAR J,ALLEN A,KLEINMAN J,et al.What frustratesscreen reader users on the web:A study of 100 blind users[J].International Journal of human-computer interaction,2007,22(3):247-269.
[5]YU J E,CHATTOPADHYAY D.“Maps are hard for me”:identifying how older adults struggle with mobile maps[C]//Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility(ASSETS).New York,NY:ACM,2020:1-8.
[6]ABOU-ZAHRA S,HENRY S L.Exploring web accessibility solutions in developing regions as innovations for the benefit of all[C]//Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility(W4A).New York,NY:ACM,2010:1-4.
[7]HENRY S L,ABOU-ZAHRA S,BREWER J.The role of accessibility in a universal web[C]//Proceedings of the 11th Web for all Conference(W4A).New York,NY:ACM,2014:1-4.
[8]W3C Web Accessibility Initiative(WAI).Introduction to web accessibility[EB/OL].(2022-03-31)[2022-11-25].https://www.w3.org/WAI/fundamentals/accessibility-intro/.
[9]Android Accessibility.Android accessibility overview[EB/OL].[2022-11-25].https://support.go ogle.com/accessibility/android/answer/6006564.
[10]Android Developers.Android Accessibility Developer Guidelines[EB/OL].(2022-10-24)[2022-11-25].https://developer.android.com/guide/topics/ui/accessibility.
[11]MILNE S,DICKINSON A,CARMICHAEl A,et al.Are guidelines enough? An introduction to designing Web sites accessible to older people[J].IBM Systems Journal,2005,44(3):557-571.
[12]MANKOFF J,FAIT H,TRAN T.Is your web page accessible? A comparative study of methods for assessing web page accessibility for the blind[C]//Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI).New York,NY:ACM,2005:41-50.
[13]CLEGG-VINELL R,BAILEY C,GKATZIDOU V.Investiga-ting the appropriateness and relevance of mobile web accessibility guidelines[C]//Proceedings of the 11th Web for All Confe-rence(W4A).New York,NY:ACM,2014:1-4.
[14]ROSS A S,ZHANG X,FOGARTY J,et al.Epidemiology as a framework for large-scale mobile application accessibility assessment[C]//Proceedings of the 19th international ACM SIGACCESS Conference on Computers and Accessibility(ASSETS).New York,NY:ACM,2017:2-11.
[15]YAN S,RAMACHANDRAN P G.The current status of accessibility in mobile apps[J].ACM Transactions on Accessible Computing(TACCESS),2019,12(1):1-31.
[16]ZHANG X,ROSS A S,FOGARTY J.Robust annotation of mobile application interfaces in methods for accessibility repair and enhancement[C]//Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology(UIST).New York,NY:ACM,2018:609-621.
[17]ZHANG X,DE GREEF L,SWEARNGIN A,et al.Screen recognition:Creating accessibility metadata for mobile applications from pixels[C]//Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems(CHI).New York,NY:ACM,2021:1-15.
[18]ALSHAYBAN A,AHMED I,MALEK S.Accessibility issues in android apps:State of affairs,sentiments,and ways forward[C]//Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering(ICSE).New York,NY:ACM,2020:1323-1334.
[19]ANUPAM V,FREIRE J,KUMAR B,et al.Automating Webnavigation with the WebVCR[J].Computer Networks,2000,33(1/2/3/4/5/6):503-517.
[20]ALLEN J,CHAMBERS N,FERGUSON G,et al.Plow:A collaborative task learning agent[C]//Proceedings of the 22nd National Conference on Artificial Intelligence(AAAI).Menlo Park,CA:AAAI Press,2007:1514-1519.
[21]LAU T,CERRUTI J,MANZATO G,et al.A conversational interface to web automation[C]//Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology(UIST).New York,NY:ACM,2010:229-238.
[22]LI I,NICHOLS J,LAU T,et al.Here's what I did:Sharing andreusing web activity with ActionShot[C]//Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI).New York,NY:ACM,2010:723-732.
[23]FOK R,ZHONG M,ROSS A S,et al.A Large-Scale Longitudinal Analysis of Missing Label Accessibility Failures in Android Apps[C]//Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems(CHI).New York,NY:ACM,2022:1-16.
[24]VENDOME C,SOLANO D,LIÑÁN S,et al.Can everyone use my app? an empirical study on accessibility in android apps[C]//Proceedings of the 2019 IEEE International Conference on Software Maintenance and Evolution(ICSME).Piscataway,NJ:IEEE Press,2019:41-52.
[25]ELER M M,ROJAS J M,GE Y,et al.Automated accessibility testing of mobile apps[C]//Proceedings of the 2018 IEEE 11th International Conference on Software Testing,Verification and Validation(ICST).Piscataway,NJ:IEEE Press,2018:116-126.
[26]MORAN K,LINARES-VÁSQUEZ M,BERNAL-CÁRDENASC,et al.Fusion:A tool for facilitating and augmenting android bug reporting[C]//Proceedings of the 38th International Conference on Software Engineering Companion(ICSE).New York,NY:ACM,2016:609-612.
[27]KONG P,LI L,GAO J,et al.Automated testing of androidapps:A systematic literature review[J].IEEE Transactions on Reliability,2018,68(1):45-66.
[28]BEKRAR S,BEKRAR C,GROZ R,et al.Finding software vulnerabilities by smart fuzzing[C]//Proceedings of the 2011 Fourth IEEE International Conference on Software Testing,Verification and Validation(ICST).Piscataway,NJ:IEEE Press,2011:427-430.
[29]LINARES-VÁSQUEZ M,BERNAL-CÁRDENAS C,MORANK,et al.How do developers test android applications?[C]//Proceedings of the 2017 IEEE International Conference on Software Maintenance and Evolution(ICSME).Piscataway,NJ:IEEE Press,2017:613-622.
[30]JOORABCHI M E,MESBAH A,KRUCHTEN P.Real challenges in mobile app development[C]//Proceedings of the 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement(ESEM).Piscataway,NJ:IEEE Press,2013:15-24.
[31]LAM W,WU Z,LI D,et al.Record and replay for android:Are we there yet in industrial cases?[C]//Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering(ESEC/FSE).New York,NY:ACM,2017:854-859.
[32]SAHIN O,ALIYEVA A,MATHAVAN H,et al.Randr:Record and replay for android applications via targeted runtime instrumentation[C]//Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering(ASE).Piscataway,NJ:IEEE Press,2019:128-138.
[33]GOMEZ L,NEAMTIU I,AZIM T,et al.Reran:Timing-andtouch-sensitive record and replay for android[C]//Proceedings of the 2013 International Conference on Software Engineering(ICSE).Piscataway,NJ:IEEE Press,2013:72-81.
[34]GUO J,LI S,LOU J G,et al.Sara:self-replay augmented record and replay for Android in industrial cases[C]//Proceedings of the 28th ACM Sigsoft International Symposiumon Software Testing and Analysis(ISSTA).New York,NY:ACM,2019:90-100.
[35]Google Play.HiroMacro Auto-Touch Macro[EB/OL].(2017-08-20)[2023-01-28].https://play.google.com/store/apps/details?id=com.prohiro.macro&pli=1.
[36]Uptodown.RepetiTouch Free[EB/OL].(2021-02-24)[2023-01-30].https://repetitouch-free.cn.uptodown.com/android.
[37]Github.culebra[EB/OL].(2018-06-01)[2023-01-30].https://github.com/dtmilano/AndroidViewClient/wiki/culebra.
[38]Android Developers.Create UI tests with Espresso Test Re-corder[EB/OL].(2023-03-08)[2023-03-10].https://developer.android.com/studio/test/other-testing-tools/espresso-test-recorder.
[39]Android Developers.monkeyrunner[EB/OL].(2022-12-09)[2023-01-30].https://developer.android.com/studio/test/monkey-runner/index.html.
[40]Ranorex.Mobile Testing Tools[EB/OL].[2023-01-30].https://www.ranorex.com/mobile-automation-testing.
[41]HU Y,AZIM T,NEAMTIU I.Versatile yet lightweight record-and-replay for android[C]//Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming,Systems,Languages,and Applications(OOPSLA).New York,NY:ACM,2015:349-366.
[42]FAZZINI M,FREITAS E N D A,CHOUDHARY S R,et al.Barista:A technique for recording,encoding,and running platform independent android tests[C]//Proceedings of the 2017 IEEE International Conference on Software Testing,Verification and Validation(ICST).Piscataway,NJ:IEEE Press,2017:149-160.
[43]QIN Z,TANG Y,NOVAK E,et al.Mobiplay:A remote execution based record-and-replay tool for mobile applications[C]//Proceedings of the 38th International Conference on Software Engineering(ICSE).New York,NY:ACM,2016:571-582.
[44]BERNAL-CÁRDENAS C,COOPER N,MORAN K,et al.Translating video recordings of mobile app usages into replayable scenarios[C]//Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering(ICSE).New York,NY:ACM,2020:309-321.
[45]HALPERN M,ZHU Y,PERI R,et al.Mosaic:cross-platform user-interaction record and replay for the fragmented android ecosystem[C]//Proceedings of the 2015 IEEE International Symposium on Performance Analysis of Systems and Software(ISPASS).Piscataway,NJ:IEEE Press,2015:215-224.
[46]Github.replaykit[EB/OL].(2018-06-16)[2022-11-27].https://github.com/appetizerio/replaykit.
[47]TALEBIPOUR S,ZHAO Y,DOJCILOVIĆ L,et al.UI test migration across mobile platforms[C]//Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering(ASE).Piscataway,NJ:IEEE Press,2021:756-767.
[48]QIN X,ZHONG H,WANG X.Testmig:Migrating gui test cases from ios to android[C]//Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis(ISSTA).New York,NY:ACM,2019:284-295.
[49]YU S,FANG C,YUN Y,et al.Layout and image recognition driving cross-platform automated mobile testing[C]//Procee-dings of the 43rd International Conference on Software Enginee-ring(ICSE).Piscataway,NJ:IEEE Press,2021:1561-1571.
[50]LUKIĆ N,TALEBIPOUR S,MEDVIDOVIĆ N.Remote control of ios devices via accessibility features[C]//Proceedings of the 2020 ACM Workshop on Forming an Ecosystem Around Software Transformation(FEAST).New York,NY:ACM,2020:35-40.
[51]AOSP.Android 4.0 Compatibility Definition Document[EB/OL].(2013-04-21)[2023-02-28].https://developer.android.com/guide/topics/ui/accessibility/service.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed