Computer Science

Computer Science ›› 2023, Vol. 50 ›› Issue (12): 32-48.doi: 10.11896/jsjkx.230300164

• Computer Software • Previous Articles     Next Articles

Mobile Application Accessibility Enhancement Method Based on Recording and Playback

LI Xiangmin, SHEN Liwei, DONG Zhen   

  1. School of Computer Science and Technology,Fudan University,Shanghai 200438,China
    Shanghai Key Laboratory of Data Science(Fudan University),Shanghai 200438,China
  • Received:2023-03-21 Revised:2023-05-29 Online:2023-12-15 Published:2023-12-07
  • About author:LI Xiangmin,born in 1998,postgra-duate.His main research interests is intelligent software engineering and systems.
    SHEN Liwei,born in 1982,Ph.D,asso-ciate professor.His main research inte-rests include cyber physical social,ubi-quitous computing and cloud computing.

PDF (PC)

1565

Abstract: Accessibility of mobile applications refers to the ability to use mobile applications conveniently without being affected by physical and cognitive impairments,which is of great significance to the elderly and disabled groups.Shortening the interaction path(reducing the number of steps) during the use of an application is an important way to enhance the accessibility of mobile applications.The recording and playback technology automatically executes fixed operations in the interactive process based on the recorded scripts to reduce the interactive operations.However,existing recording and playback tools still have limitations,including relying on ROOT permissions or using intrusive means to record and realize script migration.In addition,scripts recorded by existing tools do not support parameterized operations.In response to these problems,this paper proposes a mobile application accessibility enhancement method based on recording and playback.In the recording process of this method,the accessibility service is used as the medium to avoid applying for ROOT permission or using intrusive means.The path index algorithm is designed to ensure the portability of scripts,and the script parameterization algorithm is designed to record parameterized operations,so as to generate application execution scripts with terminal mobility and operation data generalization.Based on the proposed method,a prototype tool RRA for recording and playback is developed and 50 common execution scripts for 10 popular applications are constructed.The playback success rate on the same device using these scripts is 80%,which is comparable to the comparison method SARA.Parameterized recording of 5 scripts out of 40 scripts successfully played back by RRA with a 100% playback success rate on the same device.The 29 scripts and 5 parameterized scripts that can be successfully recorded by both methods are migrated and executed,and the playback success rate of RRA is 94%,which is higher than that of SARA.

Key words: Android, Mobile application, Accessibility, Recording and playback, Transferable

CLC Number: 

  • TP311
[1]IWARSSON S,STÅHL A.Accessibility,usability and universal design－positioning and definition of concepts describing person-environment relationships[J].Disability and rehabilitation,2003,25(2):57-66.
[2]BIGHAM J P,LIN I,SAVAGE S.The Effects of “Not Knowing What You Don't Know” on Web Accessibility for Blind Web Users[C]//Proceedings of the 19thInternational ACM SIGACCESS Conference on Computers and Accessibility(ASSETS).New York,NY:ACM,2017:101-109.
[3]ONEY S,LUNDGARD A,KROSNICK R,et al.Arboretum and arbility:Improving web accessibility through a shared browsing architecture[C]//Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology(UIST).New York,NY:ACM,2018:937-949.
[4]LAZAR J,ALLEN A,KLEINMAN J,et al.What frustratesscreen reader users on the web:A study of 100 blind users[J].International Journal of human-computer interaction,2007,22(3):247-269.
[5]YU J E,CHATTOPADHYAY D.“Maps are hard for me”:identifying how older adults struggle with mobile maps[C]//Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility(ASSETS).New York,NY:ACM,2020:1-8.
[6]ABOU-ZAHRA S,HENRY S L.Exploring web accessibility solutions in developing regions as innovations for the benefit of all[C]//Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility(W4A).New York,NY:ACM,2010:1-4.
[7]HENRY S L,ABOU-ZAHRA S,BREWER J.The role of accessibility in a universal web[C]//Proceedings of the 11th Web for all Conference(W4A).New York,NY:ACM,2014:1-4.
[8]W3C Web Accessibility Initiative(WAI).Introduction to web accessibility[EB/OL].(2022-03-31)[2022-11-25].https://www.w3.org/WAI/fundamentals/accessibility-intro/.
[9]Android Accessibility.Android accessibility overview[EB/OL].[2022-11-25].https://support.go ogle.com/accessibility/android/answer/6006564.
[10]Android Developers.Android Accessibility Developer Guidelines[EB/OL].(2022-10-24)[2022-11-25].https://developer.android.com/guide/topics/ui/accessibility.
[11]MILNE S,DICKINSON A,CARMICHAEl A,et al.Are guidelines enough? An introduction to designing Web sites accessible to older people[J].IBM Systems Journal,2005,44(3):557-571.
[12]MANKOFF J,FAIT H,TRAN T.Is your web page accessible? A comparative study of methods for assessing web page accessibility for the blind[C]//Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI).New York,NY:ACM,2005:41-50.
[13]CLEGG-VINELL R,BAILEY C,GKATZIDOU V.Investiga-ting the appropriateness and relevance of mobile web accessibility guidelines[C]//Proceedings of the 11th Web for All Confe-rence(W4A).New York,NY:ACM,2014:1-4.
[14]ROSS A S,ZHANG X,FOGARTY J,et al.Epidemiology as a framework for large-scale mobile application accessibility assessment[C]//Proceedings of the 19th international ACM SIGACCESS Conference on Computers and Accessibility(ASSETS).New York,NY:ACM,2017:2-11.
[15]YAN S,RAMACHANDRAN P G.The current status of accessibility in mobile apps[J].ACM Transactions on Accessible Computing(TACCESS),2019,12(1):1-31.
[16]ZHANG X,ROSS A S,FOGARTY J.Robust annotation of mobile application interfaces in methods for accessibility repair and enhancement[C]//Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology(UIST).New York,NY:ACM,2018:609-621.
[17]ZHANG X,DE GREEF L,SWEARNGIN A,et al.Screen recognition:Creating accessibility metadata for mobile applications from pixels[C]//Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems(CHI).New York,NY:ACM,2021:1-15.
[18]ALSHAYBAN A,AHMED I,MALEK S.Accessibility issues in android apps:State of affairs,sentiments,and ways forward[C]//Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering(ICSE).New York,NY:ACM,2020:1323-1334.
[19]ANUPAM V,FREIRE J,KUMAR B,et al.Automating Webnavigation with the WebVCR[J].Computer Networks,2000,33(1/2/3/4/5/6):503-517.
[20]ALLEN J,CHAMBERS N,FERGUSON G,et al.Plow:A collaborative task learning agent[C]//Proceedings of the 22nd National Conference on Artificial Intelligence(AAAI).Menlo Park,CA:AAAI Press,2007:1514-1519.
[21]LAU T,CERRUTI J,MANZATO G,et al.A conversational interface to web automation[C]//Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology(UIST).New York,NY:ACM,2010:229-238.
[22]LI I,NICHOLS J,LAU T,et al.Here's what I did:Sharing andreusing web activity with ActionShot[C]//Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI).New York,NY:ACM,2010:723-732.
[23]FOK R,ZHONG M,ROSS A S,et al.A Large-Scale Longitudinal Analysis of Missing Label Accessibility Failures in Android Apps[C]//Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems(CHI).New York,NY:ACM,2022:1-16.
[24]VENDOME C,SOLANO D,LIÑÁN S,et al.Can everyone use my app? an empirical study on accessibility in android apps[C]//Proceedings of the 2019 IEEE International Conference on Software Maintenance and Evolution(ICSME).Piscataway,NJ:IEEE Press,2019:41-52.
[25]ELER M M,ROJAS J M,GE Y,et al.Automated accessibility testing of mobile apps[C]//Proceedings of the 2018 IEEE 11th International Conference on Software Testing,Verification and Validation(ICST).Piscataway,NJ:IEEE Press,2018:116-126.
[26]MORAN K,LINARES-VÁSQUEZ M,BERNAL-CÁRDENASC,et al.Fusion:A tool for facilitating and augmenting android bug reporting[C]//Proceedings of the 38th International Conference on Software Engineering Companion(ICSE).New York,NY:ACM,2016:609-612.
[27]KONG P,LI L,GAO J,et al.Automated testing of androidapps:A systematic literature review[J].IEEE Transactions on Reliability,2018,68(1):45-66.
[28]BEKRAR S,BEKRAR C,GROZ R,et al.Finding software vulnerabilities by smart fuzzing[C]//Proceedings of the 2011 Fourth IEEE International Conference on Software Testing,Verification and Validation(ICST).Piscataway,NJ:IEEE Press,2011:427-430.
[29]LINARES-VÁSQUEZ M,BERNAL-CÁRDENAS C,MORANK,et al.How do developers test android applications?[C]//Proceedings of the 2017 IEEE International Conference on Software Maintenance and Evolution(ICSME).Piscataway,NJ:IEEE Press,2017:613-622.
[30]JOORABCHI M E,MESBAH A,KRUCHTEN P.Real challenges in mobile app development[C]//Proceedings of the 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement(ESEM).Piscataway,NJ:IEEE Press,2013:15-24.
[31]LAM W,WU Z,LI D,et al.Record and replay for android:Are we there yet in industrial cases?[C]//Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering(ESEC/FSE).New York,NY:ACM,2017:854-859.
[32]SAHIN O,ALIYEVA A,MATHAVAN H,et al.Randr:Record and replay for android applications via targeted runtime instrumentation[C]//Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering(ASE).Piscataway,NJ:IEEE Press,2019:128-138.
[33]GOMEZ L,NEAMTIU I,AZIM T,et al.Reran:Timing-andtouch-sensitive record and replay for android[C]//Proceedings of the 2013 International Conference on Software Engineering(ICSE).Piscataway,NJ:IEEE Press,2013:72-81.
[34]GUO J,LI S,LOU J G,et al.Sara:self-replay augmented record and replay for Android in industrial cases[C]//Proceedings of the 28th ACM Sigsoft International Symposiumon Software Testing and Analysis(ISSTA).New York,NY:ACM,2019:90-100.
[35]Google Play.HiroMacro Auto-Touch Macro[EB/OL].(2017-08-20)[2023-01-28].https://play.google.com/store/apps/details?id=com.prohiro.macro&pli=1.
[36]Uptodown.RepetiTouch Free[EB/OL].(2021-02-24)[2023-01-30].https://repetitouch-free.cn.uptodown.com/android.
[37]Github.culebra[EB/OL].(2018-06-01)[2023-01-30].https://github.com/dtmilano/AndroidViewClient/wiki/culebra.
[38]Android Developers.Create UI tests with Espresso Test Re-corder[EB/OL].(2023-03-08)[2023-03-10].https://developer.android.com/studio/test/other-testing-tools/espresso-test-recorder.
[39]Android Developers.monkeyrunner[EB/OL].(2022-12-09)[2023-01-30].https://developer.android.com/studio/test/monkey-runner/index.html.
[40]Ranorex.Mobile Testing Tools[EB/OL].[2023-01-30].https://www.ranorex.com/mobile-automation-testing.
[41]HU Y,AZIM T,NEAMTIU I.Versatile yet lightweight record-and-replay for android[C]//Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming,Systems,Languages,and Applications(OOPSLA).New York,NY:ACM,2015:349-366.
[42]FAZZINI M,FREITAS E N D A,CHOUDHARY S R,et al.Barista:A technique for recording,encoding,and running platform independent android tests[C]//Proceedings of the 2017 IEEE International Conference on Software Testing,Verification and Validation(ICST).Piscataway,NJ:IEEE Press,2017:149-160.
[43]QIN Z,TANG Y,NOVAK E,et al.Mobiplay:A remote execution based record-and-replay tool for mobile applications[C]//Proceedings of the 38th International Conference on Software Engineering(ICSE).New York,NY:ACM,2016:571-582.
[44]BERNAL-CÁRDENAS C,COOPER N,MORAN K,et al.Translating video recordings of mobile app usages into replayable scenarios[C]//Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering(ICSE).New York,NY:ACM,2020:309-321.
[45]HALPERN M,ZHU Y,PERI R,et al.Mosaic:cross-platform user-interaction record and replay for the fragmented android ecosystem[C]//Proceedings of the 2015 IEEE International Symposium on Performance Analysis of Systems and Software(ISPASS).Piscataway,NJ:IEEE Press,2015:215-224.
[46]Github.replaykit[EB/OL].(2018-06-16)[2022-11-27].https://github.com/appetizerio/replaykit.
[47]TALEBIPOUR S,ZHAO Y,DOJCILOVIĆ L,et al.UI test migration across mobile platforms[C]//Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering(ASE).Piscataway,NJ:IEEE Press,2021:756-767.
[48]QIN X,ZHONG H,WANG X.Testmig:Migrating gui test cases from ios to android[C]//Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis(ISSTA).New York,NY:ACM,2019:284-295.
[49]YU S,FANG C,YUN Y,et al.Layout and image recognition driving cross-platform automated mobile testing[C]//Procee-dings of the 43rd International Conference on Software Enginee-ring(ICSE).Piscataway,NJ:IEEE Press,2021:1561-1571.
[50]LUKIĆ N,TALEBIPOUR S,MEDVIDOVIĆ N.Remote control of ios devices via accessibility features[C]//Proceedings of the 2020 ACM Workshop on Forming an Ecosystem Around Software Transformation(FEAST).New York,NY:ACM,2020:35-40.
[51]AOSP.Android 4.0 Compatibility Definition Document[EB/OL].(2013-04-21)[2023-02-28].https://developer.android.com/guide/topics/ui/accessibility/service.
[1] YUAN Jiangfeng, LI Haoxiang, YOU Wei, HUANG Jianjun, SHI Wenchang, LIANG Bin. Locating Third-party Library Functions in Obfuscated Applications [J]. Computer Science, 2023, 50(7): 293-301.
[2] FU Xiong, NIE Xiaohan, WANG Junchang. Study on Android Fake Application Detection Method Based on Interface Similarity [J]. Computer Science, 2023, 50(6A): 220300114-7.
[3] YU Xingzhan, LU Tianliang, DU Yanhui, WANG Xirui, YANG Cheng. Android Malware Family Classification Method Based on Synthetic Image and Xception Improved Model [J]. Computer Science, 2023, 50(4): 351-358.
[4] WANG Xi, ZHAO Chunlei, BU Zhiliang, YANG Yi. Automated Testing Method of Android Applications Based on SA-UCB Algorithm [J]. Computer Science, 2023, 50(11A): 221200145-7.
[5] DING Xuhui, ZHANG Linlin, ZHAO Kai, WANG Xusheng. Android Application Privacy Disclosure Detection Method Based on Static and Dynamic Combination [J]. Computer Science, 2023, 50(10): 327-335.
[6] YAO Ye, ZHU Yi-an, QIAN Liang, JIA Yao, ZHANG Li-xiang, LIU Rui-liang. Android Malware Detection Method Based on Heterogeneous Model Fusion [J]. Computer Science, 2022, 49(6A): 508-515.
[7] WANG Yi, LI Zheng-hao, CHEN Xing. Recommendation of Android Application Services via User Scenarios [J]. Computer Science, 2022, 49(6A): 267-271.
[8] WANG Yi, CHEN Ying-ren, CHEN Xing, LIN Bin, MA Yun. Automating Release of Android APIs Based on Computational Reflection [J]. Computer Science, 2022, 49(12): 136-145.
[9] YANG Yi, WANG Xi, ZHAO Chun-lei, BU Zhi-liang. Overview of Android GUI Automated Testing [J]. Computer Science, 2022, 49(11A): 210900231-10.
[10] SUN Zhi-qiang, WAN Liang, DING Hong-wei. Android Malware Detection Method Based on Deep Autoencoder Network [J]. Computer Science, 2020, 47(4): 298-304.
[11] SUN Zhi-gang, WANG Guo-tao, JIANG Ai-ping, GAO Meng-meng, LIU Jin-gang. Monitoring System of Traffic Safety Based on Information Fusion Technology [J]. Computer Science, 2020, 47(11A): 642-650.
[12] ZHANG Jing, LI Rui-xuan, TANG Jun-wei, HAN Hong-mu, GU Xi-wu. Collusion Behavior Detection Towards Android Third-party Libraries [J]. Computer Science, 2019, 46(5): 83-91.
[13] XIE Nian-nian, ZENG Fan-ping, ZHOU Ming-song, QIN Xiao-xia, LV Cheng-cheng, CHEN Zhao. Android Malware Detection with Multi-dimensional Sensitive Features [J]. Computer Science, 2019, 46(2): 95-101.
[14] ZHANG Zong-mei, GUI Sheng-lin, REN Fei. Android Malware Detection Based on N-gram [J]. Computer Science, 2019, 46(2): 145-151.
[15] HOU Yu-chen, WU Wei. Design and Implementation of Crowdsourcing System for Still Image Activity Annotation [J]. Computer Science, 2019, 46(11A): 580-583.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.