Computer Science ›› 2024, Vol. 51 ›› Issue (12): 223-233.doi: 10.11896/jsjkx.240400077

• Artificial Intelligence • Previous Articles     Next Articles

Large Language Model-based Method for Mobile App Accessibility Enhancement

MA Qimin1, LI Xiangmin1,2, ZHOU Yaqian1   

  1. 1 School of Computer Science and Technology, Fudan University, Shanghai 200438, China
    2 Shanghai Key Laboratory of Data Science(Fudan University), Shanghai 200438, China
  • Received:2024-04-11 Revised:2024-08-19 Online:2024-12-15 Published:2024-12-10
  • About author:MA Qimin,born in 1998,postgraduate.Her main research interests include na-tural language processing and information search.
    ZHOU Yaqian,born in 1976,Ph.D,associate professor,is a member of CCF(No.14944M).Her main research interests include natural language processing and multimodal language mo-del.

Abstract: Mobile application accessibility refers to the degree to which mobile applications are designed and implemented to ensure that any user can easily access the application.However,only a small fraction of the vast number of applications in the domestic mobile application market support accessibility features,which contradicts to the vision of breaking the digital divide and enjoying the benefits of the digital age for the growing elderly and visually impaired population.Large language models(LLMs) have demonstrated significant potential for achieving human-level intelligence.Through prompts guidance,they can engage in simple logical reasoning and decision-making.In addition,shortening the interactive pathway is an intuitive strategy for enhancing mobile application accessibility.Inspired by the aforementioned facts,we propose an innovative method for enhancing mobile application accessibility based on LLMs.This method creatively applies accessibility services and LLMs,aiming to improve security,automation,and intelligence.We have implemented a mobile application accessibility tool called AccessLink.Under the premise of non-invasiveness and user authorization,AccessLink perceives and interacts with the graphical user interface of mobile applications.Additionally,we have developed a dataset construction approach based on automated methods.Experimental validation is conducted using the constructed dataset with large models such as GPT-3.5,GPT-4.0,QianWen and Baichuan,demonstrating the effectiveness of the proposed method.

Key words: Large language model, Android, Mobile application, Accessibility, Natural language processing

CLC Number: 

  • TP311
[1]卜佳俊,唐李真.无障碍与信息技术[M].沈阳:辽宁人民出版社,2019.
[2]YU J E,CHATTOPADHYAY D.“Maps are hard for me”:identifying how older adults struggle with mobile maps[C]//Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility.2020:1-8.
[3]中国信息通信研究院.信息无障碍白皮书(2022年)[R/OL].(2022-05-19)[2024-04-09].http://www.caict.ac.cn/kxyj/qwfb/bps/202205/P020220518510041281463.pdf.
[4]LI X M,SHEN L W,DONG Z.Mobile Application Accessibility Enhancement Method Based on Recording and Playback[J].Computer Science,2023,50(12):32-48.
[5]ONEY S,LUNDGARD A,KROSNICK R,et al.Arboretum and arbility:Improving web accessibility through a shared browsing architecture[C]//Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology.2018:937-949.
[6]LUKIĆ N,TALEBIPOUR S,MEDVIDOVIĆ N.Remote control of ios devices via accessibility features[C]//Proceedings of the 2020 ACM Workshop on Forming an Ecosystem Around Software Transformation.2020:35-40.
[7]ZHANG X,ROSS A S,FOGARTY J.Robust annotation of mobile application interfaces in methods for accessibility repair and enhancement[C]//Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology.2018:609-621.
[8]ZHANG X,DE GREEF L,SWEARNGIN A,et al.Screen recognition:Creating accessibility metadata for mobile applications frompixels[C]//Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems.2021:1-15.
[9]ELER M M,ROJAS J M,GE Y,et al.Automated accessibility testing of mobile apps[C]//2018 IEEE 11th International Conference on Software Testing,Verification and Validation(ICST).IEEE,2018:116-126.
[10]VENDOME C,SOLANO D,LIÑÁN S,et al.Can everyone use my app? an empirical study on accessibility in android apps[C]//2019 IEEE International Conference on Software Maintenance and Evolution(ICSME).IEEE,2019:41-52.
[11]ALSHAYBAN A,AHMED I,MALEK S.Accessibility issues in android apps:state of affairs,sentiments,and ways forward[C]//Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering.2020:1323-1334.
[12]FOK R,ZHONG M,ROSS A S,et al.A Large-Scale Longitudinal Analysis of Missing Label Accessibility Failures in Android Apps[C]//Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems.2022:1-16.
[13]VASWANI A,SHAZEER N,PARMAR N,et al.Attention is all you need[J].arXiv:1706.03762,2017.
[14]PETERS M,NEUMANN M,IYYER M,et al.Deep Contextua-lized Word Representations[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.2018.
[15]DEVLIN J,CHANG M W,LEE K,et al.Bert:Pre-training of deep bidirectional transformers for language understanding[J].arXiv:1810.04805,2018.
[16]RADFORD A,WU J,CHILD R,et al.Language models are unsupervised multitask learners[J].OpenAI blog,2019,1(8):9.
[17]BROWN T,MANN B,RYDER N,et al.Language models are few-shot learners[J].Advances in Neural Information Proces-sing Systems,2020,33:1877-1901.
[18]张奇,郑锐,黄萱菁.大规模语言模型:从理论到实践[M].北京:电子工业出版社,2024.
[19]WEI J,WANG X,SCHUURMANS D,et al.Chain-of-thought prompting elicits reasoning in large language models[J].Advances in Neural Information Processing Systems,2022,35:24824-24837.
[20]KOJIMA T,GU S S,REID M,et al.Large language models are zero-shot reasoners[J].Advances in Neural Information Processing Systems,2022,35:22199-22213.
[21]YAO S,YU D,ZHAO J,et al.Tree of thoughts:Deliberateproblem solving with large language models[J].arXiv:2305.10601,2023.
[22]MADAAN A,TANDON N,GUPTA P,et al.Self-refine:Iterative refinement with self-feedback[J].arXiv:2303.17651,2023.
[23]YAO S,ZHAO J,YU D,et al.React:Synergizing reasoning and acting in language models[J].arXiv:2210.03629,2022.
[24]SHINN N,CASSANO F,LABASH B,et al.Reflexion:Lan-guage Agents with Verbal Reinforcement Learning[J].arXiv:2303.11366,2023.
[25]YANG Z,LIU J,HAN Y,et al.Appagent:Multimodal agents as smartphone users[J].arXiv:2312.13771,2023.
[26]FURUTA H,NACHUM O,LEE K H,et al.Multimodal web navigation with instruction-finetuned foundation models[J].arXiv:2305.11854,2023.
[27]ZHAN Z,ZHANG A.You only look at screens:Multimodalchain-of-action agents[J].arXiv:2309.11436,2023.
[1] GUO Zhiqiang, GUAN Donghai, YUAN Weiwei. Word-Character Model with Low Lexical Information Loss for Chinese NER [J]. Computer Science, 2024, 51(8): 272-280.
[2] LIU Yumeng, ZHAO Yijing, WANG Bicong, WANG Chao, ZHANG Baomin. Advances in SQL Intelligent Synthesis Technology [J]. Computer Science, 2024, 51(7): 40-48.
[3] LI Bin, WANG Haochang. Implementation and Application of Chinese Grammatical Error Diagnosis System Based on CRF [J]. Computer Science, 2024, 51(6A): 230900073-6.
[4] YANG Binxia, LUO Xudong, SUN Kaili. Recent Progress on Machine Translation Based on Pre-trained Language Models [J]. Computer Science, 2024, 51(6A): 230700112-8.
[5] WANG Yingjie, ZHANG Chengye, BAI Fengbo, WANG Zumin. Named Entity Recognition Approach of Judicial Documents Based on Transformer [J]. Computer Science, 2024, 51(6A): 230500164-9.
[6] LI Minzhe, YIN Jibin. TCM Named Entity Recognition Model Combining BERT Model and Lexical Enhancement [J]. Computer Science, 2024, 51(6A): 230900030-6.
[7] PENG Bo, LI Yaodong, GONG Xianfu, LI Hao. Method for Entity Relation Extraction Based on Heterogeneous Graph Neural Networks and TextSemantic Enhancement [J]. Computer Science, 2024, 51(6A): 230700071-5.
[8] ZHANG Mingdao, ZHOU Xin, WU Xiaohong, QING Linbo, HE Xiaohai. Unified Fake News Detection Based on Semantic Expansion and HDGCN [J]. Computer Science, 2024, 51(4): 299-306.
[9] TU Xin, ZHANG Wei, LI Jidong, LI Meijiao , LONG Xiangbo. Study on Automatic Classification of English Tense Exercises for Intelligent Online Teaching [J]. Computer Science, 2024, 51(4): 353-358.
[10] HE Jiaojun, CAI Manchun, LU Tianliang. Android Malware Detection Method Based on GCN and BiLSTM [J]. Computer Science, 2024, 51(4): 388-395.
[11] ZHENG Cheng, SHI Jingwei, WEI Suhua, CHENG Jiaming. Dual Feature Adaptive Fusion Network Based on Dependency Type Pruning for Aspect-basedSentiment Analysis [J]. Computer Science, 2024, 51(3): 205-213.
[12] LI Zhanqi, WU Xinwei, ZHANG Lei, LIU Quanzhou, XIE Hui, XIONG Deyi. Automatic Test Case Generation Method for Automotive Electronic Control System Verification [J]. Computer Science, 2024, 51(12): 63-70.
[13] BAI Jianghao, PIAO Yong. Ensemble Learning Based Open Source License Detection and Compatibility Assessment [J]. Computer Science, 2024, 51(12): 79-86.
[14] ZHU Yangfu, LI Meiling, TAN Jiachen, WU Bin. Study on Text-based Personality Detection-A Review [J]. Computer Science, 2024, 51(12): 209-222.
[15] LIU Dongxu, DUAN Liguo, CUI Juanjuan, CHANG Xuanwei. Short Text Semantic Matching Strategy Fusing Sememe Similarity Matrix and Dual-channel of Char-Word Vectors [J]. Computer Science, 2024, 51(12): 250-258.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!