Computer Science ›› 2026, Vol. 53 ›› Issue (10): 1-17.doi: 10.11896/jsjkx.260300074
• Large Models and Multimodal Intelligent Perception • Previous Articles Next Articles
LIU Zhenyu1, YAO Dengfeng1,2, ZHAO Yuan1, MENG Ziwen1, XU Cheng1, LI Yan1
CLC Number:
| [1] XUAN H Y,CHEN Q,WU Z L,et al.A survey on audio-visual multimodal learning[J/OL].Acta Automatica Sinica,2026.DOI:10.16383/j.aas.c250341. [2] WANG J Q,YANG B Y.Review and reflections on the develop-ment of multimodal large models [J].Radio Engineering,2025,55(9):1727-1742. [3] GEMMEKE J F,ELLIS D P W,FREEDMAN D,et al.AudioSet:An ontology and human-labeled dataset for audio events[C]//Proceedings of the IEEE International Conference on Acoustics,Speech and Signal Processing.IEEE,2017:776-780. [4] CHEN H,XIE W,VEDALDI A,et al.VGGSound:A large-scaleaudio-visual dataset[C]//Proceedings of the IEEE International Conference on Acoustics,Speech and Signal Processing.IEEE,2020:721-725. [5] XIAO F,LEE Y J,GRAUMAN K,et al.Audiovisual SlowFast networks for video recognition[PP/OL].arXiv [2026-08-04].https://arxiv.org/abs/2001.08740. [6] YANG P,WANG X,DUAN X,et al.AVQA:A dataset for audio-visual question answering on videos[C]//Proceedings of the 30th ACM International Conference on Multimedia.New York:ACM,2022:3480-3491. [7] LI G,WEI Y,TIANY,et al.Learning to answer questions in dynamic audio-visual scenarios[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2022:19108-19118. [8] ZHANG H,LI X,BING L.Video-LLaMA:An instruction-tunedaudio-visual language model for video understanding[C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing:System Demonstrations.ACL,2023:543-553. [9] WANG Y,LI K,LI X,et al.InternVideo2:Scaling foundationmodels for multimodal video understanding[C]//Proceedings of the European Conference on Computer Vision.Cham:Springer,2024:396-416. [10] CHENG Z,LENG S,ZHANG H,et al.VideoLLaMA 2:Advancing spatial-temporal modeling and audio understanding in video-LLMs[PP/OL].arXiv[2026-08-05].https://arxiv.org/abs/2406.07476. [11] CHEN Z,LIU H,YU W,et al.M3AV:A multimodal,multigenre,and multipurpose audio-visual academic lecture dataset[C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics.ACL,2024:9041-9060. [12] FU C,DAI Y,LUO Y,et al.Video-MME:The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2025:24108-24118. [13] WU H,LI D,CHEN B,et al.LongVideoBench:A benchmarkfor long-context interleaved video-language understanding[C]//Advances in Neural Information Processing Systems.Vancouver:Curran Associates,2024. [14] FANG X,MAO K,DUAN H,et al.MMBench-Video:A long-form multi-shot benchmark for holistic video understanding[C]//Advances in Neural Information Processing Systems.Vancouver:Curran Associates,2024:89098-89124. [15] CHU Y,XU J,YANG Q,et al.Qwen2-Audio technical report[PP/OL].arXiv [2026-08-04].https://arxiv.org/abs/2407.10759. [16] XU J,GUO Z,HE J,et al.Qwen2.5-Omni technical report[PP/OL].arXiv [2026-08-04].https://arxiv.org/abs/2503.20215. [17] WANG Y,LIAO Y,LIU H,et al.MM-SAP:A comprehensive benchmark for assessing self-awareness of multimodal large language models in perception[C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics.ACL,2024:9192-9205. [18] ZHOU J,WANG J,ZHANG J,et al.Audio-visual segmentation[C]//Proceedings of the European Conference on Computer Vision.Cham:Springer,2022:386-403. [19] CHEN Y,LIU Y,WANG H,et al.Unraveling instance associations:A closer look for audio-visual segmentation[C]//Procee-dings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2024:26445-26455. [20] FAN Y,DU B,LIN Y,et al.Revisit weakly-supervised audio-visual video parsing from the language perspective[C]//Advances in Neural Information Processing Systems.New Orleans:Curran Associates,2023:40610-40622. [21] RACHAVARAPU K K,RAMAKRISHNAN K,RAJAGOPA-LAN A N.Weakly-supervised audio-visual video parsing with prototype-based pseudo-labeling[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2024:26407-26416. [22] LIU J,LIU Y,ZHANG F,et al.Audio-visual segmentation via unlabeled frame exploitation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2024:26328-26339. [23] YANG Q,NIE X,LI T,et al.Cooperation does matter:Exploring multi-order bilateral relations for audio-visual segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2024:26361-26371. [24] LIU Y,XIA J,ZHANG S,et al.Long video audible object understanding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2025:13926-13936. [25] MA J,HU M,WANG P,et al.Look,listen,and answer:Overcoming biases for audio-visual question answering[C]//Advances in Neural Information Processing Systems.Vancouver:Curran Associates,2024:9507-9531. [26] LIU J,CHEN S,HE X,et al.VALOR:Vision-audio-languageomni-perception pretraining model and dataset[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2024,46(8):5369-5385. [27] ZHAO P,ZHOU J,ZHAO Y,et al.Multimodal class-aware semantic enhancement network for audio-visual video parsing[C]//Proceedings of the AAAI Conference on Artificial Intelligence.AAAI,2025:10448-10456. [28] BAO C Z,DING K,DONG J F,et al.Research progress of videoquestion answering technologies [J].Journal of Computer Research and Development,2024,61(3):639-673. [29] TIAN Y,SHI J,LI B,et al.Audio-visual event localization inunconstrained videos[C]//Proceedings of the European Confe-rence on Computer Vision.Cham:Springer,2018:247-263. [30] TIAN Y,LI D,XU C.Unified multisensory perception:Weakly-supervised audio-visual video parsing[C]//Proceedings of the European Conference on Computer Vision.Cham:Springer,2020:436-454. [31] GE J,LI S,CHEN Z,et al.Towards open-vocabulary audio-vi-sual event localization[C]//Proceedings of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition.IEEE,2025:4962-4972. [32] GUO R,QU L,NIU D,et al.Open-vocabulary audio-visual semantic segmentation[C]//Proceedings of the 32nd ACM International Conference on Multimedia.New York:ACM,2024:7533-7541. [33] CHEN S,LI H,WANG Q,et al.VAST:A vision-audio-subtitle-text omni-modality foundation model and dataset[C]//Advances in Neural Information Processing Systems.New Orleans:Curran Associates,2023:72842-72866. [34] YANG Q,XU J,LIU W,et al.AIR-Bench:Benchmarking large audio-language models via generative comprehension[C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics.ACL,2024:1979-1998. [35] WANG B,ZOU X,LIN G,et al.AudioBench:A universalbenchmark for audio large language models[C]//Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics:Human Language Technologies.ACL,2025:4297-4316. [36] SAKSHI S,TYAGI U,KUMAR S,et al.MMAU:A massive multi-task audio understanding andreasoning benchmark[C]//Proceedings of the International Conference on Learning Representations.Singapore:OpenReview,2025. [37] CHEN S,YIN Y,CAO J,et al.OpenAVS:Training-free open-vocabulary audio visual segmentation with foundational models[PP/OL].arXiv [2026-08-05].https://arxiv.org/abs/2505.01448. [38] WEI C,FAN H,XIE S,et al.Masked feature prediction for self-supervised visual pre-training[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2022:14668-14678. [39] TONG Z,SONG Y,WANG J,et al.VideoMAE:Masked au-toencoders are data-efficient learners for self-supervised video pre-training[C]//Advances in Neural Information Processing Systems.New Orleans:Curran Associates,2022:10078-10093. [40] LI K,WANG Y,HE Y,et al.UniFormerV2:Unlocking the potential of image ViTs for video understanding[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.IEEE,2023:1632-1643. [41] ZHAO L,GUNDAVARAPU N B,YUAN L,et al.VideoPrism:A foundational visual encoder for video understanding[C]//Proceedings of the 41st International Conference on Machine Lear-ning.Vienna:PMLR,2024:60785-60811. [42] LI K,LI X,WANG Y,et al.VideoMamba:State space model for efficient video understanding[C]//Proceedings of the European Conference on Computer Vision.Cham:Springer,2024:237-255. [43] CHEN K,DU X,ZHU B,et al.HTS-AT:A hierarchical token-semantic audio transformer for sound classification and detection[C]//Proceedings of the IEEE International Conference on Acoustics,Speech and Signal Processing.IEEE,2022:646-650. [44] HUANG P Y,XU H,LI J,et al.Masked autoencoders that listen[C]//Advances in Neural Information Processing Systems.New Orleans:Curran Associates,2022:28708-28720. [45] CHEN S,WU Y,WANG C,et al.BEATs:Audio pre-trainingwith acoustic tokenizers[C]//Proceedings of the 40th International Conference on Machine Learning.Honolulu:PMLR,2023:5178-5193. [46] CHEN W,LIANG Y,MA Z,et al.EAT:Self-supervised pre-trainingwith efficient audio transformer[C]//Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence.Jeju:IJCAI,2024:3807-3815. [47] GONG Y,ROUDITCHENKO A,LIU A H,et al.Contrastive audio-visual masked autoencoder[C]//Proceedings of the International Conference on Learning Representations.Kigali:OpenReview,2023. [48] HUANG P Y,SHARMA V,XU H,et al.MAViL:Masked audio-video learners[C]//Advances in Neural Information Processing Systems.New Orleans:Curran Associates,2023:20371-20393. [49] GIRDHAR R,EL-NOUBY A,LIU Z,et al.ImageBind:One embedding space to bind them all[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2023:15180-15190. [50] ARAUJO E,ROUDITCHENKO A,GONG Y,et al.CAV-MAE Sync:Improving contrastive audio-visual mask autoencoders via fine-grained alignment[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2025:18794-18803. [51] SHI B,HSU W N,LAKHOTIA K,et al.Learning audio-visualspeech representation by masked multimodal cluster prediction[C]//Proceedings of the International Conference on Learning Representations.2022. [52] SEO P H,NAGRANI A,SCHMID C.AVFormer:Injecting vision into frozen speech models for zero-shot AV-ASR[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2023:22922-22931. [53] RACHAVARAPU K K,RAJAGOPALAN A N.Boosting positive segments for weakly-supervised audio-visual video parsing[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.IEEE,2023:10192-10202. [54] ZHOU J,GUO D,MAO Y,et al.Label-anticipated event disentanglement for audio-visual video parsing[C]//Proceedings of the European Conference on Computer Vision.Cham:Springer,2024:35-51. [55] SUN W,ZHANG J,WANG J,et al.Learning audio-visualsource localization via false negative aware contrastive learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2023:6420-6429. [56] MO S,TIAN Y.Audio-visual grouping network for sound localization from mixtures[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2023:10565-10574. [57] MAHMUD T,TIAN Y,MARCULESCU D.T-VSL:Text-guided visual sound source localization in mixtures[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Seattle:IEEE Press,2024:26742-26751. [58] KIM D,UM S J,LEE S,et al.Learning tovisually localize sound sources from mixtures without prior source knowledge[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2024:4864-4873. [59] MAO Y,ZHANG J,XIANG M,et al.Multimodal variationalauto-encoder based audio-visual segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.IEEE,2023:954-965. [60] LUO Z,LIU N,YANG X,et al.TAViS:Text-bridged audio-vi-sual segmentation with foundation models[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.IEEE,2025:24014-24023. [61] LIN J A,BAO C Z,DONG J F,et al.Multilingual text-vi-deo cross-modal retrieval model via multilingual-visual common space learning[J].Chinese Journal of Computers,2024,47(9):2195-2210. [62] LEE K,ZHANG Y,DUAN Z.Audio visual segmentationthrough text embeddings[C]//Proceedings of the IEEE International Conference on Image Processing.IEEE,2025. [63] LI G,WANG X,ZHU W.AV-Unified:A unified framework for audio-visual scene understanding[PP/OL].arXiv [2026-08-05].https://arxiv.org/abs/2603.06530. [64] CHENG Y,LI Y,HE J,et al.Mixtures of expertsfor audio-visual learning[C]//Advances in Neural Information Processing Systems.Vancouver:Curran Associates,2024:219-243. [65] DUAN H,XIA Y,ZHOU M,et al.Cross-modal prompts:Adapting large pre-trained models for audio-visual downstream tasks[C]//Advances in Neural Information Processing Systems.New Orleans:Curran Associates,2023:12078-12100. [66] MEI X,MENG C,LIU H,et al.WavCaps:A ChatGPT-assisted weakly-labelled audio captioning dataset for audio-language multimodal research[PP/OL].arXiv [2026-08-05].https://arxiv.org/abs/2303.17395. [67] ZELLERS R,LU J,LU X,et al.MERLOT Reserve:Neuralscript knowledge through vision and language and sound[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2022:16354-16366. [68] WANG L,HUANG B,ZHAO Z,et al.VideoMAE V2:Scaling video masked autoencoders with dual masking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2023:14549-14560. [69] LI K,WANG Y,LI Y,et al.Unmasked teacher:Towards trai-ning-efficient video foundation models[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.IEEE,2023:19948-19960. [70] WANG Y,LI K,LI Y,et al.InternVideo:General video foundation models via generative and discriminative learning[PP/OL].arXiv [2026-08-05].https://arxiv.org/abs/2212.03191. [71] LIU T Y,WU Z X,CHEN J J,et al.Multimodal pre-trainingmethods for vision-language understanding and generation [J].Journal of Software,2023,34(5):2024-2034. [72] ZHU B,LIN B,NING M,et al.LanguageBind:Extending video-language pretraining to N-modality by language-based semantic alignment[C]//Proceedingsof the International Conference on Learning Representations.Vienna:OpenReview,2024. [73] WU Y,CHEN K,ZHANG T,et al.Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation[C]//Proceedings of the IEEE International Conference on Acoustics,Speech and Signal Processing.IEEE,2023:1-5. [74] DESHMUKH S,ELIZALDE B,SINGH R,et al.Pengi:An audio language model for audio tasks[C]//Advances in Neural Information Processing Systems.New Orleans:Curran Associates,2023:18090-18108. [75] KONG Z,GOEL A,BADLANI R,et al.Audio Flamingo:A novel audio language model with few-shot learning and dialogue abilities[C]//Proceedings of the International Conference on Machine Learning.Vienna:PMLR,2024:25325-25348. [76] PIERGIOVANNI A J,NOBLE I,KIM D,et al.Mirasol3B:A multimodal autoregressive model for time-aligned and contextual modalities[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2024:26805-26815. [77] MAAZ M,RASHEED H,KHAN S,et al.Video-ChatGPT:Towards detailed video understanding via large vision and language models[C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics.ACL,2024:12585-12602. [78] REN S,YAO L,LI S,et al.TimeChat:A time-sensitive multimodal large language model for long video understanding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2024:14313-14323. [79] HUANG R,LI M,YANG D,et al.AudioGPT:Understanding and generating speech,music,sound,and talking head[C]//Proceedings of the AAAI Conference on Artificial Intelligence.AAAI,2024:23802-23804. [80] GONG Y,LUO H,LIU A H,et al.Listen,think,and understand[C]//Proceedings of the International Conference on Learning Representations.Vienna:OpenReview,2024. [81] TANG C,YU W,SUN G,et al.SALMONN:Towards generic hearing abilities for large language models[C]//Proceedings of the International Conference on Learning Representations.Vienna:OpenReview,2024. [82] CHU Y,XU J,ZHOU X,et al.Qwen-Audio:Advancing universal audio understanding via unified large-scale audio-language models[PP/OL].arXiv [2026-08-05].https:arxiv.org/abs/2311.07919. [83] SUN G,YU W,TANG C,et al.video-SALMONN:Speech-enhanced audio-visual large language models[C]//Proceedings of the 41st International Conference on Machine Learning.Vienna:PMLR,2024. [84] HAN J,GONG K,ZHANG Y,et al.OneLLM:One frameworkto align all modalities with language[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2024. [85] LU J,CLARK C,LEE S,et al.Unified-IO 2:Scaling autoregressive multimodal models with vision,language,audio,and action[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2024:26439-26455. [86] WU S,FEI H,QU L,et al.NExT-GPT:Any-to-any multimodal LLM[C]//Proceedings of the 41st International Conference on Machine Learning.Vienna:PMLR,2024. [87] WANG H,HUANG L.Grounded-VideoLLM:Sharpening fine-grained temporal grounding in video large language models[C]//Findings of the Association for Computational Linguistics:EMNLP.ACL,2025. [88] GUAN T,LIU F,WU X,et al.HallusionBench:An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2024:14375-14385. [89] WANG Y,WANG Y,ZHAO D,et al.VideoHallucer:Evaluating intrinsic and extrinsic hallucinations in large video-language models[PP/OL].arXiv [2026-08-05].https://arxiv.org/abs/2406.16338. [90] LI C,IM E W,FAZLI P.VidHalluc:Evaluating temporal hallucinations in multimodal large language models for video understanding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2025:13723-13733. [91] KIM S B,OH H B,LEE J,et al.AVHBench:A cross-modalhallucination benchmark for audio-visual large language models[C]//Proceedings of the International Conference on Learning Representations.Singapore:OpenReview,2025. [92] CHENG X,FU D,WEN C,et al.AHa-Bench:Benchmarkingaudio hallucinations in large audio-language models[C]//Advances in Neural Information Processing Systems,Datasets and Benchmarks Track.San Diego:Curran Associates,2025. |
| [1] | HE Jiaojun, LI Xin. Review of Graph Learning Based on Large Language Models:Methods,Benchmarks and Advances [J]. Computer Science, 2026, 53(9): 1-15. |
| [2] | FENG Guang, SUN Xiangli, LIN Yibao, LIU Xinting, CAO Yuqiao, HUANG Junhui, LIAO Beirong. Multimodal Sentiment Analysis Based on Prompt Learning and Guided Gated Fusion Mechanism [J]. Computer Science, 2026, 53(9): 299-308. |
| [3] | LIU Jing. Review of Music Artificial Intelligence Driven by Large Language Models [J]. Computer Science, 2026, 53(8): 229-244. |
| [4] | ZHU Rong, HU Mengyao, DAI Lingyun, LI Feng. Single-cell Multi-omics Clustering Method Based on Missing Value Imputation and Cross-modalAlignment [J]. Computer Science, 2026, 53(7): 289-297. |
| [5] | CHENG Cheng, ZHONG Han. Hateful Meme Detection Framework Enhanced by Large Models and Knowledge Graphs [J]. Computer Science, 2026, 53(10): 77-91. |
| [6] | LIU Jian, YAO Renyuan, GAO Nan, LIANG Ronghua, CHEN Peng. VSRI:Visual Semantic Relational Interactor for Image Caption [J]. Computer Science, 2025, 52(8): 222-231. |
| [7] | CHEN Jinyin, XI Changkun, ZHENG Haibin, GAO Ming, ZHANG Tianxin. Survey of Security Research on Multimodal Large Language Models [J]. Computer Science, 2025, 52(7): 315-341. |
| [8] | SONG Xinyang, YAN Zhiyuan, SUN Muyi, DAI Linlin, LI Qi, SUN Zhenan. Review of Talking Face Generation [J]. Computer Science, 2023, 50(8): 68-78. |
| [9] | NIE Xiu-shan, PAN Jia-nan, TAN Zhi-fang, LIU Xin-fang, GUO Jie, YIN Yi-long. Overview of Natural Language Video Localization [J]. Computer Science, 2022, 49(9): 111-122. |
|
||