Computer Science

Computer Science ›› 2026, Vol. 53 ›› Issue (10): 51-67.doi: 10.11896/jsjkx.260500084

• Large Models and Multimodal Intelligent Perception • Previous Articles     Next Articles

Multi-stage Time-spectral Preserving Diffusion Network for Multimodal Missing ModalityRecovery

DAI Ruiting1,2, MENG Haoran1, LU Tianqi1, WANG Xinyu1, LI Chenxi1, HE Tao2,3, MO Lisi4   

  1. 1 School of Information and Software Engineering,University of Electronic Science and Technology of China,Chengdu 610054, China
    2 Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province,University of Electronic Science and Technology of China,Chengdu 611731, China
    3 Institute of Intelligent Computing,University of Electronic Science and Technology of China,Chengdu 611731, China
    4 School of Information and Communication Engineering,University of Electronic Science and Technology of China,Chengdu 611731, China
  • Received:2026-05-15 Revised:2026-08-12 Published:2026-10-13
  • About author:DAI Ruiting,born in 1990,Ph.D candidate,engineer.Her main research in-terests include multimodal learning and machine reasoning.
    MO Lisi,born in 1986,master,senior engineer.His main research interests include network behavior and machine learning.
  • Supported by:
    National Natural Science Foundation of China for Young Scientists(62306064,62406057) and Sichuan Science and Technology Program(2024ZDZX0011).

PDF (PC)

80

Abstract: Incomplete multimodal learning(IML) relies on the complementary fusion of heterogeneous modalities such as vision,speech,and text.However,in real-world scenarios,modalities are often missing due to sensor failures,bandwidth constraints,or privacy restrictions,which severely weakens model robustness.Existing conditional diffusion-based recovery methods mainly perform generation in a single time domain,making it difficult to preserve global spectral structures and prone to locality bias.Meanwhile,static condition injection may cause condition over-dominance,suppressing the faithful representation of the missing moda-lity.To address these issues,this paper proposes a multi-stage time-spectral preserving diffusion(MT-SPD) model.Under joint time-frequency modeling,MT-SPD designs a three-stage generation strategy,including global structure construction,conditional spectrum alignment,and local detail refinement,to balance macroscopic consistency and local fidelity.Furthermore,a dual-path diffusion mechanism is introduced,where the intra-modal branch maintains shared spectral self-consistency and the missing-modality branch focuses on target reconstruction,with cross-guidance enabling intra-and inter-modal collaboration.Experimental results on multimodal benchmark datasets including IEMOCAP,CMU-MOSI,and CMU-MOSEI show that MT-SPD outperforms existing methods.In particular,it improves the F1 score by 3.92 percentage points over the second-best model in the {A,V} scenario on CMU-MOSI and demonstrates strong robustness under varying missing rates.

Key words: Multimodal intelligent perception, Missing modality recovery, Multi-source data fusion, Large-scale generative diffusion model, Time-frequency joint modeling, Cross-modal robust sensing

CLC Number: 

  • TP391.4
[1] KAZEMI V,ELQURSH A.Show,ask,attend,and answer:A strong baseline for visual question answering[J].arXiv:1704.03162,2017.
[2] ABDULLAH S M S A,AMEEN S Y A,SADEEQ M A M,et al.Multimodal emotion recognition using deep learning[J].Journal of Applied Science and Technology Trends,2021,2(1):73-79.
[3] NODA K,ARIE H,SUGA Y,et al.Multimodal integrationlearning of robot behavior using deep neural networks[J].Robotics and Autonomous Systems,2014,62(6):721-736.
[4] WANG H,WU Q,SHEN C.Soft expert reward learning for vision-and-language navigation[C]//European Conference on Computer Vision.Cham:Springer,2020:126-141.
[5] NI J,MUHLSTEIN L,MCAULEY J.Modeling heart rate andactivity data for personalized fitness recommendation[C]//The World Wide Web Conference.2019:1343-1353.
[6] WANG S,CAESAR H,NAN L,et al.UniBEV:Multi-modal 3Dobject detection with uniform BEV encoders for robustness against missing sensor modalities[C]//2024 IEEE Intelligent Vehicles Symposium.IEEE,2024:2776-2783.
[7] JANG J H,CHOI J,ROH H W,et al.Deep learning approachfor imputation of missing values in actigraphy data:Algorithm development study[J].JMIR mHealth and uHealth,2020,8(7):e16113.
[8] PAN Y,LIU M,XIA Y,et al.Disease-image-specific learning fordiagnosis-oriented neuroimage synthesis with incomplete multi-modality data[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2021,44(10):6839-6853.
[9] SAMIEE K,KOVÁCS P.ECG decomposition using cascadedspline projection residual auto encoders[C]//2023 Computing in Cardiology.IEEE,2023,50:1-4.
[10] LIAN Z,CHEN L,SUN L,et al.GCNet:Graph completion network for incomplete multimodal learning in conversation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2023,45(7):8419-8432.
[11] GONG R F,HUANG L Y,ZHU Q,et al.Graph learning based missing brain network generation and multimodal fusion diagnosis method[J].Journal of Data Acquisition and Processing,2024,39(4):843-862.
[12] WANG Y,LI Y,CUI Z.Incomplete multimodality-diffused emo-tion recognition[C]//Advances in Neural Information Proces-sing Systems.2023:17117-17128.
[13] TASHIRO Y,SONG J,SONG Y,et al.CSDI:Conditional score-based diffusion models for probabilistic time series imputation[C]//Advances in Neural Information Processing Systems.2021:24804-24816.
[14] ALCARAZ J L,STRODTHOFF N.Diffusion-based time series imputation and forecasting with structured state space models[J].arXiv:2208.09399,2022.
[15] SAHARIA C,CHAN W,CHANG H,et al.Palette:Image-to-image diffusion models[C]//ACM SIGGRAPH 2022 Confe-rence Proceedings.2022:1-10.
[16] ROMBACH R,BLATTMANN A,LORENZ D,et al.High-re-solution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:10684-10695.
[17] CHEN J,ZHANG A.HGMF:Heterogeneous graph-based fu-sion for multimodal data with incompleteness[C]//Proceedings of the 26th ACM SIGKDD International Conference on Know-ledge Discovery and Data Mining.2020:1295-1305.
[18] CAI L,WANG Z,GAO H,et al.Deep adversarial learning for multi-modality missing data completion[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.2018:1158-1166.
[19] NGIAM J,KHOSLA A,KIM M,et al.Multimodal deep learning[C]//Proceedings of the 28th International Conference on Machine Learning.2011:689-696.
[20] DING Z H,XIE J W,WANG B.Deep learning based FDA-MIMO radar covariance matrix missing data recovery method[J].Journal of Radars,2023,12(5):1112-1124.
[21] ENDERS C K.Applied Missing Data Analysis[M].New York:Guilford Press,2022.
[22] FU B,YU F,LIU A,et al.Generate like experts:Multi-stage font generation by incorporating font transfer process into diffusion models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2024:6892-6901.
[23] ZHOU J,LI J,ZHENG G,et al.MTSCI:A conditional diffusion model for multivariate time series consistent imputation[C]//Proceedings of the 33rd ACM International Conference on Information and Knowledge Management.2024:3474-3483.
[24] HO J,JAIN A,ABBEEL P.Denoising diffusion probabilisticmodels[C]//Advances in Neural Information Processing Systems.2023:6840-6851.
[25] LIU Q,LEE J,JORDAN M.A kernelized Stein discrepancy for goodness-of-fit tests[C]//International Conference on Machine Learning.PMLR,2016:276-284.
[26] SONG Y,ERMON S.Generative modeling by estimating gra-dients of the data distribution[C]//Advances in Neural Infor-mation Processing Systems.2019.
[27] SONG Y,SOHL-DICKSTEIN J,KINGMA D P,et al.Score-based generative modeling through stochastic differential equations[C]//International Conference on Learning Representations.2021.
[28] ZHENG S,CHAROENPHAKDEE N.Diffusion models formissing value imputation in tabular data[C]//NeurIPS 2022 First Table Representation Workshop.2022.
[29] CRABBÉ J,HUYNH N,STANCZUK J P,et al.Time series diffusion in the frequency domain[C]//Proceedings of the 41st International Conference on Machine Learning.PMLR,2024,235:9407-9438.
[30] ZHANG H,WANG W,YU T.Towards robust multimodal sentiment analysis with incomplete data[C]//Advances in Neural Information Processing Systems.2024.
[31] DEVLIN J,CHANG M W,LEE K,et al.BERT:Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics.2019:4171-4186.
[32] RAHMAN W,HASAN M K,LEE S,et al.Integrating multi-modal information in large pretrained transformers[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.2020:2359-2369.
[33] LAMBA H,MATTINGLY J C,STUART A M.An adaptive Euler-Maruyama scheme for SDEs:Convergence and stability[J].IMA Journal of Numerical Analysis,2007,27(3):479-506.
[34] HYVÄRINEN A,DAYAN P.Estimation of non-normalized sta-tistical models by score matching[J].Journal of Machine Lear-ning Research,2005,6(4):695-709.
[35] BUSSO C,BULUT M,LEE C,et al.IEMOCAP:Interactiveemotional dyadic motion capture database[J].Language Resources and Evaluation,2008,42(4):335-359.
[36] ZADEH A,ZELLERS R,PINCUS E,et al.Multimodal sentiment intensity analysis in videos:Facial gestures and verbal messages[J].IEEE Intelligent Systems,2016,31(6):82-88.
[37] ZADEH A B,LIANG P P,PORIA S,et al.Multimodal language analysis in the wild:CMU-MOSEI dataset and interpretable dynamic fusion graph[C]//Proceedings of the 56th Annual Mee-ting of the Association for Computational Linguistics.2018:2236-2246.
[38] ZENG J,LIU T,ZHOU J.Tag-assisted multimodal sentimentanalysis under uncertain missing modalities[C]//Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval.2022:1545-1554.
[39] LIN R,HU H.MissModal:Increasing robustness to missingmodality in multimodal sentiment analysis[J].Transactions of the Association for Computational Linguistics,2023,11:1686-1702.
[40] GUO Z,JIN T,ZHAO Z.Multimodal prompt learning withmissing modalities for sentiment analysis and emotion recognition[C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics.2024:1726-1736.
[41] WANG Y,CUI Z,LI Y.Distribution-consistent modal recovering for incomplete multimodal learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2023:21968-21977.
[42] KE G,HE S,WANG X L,et al.Knowledge bridger:Towardstraining-free missing multi-modality completion[J].arXiv:2502.19834,2025.
[43] STÖCKLI S,SCHULTE-MECKLENBECK M,BORER S,et al.Facial expression analysis with AFFDEX and FACET:A validation study[J].Behavior Research Methods,2018,50(4):1446-1460.
[44] DEGOTTEX G,KANE J,DRUGMAN T,et al.COVAREP－A collaborative voice analysis repository for speech technologies[C]//2014 IEEE International Conference on Acoustics,Speech and Signal Processing.IEEE,2014:960-964.
[1] HUO Tian-yuan, GU Jing-jing. Dynamic and Static Relationship Fusion of Multi-source Health Perception Data for Disease Diagnosis [J]. Computer Science, 2022, 49(11A): 211100241-9.
[2] JU Chun-hua, ZOU Jiang-bo, FU Xiao-kang. Design and Application of Big Data Credit Reporting Platform Integrating Blockchain Technology [J]. Computer Science, 2018, 45(11A): 522-526.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.