计算机科学

计算机科学 ›› 2026, Vol. 53 ›› Issue (9): 291-298.doi: 10.11896/jsjkx.250900090

• 人工智能 • 上一篇    下一篇

基于跨模态一致性约束的多模态概率生成模型

陈亚瑞1, 洪乐涵1, 邵建林1, 杨剑宁2, 廖芸1, 史艳翠1   

  1. 1 天津科技大学人工智能学院 天津 300457
    2 西安邮电大学通信与信息工程学院 西安 710121
  • 收稿日期:2025-09-15 修回日期:2025-11-28 出版日期:2026-09-15 发布日期:2026-09-10
  • 通讯作者: 史艳翠(syc@tust.edu.cn)
  • 作者简介:(yrchen@tust.edu.cn)
  • 基金资助:
    国家自然科学基金(61976156)

Multimodal Probabilistic Generative Model Based on Cross-modal Consistency Constraints

CHEN Yarui1, HONG Lehan1, SHAO Jianlin1, YANG Jianning2, LIAO Yun1, SHI Yancui1   

  1. 1 College of Artificial Intelligence,Tianjin University of Science and Technology,Tianjin 300457,China
    2 School of Communications and Information Engineering,Xi'an University of Posts and Telecommunications,Xi'an 710121,China
  • Received:2025-09-15 Revised:2025-11-28 Published:2026-09-15 Online:2026-09-10
  • About author:CHEN Yarui,born in 1982,Ph.D,associate professor,is a member of CCF(No.47164S).Her main research in-terests include deep learning,image processing and video processing.
    SHI Yancui,born in 1982,Ph.D,asso-ciate professor,is a member of CCF(No.39889M).Her main research interests include Internet of Things applications,recommendation system and social network analysis.
  • Supported by:
    National Natural Science Foundation of China(61976156).

PDF (PC)

271

摘要： 随着跨模态生成与理解任务的不断发展,模态间一致性问题已成为多模态深度概率生成模型研究中的关键挑战。针对此问题,提出了基于跨模态一致性约束的多模态概率生成模型(CCVAE)。该模型将隐空间解耦为模态共享与私有隐空间,利用专家积机制融合各模态共享信息。在共享隐空间中,通过对齐约束,在无监督条件下利用对比学习引导各模态共享表示对齐,缩小不同模态的分布差异;设计维度方差约束,抑制共享空间各维度均值的模态差异,实现模态间的特征结构对齐,提升跨模态共享表示的稳定性与一致性。在多个多模态数据集上进行了对比实验,结果表明,CCVAE在跨模态数据交叉生成与转换生成任务中表现优越,显著提升了生成结果的质量。隐向量的可视化分析进一步验证了该模型在共享表示对齐及跨模态判别能力方面的有效性。

关键词: 多模态表示学习, 变分自编码器, 解耦, 生成模型, 无监督学习

Abstract: With the continuous development of cross-modal generation and understanding tasks,the issue of inter modal consistency has become a key challenge in the research of multimodal deep probability generation models.To address this issue,this paper proposes a multimodal probability generation model(CCVAE) based on cross-modal consistency constraints.This model decouples the hidden space into modal sharing and private hidden space,and uses the expert product mechanism to fuse the shared information of each modality.In the shared latent space,unsupervised alignment constraints are used to guide the alignment of shared representations among different modalities through contrastive learning,reducing the distribution differences between different modalities.Dimensional variance constraints are designed to suppress modal differences in the mean values of various dimensions in the shared space,achieve feature structure alignment between modalities,and improve the stability and consistency of cross-modal shared representations.Comparative experimental results on multiple multimodal datasets show that CCVAE performs superior in cross-modal data cross generation and transformation generation tasks,significantly improving the quality of generated results.The visualization analysis of latent vectors further validates the effectiveness of this model in shared representation alignment and cross-modal discriminative ability.

Key words: Multimodal representation learning, Variational autoencoder, Decoupling, Generate models, Unsupervised learning

中图分类号: 

  • TP183
[1] BALTRUŠAITIS T,AHUJA C,MORENCY L P.Multimodal machine learning:a survey and taxonomy[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2018,41(2):423-443.
[2] SUZUKI M,MATSUO Y.A survey of multimodal deep generative models[J].Advanced Robotics,2022,36(5／6):261-278.
[3] LIU H,YU S,HOU Q,et al.Cross-Modality Image Transformation Using Generative Adversarial Network[C] //International Conference on Man-Machine-Environment System Engineering.Singapore:Springer,2024:463-468.
[4] ZHU Z,LI Y,LYU W,et al.Consistent multimodal generation via a unified GAN framework[C] //Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.2024:5048-5057.
[5] MENG D,TZELEPIS C,PATRAS I,et al.MM2Latent:Text-to-facial image generation and editing in GANs with multimodal assistance[C] //European Conference on Computer Vision.Cham:Springer,2025:88-106.
[6] ZHANG H,KOH J Y,BALDRIDGE J,et al.Cross-modal con-trastive learning for text-to-image generation[C] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:833-842.
[7] ZHOU D,ZHANG H,MA J,et al.BC-GAN:A Generative Adversarial Network for Synthesizing a Batch of Collocated Clo-thing[J].IEEE Transactions on Circuits and Systems for Video Technology,2023,34(5):3245-3259.
[8] SUZUKI M,NAKAYAMA K,MATSUO Y.Joint multimodal learning with deep generative models[J].arXiv:1611.01891,2016.
[9] WU M,GOODMAN N.Multimodal generative models for scal-able weakly-supervised learning[C] //Advances in Neural Information Processing Systems.2018.
[10] SHI Y,PAIGE B,TORR P.Variational mixture-of-experts autoencoders for multi-modal deep generative models[C] //Advances in Neural Information Processing Systems.2019.
[11] LEE M,PAVLOVIC V.Private-shared disentangled multimodal vae for learning of latent representations[C] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:1692-1700.
[12] SUTTER T M,MENG Y,AGOSTINI A,et al.Unity by diversity:improved representation learning in multimodal vaes[J].arXiv:2403.05300,2024.
[13] DAUNHAWER I,SUTTER T M,MARCINKEVIČS R,et al.Self-supervised disentanglement of modality-specific and shared factors improves multimodal generative models[C] //DAGM German Conference on Pattern Recognition.Cham:Springer,2020:459-473.
[14] VAN DEN OORD A,LI Y,VINYALS O.Representation lear-ning with contrastive predictive coding[J].arXiv:1807.03748,2018.
[15] PALUMBO E,DAUNHAWER I,VOGT J E.MMVAE+:Enhancing the generative quality of multimodal VAEs without compromises[C] //The Eleventh International Conference on Learning Representations.OpenReview,2023.
[16] MONDAL A K,SAILOPAL A,SINGLA P,et al.SSDMM-VAE:variational multi-modal disentangled representation learning[J].Applied Intelligence,2023,53(7):8467-8481.
[17] MÄRTENS K,YAU C.Disentangling shared and private latent factors in multimodal Variational Autoencoders[C] //Machine Learning in Computational Biology.PMLR,2024:60-75.
[18] ZHOU X,MIAO C.Disentangled graph variational auto-encoder for multimodal recommendation with interpretability[J].IEEE Transactions on Multimedia,2024,26:7543-7554.
[19] MAHAJAN S,BOTSCHEN T,GUREVYCH I,et al.Joint wasserstein autoencoders for aligning multimodal embeddings[C] //Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops.2019.
[20] JIANG Q,CHEN C,ZHAO H,et al.Understanding and constructing latent modality structures in multi-modal representation learning[C] //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2023:7661-7671.
[21] MOLINO D,DI FEOLA F,SHEN L,et al.Any-to-Any Vision-Language Model for Multimodal X-ray Imaging and Radiological Report Generation[J].arXiv:2505.01091,2025.
[22] RADFORD A,KIM J W,HALLACY C,et al.Learning transferable visual models from natural language supervision[C] //International Conference on Machine Learning.PMLR,2021:8748-8763.
[23] JIA C,YANG Y,XIA Y,et al.Scaling up visual and vision-language representation learning with noisy text supervision[C] //International Conference on Machine Learning.PMLR,2021:4904-4916.
[24] CHEN Y C,LI L,YU L,et al.Uniter:Universal image-text representation learning[C] //European Conference on Computer Vision.Cham:Springer,2020:104-120.
[25] GAN Z,CHEN Y C,LI L,et al.Large-scale adversarial training for vision-and-language representation learning[J].Advances in Neural Information Processing Systems,2020,33:6616-6628.
[26] FAGHRI F,FLEET D J,KIROS J R,et al.Vse++:Improving visual-semantic embeddings with hard negatives[J].arXiv:1707.05612,2017.
[27] LI J,SELVARAJU R,GOTMARE A,et al.Align before fuse:Vision and language representation learning with momentum distillation[J].Advances in Neural Information Processing Systems,2021,34:9694-9705.
[28] GONZALEZ-GARCIA A,VAN DE WEIJER J,BENGIO Y.Image-to-image translation for cross-domain disentanglement[J].arXiv:1805.09730,2018.
[29] SUTTER T M,DAUNHAWER I,VOGT J E.Generalized multimodal ELBO[J].arXiv:2105.02470,2021.
[30] NETZER Y,WANG T,COATES A,et al.Reading digits in na-tural images with unsupervised feature learning[C] //NIPS Workshop on Deep Lear-ning and Unsupervised Feature Lear-ning.2011:4.
[31] QIU P,ZHU W,KUMAR S,et al.Multimodal Variational Autoencoder:A Barycentric View[C] //Proceedings of the AAAI Conference on Artificial Intelligence.2025:20060-20068.
[32] VAN DER MAATEN L,HINTON G.Visualizing data using t-SNE[J].Journal of Machine Learning Research,2008,9(11):2579-2605.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市两江新区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发