Abstract: With the continuous development of cross-modal generation and understanding tasks,the issue of inter modal consistency has become a key challenge in the research of multimodal deep probability generation models.To address this issue,this paper proposes a multimodal probability generation model(CCVAE) based on cross-modal consistency constraints.This model decouples the hidden space into modal sharing and private hidden space,and uses the expert product mechanism to fuse the shared information of each modality.In the shared latent space,unsupervised alignment constraints are used to guide the alignment of shared representations among different modalities through contrastive learning,reducing the distribution differences between different modalities.Dimensional variance constraints are designed to suppress modal differences in the mean values of various dimensions in the shared space,achieve feature structure alignment between modalities,and improve the stability and consistency of cross-modal shared representations.Comparative experimental results on multiple multimodal datasets show that CCVAE performs superior in cross-modal data cross generation and transformation generation tasks,significantly improving the quality of generated results.The visualization analysis of latent vectors further validates the effectiveness of this model in shared representation alignment and cross-modal discriminative ability.

Key words: Multimodal representation learning, Variational autoencoder, Decoupling, Generate models, Unsupervised learning