Abstract: In the process of learning the overall loss of multimodal representations,the dependence of reconstruction loss on the model is relatively less,resulting in hidden representations that cannot effectively capture the details of their respective modalities.This paper proposes a multi-subspace sentiment analysis framework.Firstly,the framework projects each modality to two distinct utterance representations:modality-invariant and modality-specific.We construct the main shared subspace and the auxi-liary shared subspace that helps the main subspace to reduce the modality gap in the modality-invariant representation.Also,construct the private subspaces in the modality-specific representation to capture the characteristic features of each modality.We take the hidden vectors in all subspaces as the input of the decoder function and reconstruct the modal vector to achieve optimization of reconstruction loss.Secondly,in the fusion procedure,we perform a multi-headed self-attention based on Transformer on these representations,so that each cross-modal representation can induce potential information from fellow representations that have a synergistic effect on the overall emotional orientation.Finally,we construct a joint-vector by using concatenation and use fully connected layers to generate task predictions.Experimental results on both MOSI and MOSEI datasets show that the proposed framework outperforms the baselines in most evaluation criteria.

Key words: Multimodal representation, Sentiment analysis, Transformer, Self-attention, Cross-modality