Abstract: The goal of multimodal sentiment analysis is to perceive and understand human emotions through various heteroge-neous modalities,such as language,video,and audio.However,there are complex correlations between different modalities.Most existing methods directly fuse multiple modality features,they overlook the fact that asynchronous modality fusion representations contribute differently to sentiment analysis.To address the above issues,this paper proposes a sentiment classification me-thod based on stepwise collaborative fusion representation.Firstly,a denoising bottleneck model is used to filter out noise and redundancy in the audio and video,and the two modalities are fused through Transformer,establishing a low-level feature representation of the audio-video fusion.Then,a cross-modal attention mechanism is utilized to enhance the audio-video modalities with the text modality,constructing a high-level feature representation of the audio-video fusion.Secondly,a novel multimodal fusion layer is designed to incorporate multi-level feature representations into the pre-trained T5 model,establishing a text-centric multimodal fusion representation.Finally,the low-level feature representation,high-level feature representation,and text-centric feature fusion representation are combined to achieve sentiment classification of multimodal data.Experimental results on two public datasets,CMU-MOSI and CMU-MOSEI indicate that the proposed method improves the Acc-7 metric by 0.1 and 0.17 compared to the existing baseline model ALMT,demonstrating that stepwise collaborative fusion representation can enhance multimodal sentiment classification performance.

Key words: Multimodal fusion, Sentiment analysis, Bottleneck mechanism, Attention mechanism, Pre-trained model