Abstract: Aiming at the limitations such as single modality of information source,poor anti-interference,high computational cost,and small attention to temporal features in the field of emotion recognition,this paper propose a hybrid emotion recognition method FB-TimesNet based on the improvement of TimesNet for facial expression and body gesture.Firstly,the human body key-point coordinates of the video frames were extracted,and the facial key-point coordinates were used as the original information features respectively,with the change value of facial key-point coordinates relative to the natural state of the change values of facial keypoint coordinates relative to the natural state,and body posture keypoint coordinates as the raw information features of facial expression and body posture,thus reducing the dimensionality of the data and computational cost.Second,the periodic changes of the input data were captured using the fast Fourier variation,which transforms the one-dimensional data into two-dimensional data,and then two-dimensional convolution kernels were used to encode and extract spatio-temporal features for the two sets of features separately to enhance the characterization ability of the data.Finally,the fusion algorithm was used to dynamically allocate the weights of each modality to obtain the best fusion effect.In this paper,extensive comparative experiments have been conducted on two common sentiment datasets,and the experimental results show that FB-TimesNe improves the classification accuracy by 4.89% compared to the baseline model on the BRED dataset.

Key words: Video emotion recognition, Spatio-temporal features, Expression recognition, Body posture, Multimodal feature fusion