Abstract: Spoken English occupies a crucial position in English learning.Addressing the scarcity of existing datasets for evaluating emotional expression in spoken English and the inadequate utilization of multimodal information,this paper introduces a novel dataset named the English spoken multimodal emotion dataset(ESMED).This dataset is annotated with continuous emotions(arousal,valence) and emotional quality scores.Additionally,an innovative network model for evaluating spoken English emotions is proposed.The model initially compresses and fuses continuous emotional information through perception resampling and multimodal fusion modules to predict arousal and valence.Subsequently,it performs specific transformations on the features through learnable bottleneck and joint decoding layers.The emotional quality evaluation module then jointly decodes arousal,valence,and transformed features to obtain the final quantified emotional quality score.Experimental results demonstrate that the proposed model achieves a concordance correlation coefficient(CCC) of 0.500 3 and a mean absolute error(MAE) of 0.635 4 on the ESMED dataset,verifying the effectiveness and accuracy of the proposed method.

Key words: Emotion recognition, Perceiver resampling, Multimodal fusion, Joint decoding, Emotion quality evaluation