Abstract: The key challenge in audio-driven digital face video generation lies in aligning the information from two different modalities,audio and video,to achieve lip synchronization.Existing technologies have primarily been developed using English datasets.However,due to the phonetic differences between Chinese and English,directly applying these methods to Chinese audio-driven face video generation results in issues such as blurred teeth and insufficient video clarity.This paper proposes M-CSAWav2Lip,an audio-driven digital face video generation method based on a GAN framework and enhanced by an attention mechanism.The method combines MFCC and Mel Spectrograms for audio feature extraction.By leveraging the temporal dynamics of MFCC and the frequency resolution of Mel Spectrograms,the method captures subtle variations in speech information comprehensively.During the digital face generation process,a network architecture based on attention mechanisms and residual connections is employed.This architecture uses weighted channel and spatial attention mechanisms to enhance the importance of features,improving the ability to extract key audio and video features.This allows for the effective encoding and fusion of Chinese audio-video information,generating lip movements and facial videos that are consistent with the audio content.Finally,the model is trained and tested on both a custom Chinese dataset and a general dataset.Experimental results demonstrate that the generated lip-synced digital face videos show improvements in both accuracy and quality.

Key words: Audio-driven, Lip synchronization, Audio feature extraction, Digital face generation, Attention mechanism