计算机科学

计算机科学 ›› 2026, Vol. 53 ›› Issue (10): 198-206.doi: 10.11896/jsjkx.250900045

• 计算机图形学 & 多媒体 • 上一篇    下一篇

基于4D Gaussian Splatting的语音驱动数字人脸生成方法

邵明慧, 鹿浩然, 王国栋   

  1. 青岛大学计算机科学技术学院 山东 青岛 266071
  • 收稿日期:2025-09-05 修回日期:2025-11-23 发布日期:2026-10-13
  • 通讯作者: 王国栋(doctorwgd@gmail.com)
  • 作者简介:(smhofficial@163.com)
  • 基金资助:
    青岛市自然科学基金(23-2-1-163-zyyd-jch);青岛大学纺织+联合研究项目(FZ2024101)

4D Gaussian Splatting for Speech-driven Digital Human Head Avatar Generation

SHAO Minghui, LU Haoran, WANG Guodong   

  1. College of Computer Science and Technology,Qingdao University,Qingdao,Shandong 266071, China
  • Received:2025-09-05 Revised:2025-11-23 Online:2026-10-13
  • About author:SHAO Minghui,born in 2000,postgra-duate.Her main research interests include 3DGS and 3D human face reconstruction.
    WANG Guodong,born in 1980,Ph.D,professor,is a senior member of CCF(No.16234M).His main research interests include computer graphics and artificial intelligence.
  • Supported by:
    Qingdao Natural Science Foundation(23-2-1-163-zyyd-jch) and Qingdao University Textile Plus Joint Research Program(FZ2024101).

PDF (PC)

95

摘要： 在语音驱动数字人的头部生成中,面部和躯干动作的自然性与逼真度问题尤为重要。为了解决这一问题,提出了一种基于4D Gaussian Splatting(4DGS)和可学习位置扰动模块的语音驱动数字人脸视频生成方法,采用4DGS代替3D Gaussian Splatting(3DGS),在时空建模中引入时间维度,实现了对动态面部表情的精确建模和高效渲染。同时,引入多分辨率六平面位置编码,对三维空间特征和时间信息进行高效编码,以捕捉人脸细节的多层次信息,并通过解码为高斯属性而非颜色和密度,实现了高效的高斯渲染。该方法首次将多分辨率六平面编码面向语音驱动场景进行重构,并针对性设计了一种可学习的位置扰动模块,该模型能够自适应调整位置编码,确保语音驱动下面部表情的精确同步与自然还原。实验结果表明,该方法在面部动作同步性、表情细节还原以及渲染效率方面均表现出色,能够在动态场景中高效捕捉复杂的时空变化和面部表情的细节变化。

关键词: 4DGS, 语音驱动数字人, 人脸重建, 多分辨率六平面

Abstract: In speech-driven digital human head avatar generation,the naturalness and realism of facial and torso movements are particularly important.To address this issue,this paper proposes a speech-driven digital human video generation method based on 4DGS(4D Gaussian Splatting) and a learnable position perturbation module.By replacing 3D Gaussian Splatting(3DGS) with 4DGS,the method introduces a temporal dimension in the spatiotemporal modeling,enabling precise modeling and efficient rendering of dynamic facial expressions.Additionally,a multi-resolution six-plane positional encoding is introduced to efficiently encode both 3D spatial features and temporal information,capturing multi-level details of the face and achieving efficient Gaussian rendering by decoding into Gaussian attributes instead of color and density.This method is the first to reconstruct multi-resolution six-plane encoding for speech-driven scenarios and specifically designs a learnable position perturbation module.The model can adaptively adjust the position encoding,ensuring accurate synchronization and natural restoration of facial expressions under speech driving.Experimental results show that this method excels in facial action synchronization,expression detail restoration,and rendering efficiency,efficiently capturing complex spatiotemporal changes and subtle variations in facial expressions in dynamic scenes.

Key words: 4DGS, Speech-driven digital human, Head avatar, Multi-resolution six-plane

中图分类号: 

  • TP391
[1] SONG X Y,YAN Z Y,SUN M Y,et al.Review of Talking Face Generation[J].Computer Science,2023,50(8):68-78.
[2] MILDENHALL B,SRINIVASAN P P,TANCIK M,et al.NeRF:representing scenes as neural radiance fields for view synthesis[J].Communications of the ACM,2021,65(1):99-106.
[3] CHAKKERA S T R,CHATZIAGAPI A,SAMARAS D.JEAN:Joint Expression and Audio-guided NeRF-based Talking Face Generation[J].arXiv:2409.12156,2024.
[4] GUO Y,CHEN K,LIANG S,et al.AD-NeRF:Audio Driven Neural Radiance Fields for Talking Head Synthesis[C]//2021 IEEE/CVF International Conference on Computer Vision(ICCV).Montreal,QC,Canada:IEEE,2021:5764-5774.
[5] GUO L,NING X,FU Y,et al.Rad-NeRF:Ray-decoupled Training of Neural Radiance Field[J].Advances in Neural Information Processing Systems,2024,37:113742-113771.
[6] KIM G,SEO K,CHA S,et al.NeRFFaceSpeech:One-shot Audio-driven 3D Talking Head Synthesis via Generative Prior[J].arXiv:2405.05749,2024.
[7] WANG Y,SKOROKHODOV I,WONKA P.PET-NeuS:Positional Encoding Tri-Planes for Neural Surfaces[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).Vancouver,BC,Canada:IEEE,2023:12598-12607.
[8] DU C,CHEN Q,HE T,et al.DAE-Talker:High FidelitySpeech-Driven Talking Face Generation with Diffusion Autoencoder[C]//Proceedings of the 31st ACM International Confe-rence on Multimedia.Ottawa ON Canada:ACM,2023:4281-4289.
[9] CHEN X,MIHAJLOVIC M,WANG S,et al.Morphable Diffusion:3D-Consistent Diffusion for Single-image Avatar Creation[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).Seattle,WA,USA:IEEE,2024:10359-10370.
[10] KIRSCHSTEIN T,GIEBENHAIN S,NIESSNER M.DiffusionAvatars:Deferred Diffusion for High-fidelity 3D Head Avatars[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).Seattle,WA,USA:IEEE,2024:5481-5492.
[11] SHEN S,ZHAO W,MENG Z,et al.DiffTalk:Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).Vancouver,BC,Canada:IEEE,2023:1982-1991.
[12] LI J,ZHANG J,BAI X,et al.Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis[C]//2023 IEEE/CVF International Conference on Computer Vision(ICCV).Paris,France:IEEE,2023:7534-7544.
[13] WU G,YI T,FANG J,et al.4D Gaussian Splatting for Real-Time Dynamic Scene Rendering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2024:20310-20320.
[14] WEN B,XIE H,CHEN Z,et al.3D Scene Generation:A Survey[J].arXiv:2505.05474,2025.
[15] CHEN B,HU S,CHEN Q,et al.GSTalker:Real-time Audio-Driven Talking Face Generation via Deformable Gaussian Splatting[J].arXiv:2404.19040,2024.
[16] CHO K,LEE J,YOON H,et al.GaussianTalker:Real-TimeHigh-Fidelity Talking Head Synthesis with Audio-Driven 3D Gaussian Splatting[J].arXiv:2404.16012,2024.
[17] LI J,ZHANG J,BAI X,et al.TalkingGaussian:Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting[C]//European Conference on Computer Vision.Cham:Springer Nature Switzerland,2024:127-145.
[18] ZHOU Z,FENG Q,LI H.SynGauss:Real-Time 3D Gaussian Splatting for Audio-Driven Talking Head Synthesis[J].IEEE Access,2025,13:42167-42177.
[19] FRIDOVICH-KEIL S,MEANTI G,WARBURG F R,et al.K-Planes:Explicit Radiance Fields in Space,Time,and Appearance[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).Vancouver,BC,Canada:IEEE,2023:12479-12488.
[20] SCHNEIDER S,BAEVSKI A,COLLOBERT R,et al.wav2vec:Unsupervised Pre-training for Speech Recognition[J].arXiv:1904.05862,2019.
[21] HANNUN A,CASE C,CASPER J,et al.Deep Speech:Scaling up end-to-end speech recognition[J].arXiv:1412.5567,2024.
[22] HSU W N,BOLTE B,TSAI Y H H,et al.HuBERT:Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units[J].IEEE/ACM Transactions on Audio,Speech,and Language Processing,2021,29:3451-3460.
[23] LI Z D,ZHU Z X,ZHAO S H,et al.A review of digital human technology:modeling methods and driving strategies[J].Journal of Jilin University(Engineering and Technology Edition),2026,56(2):289-312.
[24] LIU D,WANG Z,CHEN P.DSEM-NeRF:Multimodal feature fusion and global-local attention for enhanced 3D scene reconstruction[J].Information Fusion,2025,115:102752.
[25] YE Z,JIANG Z,REN Y,et al.GeneFace:Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis[J].arXiv:2301.13430,2023.
[26] KERBL B,KOPANAS G,LEIMKÜHLER T,et al.3D Gaussian Splatting for Real-Time Radiance Field Rendering[J].ACM Transactions on Graphics,2023,42(4):139:1-139:14.
[27] CHAN E R,LIN C Z,CHAN M A,et al.Efficient Geometry-aware 3D Generative Adversarial Networks[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).New Orleans,LA,USA:IEEE,2022:16102-16112.
[28] CAO A,JOHNSON J.HexPlane:A Fast Representation for Dynamic Scenes[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2023:130-141.
[29] ZHANG R,ISOLA P,EFROS A A,et al.The Unreasonable Effectiveness of Deep Features as a Perceptual Metric[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:586-595.
[30] WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[31] QI C R,SU H,MO K,et al.PointNet:Deep Learning on Point Sets for 3D Classification and Segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:652-660.
[32] SCHONBERGER J L,FRAHM J M.Structure-From-MotionRevisited[C]//Proceedings of the IEEE Conference on Compu-ter Vision and Pattern Recognition.2016:4104-4113.
[33] BALTRUSAITIS T,ROBINSON P,MORENCY L P.Open-Face:An open source facial behavior analysis toolkit[C]//2016 IEEE Winter Conference on Applications of Computer Vision(WACV).Lake Placid,NY,USA:IEEE,2016:1-10.
[34] BALTRUŠAITIS T,MAHMOUD M,ROBINSON P.Cross-dataset learning and person-specific normalisation for automatic Action Unit detection[C]//2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition(FG).IEEE,2015,6:1-6.
[35] PRAJWAL K R,MUKHOPADHYAY R,NAMBOODIRI V P,et al.A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild[C]//Proceedings of the 28th ACM International Conference on Multimedia.New York,NY,USA:Association for Computing Machinery,2020:484-492.
[36] HEUSEL M,RAMSAUER H,UNTERTHINER T,et al.GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium [C]//Proceedings of the 31st International Conference on Neural Information Processing Systems.2017:6629-6640.
[37] CHUNG J S,ZISSERMAN A.Out of Time:Automated Lip Sync in the Wild[C]//Computer Vision-ACCV 2016 Workshops.Cham:Springer International Publishing,2017:251-263.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市两江新区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发