计算机科学

计算机科学 ›› 2022, Vol. 49 ›› Issue (2): 40-50.doi: 10.11896/jsjkx.210500215

• 计算机视觉:理论与应用 • 上一篇    下一篇

基于深度学习的单幅图像三维人脸重建研究综述

何嘉玉1, 黄宏博1, 张红艳1, 孙牧野1, 刘亚辉2, 周哲海3   

  1. 1 北京信息科技大学计算机学院 北京100192
    2 北京信息科技大学信息管理学院 北京100192
    3 北京信息科技大学仪器科学与光电工程学院 北京100192
  • 收稿日期:2021-05-29 修回日期:2021-06-30 出版日期:2022-02-15 发布日期:2022-02-23
  • 通讯作者: 黄宏博(hhb@bistu.edu.cn)
  • 作者简介:hejiayu_97@163.com
  • 基金资助:
    国家自然科学基金重点项目(61931003);北京市长城学者计划(CIT&TCD20190323);北京市教委科技计划一般项目(KM201811232024)

Review of 3D Face Reconstruction Based on Single Image

HE Jia-yu1, HUANG Hong-bo1, ZHANG Hong-yan1, SUN Mu-ye1, LIU Ya-hui2, ZHOU Zhe-hai3   

  1. 1 School of Computing,Beijing Information Science and Technology University,Beijing 100192,China
    2 School of Information Management,Beijing Information Science and Technology University,Beijing 100192,China
    3 School of Instrument Science and Optoelectronic Engineering,Beijing Information Science and Technology University,Beijing 100192,China
  • Received:2021-05-29 Revised:2021-06-30 Online:2022-02-15 Published:2022-02-23
  • About author:HE Jia-yu,born in 1997,postgraduate.Her main research interests include 3D face reconstruction and face recognition.
    HUANG Hong-bo,born in 1976,Ph.D,associate professor.His main research interests include computer vision,deep learning and optimization theory.
  • Supported by:
    Key Program of the National Natural Science Foundation of China(61931003),Great Wall Scholar Program of Beijing(CIT&TCD20190323) and Beijing Education Commission for General Project of Science and Technology Plan(KM201811232024).

PDF (PC)

2006

摘要： 在计算机视觉领域中,三维人脸重建是一个具有研究价值的方向,高质量地重建出三维人脸在人脸识别、防伪、游戏娱乐、影视动画和美容医疗等领域具有重要的意义。近二十年来,虽然基于单幅图像的三维人脸重建领域已经取得很大的进展,但使用传统算法进行重建的结果仍会受到人脸表情、遮挡、环境光的影响,并且会出现重建效果精度不佳和鲁棒性不足等问题。随着深度学习进入三维人脸重建领域,各种优于传统重建算法的方法相继出现。文中首先重点介绍了基于深度学习的单幅图像三维人脸重建算法,将算法按不同的网络架构分为4类,并对各类最具有代表性的方法进行了详细阐述。然后汇总了基于单幅图像的三维人脸重建算法常用的三维人脸数据集,并在数据集上对具有代表性的方法进行了性能评估。最后对基于单幅图像的三维人脸重建领域进行了总结与展望。

关键词: 卷积神经网络, 人脸识别, 三维可变形模型, 三维人脸重建, 深度学习

Abstract: In the field of computer vision,3D face reconstruction is a valuable research direction.High quality reconstruction of 3D faces can find applications in face recognition,anti-proofing,animation and medical cosmetology.In the last two decades,although great progress has been made 3D face reconstruction based on a single image,the results of reconstruction using traditionalalgorithms are still facing the challenge of facial expression,occlusion and ambient light,and there will be problems such as poor reconstruction accuracy and robustness.With the rapid development of deep learning in 3D face reconstruction,various methods which are superior to traditional reconstruction algorithms have emerged.Firstly,this paper focuses on deep-learning-based reconstruction algorithms.The algorithms are divided into four categories according to different network architecture,and the most popular methods are described in detail.Then commonly used 3D face data sets are introduced,and performance of representative methods are evaluated.Finally,conclusions and prospects of the single-image-based 3D face reconstruction are given.

Key words: 3D face reconstruction, 3D morphable model, Convolutional neural networks, Deep learning, Face recognition

中图分类号: 

  • TP391
[1]BLANZ V,VETTER T.A Morphable Model for the Synthesis of 3D Faces[C]//Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques.ACM Press,1999:187-194.
[2]PAYSAN P,KNOTHE R,AMBERG B,et al.A 3D Face Model for Pose and Illumination Invariant Face Recognition[C]//2009 sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.IEEE,2009:296-301.
[3]BOOTH J,ROUSSOS A,PONNIAH A,et al.Large scale3D morphable models[J].International Journal of Computer Vision,2018,126(2):233-254.
[4]CAO C,WENG Y,ZHOU S,et al.Facewarehouse:a 3D facial expression database for visual computing[J].IEEE Transactions on Visualization and Computer Graphics,2013,20(3):413-425.
[5]BOOTH J,ANTONAKOS E,PLOUMPIS S,et al.3D FaceMorphable Models “In-The-Wild” [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2017:48-57.
[6]VLASIC D,BRAND M,PFISTER H,et al.Face transfer with multilinear models[J].ACM Transactions on Graphics (TOG),2005,24(3):426-433.
[7]YANG H,ZHU H,WANG Y,et al.FaceScape:A Large-ScaleHigh Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2020:601-610.
[8]KIM H,ZOLLHOFER M,TEWARI A,et al.Inversefacenet:Deep Monocular Inverse Face Rendering[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2018:4625-4634.
[9]RICHARDSON E,SELA M,OR-EL R,et al.Learning Detailed Face Reconstruction From a Single Image[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2017:1259-1268.
[10]FAN X,CHENG S,KANG H,et al.Dual neural networks coupling data regression with explicit priors for monocular 3D face reconstruction[J/OL].IEEE Transactions on Multimedia,2020.https://www.researchgate.net/publication/341364310_Dual_Neural_Networks_Coupling_Data_Regression_With_Explicit_Priors_for_Monocular_3D_Face_Reconstruction.
[11]LUO Y,TU X,XIE M.Learning Robust 3D Face Reconstruc-tion and Discriminative Identity Representation[C]//2019 IEEE 2nd International Conference on Information Communication and Signal Processing (ICICSP).IEEE,2019:317-321.
[12]DOU P,SHAH S K,KAKADIARIS I A.End-to-end 3D FaceReconstruction with Deep Neural Networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2017:5908-5917.
[13]ZHU X,LEI Z,LIU X,et al.Face Alignment Across LargePoses:A 3D Solution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2016:146-155.
[14]ZHANG G,HAN H,SHAN S,et al.Face Alignment AcrossLarge Pose Via MT-CNN Based 3D Shape Reconstruction [C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).IEEE,2018:210-217.
[15]XING Y,TEWARI R,MENDONCA P.A Self-Supervised Bootstrap Method for Single-Image 3D Face Reconstruction[C]//2019 IEEE Winter Conference on Applications of Computer Vision (WACV).IEEE,2019:1014-1023.
[16]TRAN A T,HASSNER T,MASI I,et al.Regressing Robustand Discriminative 3D Morphable Models with a Very Deep Neural Network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2017:5163-5172.
[17]CHANG F J,TRAN A T,HASSNER T,et al.ExpNet:Land-mark-Free,Deep,3D Facial Expressions[C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).IEEE,2018:122-129.
[18]CHANG F,TRAN A T,HASSNER T,et al.FacePoseNet:Making a Case for Landmark-Free Face Alignment[C]//2017 IEEE International Conference on Computer Vision Workshops (ICCVW).IEEE,2017:1599-1608.
[19]WU Y,DONG L F.3D Face Shape and Texture Reconstruction Based on Weak Supervised Learning [J].Application of Computer System,2020,29(11):183-189.
[20]TEWARI A,ZOLLHOFER M,KIM H,et al.Mofa:Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction [C]//Proceedings of the IEEE International Conference on Computer Vision Workshops.IEEE,2017:1274-1283.
[21]TEWARI A,BERNARD F,GARRIDO P,et al.Fml:FaceModel Learning from Videos[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2019:10812-10822.
[22]ZHU W,WU H T,CHEN Z,et al.Reda:Reinforced Differentiable Attribute for 3D Face Reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2020:4958-4967.
[23]CHAUDHURI B,VESDAPUNT N,WANG B.Joint Face De-tection and Facial Motion Retargeting for Multiple Faces[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2019:9719-9728.
[24]CHINAEV N,CHIGORIN A,LAPTEV I.MobileFace:3D Face Reconstruction with Efficient CNN Regression[C]//European Conference on Computer Vision.Cham:Springer,2018:15-30.
[25]GUO J Z,ZHU X Y,YANG Y,et al.Towards Fast,Accurate and Stable 3D Dense Face Alignment[C]//European Conference on Computer Vision.ECCV,2020:152-168.
[26]JACKSON A S,BULAT A,ARGYRIOU V,et al.Large Pose3D Face Reconstruction from a Single Image Via Direct Volumetric CNN Regression[C]//Proceedings of the IEEE International Conference on Computer Vision.IEEE,2017:1031-1039.
[27]RICHARDSON E,SELA M,KIMMEL R.3D Face Reconstruction by Learning from Synthetic Data[C]//2016 Fourth International Conference on 3D Vision (3DV).IEEE,2016:460-469.
[28]MCDONAGH S,KLAUDINY M,BRADLEY D,et al.Synthetic Prior Design for Real-Time Face Tracking[C]//2016 Fourth International Conference on 3D Vision (3DV).IEEE,2016:639-648.
[29]DENG Y,YANG J,XU S,et al.Accurate 3D Face Reconstruc-tion with Weakly-Supervised Learning:From Single Image to Image Set[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.IEEE,2019:285-295.
[30]RANJAN A,BOLKART T,SANYAL S,et al.Generating 3DFaces Using Convolutional Mesh Autoencoders[C]//Procee-dings of the European Conference on Computer Vision.ECCV,2018:704-720.
[31]LIN J,YUAN Y,SHAO T,et al.Towards High-Fidelity 3DFace Reconstruction from In-The-Wild Images Using Graph Convolutional Networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2020:5891-5900.
[32]QIU Y,XU X,QIU L,et al.3DCaricShop:A Dataset andA Baseline Method for Single-View 3D Caricature Face Reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.CVPR,2021:10236-10245.
[33]ABREVAYA V F,WUHRER S,BOYER E.Multilinear Au-toencoder for 3D Face Model Learning[C]//2018 IEEE Winter Conference on Applications of Computer Vision.IEEE,2018:1-9.
[34]ZHANG J,LIN L,ZHU J,et al.Weakly-Supervised Multi-Face 3D Reconstruction[J].arXiv:2101.02000,2021.
[35]GENOVA K,COLE F,MASCHINOT A,et al.UnsupervisedTraining for 3D Morphable Model Regression[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2018:8377-8386.
[36]CHEN Y,WU F,WANG Z,et al.Self-supervised learning of detailed 3d face reconstruction[J].IEEE Transactions on Image Processing,2020,29:8696-8705.
[37]YANG M,GUO J,YE J,et al.Detailed 3D face reconstructionfrom single images via self-supervised attribute learning[M]//SIGGRAPH Asia 2020 Posters.2020:1-2.
[38]YOON J S,SHIRATORI T,YU S I,et al.Self-SupervisedAdaptation of High-Fidelity Face Models for Monocular Perfor-mance Tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2019:4601-4609.
[39]DEVI P R S,BASKARAN R.SL2E-AFRE:Personalized 3Dface reconstruction using autoencoder with simultaneous subspace learning and landmark estimation[J].Applied Intelligence,2021,51(4):2253-2268.
[40]TRAN L,LIU X.Nonlinear 3D Face Morphable Model[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2018:7346-7355.
[41]TRAN L,LIU F,LIU X.Towards High-Fidelity Nonlinear 3D Face Morphable Model[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2019:1126-1135.
[42]LIU F,ZHU R,ZENG D,et al.Disentangling Features in 3DFace Shapes for Joint Face Reconstruction and Recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2018:5216-5225.
[43]LI X,WENG Z,LIANG J,et al.A Novel Two-Pathway En-coder-Decoder Network for 3D Face Reconstruction[C]//2020 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP 2020).IEEE,2020:3682-3686.
[44]ZHOU Y,DENG J,KOTSIA I,et al.Dense 3D Face DecodingOver 2500fps:Joint Texture &Shape Convolutional Mesh Deco-ders[C]//Proceedings of the IEEE/CVF Conference on Compu-ter Vision and Pattern Recognition.IEEE,2019:1097-1106.
[45]BAI Z,CUI Z,LIU X,et al.Riggable 3D Face Reconstructionvia In-Network Optimization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2021:6216-6225.
[46]FENG Y,WU F,SHAO X,et al.Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network[C]//Proceedings of the European Conference on Computer Vision(ECCV).2018:534-551.
[47]HUANG Y D,ZHOU D K,YANG X.Joint 3D Face Reconstruction and Dense Alignment Algorithm Based on Multiscale Fusion Features and Residual Attention Mechanism [J].Computer Application Research,2021,23(5):1-5.
[48]CAI H,GUO Y,PENG Z,et al.Landmark detection and 3D face reconstruction for caricature using a nonlinear parametric model[J].Graphical Models,2021,115:101-103.
[49]FENG Y,FENG H,BLACK M J,et al.Learning an Animatable Detailed 3D Face Model from In-The-Wild Images[J].ACM Transactions on Graphics (TOG),2021,40(4):1-13.
[50]SLOSSBERG R,SHAMAI G,KIMMEL R.High Quality Facial Surface and Texture Synthesis via Generative Adversarial Networks[C]//Proceedings of the European Conference on Computer Vision Workshops(ECCV).2018:498-513.
[51]ABREVAYA V F,BOUKHAYMA A,WUHRER S,et al.Agenerative 3D Facial Model By Adversarial Training[J].arXiv:1902.03619v1,2019.
[52]GECER B,PLOUMPIS S,KOTSIA I,et al.Ganfit:GenerativeAdversarial Network Fitting for High Fidelity 3D Face Reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2019:1155-1164.
[53]LATTAS A,MOSCHOGLOU S,GECER B,et al.AvatarMe:Realistically Renderable 3D Facial Reconstruction “ In-the-Wild”[C]//Proceedings of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition.IEEE,2020:760-769.
[54]PIAO J,SUN K,WANG Q,et al.Inverting Generative Adversarial Renderer for Face Reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:15619-15628.
[55]TEWARI A,ZOLLHOFER M,GARRIDO P,et al.Self-Supervised Multi-Level Face Model Learning for Monocular Reconstruction at Over 250 hz[C]//Proceedings of the IEEE Confe-rence on Computer Vision and Pattern Recognition.IEEE,2018:2549-2559.
[56]ZHANG X X,LI J L.3D parametric face model based on Gans unsupervised regression [J].Information Technology and Network Security,2020,39(11):50-55.
[57]TU X,ZHAO J,XIE M,et al.3D Face Reconstruction From a Single Image Assisted by 2D Face Images in The Wild[J].IEEE Transactions on Multimedia,2020,23:1160-1172.
[58]PIAO J,QIAN C,LI H.Semi-Supervised Monocular 3D FaceReconstruction with End-to-End Shape-Preserved Domain Transfer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.IEEE,2019:9398-9407.
[59]CHENG S,BRONSTEIN M,ZHOU Y,et al.Meshgan:Non-linear 3d morphable models of faces[J].arXiv:1903.10384,2019.
[60]KOESTINGER M,WOHLHART P,ROTH P M,et al.Annotated Facial Landmarks in the Wild:A Large-Scale,Real-World Database for Facial Landmark Localization[C]//2011 IEEE International Conference on Computer Vision Workshops(ICCV).IEEE,2011:2144-2151.
[61]YI B C,SUN Y F,WANG C Z,et al.BJUT-3D 3D face database and processing technology [J].Computer Research and Development,2009(6):1009-1018.
[62]YIN L,WEI X,SUN Y,et al.A 3D Facial Expression Database for Facial Behavior Research[C]//7th International Conference on Automatic Face and Gesture Recognition (FGR’06).IEEE,2006:211-216.
[63]ALMABDY S,ELREFAEI L.Feature extraction and fusion for face recognition systems using pre-trained convolutional neural networks[J].International Journal of Computing and Digital Systems,2021,9:1-7.
[64]BAGDANOV A D,BIMBO A D,MASI I.The Florence 2D/3D Hybrid Face Dataset[C]//Proceedings of the 2011 Joint ACM Workshop on Human Gesture and Behavior Understanding.ACM,2011:79-80.
[65]PHILLIPS P J,FLYNN P J,SCRUGGS T,et al.Overview of the Face Recognition Grand Challenge[C]//2005 IEEE Computer Society Conference on Computer Vision and Pattern Re-cognition (CVPR’05).IEEE,2005:947-954.
[66]MORENO A.GavabDB:A 3D Face Database[C]//Proc. 2ndCOST275 Workshop on Biometrics on the Internet.2004:75-80.
[1] 饶志双, 贾真, 张凡, 李天瑞.
基于Key-Value关联记忆网络的知识图谱问答方法
Key-Value Relational Memory Networks for Question Answering over Knowledge Graph
计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277
[2] 汤凌韬, 王迪, 张鲁飞, 刘盛云.
基于安全多方计算和差分隐私的联邦学习方案
Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy
计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108
[3] 周乐员, 张剑华, 袁甜甜, 陈胜勇.
多层注意力机制融合的序列到序列中国连续手语识别和翻译
Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion
计算机科学, 2022, 49(9): 155-161. https://doi.org/10.11896/jsjkx.210800026
[4] 徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺.
时序知识图谱表示学习
Temporal Knowledge Graph Representation Learning
计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204
[5] 李宗民, 张玉鹏, 刘玉杰, 李华.
基于可变形图卷积的点云表征学习
Deformable Graph Convolutional Networks Based Point Cloud Representation Learning
计算机科学, 2022, 49(8): 273-278. https://doi.org/10.11896/jsjkx.210900023
[6] 王剑, 彭雨琦, 赵宇斐, 杨健.
基于深度学习的社交网络舆情信息抽取方法综述
Survey of Social Network Public Opinion Information Extraction Based on Deep Learning
计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099
[7] 郝志荣, 陈龙, 黄嘉成.
面向文本分类的类别区分式通用对抗攻击方法
Class Discriminative Universal Adversarial Attack for Text Classification
计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077
[8] 姜梦函, 李邵梅, 郑洪浩, 张建朋.
基于改进位置编码的谣言检测模型
Rumor Detection Model Based on Improved Position Embedding
计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046
[9] 陈泳全, 姜瑛.
基于卷积神经网络的APP用户行为分析方法
Analysis Method of APP User Behavior Based on Convolutional Neural Network
计算机科学, 2022, 49(8): 78-85. https://doi.org/10.11896/jsjkx.210700121
[10] 朱承璋, 黄嘉儿, 肖亚龙, 王晗, 邹北骥.
基于注意力机制的医学影像深度哈希检索算法
Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism
计算机科学, 2022, 49(8): 113-119. https://doi.org/10.11896/jsjkx.210700153
[11] 孙奇, 吉根林, 张杰.
基于非局部注意力生成对抗网络的视频异常事件检测方法
Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection
计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[12] 檀莹莹, 王俊丽, 张超波.
基于图卷积神经网络的文本分类方法研究综述
Review of Text Classification Methods Based on Graph Convolutional Network
计算机科学, 2022, 49(8): 205-216. https://doi.org/10.11896/jsjkx.210800064
[13] 胡艳羽, 赵龙, 董祥军.
一种用于癌症分类的两阶段深度特征选择提取算法
Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification
计算机科学, 2022, 49(7): 73-78. https://doi.org/10.11896/jsjkx.210500092
[14] 张颖涛, 张杰, 张睿, 张文强.
全局信息引导的真实图像风格迁移
Photorealistic Style Transfer Guided by Global Information
计算机科学, 2022, 49(7): 100-105. https://doi.org/10.11896/jsjkx.210600036
[15] 戴朝霞, 李锦欣, 张向东, 徐旭, 梅林, 张亮.
基于DNGAN的磁共振图像超分辨率重建算法
Super-resolution Reconstruction of MRI Based on DNGAN
计算机科学, 2022, 49(7): 113-119. https://doi.org/10.11896/jsjkx.210600105
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发