Computer Science

Computer Science ›› 2022, Vol. 49 ›› Issue (2): 40-50.doi: 10.11896/jsjkx.210500215

• Computer Vision: Theory and Application • Previous Articles     Next Articles

Review of 3D Face Reconstruction Based on Single Image

HE Jia-yu1, HUANG Hong-bo1, ZHANG Hong-yan1, SUN Mu-ye1, LIU Ya-hui2, ZHOU Zhe-hai3   

  1. 1 School of Computing,Beijing Information Science and Technology University,Beijing 100192,China
    2 School of Information Management,Beijing Information Science and Technology University,Beijing 100192,China
    3 School of Instrument Science and Optoelectronic Engineering,Beijing Information Science and Technology University,Beijing 100192,China
  • Received:2021-05-29 Revised:2021-06-30 Online:2022-02-15 Published:2022-02-23
  • About author:HE Jia-yu,born in 1997,postgraduate.Her main research interests include 3D face reconstruction and face recognition.
    HUANG Hong-bo,born in 1976,Ph.D,associate professor.His main research interests include computer vision,deep learning and optimization theory.
  • Supported by:
    Key Program of the National Natural Science Foundation of China(61931003),Great Wall Scholar Program of Beijing(CIT&TCD20190323) and Beijing Education Commission for General Project of Science and Technology Plan(KM201811232024).

PDF (PC)

2026

Abstract: In the field of computer vision,3D face reconstruction is a valuable research direction.High quality reconstruction of 3D faces can find applications in face recognition,anti-proofing,animation and medical cosmetology.In the last two decades,although great progress has been made 3D face reconstruction based on a single image,the results of reconstruction using traditionalalgorithms are still facing the challenge of facial expression,occlusion and ambient light,and there will be problems such as poor reconstruction accuracy and robustness.With the rapid development of deep learning in 3D face reconstruction,various methods which are superior to traditional reconstruction algorithms have emerged.Firstly,this paper focuses on deep-learning-based reconstruction algorithms.The algorithms are divided into four categories according to different network architecture,and the most popular methods are described in detail.Then commonly used 3D face data sets are introduced,and performance of representative methods are evaluated.Finally,conclusions and prospects of the single-image-based 3D face reconstruction are given.

Key words: 3D face reconstruction, 3D morphable model, Convolutional neural networks, Deep learning, Face recognition

CLC Number: 

  • TP391
[1]BLANZ V,VETTER T.A Morphable Model for the Synthesis of 3D Faces[C]//Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques.ACM Press,1999:187-194.
[2]PAYSAN P,KNOTHE R,AMBERG B,et al.A 3D Face Model for Pose and Illumination Invariant Face Recognition[C]//2009 sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.IEEE,2009:296-301.
[3]BOOTH J,ROUSSOS A,PONNIAH A,et al.Large scale3D morphable models[J].International Journal of Computer Vision,2018,126(2):233-254.
[4]CAO C,WENG Y,ZHOU S,et al.Facewarehouse:a 3D facial expression database for visual computing[J].IEEE Transactions on Visualization and Computer Graphics,2013,20(3):413-425.
[5]BOOTH J,ANTONAKOS E,PLOUMPIS S,et al.3D FaceMorphable Models “In-The-Wild” [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2017:48-57.
[6]VLASIC D,BRAND M,PFISTER H,et al.Face transfer with multilinear models[J].ACM Transactions on Graphics (TOG),2005,24(3):426-433.
[7]YANG H,ZHU H,WANG Y,et al.FaceScape:A Large-ScaleHigh Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2020:601-610.
[8]KIM H,ZOLLHOFER M,TEWARI A,et al.Inversefacenet:Deep Monocular Inverse Face Rendering[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2018:4625-4634.
[9]RICHARDSON E,SELA M,OR-EL R,et al.Learning Detailed Face Reconstruction From a Single Image[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2017:1259-1268.
[10]FAN X,CHENG S,KANG H,et al.Dual neural networks coupling data regression with explicit priors for monocular 3D face reconstruction[J/OL].IEEE Transactions on Multimedia,2020.https://www.researchgate.net/publication/341364310_Dual_Neural_Networks_Coupling_Data_Regression_With_Explicit_Priors_for_Monocular_3D_Face_Reconstruction.
[11]LUO Y,TU X,XIE M.Learning Robust 3D Face Reconstruc-tion and Discriminative Identity Representation[C]//2019 IEEE 2nd International Conference on Information Communication and Signal Processing (ICICSP).IEEE,2019:317-321.
[12]DOU P,SHAH S K,KAKADIARIS I A.End-to-end 3D FaceReconstruction with Deep Neural Networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2017:5908-5917.
[13]ZHU X,LEI Z,LIU X,et al.Face Alignment Across LargePoses:A 3D Solution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2016:146-155.
[14]ZHANG G,HAN H,SHAN S,et al.Face Alignment AcrossLarge Pose Via MT-CNN Based 3D Shape Reconstruction [C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).IEEE,2018:210-217.
[15]XING Y,TEWARI R,MENDONCA P.A Self-Supervised Bootstrap Method for Single-Image 3D Face Reconstruction[C]//2019 IEEE Winter Conference on Applications of Computer Vision (WACV).IEEE,2019:1014-1023.
[16]TRAN A T,HASSNER T,MASI I,et al.Regressing Robustand Discriminative 3D Morphable Models with a Very Deep Neural Network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2017:5163-5172.
[17]CHANG F J,TRAN A T,HASSNER T,et al.ExpNet:Land-mark-Free,Deep,3D Facial Expressions[C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).IEEE,2018:122-129.
[18]CHANG F,TRAN A T,HASSNER T,et al.FacePoseNet:Making a Case for Landmark-Free Face Alignment[C]//2017 IEEE International Conference on Computer Vision Workshops (ICCVW).IEEE,2017:1599-1608.
[19]WU Y,DONG L F.3D Face Shape and Texture Reconstruction Based on Weak Supervised Learning [J].Application of Computer System,2020,29(11):183-189.
[20]TEWARI A,ZOLLHOFER M,KIM H,et al.Mofa:Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction [C]//Proceedings of the IEEE International Conference on Computer Vision Workshops.IEEE,2017:1274-1283.
[21]TEWARI A,BERNARD F,GARRIDO P,et al.Fml:FaceModel Learning from Videos[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2019:10812-10822.
[22]ZHU W,WU H T,CHEN Z,et al.Reda:Reinforced Differentiable Attribute for 3D Face Reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2020:4958-4967.
[23]CHAUDHURI B,VESDAPUNT N,WANG B.Joint Face De-tection and Facial Motion Retargeting for Multiple Faces[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2019:9719-9728.
[24]CHINAEV N,CHIGORIN A,LAPTEV I.MobileFace:3D Face Reconstruction with Efficient CNN Regression[C]//European Conference on Computer Vision.Cham:Springer,2018:15-30.
[25]GUO J Z,ZHU X Y,YANG Y,et al.Towards Fast,Accurate and Stable 3D Dense Face Alignment[C]//European Conference on Computer Vision.ECCV,2020:152-168.
[26]JACKSON A S,BULAT A,ARGYRIOU V,et al.Large Pose3D Face Reconstruction from a Single Image Via Direct Volumetric CNN Regression[C]//Proceedings of the IEEE International Conference on Computer Vision.IEEE,2017:1031-1039.
[27]RICHARDSON E,SELA M,KIMMEL R.3D Face Reconstruction by Learning from Synthetic Data[C]//2016 Fourth International Conference on 3D Vision (3DV).IEEE,2016:460-469.
[28]MCDONAGH S,KLAUDINY M,BRADLEY D,et al.Synthetic Prior Design for Real-Time Face Tracking[C]//2016 Fourth International Conference on 3D Vision (3DV).IEEE,2016:639-648.
[29]DENG Y,YANG J,XU S,et al.Accurate 3D Face Reconstruc-tion with Weakly-Supervised Learning:From Single Image to Image Set[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.IEEE,2019:285-295.
[30]RANJAN A,BOLKART T,SANYAL S,et al.Generating 3DFaces Using Convolutional Mesh Autoencoders[C]//Procee-dings of the European Conference on Computer Vision.ECCV,2018:704-720.
[31]LIN J,YUAN Y,SHAO T,et al.Towards High-Fidelity 3DFace Reconstruction from In-The-Wild Images Using Graph Convolutional Networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2020:5891-5900.
[32]QIU Y,XU X,QIU L,et al.3DCaricShop:A Dataset andA Baseline Method for Single-View 3D Caricature Face Reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.CVPR,2021:10236-10245.
[33]ABREVAYA V F,WUHRER S,BOYER E.Multilinear Au-toencoder for 3D Face Model Learning[C]//2018 IEEE Winter Conference on Applications of Computer Vision.IEEE,2018:1-9.
[34]ZHANG J,LIN L,ZHU J,et al.Weakly-Supervised Multi-Face 3D Reconstruction[J].arXiv:2101.02000,2021.
[35]GENOVA K,COLE F,MASCHINOT A,et al.UnsupervisedTraining for 3D Morphable Model Regression[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2018:8377-8386.
[36]CHEN Y,WU F,WANG Z,et al.Self-supervised learning of detailed 3d face reconstruction[J].IEEE Transactions on Image Processing,2020,29:8696-8705.
[37]YANG M,GUO J,YE J,et al.Detailed 3D face reconstructionfrom single images via self-supervised attribute learning[M]//SIGGRAPH Asia 2020 Posters.2020:1-2.
[38]YOON J S,SHIRATORI T,YU S I,et al.Self-SupervisedAdaptation of High-Fidelity Face Models for Monocular Perfor-mance Tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2019:4601-4609.
[39]DEVI P R S,BASKARAN R.SL2E-AFRE:Personalized 3Dface reconstruction using autoencoder with simultaneous subspace learning and landmark estimation[J].Applied Intelligence,2021,51(4):2253-2268.
[40]TRAN L,LIU X.Nonlinear 3D Face Morphable Model[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2018:7346-7355.
[41]TRAN L,LIU F,LIU X.Towards High-Fidelity Nonlinear 3D Face Morphable Model[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2019:1126-1135.
[42]LIU F,ZHU R,ZENG D,et al.Disentangling Features in 3DFace Shapes for Joint Face Reconstruction and Recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2018:5216-5225.
[43]LI X,WENG Z,LIANG J,et al.A Novel Two-Pathway En-coder-Decoder Network for 3D Face Reconstruction[C]//2020 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP 2020).IEEE,2020:3682-3686.
[44]ZHOU Y,DENG J,KOTSIA I,et al.Dense 3D Face DecodingOver 2500fps:Joint Texture &Shape Convolutional Mesh Deco-ders[C]//Proceedings of the IEEE/CVF Conference on Compu-ter Vision and Pattern Recognition.IEEE,2019:1097-1106.
[45]BAI Z,CUI Z,LIU X,et al.Riggable 3D Face Reconstructionvia In-Network Optimization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2021:6216-6225.
[46]FENG Y,WU F,SHAO X,et al.Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network[C]//Proceedings of the European Conference on Computer Vision(ECCV).2018:534-551.
[47]HUANG Y D,ZHOU D K,YANG X.Joint 3D Face Reconstruction and Dense Alignment Algorithm Based on Multiscale Fusion Features and Residual Attention Mechanism [J].Computer Application Research,2021,23(5):1-5.
[48]CAI H,GUO Y,PENG Z,et al.Landmark detection and 3D face reconstruction for caricature using a nonlinear parametric model[J].Graphical Models,2021,115:101-103.
[49]FENG Y,FENG H,BLACK M J,et al.Learning an Animatable Detailed 3D Face Model from In-The-Wild Images[J].ACM Transactions on Graphics (TOG),2021,40(4):1-13.
[50]SLOSSBERG R,SHAMAI G,KIMMEL R.High Quality Facial Surface and Texture Synthesis via Generative Adversarial Networks[C]//Proceedings of the European Conference on Computer Vision Workshops(ECCV).2018:498-513.
[51]ABREVAYA V F,BOUKHAYMA A,WUHRER S,et al.Agenerative 3D Facial Model By Adversarial Training[J].arXiv:1902.03619v1,2019.
[52]GECER B,PLOUMPIS S,KOTSIA I,et al.Ganfit:GenerativeAdversarial Network Fitting for High Fidelity 3D Face Reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.IEEE,2019:1155-1164.
[53]LATTAS A,MOSCHOGLOU S,GECER B,et al.AvatarMe:Realistically Renderable 3D Facial Reconstruction “ In-the-Wild”[C]//Proceedings of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition.IEEE,2020:760-769.
[54]PIAO J,SUN K,WANG Q,et al.Inverting Generative Adversarial Renderer for Face Reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:15619-15628.
[55]TEWARI A,ZOLLHOFER M,GARRIDO P,et al.Self-Supervised Multi-Level Face Model Learning for Monocular Reconstruction at Over 250 hz[C]//Proceedings of the IEEE Confe-rence on Computer Vision and Pattern Recognition.IEEE,2018:2549-2559.
[56]ZHANG X X,LI J L.3D parametric face model based on Gans unsupervised regression [J].Information Technology and Network Security,2020,39(11):50-55.
[57]TU X,ZHAO J,XIE M,et al.3D Face Reconstruction From a Single Image Assisted by 2D Face Images in The Wild[J].IEEE Transactions on Multimedia,2020,23:1160-1172.
[58]PIAO J,QIAN C,LI H.Semi-Supervised Monocular 3D FaceReconstruction with End-to-End Shape-Preserved Domain Transfer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.IEEE,2019:9398-9407.
[59]CHENG S,BRONSTEIN M,ZHOU Y,et al.Meshgan:Non-linear 3d morphable models of faces[J].arXiv:1903.10384,2019.
[60]KOESTINGER M,WOHLHART P,ROTH P M,et al.Annotated Facial Landmarks in the Wild:A Large-Scale,Real-World Database for Facial Landmark Localization[C]//2011 IEEE International Conference on Computer Vision Workshops(ICCV).IEEE,2011:2144-2151.
[61]YI B C,SUN Y F,WANG C Z,et al.BJUT-3D 3D face database and processing technology [J].Computer Research and Development,2009(6):1009-1018.
[62]YIN L,WEI X,SUN Y,et al.A 3D Facial Expression Database for Facial Behavior Research[C]//7th International Conference on Automatic Face and Gesture Recognition (FGR’06).IEEE,2006:211-216.
[63]ALMABDY S,ELREFAEI L.Feature extraction and fusion for face recognition systems using pre-trained convolutional neural networks[J].International Journal of Computing and Digital Systems,2021,9:1-7.
[64]BAGDANOV A D,BIMBO A D,MASI I.The Florence 2D/3D Hybrid Face Dataset[C]//Proceedings of the 2011 Joint ACM Workshop on Human Gesture and Behavior Understanding.ACM,2011:79-80.
[65]PHILLIPS P J,FLYNN P J,SCRUGGS T,et al.Overview of the Face Recognition Grand Challenge[C]//2005 IEEE Computer Society Conference on Computer Vision and Pattern Re-cognition (CVPR’05).IEEE,2005:947-954.
[66]MORENO A.GavabDB:A 3D Face Database[C]//Proc. 2ndCOST275 Workshop on Biometrics on the Internet.2004:75-80.
[1] XU Yong-xin, ZHAO Jun-feng, WANG Ya-sha, XIE Bing, YANG Kai. Temporal Knowledge Graph Representation Learning [J]. Computer Science, 2022, 49(9): 162-171.
[2] RAO Zhi-shuang, JIA Zhen, ZHANG Fan, LI Tian-rui. Key-Value Relational Memory Networks for Question Answering over Knowledge Graph [J]. Computer Science, 2022, 49(9): 202-207.
[3] TANG Ling-tao, WANG Di, ZHANG Lu-fei, LIU Sheng-yun. Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy [J]. Computer Science, 2022, 49(9): 297-305.
[4] WANG Jian, PENG Yu-qi, ZHAO Yu-fei, YANG Jian. Survey of Social Network Public Opinion Information Extraction Based on Deep Learning [J]. Computer Science, 2022, 49(8): 279-293.
[5] HAO Zhi-rong, CHEN Long, HUANG Jia-cheng. Class Discriminative Universal Adversarial Attack for Text Classification [J]. Computer Science, 2022, 49(8): 323-329.
[6] JIANG Meng-han, LI Shao-mei, ZHENG Hong-hao, ZHANG Jian-peng. Rumor Detection Model Based on Improved Position Embedding [J]. Computer Science, 2022, 49(8): 330-335.
[7] SUN Qi, JI Gen-lin, ZHANG Jie. Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection [J]. Computer Science, 2022, 49(8): 172-177.
[8] HU Yan-yu, ZHAO Long, DONG Xiang-jun. Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification [J]. Computer Science, 2022, 49(7): 73-78.
[9] CHENG Cheng, JIANG Ai-lian. Real-time Semantic Segmentation Method Based on Multi-path Feature Extraction [J]. Computer Science, 2022, 49(7): 120-126.
[10] HOU Yu-tao, ABULIZI Abudukelimu, ABUDUKELIMU Halidanmu. Advances in Chinese Pre-training Models [J]. Computer Science, 2022, 49(7): 148-163.
[11] ZHOU Hui, SHI Hao-chen, TU Yao-feng, HUANG Sheng-jun. Robust Deep Neural Network Learning Based on Active Sampling [J]. Computer Science, 2022, 49(7): 164-169.
[12] SU Dan-ning, CAO Gui-tao, WANG Yan-nan, WANG Hong, REN He. Survey of Deep Learning for Radar Emitter Identification Based on Small Sample [J]. Computer Science, 2022, 49(7): 226-235.
[13] WANG Jun-feng, LIU Fan, YANG Sai, LYU Tan-yue, CHEN Zhi-yu, XU Feng. Dam Crack Detection Based on Multi-source Transfer Learning [J]. Computer Science, 2022, 49(6A): 319-324.
[14] CHU Yu-chun, GONG Hang, Wang Xue-fang, LIU Pei-shun. Study on Knowledge Distillation of Target Detection Algorithm Based on YOLOv4 [J]. Computer Science, 2022, 49(6A): 337-344.
[15] ZHOU Zhi-hao, CHEN Lei, WU Xiang, QIU Dong-liang, LIANG Guang-sheng, ZENG Fan-qiao. SMOTE-SDSAE-SVM Based Vehicle CAN Bus Intrusion Detection Algorithm [J]. Computer Science, 2022, 49(6A): 562-570.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.