基于人脸部件掩膜的自监督三维人脸重建

doi:10.11896/jsjkx.220600035

计算机科学 ›› 2023, Vol. 50 ›› Issue (2): 214-220.doi: 10.11896/jsjkx.220600035

• 计算机图形学&多媒体 • 上一篇下一篇

基于人脸部件掩膜的自监督三维人脸重建

朱磊¹, 王善敏², 刘青山¹

1 南京信息工程大学数字取证教育部研究中心南京 210044
2 南京航空航天大学计算机学院南京 210016

收稿日期:2022-06-03 修回日期:2022-09-28 出版日期:2023-02-15 发布日期:2023-02-22
通讯作者: 刘青山(qsliu@nuist.edu.cn)
作者简介:(leizhu9702@163.com)
基金资助:
国家杰出青年基金(61825601)

Self-supervised 3D Face Reconstruction Based on Detailed Face Mask

ZHU Lei¹, WANG Shanmin², LIU Qingshan¹

1 Engineering Research Center of Digital Forensics,Ministry of Education,Nanjing University of Information Science, Technology,Nanjing210044,China
2 College of Computer Science and Technology,Nanjing University of Aeronautics and Astronautics,Nanjing 210016,China

Received:2022-06-03 Revised:2022-09-28 Online:2023-02-15 Published:2023-02-22
Supported by:
National Science Fund for Distinguished Young Scholars of China(61825601)

摘要/Abstract

摘要： 三维人脸重建旨在从二维人脸图片中恢复出三维人脸模型。自监督三维人脸重建能够缓解三维人脸数据缺乏的问题,因此成为了近年来的研究热点。现有的自监督方法通常聚焦于使用全局监督信号,对人脸的局部细节关注不足。为了更好地恢复出细节生动的精细化三维人脸,提出了一种基于人脸部件掩膜的精细化三维人脸重建方法,该方法在不需要任何三维人脸标注的情况下,可以重建出精细化三维人脸。其主要思想是在二维图片一致性损失、图片深层感知损失等基本损失函数上,通过人脸部件掩膜,给予人脸区域精细化约束,并对人脸部件掩膜进行自监督约束,从而提高重建的三维人脸局部的准确性。在AFLW2000-3D和MICC Florence数据集上进行了定性以及定量实验,验证了所提方法的有效性和优越性。

关键词: 三维人脸重建, 人脸对齐, 人脸建模, 自监督学习, 人脸渲染

Abstract: Self-supervised 3D face reconstruction can alleviate the problem of lack of 3D face data,and has therefore become a hot research topic in recent years.Existing self-supervised methods usually focus on using globally supervised signals and do not pay enough attention to the local details of faces.In order to better recover fine-grained 3D faces with vivid details,this paper proposes a fine-grained 3D face reconstruction method based on face part masks,which can reconstruct fine-grained 3D faces without any 3D face annotation.The main idea is to improve the local accuracy of the reconstructed 3D face by giving refinement constraints on the face region through the face part mask and self-supervised constraints on the face part mask on top of the basic loss functions such as 2D image consistency loss,image deep perception loss,etc.Qualitative and quantitative experiments on AFLW2000-3D and MICC Florence datasets demonstrate the effectiveness and superiority of the proposed method.

Key words: 3D face reconstruction, Face alignment, Face modeling, Self-supervised learning, Face rendering

中图分类号:

TP391

朱磊, 王善敏, 刘青山. 基于人脸部件掩膜的自监督三维人脸重建[J]. 计算机科学, 2023, 50(2): 214-220. https://doi.org/10.11896/jsjkx.220600035

ZHU Lei, WANG Shanmin, LIU Qingshan. Self-supervised 3D Face Reconstruction Based on Detailed Face Mask[J]. Computer Science, 2023, 50(2): 214-220. https://doi.org/10.11896/jsjkx.220600035

参考文献

[1]BLANZ V,ROMDHANI S,VETTER T.Face identificationacross different poses and illuminations with a 3d morphable model[C]//Proceedings of fifth IEEE International Conference on Automatic Face Gesture Recognition.IEEE,2002:202-207.
[2]BLANZ V,VETTER T.Face recognition based on fitting a 3d morphable model[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2003,25(9):1063-1074.
[3]PAYSAN P,KNOTHE R,AMBERG B,et al.A 3D face model for pose and illumination invariant face recognition[C]//2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.IEEE,2009:296-301.
[4]XIONG X,DE LA TORRE F.Global supervised descent method[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015.
[5]YU Z,LI X,NIU X,et al.Face anti-spoofing with human material perception[C]//European Conference on Computer Vision.Springer,2020:557-575.
[6]QIN Y,ZHAO C,ZHU X,et al.Learning meta model for zero-and few-shot face anti-spoofing[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2019.
[7]BLANZ V,VETTER T.A morphable model for the synthesis of3D faces[C]//Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques.1999.
[8]BOOTH J,ROUSSOS A,PONNIAH A,et al.Large scale 3dmorphable models[J].International Journal of Computer Vision,2018,126(2):233-254.
[9]YI H,LI C,CAO Q,et al.Mmface:A multi-metric regressionnetwork for unconstrained face reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019.
[10]TRÂN A T,HASSNER T,MASI I,et al.Extreme 3d face reconstruction:Seeing through occlusions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018.
[11]JIANG L,WU X J,KITTLER J.Dual attention MobDenseNet(DAMDNet) for robust 3D face alignment[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops.2019.
[12]LIN J,YUAN Y,SHAO T,et al.Towards high-fidelity 3d face reconstruction from in-the-wild images using graph convolu-tional networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020.
[13]TRAN L,LIU X.Nonlinear 3d face morphable model[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018.
[14]BAGAUTDINOV T,WU C,SARAGIH J,et al.Modeling facial geometry using compositional vaes[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018.
[15]TRAN L,LIU F,LIU X.Towards high-fidelity nonlinear 3Dface morphable model[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019.
[16]FENG Y,WU F,SHAO X,et al.Joint 3d face reconstruction and dense alignment with position map regression network[C]//Proceedings of the European Ccnference on Computer Vision(ECCV).2018.
[17]JACKSON A S,BULAT A,ARGYRIOU V,et al.Large pose 3D face reconstruction from a single image via direct volumetric CNN regression[C]//Proceedings of the IEEE International Conference on Computer Vision.2017.
[18]DENG Y,YANG J,XU S,et al.Accurate 3d face reconstruction with weakly-supervised learning:From single image to image set[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.2019.
[19]TEWARI A,ZOLLHÖFER M,GARRIDO P,et al.Self-supervised multi-level face model learning for monocular reconstruction at over 250 HZ[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018.
[20]FENG Y,FENG H,BLACK M J,et al.Learning an animatable detailed 3D face model from in-the-wild images[J].ACM Transac-tions on Graphics(TOG),2021,40(4):1-13.
[21]HE K,ZHANG X,REN S,et al.Deep residual learning forimage recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016.
[22]CAO C,WENG Y,ZHOU S,et al.Facewarehouse:A 3d facial expression database for visual computing[J].IEEE Transactions on Visualization and Computer Graphics,2013,20(3):413-425.
[23]BULAT A,TZIMIROPOULOS G.How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks)[C]//Proceedings of the IEEE Interna-tional Conference on Computer Vision.2017.
[24]GENOVA K,COLE F,MASCHINOT A,et al.Unsupervised training for 3d morphable model regression[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018.
[25]RAVI N,REIZENSTEIN J,NOVOTNY D,et al.Accelerating 3d deep learning with pytorch3d[J].arXiv:2007.08501,2020.
[26]YU C,WANG J,PENG C,et al.Bisenet:Bilateral segmentation network for real-time semantic segmentation[C]//Proceedings of the European Conference on Computer Vision(ECCV).2018.
[27]JOHNSON J,ALAHI A,FEI FEI L.Perceptual losses for real-time style transfer and super-resolution[C]//European Confe-rence on Computer Vision.Springer,2016:694-711.
[28]LEDIG C,THEIS L,HUSZÁR F,et al.Photo-realistic singleimage super-resolution using a generative adversarial network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017.
[29]CAO Q,SHEN L,XIE W,et al.Vggface2:A dataset for recognising faces across pose and age[C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition(FG 2018).IEEE,2018:67-74.
[30]ZHU X,LEI Z,LIU X,et al.Face alignment across large poses:A 3d solution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016.
[31]BAGDANOV A D,DEL BIMBO A,MASI I.The florence 2d/3d hybrid face dataset[C]//Proceedings of the 2011 Joint ACM Workshop on Human Gesture and Behavior Understanding.2011.
[32]KOESTINGER M,WOHLHART P,ROTH P M,et al.Annotated facial landmarks in the wild:A large-scale,real-world database for facial landmark localization[C]//2011 IEEE International Conference on Computer Vision Workshops(ICCV Workshops).IEEE,2011:2144-2151.
[33]TU X,ZHAO J,JIANG Z,et al.3D Face Reconstruction from A Single Image Assisted by 2D Face Images in the Wild[J].IEEE Transactions on Multimedia,2021,23:1160-1172.
[34]MCDONAGH J,TZIMIROPOULOS G.Joint face detection and alignment with a deformable hough transform model[C]//European Conference on Computer Vision.Springer,2016:569-580.
[35]ZHU X,LIU X,LEI Z,et al.Face alignment in full pose range:A 3d total solution[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,41(1):78-92.
[36]BHAGAVATULA C,ZHU C,LUU K,et al.Faster than real-time facial alignment:A 3d spatial transformer network approach in unconstrained poses[C]//Proceedings of the IEEE International Conference on Computer Vision.2017.
[37]LIU Y,JOURABLOO A,REN W,et al.Dense face alignment[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops.2017.
[38]GECER B,PLOUMPIS S,KOTSIA I,et al.Ganfit:Generativeadversarial network fitting for high fidelity 3d face reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019.
[39]TRAN A T,HASSNER T,MASI I,et al.Regressing robust and discriminative 3D morphable models with a very deep neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017.
[40]BESL P J,MCKAY N D.Method for registration of 3-D shapes[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1992,14(2):239-256.
[41]BOOTH J,ANTONAKOS E,PLOUMPIS S,et al.3d face morphable models “in-the-wild”[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017.
[42]GUO J,ZHU X,YANG Y,et al.Towards fast,accurate and stable 3d dense face alignment[C]//European Conference on Computer Vision.Springer,2020:152-168.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

基于人脸部件掩膜的自监督三维人脸重建

Self-supervised 3D Face Reconstruction Based on Detailed Face Mask

PDF (PC)

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 7

Metrics

本文评价

推荐阅读 0

[1]	王鹏宇, 台文鑫, 刘芳, 钟婷, 罗绪成, 周帆. 基于数据增强的自监督飞行航迹预测 Self-supervised Flight Trajectory Prediction Based on Data Augmentation 计算机科学, 2023, 50(2): 130-137. https://doi.org/10.11896/jsjkx.211200016
[2]	杜航原, 李铎, 王文剑. 一种面向电商网络的异常用户检测方法 Method for Abnormal Users Detection Oriented to E-commerce Network 计算机科学, 2022, 49(7): 170-178. https://doi.org/10.11896/jsjkx.210600092
[3]	何嘉玉, 黄宏博, 张红艳, 孙牧野, 刘亚辉, 周哲海. 基于深度学习的单幅图像三维人脸重建研究综述 Review of 3D Face Reconstruction Based on Single Image 计算机科学, 2022, 49(2): 40-50. https://doi.org/10.11896/jsjkx.210500215
[4]	苗壮, 王亚鹏, 李阳, 王家宝, 张睿, 赵昕昕. 一种鲁棒的双教师自监督蒸馏哈希学习方法 Robust Hash Learning Method Based on Dual-teacher Self-supervised Distillation 计算机科学, 2022, 49(10): 159-168. https://doi.org/10.11896/jsjkx.210800050
[5]	田嵩旺, 蔺素珍, 杨博. 基于多判别器的多波段图像自监督融合方法 Multi-band Image Self-supervised Fusion Method Based on Multi-discriminator 计算机科学, 2021, 48(8): 185-190. https://doi.org/10.11896/jsjkx.200600132
[6]	. 基于Boost改进的嵌入式隐马尔可夫模型的实时表情识别计算机科学, 2005, 32(11): 175-178.
[7]	王洵董兰芳等. 人脸建模与动画的研究计算机科学, 2003, 30(3): 5-11.