基于人脸部件掩膜的自监督三维人脸重建

doi:10.11896/jsjkx.220600035

Abstract

Abstract: Self-supervised 3D face reconstruction can alleviate the problem of lack of 3D face data,and has therefore become a hot research topic in recent years.Existing self-supervised methods usually focus on using globally supervised signals and do not pay enough attention to the local details of faces.In order to better recover fine-grained 3D faces with vivid details,this paper proposes a fine-grained 3D face reconstruction method based on face part masks,which can reconstruct fine-grained 3D faces without any 3D face annotation.The main idea is to improve the local accuracy of the reconstructed 3D face by giving refinement constraints on the face region through the face part mask and self-supervised constraints on the face part mask on top of the basic loss functions such as 2D image consistency loss,image deep perception loss,etc.Qualitative and quantitative experiments on AFLW2000-3D and MICC Florence datasets demonstrate the effectiveness and superiority of the proposed method.

Key words: 3D face reconstruction, Face alignment, Face modeling, Self-supervised learning, Face rendering

CLC Number:

TP391

ZHU Lei, WANG Shanmin, LIU Qingshan. Self-supervised 3D Face Reconstruction Based on Detailed Face Mask[J].Computer Science, 2023, 50(2): 214-220.

References

[1]BLANZ V,ROMDHANI S,VETTER T.Face identificationacross different poses and illuminations with a 3d morphable model[C]//Proceedings of fifth IEEE International Conference on Automatic Face Gesture Recognition.IEEE,2002:202-207.
[2]BLANZ V,VETTER T.Face recognition based on fitting a 3d morphable model[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2003,25(9):1063-1074.
[3]PAYSAN P,KNOTHE R,AMBERG B,et al.A 3D face model for pose and illumination invariant face recognition[C]//2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.IEEE,2009:296-301.
[4]XIONG X,DE LA TORRE F.Global supervised descent method[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015.
[5]YU Z,LI X,NIU X,et al.Face anti-spoofing with human material perception[C]//European Conference on Computer Vision.Springer,2020:557-575.
[6]QIN Y,ZHAO C,ZHU X,et al.Learning meta model for zero-and few-shot face anti-spoofing[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2019.
[7]BLANZ V,VETTER T.A morphable model for the synthesis of3D faces[C]//Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques.1999.
[8]BOOTH J,ROUSSOS A,PONNIAH A,et al.Large scale 3dmorphable models[J].International Journal of Computer Vision,2018,126(2):233-254.
[9]YI H,LI C,CAO Q,et al.Mmface:A multi-metric regressionnetwork for unconstrained face reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019.
[10]TRÂN A T,HASSNER T,MASI I,et al.Extreme 3d face reconstruction:Seeing through occlusions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018.
[11]JIANG L,WU X J,KITTLER J.Dual attention MobDenseNet(DAMDNet) for robust 3D face alignment[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops.2019.
[12]LIN J,YUAN Y,SHAO T,et al.Towards high-fidelity 3d face reconstruction from in-the-wild images using graph convolu-tional networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020.
[13]TRAN L,LIU X.Nonlinear 3d face morphable model[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018.
[14]BAGAUTDINOV T,WU C,SARAGIH J,et al.Modeling facial geometry using compositional vaes[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018.
[15]TRAN L,LIU F,LIU X.Towards high-fidelity nonlinear 3Dface morphable model[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019.
[16]FENG Y,WU F,SHAO X,et al.Joint 3d face reconstruction and dense alignment with position map regression network[C]//Proceedings of the European Ccnference on Computer Vision(ECCV).2018.
[17]JACKSON A S,BULAT A,ARGYRIOU V,et al.Large pose 3D face reconstruction from a single image via direct volumetric CNN regression[C]//Proceedings of the IEEE International Conference on Computer Vision.2017.
[18]DENG Y,YANG J,XU S,et al.Accurate 3d face reconstruction with weakly-supervised learning:From single image to image set[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.2019.
[19]TEWARI A,ZOLLHÖFER M,GARRIDO P,et al.Self-supervised multi-level face model learning for monocular reconstruction at over 250 HZ[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018.
[20]FENG Y,FENG H,BLACK M J,et al.Learning an animatable detailed 3D face model from in-the-wild images[J].ACM Transac-tions on Graphics(TOG),2021,40(4):1-13.
[21]HE K,ZHANG X,REN S,et al.Deep residual learning forimage recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016.
[22]CAO C,WENG Y,ZHOU S,et al.Facewarehouse:A 3d facial expression database for visual computing[J].IEEE Transactions on Visualization and Computer Graphics,2013,20(3):413-425.
[23]BULAT A,TZIMIROPOULOS G.How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks)[C]//Proceedings of the IEEE Interna-tional Conference on Computer Vision.2017.
[24]GENOVA K,COLE F,MASCHINOT A,et al.Unsupervised training for 3d morphable model regression[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018.
[25]RAVI N,REIZENSTEIN J,NOVOTNY D,et al.Accelerating 3d deep learning with pytorch3d[J].arXiv:2007.08501,2020.
[26]YU C,WANG J,PENG C,et al.Bisenet:Bilateral segmentation network for real-time semantic segmentation[C]//Proceedings of the European Conference on Computer Vision(ECCV).2018.
[27]JOHNSON J,ALAHI A,FEI FEI L.Perceptual losses for real-time style transfer and super-resolution[C]//European Confe-rence on Computer Vision.Springer,2016:694-711.
[28]LEDIG C,THEIS L,HUSZÁR F,et al.Photo-realistic singleimage super-resolution using a generative adversarial network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017.
[29]CAO Q,SHEN L,XIE W,et al.Vggface2:A dataset for recognising faces across pose and age[C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition(FG 2018).IEEE,2018:67-74.
[30]ZHU X,LEI Z,LIU X,et al.Face alignment across large poses:A 3d solution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016.
[31]BAGDANOV A D,DEL BIMBO A,MASI I.The florence 2d/3d hybrid face dataset[C]//Proceedings of the 2011 Joint ACM Workshop on Human Gesture and Behavior Understanding.2011.
[32]KOESTINGER M,WOHLHART P,ROTH P M,et al.Annotated facial landmarks in the wild:A large-scale,real-world database for facial landmark localization[C]//2011 IEEE International Conference on Computer Vision Workshops(ICCV Workshops).IEEE,2011:2144-2151.
[33]TU X,ZHAO J,JIANG Z,et al.3D Face Reconstruction from A Single Image Assisted by 2D Face Images in the Wild[J].IEEE Transactions on Multimedia,2021,23:1160-1172.
[34]MCDONAGH J,TZIMIROPOULOS G.Joint face detection and alignment with a deformable hough transform model[C]//European Conference on Computer Vision.Springer,2016:569-580.
[35]ZHU X,LIU X,LEI Z,et al.Face alignment in full pose range:A 3d total solution[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,41(1):78-92.
[36]BHAGAVATULA C,ZHU C,LUU K,et al.Faster than real-time facial alignment:A 3d spatial transformer network approach in unconstrained poses[C]//Proceedings of the IEEE International Conference on Computer Vision.2017.
[37]LIU Y,JOURABLOO A,REN W,et al.Dense face alignment[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops.2017.
[38]GECER B,PLOUMPIS S,KOTSIA I,et al.Ganfit:Generativeadversarial network fitting for high fidelity 3d face reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019.
[39]TRAN A T,HASSNER T,MASI I,et al.Regressing robust and discriminative 3D morphable models with a very deep neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017.
[40]BESL P J,MCKAY N D.Method for registration of 3-D shapes[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1992,14(2):239-256.
[41]BOOTH J,ANTONAKOS E,PLOUMPIS S,et al.3d face morphable models “in-the-wild”[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017.
[42]GUO J,ZHU X,YANG Y,et al.Towards fast,accurate and stable 3d dense face alignment[C]//European Conference on Computer Vision.Springer,2020:152-168.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

Self-supervised 3D Face Reconstruction Based on Detailed Face Mask

PDF (PC)

Abstract

Cite this article

share this article

References

Related Articles 10

Metrics

Comments

Recommended 0

[1]	WANG Pengyu, TAI Wenxin, LIU Fang, ZHONG Ting, LUO Xucheng, ZHOU Fan. Self-supervised Flight Trajectory Prediction Based on Data Augmentation [J]. Computer Science, 2023, 50(2): 130-137.
[2]	DU Hang-yuan, LI Duo, WANG Wen-jian. Method for Abnormal Users Detection Oriented to E-commerce Network [J]. Computer Science, 2022, 49(7): 170-178.
[3]	HE Jia-yu, HUANG Hong-bo, ZHANG Hong-yan, SUN Mu-ye, LIU Ya-hui, ZHOU Zhe-hai. Review of 3D Face Reconstruction Based on Single Image [J]. Computer Science, 2022, 49(2): 40-50.
[4]	MIAO Zhuang, WANG Ya-peng, LI Yang, WANG Jia-bao, ZHANG Rui, ZHAO Xin-xin. Robust Hash Learning Method Based on Dual-teacher Self-supervised Distillation [J]. Computer Science, 2022, 49(10): 159-168.
[5]	TIAN Song-wang, LIN Su-zhen, YANG Bo. Multi-band Image Self-supervised Fusion Method Based on Multi-discriminator [J]. Computer Science, 2021, 48(8): 185-190.
[6]	WANG Qian-qing, ZHANG Jing-lei. Face Pose and Expression Correction Based on 3D Morphable Model [J]. Computer Science, 2019, 46(6): 263-269.
[7]	DENG Jian-kang, YANG Jing, SUN Yu-bao and LIU Qing-shan. Fast Face Alignment Method Based on Sparse Cascade Regression and its Application on Mobile Devices [J]. Computer Science, 2015, 42(10): 301-305.
[8]	. Face Reconstruction Algorithm Based on Small Curve Relaxation Constraint Approximation [J]. Computer Science, 2013, 40(2): 304-307.
[9]	. [J]. Computer Science, 2009, 36(3): 253-255.
[10]	. [J]. Computer Science, 2005, 32(11): 175-178.