计算机科学

计算机科学 ›› 2024, Vol. 51 ›› Issue (7): 244-256.doi: 10.11896/jsjkx.230400127

• 计算机图形学&多媒体 • 上一篇    下一篇

基于图像深度先验和鲁棒马尔可夫随机场的有遮挡人脸识别

李小薪1, 丁伟杰1,2, 方怡1, 张远成1, 王琦晖3   

  1. 1 浙江工业大学计算机科学与技术学院 杭州 310023
    2 浙江警察学院计算机与信息安全系 杭州 310053
    3 杭州师范大学钱江学院 杭州 311121
  • 收稿日期:2023-04-18 修回日期:2024-01-14 出版日期:2024-07-15 发布日期:2024-07-10
  • 通讯作者: 丁伟杰(dingweijie@zjjcxy.cn)
  • 作者简介:(mordecai@163.com)
  • 基金资助:
    浙江省自然科学基金(LGF22F020027);公安部科技强警基础工作专项项目(2020GABJC35);国家自然科学基金(62271448)

Occluded Face Recognition Based on Deep Image Prior and Robust Markov Random Field

LI Xiaoxin1, DING Weijie1,2, FANG Yi1, ZHANG Yuancheng1, WANG Qihui3   

  1. 1 College of Computer Science and Technology,Zhejiang University of Technology,Hangzhou 310023,China
    2 Department of Computer and Information Security,Zhejiang Police College,Hangzhou 310053,China
    3 Qianjiang College of Hangzhou Normal University,Hangzhou 311121,China
  • Received:2023-04-18 Revised:2024-01-14 Online:2024-07-15 Published:2024-07-10
  • About author:LI Xiaoxin,born in 1980,Ph.D,asso-ciate professor,master supervisor,is a member of CCF(No.80065M).His main research interests include image processing and pattern recognition.
    DING Weijie,born in 1981,Ph.D,professor.His main research interests include information visualization and network security.
  • Supported by:
    Natural Science Foundation of Zhejiang Province,China(LGF22F020027),Special Project of Ministry of Public Security(2020GABJC35) and National Natural Science Foundation of China(62271448).

PDF (PC)

443

摘要： 由遮挡所引发的测试数据和训练数据之间的差异,是人脸识别技术面临的重要挑战。现有的基于深度神经网络的有遮挡人脸识别方法大多需要使用大规模的有遮挡的人脸图像来训练网络模型。然而,现实世界中的任何外界物体都有可能成为遮挡,有限的训练集数据很难穷尽所有的可能性,并且使用大规模的有遮挡人脸图像训练网络模型的做法与人类视觉机制是相违背的,人眼对于遮挡区域的感知在本质上与遮挡本身并没有关系,仅依赖于无遮挡的人脸图像。为了模拟人类视觉的遮挡检测机制,将图像深度先验和鲁棒马尔可夫随机场模型结合起来,构建基于小样本数据的遮挡检测模型DIP-rMRF,并提出了一致性零填充方法以有效利用DIP-rMRF的遮挡检测结果进行后续的人脸识别。在Extended Yale B,AR和LFW这3个人脸数据库上,针对VGGFace,LCNN,PCANet,SphereFace,InterpretFR,FROM这6种CNN模型的实验结果表明,DIP-rMRF能够有效地处理遮挡以及由极端光照所引发的“类遮挡”,从而极大提升现有的深度神经网络模型对有遮挡人脸识别的性能。

关键词: 有遮挡人脸识别, 图像深度先验, 鲁棒马尔可夫随机场, 一致性零填充, 结构误差度量

Abstract: The occlusion-caused difference between test and training images is one of the most challenging issues for real-world face recognition system.Most of the existing occluded face recognition methods based on deep neural networks(DNNs) need to use large-scale occluded face images to train network models.However,any external object in the real world might become occlusions,and limited training data cannot exhaust all possible objects.Also,using large-scale occluded face images to train networks violates the human visual mechanism,the human eyes detect occlusions by only using small-scale unoccluded face images without seeing any occlusions.In order to simulate the occlusion detection mechanism of human vision,we combine the deep image prior with the robust Markov random field model to construct a novel occlusion detection model,namely DIP-rMRF,based on small-scale data,and propose a uniform zero filling method to effectively utilize the occlusion detection resultsof DIP-rMRF.Experimental resultsofsix advanced DNN-based face recognitions methods,including VGGFace,LCNN,PCANet,SphereFace,InterpretFR and FROM,on three face datasets,including Extended Yale B,AR and LFW,show that DIP-rMRF can effectively preprocess face images with occlusions and quasi-occlusions caused by extreme illuminations,and greatly improve the performance of the existing DNN models for face recognition with occlusion.

Key words: Face recognition with occlusion, Deep image prior, Robust Markov random field, Uniform zero-filling, Structural error metric

中图分类号: 

  • TP391
[1]WANG M,DENG W.Deep Face Recognition:A Survey [J].Neurocomputing,2021,429:215-244.
[2]RANJAN R,SANKARANARAYANAN S,BANSAL A,et al.Deep Learning for Understanding Faces:Machines May Be Just as Good,or Better,Than Humans [J].IEEE Signal Processing Magazine,2018,35(1):66-83.
[3]DING C,TAO D.A Comprehensive Survey on Pose-InvariantFace Recognition [J].ACM Transactions on Intelligent Systems and Technology(TIST),2016,7(3):1-42.
[4]BENDJILLALI R I,BELADGHAM M,MERIT K,et al.Illumination-Robust Face Recognition Based on Deep Convolutional Neural Networks Architectures [J].Indonesian Journal of Electrical Engineering and Computer Science,2020,18(2):1015-1027.
[5]HAN X,LIU Y,YANG H,et al.Normalization of Face Illumination with Photorealistic Texture Via Deep Image Prior Synthesis [J].Neurocomputing,2020,386:305-316.
[6]WANG K,PENG X,YANG J,et al.Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition [J].IEEE Transactions on Image Processing,2020,29:4057-4069.
[7]REVINA I M,EMMANUEL W S.A Survey on Human FaceExpression Recognition Techniques [J].Journal of King Saud University-Computer and Information Sciences,2021,33(6):619-628.
[8]LI X,LIANG R.A Review for Face Recognition with Occlu-sion:From Subspace Regression to Deep Learning [J].Chinese Journal of Computers,2018,41(1):177-207.
[9]ZENG D,VELDHUIS R,SPREEUWERS L.A Survey of Face Recognition Techniques under Occlusion [J].IET Biometrics,2021,10(6):581-606.
[10]DAMER N,GREBE J H,CHEN C,et al.The Effect of Wearing a Mask on Face Recognition Performance:An Exploratory Study [C]//Proceedings of the International Conference of the Biometrics Special Interest Group.New York:IEEE Press,2020:1-6.
[11]DAMER N,BOUTROS F,SüßMILCH M,et al.ExtendedEvaluation of the Effect of Real and Simulated Masks on Face Recognition Performance [J].IET Biometrics,2021,10(5):548-561.
[12]ZHAO F,FENG J,ZHAO J,et al.Robust Lstm-Autoencoders for Face De-Occlusion in the Wild [J].IEEE Transactions on Image Processing,2017,27(2):778-790.
[13]WAN W,CHEN J.Occlusion Robust Face Recognition Based on Mask Learning [C]//Proceedings of the IEEE/CVF International Conference on Image Processing.New York:IEEE Press,2017:3795-3799.
[14]SONG L,GONG D,LI Z,et al.Occlusion Robust Face Recognition Based on Mask Learning with Pairwise Differential Siamese Network [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.New York:IEEE Press,2019:773-782.
[15]YIN B,TRAN L,LI H,et al.Towards Interpretable Face Re-cognition [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.New York:IEEE Press,2019:9348-9357.
[16]QIU H,GONG D,LI Z,et al.End2end Occluded Face Recognition by Masking Corrupted Features [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2021,44(10):6939-6952.
[17]ZHANG Y,WANG X,SHAKEEL M S,et al.Learning Upper Patch Attention Using Dual-Branch Training Strategy for Masked Face Recognition [J].Pattern Recognition,2022,126:108522.
[18]HUANG B,WANG Z,WANG G,et al.Plface:ProgressiveLearning for Face Recognition with Mask Bias [J].Pattern Re-cognition,2023,135:109142.
[19]DENG H,FENG Z,QIAN G,et al.Mfcosface:A Masked-Face Recognition Algorithm Based on Large Margin Cosine Loss [J].Applied sciences,2021,11(16):7310.
[20]WANG C,FANG H,ZHONG Y,et al.Mlfw:A Database forFace Recognition on Masked Faces [C]//Proceedings of the Chinese Conference on Biometric Recognition.New York:Springer,2022:180-188.
[21]DU H,SHI H,LIU Y,et al.Towards Nir-Vis Masked Face Recognition [J].IEEE Signal Processing Letters,2021,28:768-772.
[22]ZHAO W,ZHU X,SHI H,et al.Consistent Sub-Decision Network for Low-Quality Masked Face Recognition [J].IEEE Signal Processing Letters,2022,29:1147-1151.
[23]LI X X,DAI D Q,ZHANG X F,et al.Structured Sparse Error Coding for Face Recognition with Occlusion [J].IEEE Transactions on Image Processing,2013,22(5):1889-1900.
[24]ULYANOV D,VEDALDI A,LEMPITSKY V.Deep ImagePrior [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2018:9446-9454.
[25]POURPANAH F,ABDAR M,LUO Y,et al.A Review of Ge-neralized Zero-Shot Learning Methods [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2022,45(4):4051-4070.
[26]DAI T,FENG Y,CHEN B,et al.Deep Image Prior Based Defense against Adversarial Examples [J].Pattern Recognition,2022,122:108249.
[27]YUAN X,HE P,ZHU Q,et al.Adversarial Examples:Attacks and Defenses for Deep Learning [J].IEEE Transactions on Neural Networks And Learning Systems,2019,30(9):2805-2824.
[28]GOODFELLOW I J,SHLENS J,SZEGEDY C.Explaining and Harnessing Adversarial Examples [C]//Proceedings of the International Conference on Learning Representations.2015.
[29]JIA X,WEI X,CAO X,et al.Comdefend:An Efficient Image Compression Model to Defend Adversarial Examples [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2019:6084-6092.
[30]MATAEV G,MILANFAR P,ELAD M.Deepred:Deep Image Prior Powered by Red [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops.IEEE Press,2019.
[31]LIU J,SUN Y,XU X,et al.Image Restoration Using Total Va-riation Regularized Deep Image Prior [C]//Proceedings of the IEEE International Conference on Acoustics,Speech and Signal Processing.New York:IEEE Press,2019:7715-7719.
[32]OTE K,HASHIMOTO F,ONISHI Y,et al.List-Mode PetImage Reconstruction Using Deep Image Prior [J].arXiv:2204.13404,2023.
[33]ZHOU Q,ZHOU C,HU H,et al.Towards the Automation of Deep Image Prior [J].arXiv:191107185,2019.
[34]ZIHAN Z,WAGNER A,MOBAHI H,et al.Face Recognition with Contiguous Occlusion Using Markov Random Fields [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.New York:IEEE Press,2009:1050-1057.
[35]LI X X,LIANG R,FENG Y,et al.Robust Face Recognitionwith Occlusion by Fusing Image Gradient Orientations with Markov Random Fields [C]//Proceedings of the International Conference on Intelligent Science and Big Data Engineering.New York:Springer,2015:431-440.
[36]LI X X,HAO P,HE L,et al.Image Gradient Orientations Embedded Structural Error Coding for Face Recognition with Occlusion [J].Journal of Ambient Intelligence and Humanized Computing,2020,11(6):2349-2367.
[37]DAHUA L,XIAOOU T.Quality-Driven Face Occlusion Detection and Recovery [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2007:1-7.
[38]HE R,ZHENG W,HU B.Maximum Correntropy Criterion for Robust Face Recognition [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2011,33(8):1561-1576.
[39]TZIMIROPOULOS G,ZAFEIRIOU S,PANTIC M.SubspaceLearning from Image Gradient Orientations [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2012,34(12):2454-2466.
[40]HE R,ZHENG W S,TAN T,et al.Half-Quadratic-Based Iterative Minimization for Robust Sparse Representation [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,36(2):261-275.
[41]LONG J,SHELHAMER E,DARRELL T.Fully Convolutional Networks for Semantic Segmentation [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:3431-3440.
[42]RONNEBERGER O,FISCHER P,BROX T.U-Net:Convolu-tional Networks for Biomedical Image Segmentation [C]//Proceedings of the International Conference on Medical Image Computing and Computer-assisted Intervention.New York:Springer,2015:234-241.
[43]MAO X,SHEN C,YANG Y B.Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections [C]//Proceedings of the Advances in Neural Information Processing Systems.Cambridge:MIT Press,2016:2802-2810.
[44]MENG Y,LEI Z,JIAN Y,et al.Robust Sparse Coding for Face Recognition [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2011:625-632.
[45]KIM J,LEE J K,LEE K M.Accurate Image Super-Resolution Using Very Deep Convolutional Networks [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2016:1646-1654.
[46]ZHOU Z,WAGNER A,MOBAHI H,et al.Face Recognitionwith Contiguous Occlusion Using Markov Random Fields [C]//Proceedings of the IEEE International Conference on Computer Vision.2009:1050-1057.
[47]LIANG R,LI X X.Mixed Error Coding for Face Recognition with Mixed Occlusions [C]//Proceedings of the International Joint Conference on Artificial Intelligence.Washington,DC:AAAI,2015.
[48]PARKHI O M,VEDALDI A,ZISSERMAN A.Deep Face Re-cognition [C]//Proceedings of the British Machine Vision Conference.Durham:BMVA Press,2015.
[49]SCHROFF F,KALENICHENKO D,PHILBIN J.FaceNet:AUnified Embedding for Face Recognition and Clustering [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2015:815-823.
[50]WU X,HE R,SUN Z.A Lightened CNN for Deep Face Representation[J].arXiv:1511.02683,2015.
[51]LIU W,WEN Y,YU Z,et al.Sphereface:Deep HypersphereEmbedding for Face Recognition [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2017:212-220.
[52]LIU W,WEN Y,RAJ B,et al.Sphereface Revived:UnifyingHyperspherical Face Recognition [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2022,45(2):2458-2474.
[53]CHAN T H,JIA K,GAO S,et al.PCANet:A Simple DeepLearning Baseline for Image Classification? [J].IEEE Transactions on Image Processing,2015,24(12):5017-5032.
[54]HUANG G B,MATTAR M,BERG T,et al.Labeled Faces in the Wild:A Database for Studying Face Recognition in Unconstrained Environments[J/OL].https://inria.hal.science/inria-00321923/document.
[55]WOLF L,HASSNER T,MAOZ I.Face Recognition in Unconstrained Videos with Matched Background Similarity [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.IEEE,2011:529-534.
[56]TAIGMAN Y,YANG M,RANZATO M A,et al.DeepFace:Closing the Gap to Human-Level Performance in Face Verification [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2014:1701-1708.
[57]WANG H,WANG Y,ZHOU Z,et al.Cosface:Large MarginCosine Loss for Deep Face Recognition [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2018:5265-5274.
[58]DENG J,GUO J,XUE N,et al.Arcface:Additive Angular Margin Loss for Deep Face Recognition [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2019:4690-4699.
[59]LEE K C,HO J,KRIEGMAN D J.Acquiring Linear Subspaces for Face Recognition under Variable Lighting [J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2005(5):684-698.
[60]MARTINEZ A M.The Ar Face Database[C]//CVC.1998.
[61]LI X X,HE L,HAO P,et al.Adaptive Weberfaces for Occlusion-Robust Face Representation and Recognition [J].IET Image Processing,2017,11(11):964-975.
[62]FIDLER S,SKOCAJ D.Combining Reconstructive and Discriminative Subspace Methods for Robust Classification and Regression by Subsampling [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2006(3):337-350.
[63]MURPHY K P.Machine Learning:A Probabilistic Perspective [M].Massachusetts:MIT Press,2012.
[64]WEI X,LI C T,HU Y.Robust Face Recognition under Varying Illumination and Occlusion Considering Structured Sparsity [C]//Proceedings of the International Conference on Digital Image Computing Techniques and Applications.New York:IEEE Press,2012:1-7.
[65]MEHDIPOUR G M,KEMAL E H.A Comprehensive Analysisof Deep Learning Based Representation for Face Recognition [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.New York:IEEE Press,2016:34-41.
[66]GUILLAUMIN M,VERBEEK J,SCHMID C.Is That You?Metric Learning Approaches for Face Identification [C]//Proceedings of the IEEE International Conference on Computer Vision.New York:IEEE Press,2009:498-505.
[67]PINTO N,DICARLO J J,COX D D.How Far Can You Get with a Modern Face Recognition Test Set Using Only Simple Features? [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2009:2591-2598.
[68]SUN Y,CHEN Y,WANG X,et al.Deep Learning Face Representation by Joint Identification-Verification [C]//Proceedings of the Advances in Neural Information Processing Systems.Cambridge:MIT Press,2014:1988-1996.
[69]SUN Y,WANG X,TANG X.Deep Learning Face Representation from Predicting 10,000 Classes [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2014:1891-1898.
[70]SUN Y,WANG X,TANG X.Deeply Learned Face Representations Are Sparse,Selective,and Robust [C]//Proceedings of the IEEE/CVF Conference on Computer vision and Pattern Re-cognition.New York:IEEE Press,2015:2892-2900.
[71]SUN Y,WANG X,TANG X.Sparsifying Neural Network Connections for Face Recognition [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2016:4856-4864.
[72]HUANG G,MATTAR M,LEE H,et al.Learning to Alignfrom Scratch [C]//Proceedings of the Advances in Neural Information Processing Systems.Cambridge:MIT Press,2012:764-772.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发