Computer Science

Computer Science ›› 2024, Vol. 51 ›› Issue (7): 244-256.doi: 10.11896/jsjkx.230400127

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Occluded Face Recognition Based on Deep Image Prior and Robust Markov Random Field

LI Xiaoxin1, DING Weijie1,2, FANG Yi1, ZHANG Yuancheng1, WANG Qihui3   

  1. 1 College of Computer Science and Technology,Zhejiang University of Technology,Hangzhou 310023,China
    2 Department of Computer and Information Security,Zhejiang Police College,Hangzhou 310053,China
    3 Qianjiang College of Hangzhou Normal University,Hangzhou 311121,China
  • Received:2023-04-18 Revised:2024-01-14 Online:2024-07-15 Published:2024-07-10
  • About author:LI Xiaoxin,born in 1980,Ph.D,asso-ciate professor,master supervisor,is a member of CCF(No.80065M).His main research interests include image processing and pattern recognition.
    DING Weijie,born in 1981,Ph.D,professor.His main research interests include information visualization and network security.
  • Supported by:
    Natural Science Foundation of Zhejiang Province,China(LGF22F020027),Special Project of Ministry of Public Security(2020GABJC35) and National Natural Science Foundation of China(62271448).

PDF (PC)

443

Abstract: The occlusion-caused difference between test and training images is one of the most challenging issues for real-world face recognition system.Most of the existing occluded face recognition methods based on deep neural networks(DNNs) need to use large-scale occluded face images to train network models.However,any external object in the real world might become occlusions,and limited training data cannot exhaust all possible objects.Also,using large-scale occluded face images to train networks violates the human visual mechanism,the human eyes detect occlusions by only using small-scale unoccluded face images without seeing any occlusions.In order to simulate the occlusion detection mechanism of human vision,we combine the deep image prior with the robust Markov random field model to construct a novel occlusion detection model,namely DIP-rMRF,based on small-scale data,and propose a uniform zero filling method to effectively utilize the occlusion detection resultsof DIP-rMRF.Experimental resultsofsix advanced DNN-based face recognitions methods,including VGGFace,LCNN,PCANet,SphereFace,InterpretFR and FROM,on three face datasets,including Extended Yale B,AR and LFW,show that DIP-rMRF can effectively preprocess face images with occlusions and quasi-occlusions caused by extreme illuminations,and greatly improve the performance of the existing DNN models for face recognition with occlusion.

Key words: Face recognition with occlusion, Deep image prior, Robust Markov random field, Uniform zero-filling, Structural error metric

CLC Number: 

  • TP391
[1]WANG M,DENG W.Deep Face Recognition:A Survey [J].Neurocomputing,2021,429:215-244.
[2]RANJAN R,SANKARANARAYANAN S,BANSAL A,et al.Deep Learning for Understanding Faces:Machines May Be Just as Good,or Better,Than Humans [J].IEEE Signal Processing Magazine,2018,35(1):66-83.
[3]DING C,TAO D.A Comprehensive Survey on Pose-InvariantFace Recognition [J].ACM Transactions on Intelligent Systems and Technology(TIST),2016,7(3):1-42.
[4]BENDJILLALI R I,BELADGHAM M,MERIT K,et al.Illumination-Robust Face Recognition Based on Deep Convolutional Neural Networks Architectures [J].Indonesian Journal of Electrical Engineering and Computer Science,2020,18(2):1015-1027.
[5]HAN X,LIU Y,YANG H,et al.Normalization of Face Illumination with Photorealistic Texture Via Deep Image Prior Synthesis [J].Neurocomputing,2020,386:305-316.
[6]WANG K,PENG X,YANG J,et al.Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition [J].IEEE Transactions on Image Processing,2020,29:4057-4069.
[7]REVINA I M,EMMANUEL W S.A Survey on Human FaceExpression Recognition Techniques [J].Journal of King Saud University-Computer and Information Sciences,2021,33(6):619-628.
[8]LI X,LIANG R.A Review for Face Recognition with Occlu-sion:From Subspace Regression to Deep Learning [J].Chinese Journal of Computers,2018,41(1):177-207.
[9]ZENG D,VELDHUIS R,SPREEUWERS L.A Survey of Face Recognition Techniques under Occlusion [J].IET Biometrics,2021,10(6):581-606.
[10]DAMER N,GREBE J H,CHEN C,et al.The Effect of Wearing a Mask on Face Recognition Performance:An Exploratory Study [C]//Proceedings of the International Conference of the Biometrics Special Interest Group.New York:IEEE Press,2020:1-6.
[11]DAMER N,BOUTROS F,SüßMILCH M,et al.ExtendedEvaluation of the Effect of Real and Simulated Masks on Face Recognition Performance [J].IET Biometrics,2021,10(5):548-561.
[12]ZHAO F,FENG J,ZHAO J,et al.Robust Lstm-Autoencoders for Face De-Occlusion in the Wild [J].IEEE Transactions on Image Processing,2017,27(2):778-790.
[13]WAN W,CHEN J.Occlusion Robust Face Recognition Based on Mask Learning [C]//Proceedings of the IEEE/CVF International Conference on Image Processing.New York:IEEE Press,2017:3795-3799.
[14]SONG L,GONG D,LI Z,et al.Occlusion Robust Face Recognition Based on Mask Learning with Pairwise Differential Siamese Network [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.New York:IEEE Press,2019:773-782.
[15]YIN B,TRAN L,LI H,et al.Towards Interpretable Face Re-cognition [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.New York:IEEE Press,2019:9348-9357.
[16]QIU H,GONG D,LI Z,et al.End2end Occluded Face Recognition by Masking Corrupted Features [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2021,44(10):6939-6952.
[17]ZHANG Y,WANG X,SHAKEEL M S,et al.Learning Upper Patch Attention Using Dual-Branch Training Strategy for Masked Face Recognition [J].Pattern Recognition,2022,126:108522.
[18]HUANG B,WANG Z,WANG G,et al.Plface:ProgressiveLearning for Face Recognition with Mask Bias [J].Pattern Re-cognition,2023,135:109142.
[19]DENG H,FENG Z,QIAN G,et al.Mfcosface:A Masked-Face Recognition Algorithm Based on Large Margin Cosine Loss [J].Applied sciences,2021,11(16):7310.
[20]WANG C,FANG H,ZHONG Y,et al.Mlfw:A Database forFace Recognition on Masked Faces [C]//Proceedings of the Chinese Conference on Biometric Recognition.New York:Springer,2022:180-188.
[21]DU H,SHI H,LIU Y,et al.Towards Nir-Vis Masked Face Recognition [J].IEEE Signal Processing Letters,2021,28:768-772.
[22]ZHAO W,ZHU X,SHI H,et al.Consistent Sub-Decision Network for Low-Quality Masked Face Recognition [J].IEEE Signal Processing Letters,2022,29:1147-1151.
[23]LI X X,DAI D Q,ZHANG X F,et al.Structured Sparse Error Coding for Face Recognition with Occlusion [J].IEEE Transactions on Image Processing,2013,22(5):1889-1900.
[24]ULYANOV D,VEDALDI A,LEMPITSKY V.Deep ImagePrior [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2018:9446-9454.
[25]POURPANAH F,ABDAR M,LUO Y,et al.A Review of Ge-neralized Zero-Shot Learning Methods [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2022,45(4):4051-4070.
[26]DAI T,FENG Y,CHEN B,et al.Deep Image Prior Based Defense against Adversarial Examples [J].Pattern Recognition,2022,122:108249.
[27]YUAN X,HE P,ZHU Q,et al.Adversarial Examples:Attacks and Defenses for Deep Learning [J].IEEE Transactions on Neural Networks And Learning Systems,2019,30(9):2805-2824.
[28]GOODFELLOW I J,SHLENS J,SZEGEDY C.Explaining and Harnessing Adversarial Examples [C]//Proceedings of the International Conference on Learning Representations.2015.
[29]JIA X,WEI X,CAO X,et al.Comdefend:An Efficient Image Compression Model to Defend Adversarial Examples [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2019:6084-6092.
[30]MATAEV G,MILANFAR P,ELAD M.Deepred:Deep Image Prior Powered by Red [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops.IEEE Press,2019.
[31]LIU J,SUN Y,XU X,et al.Image Restoration Using Total Va-riation Regularized Deep Image Prior [C]//Proceedings of the IEEE International Conference on Acoustics,Speech and Signal Processing.New York:IEEE Press,2019:7715-7719.
[32]OTE K,HASHIMOTO F,ONISHI Y,et al.List-Mode PetImage Reconstruction Using Deep Image Prior [J].arXiv:2204.13404,2023.
[33]ZHOU Q,ZHOU C,HU H,et al.Towards the Automation of Deep Image Prior [J].arXiv:191107185,2019.
[34]ZIHAN Z,WAGNER A,MOBAHI H,et al.Face Recognition with Contiguous Occlusion Using Markov Random Fields [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.New York:IEEE Press,2009:1050-1057.
[35]LI X X,LIANG R,FENG Y,et al.Robust Face Recognitionwith Occlusion by Fusing Image Gradient Orientations with Markov Random Fields [C]//Proceedings of the International Conference on Intelligent Science and Big Data Engineering.New York:Springer,2015:431-440.
[36]LI X X,HAO P,HE L,et al.Image Gradient Orientations Embedded Structural Error Coding for Face Recognition with Occlusion [J].Journal of Ambient Intelligence and Humanized Computing,2020,11(6):2349-2367.
[37]DAHUA L,XIAOOU T.Quality-Driven Face Occlusion Detection and Recovery [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2007:1-7.
[38]HE R,ZHENG W,HU B.Maximum Correntropy Criterion for Robust Face Recognition [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2011,33(8):1561-1576.
[39]TZIMIROPOULOS G,ZAFEIRIOU S,PANTIC M.SubspaceLearning from Image Gradient Orientations [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2012,34(12):2454-2466.
[40]HE R,ZHENG W S,TAN T,et al.Half-Quadratic-Based Iterative Minimization for Robust Sparse Representation [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,36(2):261-275.
[41]LONG J,SHELHAMER E,DARRELL T.Fully Convolutional Networks for Semantic Segmentation [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:3431-3440.
[42]RONNEBERGER O,FISCHER P,BROX T.U-Net:Convolu-tional Networks for Biomedical Image Segmentation [C]//Proceedings of the International Conference on Medical Image Computing and Computer-assisted Intervention.New York:Springer,2015:234-241.
[43]MAO X,SHEN C,YANG Y B.Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections [C]//Proceedings of the Advances in Neural Information Processing Systems.Cambridge:MIT Press,2016:2802-2810.
[44]MENG Y,LEI Z,JIAN Y,et al.Robust Sparse Coding for Face Recognition [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2011:625-632.
[45]KIM J,LEE J K,LEE K M.Accurate Image Super-Resolution Using Very Deep Convolutional Networks [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2016:1646-1654.
[46]ZHOU Z,WAGNER A,MOBAHI H,et al.Face Recognitionwith Contiguous Occlusion Using Markov Random Fields [C]//Proceedings of the IEEE International Conference on Computer Vision.2009:1050-1057.
[47]LIANG R,LI X X.Mixed Error Coding for Face Recognition with Mixed Occlusions [C]//Proceedings of the International Joint Conference on Artificial Intelligence.Washington,DC:AAAI,2015.
[48]PARKHI O M,VEDALDI A,ZISSERMAN A.Deep Face Re-cognition [C]//Proceedings of the British Machine Vision Conference.Durham:BMVA Press,2015.
[49]SCHROFF F,KALENICHENKO D,PHILBIN J.FaceNet:AUnified Embedding for Face Recognition and Clustering [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2015:815-823.
[50]WU X,HE R,SUN Z.A Lightened CNN for Deep Face Representation[J].arXiv:1511.02683,2015.
[51]LIU W,WEN Y,YU Z,et al.Sphereface:Deep HypersphereEmbedding for Face Recognition [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2017:212-220.
[52]LIU W,WEN Y,RAJ B,et al.Sphereface Revived:UnifyingHyperspherical Face Recognition [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2022,45(2):2458-2474.
[53]CHAN T H,JIA K,GAO S,et al.PCANet:A Simple DeepLearning Baseline for Image Classification? [J].IEEE Transactions on Image Processing,2015,24(12):5017-5032.
[54]HUANG G B,MATTAR M,BERG T,et al.Labeled Faces in the Wild:A Database for Studying Face Recognition in Unconstrained Environments[J/OL].https://inria.hal.science/inria-00321923/document.
[55]WOLF L,HASSNER T,MAOZ I.Face Recognition in Unconstrained Videos with Matched Background Similarity [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.IEEE,2011:529-534.
[56]TAIGMAN Y,YANG M,RANZATO M A,et al.DeepFace:Closing the Gap to Human-Level Performance in Face Verification [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2014:1701-1708.
[57]WANG H,WANG Y,ZHOU Z,et al.Cosface:Large MarginCosine Loss for Deep Face Recognition [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2018:5265-5274.
[58]DENG J,GUO J,XUE N,et al.Arcface:Additive Angular Margin Loss for Deep Face Recognition [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2019:4690-4699.
[59]LEE K C,HO J,KRIEGMAN D J.Acquiring Linear Subspaces for Face Recognition under Variable Lighting [J].IEEE Transactions on Pattern Analysis & Machine Intelligence,2005(5):684-698.
[60]MARTINEZ A M.The Ar Face Database[C]//CVC.1998.
[61]LI X X,HE L,HAO P,et al.Adaptive Weberfaces for Occlusion-Robust Face Representation and Recognition [J].IET Image Processing,2017,11(11):964-975.
[62]FIDLER S,SKOCAJ D.Combining Reconstructive and Discriminative Subspace Methods for Robust Classification and Regression by Subsampling [J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2006(3):337-350.
[63]MURPHY K P.Machine Learning:A Probabilistic Perspective [M].Massachusetts:MIT Press,2012.
[64]WEI X,LI C T,HU Y.Robust Face Recognition under Varying Illumination and Occlusion Considering Structured Sparsity [C]//Proceedings of the International Conference on Digital Image Computing Techniques and Applications.New York:IEEE Press,2012:1-7.
[65]MEHDIPOUR G M,KEMAL E H.A Comprehensive Analysisof Deep Learning Based Representation for Face Recognition [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.New York:IEEE Press,2016:34-41.
[66]GUILLAUMIN M,VERBEEK J,SCHMID C.Is That You?Metric Learning Approaches for Face Identification [C]//Proceedings of the IEEE International Conference on Computer Vision.New York:IEEE Press,2009:498-505.
[67]PINTO N,DICARLO J J,COX D D.How Far Can You Get with a Modern Face Recognition Test Set Using Only Simple Features? [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2009:2591-2598.
[68]SUN Y,CHEN Y,WANG X,et al.Deep Learning Face Representation by Joint Identification-Verification [C]//Proceedings of the Advances in Neural Information Processing Systems.Cambridge:MIT Press,2014:1988-1996.
[69]SUN Y,WANG X,TANG X.Deep Learning Face Representation from Predicting 10,000 Classes [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2014:1891-1898.
[70]SUN Y,WANG X,TANG X.Deeply Learned Face Representations Are Sparse,Selective,and Robust [C]//Proceedings of the IEEE/CVF Conference on Computer vision and Pattern Re-cognition.New York:IEEE Press,2015:2892-2900.
[71]SUN Y,WANG X,TANG X.Sparsifying Neural Network Connections for Face Recognition [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2016:4856-4864.
[72]HUANG G,MATTAR M,LEE H,et al.Learning to Alignfrom Scratch [C]//Proceedings of the Advances in Neural Information Processing Systems.Cambridge:MIT Press,2012:764-772.
[1] HUANG Weijie, GUO Xianwei, YU Zhiyong, HUANG Fangwan. Active Sampling of Air Quality Based on Compressed Sensing Adaptive Measurement Matrix [J]. Computer Science, 2024, 51(7): 116-123.
[2] SUN Haowen, DING Jiaman, LI Bowen, JIA Lianyin. Clustering Algorithm Based on Attribute Similarity and Distributed Structure Connectivity [J]. Computer Science, 2024, 51(7): 124-132.
[3] YANG Zhenzhen, WANG Dongtao, YANG Yongpeng, HUA Renyu. Multi-embedding Fusion Based on top-N Recommendation [J]. Computer Science, 2024, 51(7): 140-145.
[4] HU Haibo, YANG Dan, NIE Tiezheng, KOU Yue. Graph Contrastive Learning Incorporating Multi-influence and Preference for Social Recommendation [J]. Computer Science, 2024, 51(7): 146-155.
[5] LI Jiaying, LIANG Yudong, LI Shaoji, ZHANG Kunpeng, ZHANG Chao. Study on Algorithm of Depth Image Super-resolution Guided by High-frequency Information ofColor Images [J]. Computer Science, 2024, 51(7): 197-205.
[6] LOU Zhengzheng, ZHANG Xin, HU Shizhe, WU Yunpeng. Foggy Weather Object Detection Method Based on YOLOX_s [J]. Computer Science, 2024, 51(7): 206-213.
[7] LI Yongjie, QIAN Yi, WEN Yimin. Image Captioning Generation Method Based on External Prior and Self-prior Attention [J]. Computer Science, 2024, 51(7): 214-220.
[8] CHEN Jie, JIN Linjiang, ZHENG Hongbo, QIN Xujia. Deep Feature Learning and Feature Clustering of Streamlines in 3D Flow Fields [J]. Computer Science, 2024, 51(7): 221-228.
[9] LEI Yongsheng, DING Meng, SHEN Yao, LI Juhao, ZHAO Dongyue, CHEN Fushi. Action Recognition Model Based on Improved Two Stream Vision Transformer [J]. Computer Science, 2024, 51(7): 229-235.
[10] CAI Wenliang, HUANG Jun. Lane Detection Method Based on RepVGG [J]. Computer Science, 2024, 51(7): 236-243.
[11] MAO Xingjing, WEI Yong, YANG Yurui, JU Shenggen. KHGAS:Keywords Guided Heterogeneous Graph for Abstractive Summarization [J]. Computer Science, 2024, 51(7): 278-286.
[12] YAN Jingtao, LI Yang, WANG Suge, PAN Bangze. Overlap Event Extraction Method with Language Granularity Fusion Based on Joint Learning [J]. Computer Science, 2024, 51(7): 287-295.
[13] WENG Yu, LUO Haoyu, Chaomurilige, LIU Xuan , DONG Jun, LIU Zheng. CINOSUM:An Extractive Summarization Model for Low-resource Multi-ethnic Language [J]. Computer Science, 2024, 51(7): 296-302.
[14] ZHANG Hui, ZHANG Xiaoxiong, DING Kun, LIU Shanshan. Device Fault Inference and Prediction Method Based on Dynamic Graph Representation [J]. Computer Science, 2024, 51(7): 310-318.
[15] SHI Dianxi, GAO Yunqi, SONG Linna, LIU Zhe, ZHOU Chenlei, CHEN Ying. Deep-Init:Non Joint Initialization Method for Visual Inertial Odometry Based on Deep Learning [J]. Computer Science, 2024, 51(7): 327-336.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.