Computer Science ›› 2019, Vol. 46 ›› Issue (3): 53-62.doi: 10.11896/j.issn.1002-137X.2019.03.007

Special Issue: Face Recognition

• Surveys • Previous Articles     Next Articles

Survey on Non-frontal Facial Expression Recognition Methods

JIANG Bin1,GAN Yong1,ZHANG Huan-long2,ZHANG Qiu-wen1   

  1. (College of Computer and Communication Engineering,Zhengzhou University of Light Industry,Zhengzhou 450002,China)1
    (College of Electric and Information Engineering,Zhengzhou University of Light Industry,Zhengzhou 450002,China)2
  • Received:2018-05-21 Revised:2018-09-16 Online:2019-03-15 Published:2019-03-22

Abstract: Facial expression recognition is an important part in biometric feature recognition,and it is also a key techno-logy of human-machine interaction.However,most methods only focus on the frontal or nearly frontal facial images and videos,and restrict the normal head movements,so it is bad for intelligent development of facial expression recognition.To handle this problem,firstly,the face detection,head pose estimation,facial expression feature extraction and classification methods were introduced for exploring the development of non-frontal facial expression recognition system.Se-condly,the non-frontal facial expression feature extraction and classification methods were emphatically introduced,and the comparison and analysis of the facial key points-based non-frontal facial expression recognition algorithm,the appea-rance feature-based non-frontal facial expression recognition algorithm and the pose-depend-based non-frontal facial expression recognition algorithm were carried out.Finally,the current research on the non-frontal facial expression recognition was summarized,and the future research and development direction were prosected.

Key words: Appearance feature, Facial key points, Non-frontal facial expression recognition, Pose-depend

CLC Number: 

  • TP391
[1]EKMAN P,SORENSON E,FRIESEN W.Pan-Cultural Ele-
ments in Facial Displays of Emotion[J].Science,1969,164(3875):86-88.
[2]AVIEZER H,TROPE Y,TODOROV A.Body Cues,Not Facial Expressions,Discriminate between Intense Positive and Negative Emotions[J].Science,2012,338(6111):1225-1229.
[3]PANTIC M,ROTHKRANTZ L.Automatic Analysis of Facial Expressions:The State of The Art[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2000,22(12):1424-1445.
[4]TIAN Y L,KANADE T,COHN J.Recognizing Action Units
for Facial Expression Analysis[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2001,23(2):97-115.
[5]GAO W,JIN H.Analysis and research of facial expression ima-
ge[J].Chinese Journal of Computers,1997,20(9):782-789.(in Chinese)
高文,金辉.面部表情图像的分析与识别[J].计算机学报,1997,20(9):782-789.
[6]EKMAN P,COHEN L,MOOS R,et al.Divergent Reactions to The Threat of War[J].Science,1963,139(3550):88-94.
[7]YANG S F,KAFAI M,AN L,et al.Zapping Index:Using Smile to Measure Advertisement Zapping Likelihood[J].IEEE Transactions on Affective Computing,2014,5(4):432-444.
[8]TEIJEIRO-MOSQUERA L,BIEL J,ALBA-CASTRO J,et al.What Your Face Vlogs About:Expressions of Emotion and Big-Five Traits Impressions in YouTube[J].IEEE Transactions on Affective Computing,2015,6(2):193-205.
[9]RADHAMANI V,DALIN G.A Supporting Survey to Step into a Novel Approach for Providing Automated Emotion Recognition Service in Mobile Phones[C]∥Proceedings of the Second International Conference on Inventive Systems and Control.New York:IEEE Press,2018:35-39.
[10]SARIYANIDI E,GUNES H,CAVALLARO A.Automatic
Analysis of Facial Affect:A Survey of Registration,Representation,and Recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,37(6):1113-1133.
[11]LYONS M J,CAMPBELL R,PLANTE A,et al.The Noh Mask Effect Vertical Viewpoint Dependence of Facial Expression Perception[C]∥Proceedings of the Royal Society B,Biological Scien-ces.London:Royal Society of London,2000:2239-2245.
[12]NICHOLLS R,WOLFGANG J,CLODE D,et al.The Effect of Left and Right Poses on the Expression of Facial Emotion[J].Neuropsychologia,2002,40(10):1662-1665.
[13]SAMANTA A,GUHA T.On The Role of Head Motion in Affective Expression[C]∥IEEE International Conference on Acoustics,Speech and Signal Processing.New York:IEEE Press,2017:2886-2890.
[14]ZENG Z H,PANTIC M,ROISMAN I,et al.A Survey of Affect Recognition Methods:Audio,Visual,and Spontaneous Expressions[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2009,31(1):39-58.
[15]CORNEANU A,OLIU M,COHN F,et al.Survey on RGB,3D,Thermal,and Multimodal Approaches for Facial Expression Recognition:History,Trends,and Affect-Related Applications[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(8):1548-1568.
[16]EKMAN P,FRIESEN W.Facial Action Coding System:A
Technique for The Measurement of Facial Movement[M].Palo Alto:Consulting Psychologists Press,1978.
[17]PANTIC M,VALSTAR M,RADEMAKER R,et al.Web-Based Database for Facial Expression Analysis[C]∥IEEE Internatio-nal Conference on Multimedia and Expo.Amsterdam,Netherlands,New York:IEEE Press,2005:317-321.
[18]XUE Y L,MAO X,ZHANG F.Beihang University Facial Expression Database and Multiple Facial Expression Recognition[C]∥IEEE International Conference on Machine Learning and Cybernetics.New York:IEEE Press,2006:3282-3287.
[19]TONG Y,LIAO W H,JI Q.Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2007,29(10):1683-1699.
[20]GAO W,CAO B,SHAN S G,et al.The CAS-PEAL Large-Scale Chinese Face Database and Baseline Evaluations[J].IEEE Transactions on Systems,Man and Cybernetics,Part A,2008,38(1):149-161.
[21]LANGNER O,DOTSCH R,BIJLSTRA G,et al.Presentation and Validation of The Radboud Faces Database[J].Cognition & Emotion,2010,24(8):1377-1388.
[22]LUNDQVIST D,FLYKT A,HMAN A.The Karolinska Directed Emotional Faces-KDEF[R].Stockholm:Department of Clinical Neuroscience,Psychology section,Karolinska Institutet,1998.
[23]DHALL A,MURTHY O V,GOECKE R,et al.Video and Ima-
ge based Emotion Recognition Challenges in the Wild:EmotiW 2015[C]∥ACM International Conference on Multimodal Intera-ction.New York:ACM Press,2015:423-426.
[24]YIN L J,WEI X Z,SUN Y,et al.A 3D Facial Expression Database for Facial Behavior Research[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2006:211-216.
[25]ZHANG X,YIN L J,COHN J,et al.A High-Resolution 3D Dynamic Facial Expression Database[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2008:1-6.
[26]SAVRAN A,ALYZ N,DIBEKLIOGˇLU H.Bosphorus Data-
base for 3D Face Analysis[C]∥Proceedings of the First European Workshop on Biometrics & Identity Management.New York:IEEE Press,2008:47-56.
[27]ZHU X X,RAMANAN D.Face Detection,Pose Estimation and Landmark Localization in the Wild[C]∥IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2012:2879-2886.
[28]SETTY S,HUSAIN M,BEHAM P,et al.Indian Movie Face Database:A Benchmark for Face Recognition Under WideVariations[C]∥IEEE International Conference on Computer Vision,Pattern Recognition,Image Processing and Graphics.New York:IEEE Press,2013:1-5.
[29]VIOLA P,JONES M.Robust Real-Time Face Detection[J].International Journal of Computer Vision,2004,57(2):137-154.
[30]YACOOB Y,DAVIS L,BLACK M,et al.Looking at People in Action[M]∥Cipolla R,Pentland A,Eds.Computer Vision for Human-Machine Interaction.Cambridge:Cambridge University Press,1998:171-187.
[31]CECH J,FRANC V,MATAS J.A 3D Approach to Facial Landmarks:Detection,Refinement,and Tracking[C]∥IEEE International Conference on Pattern Recognition.New York:IEEE Press,2014:2173-2178.
[32]ZHANG H W,LI Q,SUN Z,et al.Combining Data-Driven and Model-Driven Methods for Robust Facial Landmark Detection[J].IEEE Transactions on Information Forensics and Security,arXiv:1611.10152.
[33]PANTIC M,PATRAS I.Dynamics of Facial Expression:Recognition of Facial Actions and Their Temporal Segments from Face Profile Image Sequences[J].IEEE Transactions on Systems,Man,and Cybernetics,Part B,2006,36(2):433-449.
[34]ROTH M,BAUML M,NEVATIA R,et al.Robust Multi-Pose Face Tracking by Multi-Stage Tracklet Association[C]∥IEEE International Conference on Pattern Recognition.New York:IEEE Press,2012:1012-1016.
[35]KBLBECK C,ERNST A.Face Detection and Tracking in Vi-
deo Sequences Using The Modified Census Transformation[J].Image & Vision Computing,2006,24(6):564-572.
[36]TANG C G,ZHENG W M,YAN J W,et al.View-Independent Facial Action Unit Detection[C]∥IEEE InternationalConfe-rence on Automatic Face & Gesture Recognition.New York:IEEE Press,2017:878-882.
[37]BATISTA C,ALBIERO V,BELLON P,et al.AUMPNet:Simul-
taneous Action Units Detection and Intensity Estimation on Multipose Facial Images Using A Single Convolutional Neural Network[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2017:866-871.
[38]BAN Y,KIM S K,KIM S,et al.Face Detection Based on Skin Color Likelihood[J].Pattern Recognition,2014,47(4):1573-1585.
[39]CHEN Y D,LI R F.A hierarchical model for joint object detection and pose estimation[J].Journal of Computer Research and Development,2015,52(11):2431-2440.(in Chinese)
陈耀东,李仁发.一种层次化的联合识别模型[J].计算机研究与发展,2015,52(11):2431-2440.
[40]MURPHY-CHUTORIAN E,TRIVEDI M.Head Pose Estimation in Computer Vision:A Survey[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2009,31(4):607-626.
[41]MA B P,SHAN S G,CHEN X L,et al.Robust appearance-
based method for head pose estimation[J].Journal of Software,2009,20(6):1651-1663.(in Chinese)
马丙鹏,山世光,陈熙霖,等.一种基于图像表观的鲁棒姿态估计方法[J].软件学报,2009,20(6):1651-1663.
[42]FLOHR F,DUMITRU-GUZU M,KOOIJ J,et al.A Probabilistic Framework for Joint Pedestrian Head and Body Orientation Estimation[J].IEEE Transactions on Intelligent Transportation Systems,2015,16(4):1872-1882.
[43]YAN Y,RICC E,SUBRAMANIAN R,et al.A Multi-Task
Learning Framework for Head Pose Estimation Under Target Motion[J].IEEE Transactions on Intelligent Transportation Systems,2016,38(6):1070-1083.
[44]KONG G S,MBOUNA R.Head Pose Estimation from A 2D
Face Image Using 3D Face Morphing with Depth Parameters[J].IEEE Transactions on Image Processing,2015,24(6):1801-1808.
[45]MUKHERJEE S,ROBERTSON N.Deep Head Pose:Gaze-Direction Estimation in Multimodal Video[J].IEEE Transactions on Multimedia,2015,17(11):2094-2107.
[46]PANTIC M,PATRAS I,ROTHKRANTZ L.Facial Mimics
Recognition from Face Profile Image Sequences[R].Delft:Delft University of Technology,2002.
[47]HU Y X,ZENG Z H,YIN L J,et al.A Study of Non-Frontal-View Facial Expressions Recognition[C]∥IEEE International Conference on Pattern Recognition.New York:IEEE Press,2008:1-4.
[48]SUNG J,KIM D.Pose-Robust Facial Expression Recognition
Using View-Based 2D+3D AAM[J].IEEE Transactions on Systems,Man,and Cybernetics,Part A,2008,38(4):852-866.
[49]HESSE N,GEHRIG T,GAO H,et al.Multi-View Facial Ex
pression Recognition Using Local Appearance Features[C]∥IEEE International Conference on Pattern Recognition.New York:IEEE Press,2012:3533-3536.
[50]TONG Y,CHEN J X,JI Q.A Unified Probabilistic Framework for Spontaneous Facial Action Modeling and Understanding[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2010,32(2):258-273.
[51]ZHENG W M,TANG H,LIN Z C,et al.A Novel Approach to Expression Recognition from Non-Frontal Face Images[C]∥IEEE International Conference on Computer Vision.New York:IEEE Press,2009:1901-1908.
[52]WU Y,WANG Z G,JI Q.Facial Feature Tracking Under Varying Facial Expressions and Face Poses Based on Restricted Boltzmann Machines[C]∥IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2013:3452-3459.
[53]ZHANG Z P,ZHANG W,DING H J,et al.Hierarchical Facial Landmark Localization via Cascaded Random Binary Patterns[J].Pattern Recognition,2014,48(4):1277-1288.
[54]SMITH B M,BRANDT J,LIN Z,et al.Nonparametric Context Modeling of Local Appearance for Pose- and Expression-Robust Facial Landmark Localization[C]∥IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2014:1741-1748.
[55]WU Y,WANG Z H,JI Q.A Hierarchical Probabilistic Model for Facial Feature Detection[C]∥IEEE Conference onCompu-ter Vision and Pattern Recognition.New York:IEEE Press,2014:1781-1788.
[56]KUMANO S,OTSUKA K,YAMATO J,et al.Pose-Invariant Facial Expression Recognition Using Variable-Intensity Templates[C]∥Proceedings of the 8th Asian Conference on Computer Vision.New York:IEEE Press,2007:324-334.
[57]HU Y X,ZENG Z H,YIN L J,et al.Multi-View Facial Expression Recognition[C]∥Proceedings of the 8th International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2008:1-6.
[58]ZHENG W M.Multi-View Facial Expression Recognition Based on Group Sparse Reduced-Rank Regression[J].IEEE Transactions on Affective Computing,2014,5(1):71-85.
[59]RUDOVIC O,PATRAS I,PANTIC M.Coupled Gaussian
Process Regression for Pose Invariant Facial Expression Reco-gnition[C]∥Proceedings of the 11th European Conference on Computer Vision.New York:IEEE Press,2010:350-363.
[60]ELEFTHERIADIS S,RUDOVIC O,PANTIC M.Discriminative
Shared Gaussian Processes for Multiview and View-Invariant Facial Expression Recognition[J].IEEE Transactions on Image Processing,2015,24(1):189-204.
[61]MOORE S,BOWDEN R.Local Binary Patterns for Multi-View Facial Expression Recognition[J].Computer Vision Image Understand,2011,115(4):541-558.
[62]ZHENG W M,TANG H,LIN Z C,et al.Emotion Recognition from Arbitrary View Facial Images[C]∥Proceedings of the 11th European Conference on Computer Vision.New York:IEEE Press,2010:490-503.
[63]WU J,LIN Z,ZHENG W,et al.Locality-Constrained Linear
Coding Based Bi-layer Model for Multi-view Facial Expression Recognition[J].Neurocomputing,2017,239(C):143-152.
[64]YUN W,KIM D,PARK C,et al.Multi-View Facial Expression Recognition Using Parametric Kernel Eigenspace Method Based on Class Features[C]∥IEEE International Conference on Systems,Man,and Cybernetics.New York:IEEE Press,2013:2689-2693.
[65]MAO Q R,RAO Q Y,YU Y B,et al.Hierarchical Bayesian
Theme Models for Multipose Facial Expression Recognition[J].IEEE Transactions on Multimedia,2017,19(4):861-873.
[66]DAPOGNY A,BAILLY K,DUBUISSON S.Dynamic Pose-Robust Facial Expression Recognition by Multi-View Pairwise Conditional Random Forests[OL].https://ieeexplore.ieee.org/document/7934067.
[67]VIERIU L,TULYAKOV S,SEMENIUTA S,et al.Facial Expression Recognition under a Wide Range of Head Poses[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2015:1-7.
[68]YANG B,CAO J M,NI R R,et al.Facial Expression Recognition Using Weighted Mixture Deep Neural Network Based on Double-Channel Facial Images[J].IEEE Access,2018,6:4630-4640.
[69]WU B F,LIN C.Adaptive Feature Mapping for Customizing
Deep Learning Based Facial Expression Recognition Model[J].IEEE Access,2018,6:12451-12461.
[70]LIU Y Y,ZENG J B,SHAN S G,et al.Multi-channel Pose-
aware Convolution Neural Networks for Multi-view Facial Expression Recognition[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2018:458-465.
[71]JAMPOUR M,MAUTHNER T,BISCHOF H.Pairwise Linear Regression:An Efficient and Fast Multi-View Facial Expression Recognition[C]∥Proceedings of the 11th IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2015.
[72]JAMPOUR M,LEPETIT V,MAUTHNER T,et al.Pose-specific Non-linear Mappings in Feature Space towards Multiview Facial Expression Recognition[J].Image & Vision Computing,2017,58:38-46.
[73]RUDOVIC O,PATRAS I,PANTIC M.Regression-Based Multi-View Facial Expression Recognition[C]∥Proceedings of the 20th International Conference on Pattern Recognition.New York:IEEE Press,2010:4121-4124.
[74]RUDOVIC O,PANTIC M,PATRAS I.Coupled Gaussian
Processes for Pose-Invariant Facial Expression Recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(6):1357-1369.
[75]LAI Y H,LAI S H.Emotion-Preserving Representation Lear-
ning via Generative Adversarial Network for Multi-view Facial Expression Recognition[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2018:263-270.
[76]GNEY F,ARAR N M,FISCHER M,et al.Cross-Pose Facial Expression Recognition[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2013.
[77]TARIQ U,YANG J C,HUANG T S.Supervised Super-Vector Encoding for Facial Expression Recognition[J].Pattern Recognition Letters,2014,46(3):89-95.
[78]HOSOI T.Head Pose and Expression Transfer Using Facial
Status Score[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2017:573-580.
[79]ZHOU Y,PI J,SHI B E.Pose-Independent Facial Action Unit Intensity Regression Based on Multi-Task Deep Transfer Lear-ning[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2017:872-877.
[80]ZHENG W M,ZONG Y,ZHOU X Y,et al.Cross-Domain Color Facial Expression Recognition Using Transductive Transfer Subspace Learning[J].IEEE Transactions on Affective Computing,2018,9(1):21-37.
[81]KAYA H,GRPINAR F,SALAH A A.Video-Based Emotion Recognition in the Wild using Deep Transfer Learning and Score Fusion[J].Image & Vision Computing,2017,65:66-75.
[82]WEBER R,BARRIELLE V,SOLADI C,et al.Unsupervised
Adaptation of A Person-Specific Manifold of Facial Expressions[J].IEEE Transactions on Affective Computing,2018.doi:10.1109/TAFFC.2018.2807430.
[1] CHEN Zhi-qiang, HAN Meng, LI Mu-hang, WU Hong-xin, ZHANG Xi-long. Survey of Concept Drift Handling Methods in Data Streams [J]. Computer Science, 2022, 49(9): 14-32.
[2] WANG Ming, WU Wen-fang, WANG Da-ling, FENG Shi, ZHANG Yi-fei. Generative Link Tree:A Counterfactual Explanation Generation Approach with High Data Fidelity [J]. Computer Science, 2022, 49(9): 33-40.
[3] ZHANG Jia, DONG Shou-bin. Cross-domain Recommendation Based on Review Aspect-level User Preference Transfer [J]. Computer Science, 2022, 49(9): 41-47.
[4] ZHOU Fang-quan, CHENG Wei-qing. Sequence Recommendation Based on Global Enhanced Graph Neural Network [J]. Computer Science, 2022, 49(9): 55-63.
[5] SONG Jie, LIANG Mei-yu, XUE Zhe, DU Jun-ping, KOU Fei-fei. Scientific Paper Heterogeneous Graph Node Representation Learning Method Based onUnsupervised Clustering Level [J]. Computer Science, 2022, 49(9): 64-69.
[6] CHAI Hui-min, ZHANG Yong, FANG Min. Aerial Target Grouping Method Based on Feature Similarity Clustering [J]. Computer Science, 2022, 49(9): 70-75.
[7] ZHENG Wen-ping, LIU Mei-lin, YANG Gui. Community Detection Algorithm Based on Node Stability and Neighbor Similarity [J]. Computer Science, 2022, 49(9): 83-91.
[8] LYU Xiao-feng, ZHAO Shu-liang, GAO Heng-da, WU Yong-liang, ZHANG Bao-qi. Short Texts Feautre Enrichment Method Based on Heterogeneous Information Network [J]. Computer Science, 2022, 49(9): 92-100.
[9] XU Tian-hui, GUO Qiang, ZHANG Cai-ming. Time Series Data Anomaly Detection Based on Total Variation Ratio Separation Distance [J]. Computer Science, 2022, 49(9): 101-110.
[10] NIE Xiu-shan, PAN Jia-nan, TAN Zhi-fang, LIU Xin-fang, GUO Jie, YIN Yi-long. Overview of Natural Language Video Localization [J]. Computer Science, 2022, 49(9): 111-122.
[11] CAO Xiao-wen, LIANG Mei-yu, LU Kang-kang. Fine-grained Semantic Reasoning Based Cross-media Dual-way Adversarial Hashing Learning Model [J]. Computer Science, 2022, 49(9): 123-131.
[12] ZHOU Xu, QIAN Sheng-sheng, LI Zhang-ming, FANG Quan, XU Chang-sheng. Dual Variational Multi-modal Attention Network for Incomplete Social Event Classification [J]. Computer Science, 2022, 49(9): 132-138.
[13] DAI Yu, XU Lin-feng. Cross-image Text Reading Method Based on Text Line Matching [J]. Computer Science, 2022, 49(9): 139-145.
[14] QU Qian-wen, CHE Xiao-ping, QU Chen-xin, LI Jin-ru. Study on Information Perception Based User Presence in Virtual Reality [J]. Computer Science, 2022, 49(9): 146-154.
[15] ZHOU Le-yuan, ZHANG Jian-hua, YUAN Tian-tian, CHEN Sheng-yong. Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion [J]. Computer Science, 2022, 49(9): 155-161.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!