计算机科学

计算机科学 ›› 2019, Vol. 46 ›› Issue (3): 53-62.doi: 10.11896/j.issn.1002-137X.2019.03.007

所属专题： 人脸识别

• 综述 • 上一篇    下一篇

非正面人脸表情识别方法综述

蒋斌1,甘勇1,张焕龙2,张秋闻1   

  1. (郑州轻工业学院计算机与通信工程学院 郑州 450002)1
    (郑州轻工业学院电气信息工程学院 郑州 450002)2
  • 收稿日期:2018-05-21 修回日期:2018-09-16 出版日期:2019-03-15 发布日期:2019-03-22
  • 作者简介:蒋斌(1983-),男,博士,讲师,主要研究方向为图像处理与模式识别,E-mail:jiangbin@zzuli.edu.cn(通信作者);甘勇(1965-),男,博士,教授,硕士生导师,主要研究方向为图像处理与模式识别、计算机网络及应用;张焕龙(1981-),男,博士,讲师,硕士生导师,主要研究方向为图像处理与模式识别、智能监控系统设计与开发、公共区域异常行为检测与分析、多源传感器信息融合技术;张秋闻(1982-),男,博士,副教授,硕士生导师,主要研究方向为图像处理与模式识别。
  • 基金资助:
    国家自然科学基金项目(61702464,61503173,61771432,61302118,61502435),河南省高等学校重点科研项目(16A520028,17B510011,14A520034),河南省科技攻关项目(182102210156),河南省高校科技创新人才项目(17HASTIT022),河南省青年骨干教师计划(2016GGJS-087),郑州轻工业学院博士科研基金项目(2014BSJJ077)资助

Survey on Non-frontal Facial Expression Recognition Methods

JIANG Bin1,GAN Yong1,ZHANG Huan-long2,ZHANG Qiu-wen1   

  1. (College of Computer and Communication Engineering,Zhengzhou University of Light Industry,Zhengzhou 450002,China)1
    (College of Electric and Information Engineering,Zhengzhou University of Light Industry,Zhengzhou 450002,China)2
  • Received:2018-05-21 Revised:2018-09-16 Online:2019-03-15 Published:2019-03-22

PDF (PC)

1454

摘要： 人脸表情识别是生物特征识别的重要组成部分,也是人机交互领域的一项关键技术。然而多数方法只关注正面或接近正面的人脸图像及视频,并限制了正常的头部运动,因此不利于人脸表情识别的智能化发展。有鉴于此,首先介绍了人脸检测、头部姿态估计、特征提取和分类的方法,以探索非正面人脸表情识别系统的发展;然后重点介绍了非正面人脸表情特征提取和分类的方法,并对基于人脸关键点的非正面人脸表情识别方法、基于外貌特征的非正面人脸表情识别方法及基于姿态相关的非正面人脸表情识别方法进行了比较分析;最后总结了非正面人脸表情识别方法的研究现状,并对进一步的研究和发展方向进行了展望。

关键词: 非正面人脸表情识别, 人脸关键点, 外貌特征, 姿态相关

Abstract: Facial expression recognition is an important part in biometric feature recognition,and it is also a key techno-logy of human-machine interaction.However,most methods only focus on the frontal or nearly frontal facial images and videos,and restrict the normal head movements,so it is bad for intelligent development of facial expression recognition.To handle this problem,firstly,the face detection,head pose estimation,facial expression feature extraction and classification methods were introduced for exploring the development of non-frontal facial expression recognition system.Se-condly,the non-frontal facial expression feature extraction and classification methods were emphatically introduced,and the comparison and analysis of the facial key points-based non-frontal facial expression recognition algorithm,the appea-rance feature-based non-frontal facial expression recognition algorithm and the pose-depend-based non-frontal facial expression recognition algorithm were carried out.Finally,the current research on the non-frontal facial expression recognition was summarized,and the future research and development direction were prosected.

Key words: Appearance feature, Facial key points, Non-frontal facial expression recognition, Pose-depend

中图分类号: 

  • TP391
[1]EKMAN P,SORENSON E,FRIESEN W.Pan-Cultural Ele-
ments in Facial Displays of Emotion[J].Science,1969,164(3875):86-88.
[2]AVIEZER H,TROPE Y,TODOROV A.Body Cues,Not Facial Expressions,Discriminate between Intense Positive and Negative Emotions[J].Science,2012,338(6111):1225-1229.
[3]PANTIC M,ROTHKRANTZ L.Automatic Analysis of Facial Expressions:The State of The Art[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2000,22(12):1424-1445.
[4]TIAN Y L,KANADE T,COHN J.Recognizing Action Units
for Facial Expression Analysis[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2001,23(2):97-115.
[5]GAO W,JIN H.Analysis and research of facial expression ima-
ge[J].Chinese Journal of Computers,1997,20(9):782-789.(in Chinese)
高文,金辉.面部表情图像的分析与识别[J].计算机学报,1997,20(9):782-789.
[6]EKMAN P,COHEN L,MOOS R,et al.Divergent Reactions to The Threat of War[J].Science,1963,139(3550):88-94.
[7]YANG S F,KAFAI M,AN L,et al.Zapping Index:Using Smile to Measure Advertisement Zapping Likelihood[J].IEEE Transactions on Affective Computing,2014,5(4):432-444.
[8]TEIJEIRO-MOSQUERA L,BIEL J,ALBA-CASTRO J,et al.What Your Face Vlogs About:Expressions of Emotion and Big-Five Traits Impressions in YouTube[J].IEEE Transactions on Affective Computing,2015,6(2):193-205.
[9]RADHAMANI V,DALIN G.A Supporting Survey to Step into a Novel Approach for Providing Automated Emotion Recognition Service in Mobile Phones[C]∥Proceedings of the Second International Conference on Inventive Systems and Control.New York:IEEE Press,2018:35-39.
[10]SARIYANIDI E,GUNES H,CAVALLARO A.Automatic
Analysis of Facial Affect:A Survey of Registration,Representation,and Recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,37(6):1113-1133.
[11]LYONS M J,CAMPBELL R,PLANTE A,et al.The Noh Mask Effect Vertical Viewpoint Dependence of Facial Expression Perception[C]∥Proceedings of the Royal Society B,Biological Scien-ces.London:Royal Society of London,2000:2239-2245.
[12]NICHOLLS R,WOLFGANG J,CLODE D,et al.The Effect of Left and Right Poses on the Expression of Facial Emotion[J].Neuropsychologia,2002,40(10):1662-1665.
[13]SAMANTA A,GUHA T.On The Role of Head Motion in Affective Expression[C]∥IEEE International Conference on Acoustics,Speech and Signal Processing.New York:IEEE Press,2017:2886-2890.
[14]ZENG Z H,PANTIC M,ROISMAN I,et al.A Survey of Affect Recognition Methods:Audio,Visual,and Spontaneous Expressions[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2009,31(1):39-58.
[15]CORNEANU A,OLIU M,COHN F,et al.Survey on RGB,3D,Thermal,and Multimodal Approaches for Facial Expression Recognition:History,Trends,and Affect-Related Applications[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(8):1548-1568.
[16]EKMAN P,FRIESEN W.Facial Action Coding System:A
Technique for The Measurement of Facial Movement[M].Palo Alto:Consulting Psychologists Press,1978.
[17]PANTIC M,VALSTAR M,RADEMAKER R,et al.Web-Based Database for Facial Expression Analysis[C]∥IEEE Internatio-nal Conference on Multimedia and Expo.Amsterdam,Netherlands,New York:IEEE Press,2005:317-321.
[18]XUE Y L,MAO X,ZHANG F.Beihang University Facial Expression Database and Multiple Facial Expression Recognition[C]∥IEEE International Conference on Machine Learning and Cybernetics.New York:IEEE Press,2006:3282-3287.
[19]TONG Y,LIAO W H,JI Q.Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2007,29(10):1683-1699.
[20]GAO W,CAO B,SHAN S G,et al.The CAS-PEAL Large-Scale Chinese Face Database and Baseline Evaluations[J].IEEE Transactions on Systems,Man and Cybernetics,Part A,2008,38(1):149-161.
[21]LANGNER O,DOTSCH R,BIJLSTRA G,et al.Presentation and Validation of The Radboud Faces Database[J].Cognition & Emotion,2010,24(8):1377-1388.
[22]LUNDQVIST D,FLYKT A,HMAN A.The Karolinska Directed Emotional Faces-KDEF[R].Stockholm:Department of Clinical Neuroscience,Psychology section,Karolinska Institutet,1998.
[23]DHALL A,MURTHY O V,GOECKE R,et al.Video and Ima-
ge based Emotion Recognition Challenges in the Wild:EmotiW 2015[C]∥ACM International Conference on Multimodal Intera-ction.New York:ACM Press,2015:423-426.
[24]YIN L J,WEI X Z,SUN Y,et al.A 3D Facial Expression Database for Facial Behavior Research[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2006:211-216.
[25]ZHANG X,YIN L J,COHN J,et al.A High-Resolution 3D Dynamic Facial Expression Database[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2008:1-6.
[26]SAVRAN A,ALYZ N,DIBEKLIOGˇLU H.Bosphorus Data-
base for 3D Face Analysis[C]∥Proceedings of the First European Workshop on Biometrics & Identity Management.New York:IEEE Press,2008:47-56.
[27]ZHU X X,RAMANAN D.Face Detection,Pose Estimation and Landmark Localization in the Wild[C]∥IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2012:2879-2886.
[28]SETTY S,HUSAIN M,BEHAM P,et al.Indian Movie Face Database:A Benchmark for Face Recognition Under WideVariations[C]∥IEEE International Conference on Computer Vision,Pattern Recognition,Image Processing and Graphics.New York:IEEE Press,2013:1-5.
[29]VIOLA P,JONES M.Robust Real-Time Face Detection[J].International Journal of Computer Vision,2004,57(2):137-154.
[30]YACOOB Y,DAVIS L,BLACK M,et al.Looking at People in Action[M]∥Cipolla R,Pentland A,Eds.Computer Vision for Human-Machine Interaction.Cambridge:Cambridge University Press,1998:171-187.
[31]CECH J,FRANC V,MATAS J.A 3D Approach to Facial Landmarks:Detection,Refinement,and Tracking[C]∥IEEE International Conference on Pattern Recognition.New York:IEEE Press,2014:2173-2178.
[32]ZHANG H W,LI Q,SUN Z,et al.Combining Data-Driven and Model-Driven Methods for Robust Facial Landmark Detection[J].IEEE Transactions on Information Forensics and Security,arXiv:1611.10152.
[33]PANTIC M,PATRAS I.Dynamics of Facial Expression:Recognition of Facial Actions and Their Temporal Segments from Face Profile Image Sequences[J].IEEE Transactions on Systems,Man,and Cybernetics,Part B,2006,36(2):433-449.
[34]ROTH M,BAUML M,NEVATIA R,et al.Robust Multi-Pose Face Tracking by Multi-Stage Tracklet Association[C]∥IEEE International Conference on Pattern Recognition.New York:IEEE Press,2012:1012-1016.
[35]KBLBECK C,ERNST A.Face Detection and Tracking in Vi-
deo Sequences Using The Modified Census Transformation[J].Image & Vision Computing,2006,24(6):564-572.
[36]TANG C G,ZHENG W M,YAN J W,et al.View-Independent Facial Action Unit Detection[C]∥IEEE InternationalConfe-rence on Automatic Face & Gesture Recognition.New York:IEEE Press,2017:878-882.
[37]BATISTA C,ALBIERO V,BELLON P,et al.AUMPNet:Simul-
taneous Action Units Detection and Intensity Estimation on Multipose Facial Images Using A Single Convolutional Neural Network[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2017:866-871.
[38]BAN Y,KIM S K,KIM S,et al.Face Detection Based on Skin Color Likelihood[J].Pattern Recognition,2014,47(4):1573-1585.
[39]CHEN Y D,LI R F.A hierarchical model for joint object detection and pose estimation[J].Journal of Computer Research and Development,2015,52(11):2431-2440.(in Chinese)
陈耀东,李仁发.一种层次化的联合识别模型[J].计算机研究与发展,2015,52(11):2431-2440.
[40]MURPHY-CHUTORIAN E,TRIVEDI M.Head Pose Estimation in Computer Vision:A Survey[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2009,31(4):607-626.
[41]MA B P,SHAN S G,CHEN X L,et al.Robust appearance-
based method for head pose estimation[J].Journal of Software,2009,20(6):1651-1663.(in Chinese)
马丙鹏,山世光,陈熙霖,等.一种基于图像表观的鲁棒姿态估计方法[J].软件学报,2009,20(6):1651-1663.
[42]FLOHR F,DUMITRU-GUZU M,KOOIJ J,et al.A Probabilistic Framework for Joint Pedestrian Head and Body Orientation Estimation[J].IEEE Transactions on Intelligent Transportation Systems,2015,16(4):1872-1882.
[43]YAN Y,RICC E,SUBRAMANIAN R,et al.A Multi-Task
Learning Framework for Head Pose Estimation Under Target Motion[J].IEEE Transactions on Intelligent Transportation Systems,2016,38(6):1070-1083.
[44]KONG G S,MBOUNA R.Head Pose Estimation from A 2D
Face Image Using 3D Face Morphing with Depth Parameters[J].IEEE Transactions on Image Processing,2015,24(6):1801-1808.
[45]MUKHERJEE S,ROBERTSON N.Deep Head Pose:Gaze-Direction Estimation in Multimodal Video[J].IEEE Transactions on Multimedia,2015,17(11):2094-2107.
[46]PANTIC M,PATRAS I,ROTHKRANTZ L.Facial Mimics
Recognition from Face Profile Image Sequences[R].Delft:Delft University of Technology,2002.
[47]HU Y X,ZENG Z H,YIN L J,et al.A Study of Non-Frontal-View Facial Expressions Recognition[C]∥IEEE International Conference on Pattern Recognition.New York:IEEE Press,2008:1-4.
[48]SUNG J,KIM D.Pose-Robust Facial Expression Recognition
Using View-Based 2D+3D AAM[J].IEEE Transactions on Systems,Man,and Cybernetics,Part A,2008,38(4):852-866.
[49]HESSE N,GEHRIG T,GAO H,et al.Multi-View Facial Ex
pression Recognition Using Local Appearance Features[C]∥IEEE International Conference on Pattern Recognition.New York:IEEE Press,2012:3533-3536.
[50]TONG Y,CHEN J X,JI Q.A Unified Probabilistic Framework for Spontaneous Facial Action Modeling and Understanding[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2010,32(2):258-273.
[51]ZHENG W M,TANG H,LIN Z C,et al.A Novel Approach to Expression Recognition from Non-Frontal Face Images[C]∥IEEE International Conference on Computer Vision.New York:IEEE Press,2009:1901-1908.
[52]WU Y,WANG Z G,JI Q.Facial Feature Tracking Under Varying Facial Expressions and Face Poses Based on Restricted Boltzmann Machines[C]∥IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2013:3452-3459.
[53]ZHANG Z P,ZHANG W,DING H J,et al.Hierarchical Facial Landmark Localization via Cascaded Random Binary Patterns[J].Pattern Recognition,2014,48(4):1277-1288.
[54]SMITH B M,BRANDT J,LIN Z,et al.Nonparametric Context Modeling of Local Appearance for Pose- and Expression-Robust Facial Landmark Localization[C]∥IEEE Conference on Computer Vision and Pattern Recognition.New York:IEEE Press,2014:1741-1748.
[55]WU Y,WANG Z H,JI Q.A Hierarchical Probabilistic Model for Facial Feature Detection[C]∥IEEE Conference onCompu-ter Vision and Pattern Recognition.New York:IEEE Press,2014:1781-1788.
[56]KUMANO S,OTSUKA K,YAMATO J,et al.Pose-Invariant Facial Expression Recognition Using Variable-Intensity Templates[C]∥Proceedings of the 8th Asian Conference on Computer Vision.New York:IEEE Press,2007:324-334.
[57]HU Y X,ZENG Z H,YIN L J,et al.Multi-View Facial Expression Recognition[C]∥Proceedings of the 8th International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2008:1-6.
[58]ZHENG W M.Multi-View Facial Expression Recognition Based on Group Sparse Reduced-Rank Regression[J].IEEE Transactions on Affective Computing,2014,5(1):71-85.
[59]RUDOVIC O,PATRAS I,PANTIC M.Coupled Gaussian
Process Regression for Pose Invariant Facial Expression Reco-gnition[C]∥Proceedings of the 11th European Conference on Computer Vision.New York:IEEE Press,2010:350-363.
[60]ELEFTHERIADIS S,RUDOVIC O,PANTIC M.Discriminative
Shared Gaussian Processes for Multiview and View-Invariant Facial Expression Recognition[J].IEEE Transactions on Image Processing,2015,24(1):189-204.
[61]MOORE S,BOWDEN R.Local Binary Patterns for Multi-View Facial Expression Recognition[J].Computer Vision Image Understand,2011,115(4):541-558.
[62]ZHENG W M,TANG H,LIN Z C,et al.Emotion Recognition from Arbitrary View Facial Images[C]∥Proceedings of the 11th European Conference on Computer Vision.New York:IEEE Press,2010:490-503.
[63]WU J,LIN Z,ZHENG W,et al.Locality-Constrained Linear
Coding Based Bi-layer Model for Multi-view Facial Expression Recognition[J].Neurocomputing,2017,239(C):143-152.
[64]YUN W,KIM D,PARK C,et al.Multi-View Facial Expression Recognition Using Parametric Kernel Eigenspace Method Based on Class Features[C]∥IEEE International Conference on Systems,Man,and Cybernetics.New York:IEEE Press,2013:2689-2693.
[65]MAO Q R,RAO Q Y,YU Y B,et al.Hierarchical Bayesian
Theme Models for Multipose Facial Expression Recognition[J].IEEE Transactions on Multimedia,2017,19(4):861-873.
[66]DAPOGNY A,BAILLY K,DUBUISSON S.Dynamic Pose-Robust Facial Expression Recognition by Multi-View Pairwise Conditional Random Forests[OL].https://ieeexplore.ieee.org/document/7934067.
[67]VIERIU L,TULYAKOV S,SEMENIUTA S,et al.Facial Expression Recognition under a Wide Range of Head Poses[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2015:1-7.
[68]YANG B,CAO J M,NI R R,et al.Facial Expression Recognition Using Weighted Mixture Deep Neural Network Based on Double-Channel Facial Images[J].IEEE Access,2018,6:4630-4640.
[69]WU B F,LIN C.Adaptive Feature Mapping for Customizing
Deep Learning Based Facial Expression Recognition Model[J].IEEE Access,2018,6:12451-12461.
[70]LIU Y Y,ZENG J B,SHAN S G,et al.Multi-channel Pose-
aware Convolution Neural Networks for Multi-view Facial Expression Recognition[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2018:458-465.
[71]JAMPOUR M,MAUTHNER T,BISCHOF H.Pairwise Linear Regression:An Efficient and Fast Multi-View Facial Expression Recognition[C]∥Proceedings of the 11th IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2015.
[72]JAMPOUR M,LEPETIT V,MAUTHNER T,et al.Pose-specific Non-linear Mappings in Feature Space towards Multiview Facial Expression Recognition[J].Image & Vision Computing,2017,58:38-46.
[73]RUDOVIC O,PATRAS I,PANTIC M.Regression-Based Multi-View Facial Expression Recognition[C]∥Proceedings of the 20th International Conference on Pattern Recognition.New York:IEEE Press,2010:4121-4124.
[74]RUDOVIC O,PANTIC M,PATRAS I.Coupled Gaussian
Processes for Pose-Invariant Facial Expression Recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(6):1357-1369.
[75]LAI Y H,LAI S H.Emotion-Preserving Representation Lear-
ning via Generative Adversarial Network for Multi-view Facial Expression Recognition[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2018:263-270.
[76]GNEY F,ARAR N M,FISCHER M,et al.Cross-Pose Facial Expression Recognition[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2013.
[77]TARIQ U,YANG J C,HUANG T S.Supervised Super-Vector Encoding for Facial Expression Recognition[J].Pattern Recognition Letters,2014,46(3):89-95.
[78]HOSOI T.Head Pose and Expression Transfer Using Facial
Status Score[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2017:573-580.
[79]ZHOU Y,PI J,SHI B E.Pose-Independent Facial Action Unit Intensity Regression Based on Multi-Task Deep Transfer Lear-ning[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.New York:IEEE Press,2017:872-877.
[80]ZHENG W M,ZONG Y,ZHOU X Y,et al.Cross-Domain Color Facial Expression Recognition Using Transductive Transfer Subspace Learning[J].IEEE Transactions on Affective Computing,2018,9(1):21-37.
[81]KAYA H,GRPINAR F,SALAH A A.Video-Based Emotion Recognition in the Wild using Deep Transfer Learning and Score Fusion[J].Image & Vision Computing,2017,65:66-75.
[82]WEBER R,BARRIELLE V,SOLADI C,et al.Unsupervised
Adaptation of A Person-Specific Manifold of Facial Expressions[J].IEEE Transactions on Affective Computing,2018.doi:10.1109/TAFFC.2018.2807430.
[1] 邓健康,王灿田,刘青山.
基于层进模型的快速人脸配准方法及其在智能移动设备上的应用
Fast Face Alignment Method Based on Hierarchical Model and it’s Application on Mobile Device
计算机科学, 2014, 41(12): 297-302. https://doi.org/10.11896/j.issn.1002-137X.2014.12.064
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发