结合特征融合和注意力机制的微表情识别方法

doi:10.11896/jsjkx.210900028

摘要/Abstract

摘要： 微表情指当人们试图隐藏或抑制自己的真实情感时,脸上出现的一种无法控制的肌肉运动。此类情绪面部表情由于具有持续时间短、动作幅度小、难以掩饰和抑制的特点,因此其识别精度受到了制约。为了应对这些挑战,文中提出一种结合特征融合和注意力机制的微表情识别方法,同时考虑了光流特征和人脸特征,通过进一步加入注意力机制来提升识别性能。该网络由3个部分组成:1)提取每个微表情片段中Onset到Apex的光流与光学应变,将垂直光流、水平光流、光学应变输入到一个浅层3DCNN中,以提取光流特征;2)以深度卷积神经网络ResNet-10为迁移模型,加入卷积注意力模块以提取人脸特征;3)将两个特征向量拼接起来进行分类。利用所提方法在3个自发微表情数据集中进行实验,结果表明,所提方法在微表情识别方面优于传统方法和现有深度学习方法。

关键词: 迁移学习, 深度学习, 特征融合, 微表情识别, 注意力机制

Abstract: Micro-expression refers to an uncontrollable muscle movement on the face when people try to hide or suppress their true emotions.Due to the short duration,small motion range,and difficulty in concealing and restraining,the recognition accuracy of such emotional facial expressions is restricted.In order to cope with these challenges,this paper proposes a novel micro-expression recognition method combining feature fusion and attention mechanism,considering optical flow features and face features,and further adding attention mechanism to improve the recognition performance.The processing steps of this method are as follows:1)Extract the optical flow and optical strain from Onset to Apex in each micro-expression segment,input the vertical optical flow,horizontal optical flow and optical strain into a shallow 3DCNN,and extract the optical flow features.2)Taking the deep convolution neural network ResNet-10 as the backbone network,the convolution attention module is added to extract face features.3)Combine the two feature vectors for classification.The experimental results reveal that the proposed method is superior to the traditional methods and existing deep learning methods in micro-expression recognition.

Key words: Attention mechanism, Deep learning, Feature fusion, Micro-expression recognition, Transfer learning

中图分类号:

TP183

李星燃, 张立言, 姚树婧. 结合特征融合和注意力机制的微表情识别方法[J]. 计算机科学, 2022, 49(2): 4-11. https://doi.org/10.11896/jsjkx.210900028

LI Xing-ran, ZHANG Li-yan, YAO Shu-jing. Micro-expression Recognition Method Combining Feature Fusion and Attention Mechanism[J]. Computer Science, 2022, 49(2): 4-11. https://doi.org/10.11896/jsjkx.210900028

参考文献

[1]EKMAN P.Emotions Revealed:Understanding Faces and Fee-lings[M].Weidenfeld & Nicolson,2004.
[2]TAKALKAR M,XU M,WU Q,et al.A survey:facial micro-expression recognition [J].Multimedia Tools and Applications,2018,77(15):1-25.
[3]EKMA P.Lie Catching and Microexpressions[M]//The Philo-sophy of Deception.2009:118-133.
[4]FRANK M,HERBASZ M,SINUK K,et al.I see how you feel:Training laypeople and professionals to recognize fleeting emotions[C]//The Annual Meeting of the International Communication Association.New York,2009:1-35.
[5]HOUSE C,MEYER R.Preprocessing and descriptor featuresfor facial micro-expression recognition [OL].[2016-10-15].https://web.stanford.edu/class/ee368/Project_Spring_1415/Reports/House_Meyer.pdf.
[6]ZHANG M,FU Q,CHEN Y H,et al.Emotional Context Influences Micro-Expression Recognition [J].PLoS ONE,2014,9(4):1-7.
[7]OJALA T,PIETIKÄINEN M,HARWOOD D.A Comparative Study of Texture Measures with Classification Based on Feature Distributions [J].Pattern Recognition,1996,29(1):51-59.
[8]JI C M,SONG T C.Sparse Representation-Based Classification Under Optimization Forms for Face Recognition[J].Journal of Chongqing University of Technology(Natural Science),2020,34(2):120-126.
[9]RAO W J,GU Y H,ZHU T T,et al.Intelligent license platerecognition method in complex environment[J].Journal of Chongqing University of Technology (Natural Science),2021,35(3):119-127.
[10]KIM D H,BADDAR W J,RO Y M.Micro-expression recognition with expression-state constrained spatio-temporal feature representations[C]//Proceedings of the 24th ACM InternationalConference on Multimedia.2016:382-386.
[11]PENG M,WANG C Y,CHEN T,et al.Dual Temporal Scale Convolutional Neural Network for Micro-Expression Recognition [J].Frontiers in Psychology,2017,8:1-12.
[12]LI J,WANG Y,SEE J,et al.Micro-expression recognition basedon 3D flow convolutional neural network[J].Pattern Analysis and Applications,2019,22(4):1331-1339.
[13]LIANG Z Y,HE J L,SUN Y.An evolutionary method of three-dimensional convolutional neural networks for automatic recognition of micro expressions [J].Computer Science,2020,47(8):227-232.
[14]MERGHANI W,DAVISON A K,YAP M H.A review on facial micro-expressions analysis:datasets,features and metrics[J].arXiv:1805.02397,2018.
[15]LIONG S T,GAN Y S,SEE J,et al.Shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition[C]//IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).IEEE,2019:1-5.
[16]LIU Y J,ZHANG J K,YAN W J,et al.A Main DirectionalMean Optical Flow Feature for Spontaneous Micro-Expression Recognition [J].IEEE Transactions on Affective Computing,2016,7(4):299-310.
[17]CHEN B,ZHANG Z,LIU N,et al.Spatiotemporal Convolutio-nal Neural Network with Convolutional Block Attention Module for Micro-Expression Recognition[J].Information(Switzerland),2020,11(8):380.
[18]PFISTER T,LI X,ZHAO G,et al.Recognising spontaneous facial micro-expressions[C]//International Conference on Computer Vision.IEEE,2011:1449-1456.
[19]XU F,ZHANG J,WANG J Z.Microexpression identificationand categorization using a facial dynamics map[J].IEEE Tran-sactions on Affective Computing,2017,8(2):254-267.
[20]LIONG S T,SEE J,WONG K S,et al.Less is more:Micro-expression recognition from video using apex frame[J].Signal Processing:Image Communication,2018,62:82-92.
[21]LIU Y J,LI B J,LAI Y K.Sparse MDMO:Learning a discriminative feature for micro-expression recognition[J].IEEE Tran-sactions on Affective Computing,2018,12(1):254-261.
[22]DALAL N,TRIGGS B.Histograms of oriented gradients forhuman detection[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.IEEE,2005,1:886-893.
[23]POLIKOVSKY S,KAMEDA Y,OHTA Y.Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor[C]//International Conference on Crime Detection & Prevention.IET,2010.
[24]CHEN M,MA H T,LI J,et al.Emotion recognition using fixed length micro-expressions sequence and weighting method[C]//IEEE International Conference on Real-time Computing and Robotics (RCAR).IEEE,2016:427-430.
[25]REDDY S,KARRI S T,DUBEY S R,et al.Spontaneous Facial Micro-Expression Recognition using 3D Spatiotemporal Convolutional Neural Networks[C]//International Joint Conference on Neural Networks (IJCNN).IEEE,2019:1-8.
[26]ZHAO Y,XU J.A convolutional neural network for compound micro-expression recognition[J].Sensors,2019,19(24):5553.
[27]WANG C,PENG M,BI T,et al.Micro-attention for micro-expression recognition[J].Neurocomputing,2020,410:354-362.
[28]QUANG N V,CHUN J,TOKUYAMA T.CapsuleNet for Micro-Expression Recognition[C]//IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).IEEE,2019:1-7.
[29]SUN B,CAO S,LI D,et al.Dynamic Micro-Expression Recognition Using Knowledge Distillation[C]//IEEE Transactions on Affective Computing.2020.
[30]ZHI R,XU H,WAN M,et al.Combining 3D convolutional neural networks with transfer learning by supervised pre-training for facial micro-expression recognition[J].IEICE Transactions on Information and Systems,2019,102(5):1054-1064.
[31]WU C,GUO F.TSNN:Three-Stream Combin-ing 2D and 3D Convolutional Neural Network for Micro-Expression Recognition[J].IEEJ Transactions on Electrical and Electronic Engineering,2021,16(1):98-107.
[32]GAN Y S,LIONG S T,YAU W C,et al.OFF-ApexNet on micro-expression recognition system[J].Signal Processing:Image Communication,2019,74:129-139.
[33]LIONG S T,SEE J,WONG K S,et al.Automatic Apex Frame Spotting in Micro-expression Database[C]//IAPR Asian Conference on Pattern Recognition.IEEE,2015:665-669.
[34]PENG M,WU Z,ZHANG Z,et al.From macro to micro expression recognition:Deep learning on small datasets using transfer learning[C]//IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).IEEE,2018:657-661.
[35]LUCEY P,COHN J F,KANADE T,et al.The Extended Cohn-Kanade Dataset (CK+):A complete dataset for action unit and emotion-specified expression[C]//Computer Vision & Pattern Recognition Workshops.IEEE,2010:94-101.
[36]ZHAO G,HUANG X,TAINI M,et al.Facial expression recognition from near-infrared videos[J].Image and Vision Computing,2011,29(9):607-619.
[37]LYONS M,AKAMATSU S,KAMACHI M,et al.Coding facial expressions with gabor wavelets[C]//Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.IEEE,1998:200-205.
[38]AIFANTI N,PAPACHRISTOU C,DELOPOU-LOS A.TheMUG facial expression database[C]//International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10.IEEE,2010:1-4.
[39]YAN W J,LI X,WANG S J,et al.CASME II:an improved spontaneous micro-expression database and the baseline evaluation[J].PLoS one,2014,9(1):1-8.
[40]DAVISON A K,LANSLEY C,COSTEN N,et al.SAMM:A Spontaneous Micro-Facial Movement Dataset[J].IEEE Transactions on Affective Computing,2018,9(99):116-129.
[41]LI X,PFISTER T,HUANG X,et al.A Spontaneous Micro-expression Database:Inducement,collection and baseline[C]//IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).IEEE,2013:1-6.
[42]WOO S,PARK J,LEE J Y,et al.CBAM:Convolutional Block Attention Module[C]//Proceedings of the European Conference on Computer Vision (ECCV).2018:3-19.
[43]SEE J,YAP M H,LI J,et al.MEGC 2019－the second facial micro-expressions grand challenge[C]//IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).IEEE,2019:1-5.
[44]PENG M,WANG C,BI T,et al.A novel apex-time network for cross-dataset micro-expression recognition[C]//2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII).IEEE,2019:1-6.
[45]ZHOU L,MAO Q,XUE L.Dual-inception network for cross-database micro-expression recognition[C]//IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).IEEE,2019:1-5.

相关文章 15

[1]	周芳泉, 成卫青. 基于全局增强图神经网络的序列推荐 Sequence Recommendation Based on Global Enhanced Graph Neural Network 计算机科学, 2022, 49(9): 55-63. https://doi.org/10.11896/jsjkx.210700085
[2]	戴禹, 许林峰. 基于文本行匹配的跨图文本阅读方法 Cross-image Text Reading Method Based on Text Line Matching 计算机科学, 2022, 49(9): 139-145. https://doi.org/10.11896/jsjkx.220600032
[3]	周乐员, 张剑华, 袁甜甜, 陈胜勇. 多层注意力机制融合的序列到序列中国连续手语识别和翻译 Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion 计算机科学, 2022, 49(9): 155-161. https://doi.org/10.11896/jsjkx.210800026
[4]	徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺. 时序知识图谱表示学习 Temporal Knowledge Graph Representation Learning 计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204
[5]	熊丽琴, 曹雷, 赖俊, 陈希亮. 基于值分解的多智能体深度强化学习综述 Overview of Multi-agent Deep Reinforcement Learning Based on Value Factorization 计算机科学, 2022, 49(9): 172-182. https://doi.org/10.11896/jsjkx.210800112
[6]	饶志双, 贾真, 张凡, 李天瑞. 基于Key-Value关联记忆网络的知识图谱问答方法 Key-Value Relational Memory Networks for Question Answering over Knowledge Graph 计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277
[7]	汤凌韬, 王迪, 张鲁飞, 刘盛云. 基于安全多方计算和差分隐私的联邦学习方案 Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy 计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108
[8]	方义秋, 张震坤, 葛君伟. 基于自注意力机制和迁移学习的跨领域推荐算法 Cross-domain Recommendation Algorithm Based on Self-attention Mechanism and Transfer Learning 计算机科学, 2022, 49(8): 70-77. https://doi.org/10.11896/jsjkx.210600011
[9]	朱承璋, 黄嘉儿, 肖亚龙, 王晗, 邹北骥. 基于注意力机制的医学影像深度哈希检索算法 Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism 计算机科学, 2022, 49(8): 113-119. https://doi.org/10.11896/jsjkx.210700153
[10]	孙奇, 吉根林, 张杰. 基于非局部注意力生成对抗网络的视频异常事件检测方法 Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection 计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[11]	闫佳丹, 贾彩燕. 基于双图神经网络信息融合的文本分类方法 Text Classification Method Based on Information Fusion of Dual-graph Neural Network 计算机科学, 2022, 49(8): 230-236. https://doi.org/10.11896/jsjkx.210600042
[12]	王剑, 彭雨琦, 赵宇斐, 杨健. 基于深度学习的社交网络舆情信息抽取方法综述 Survey of Social Network Public Opinion Information Extraction Based on Deep Learning 计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099
[13]	郝志荣, 陈龙, 黄嘉成. 面向文本分类的类别区分式通用对抗攻击方法 Class Discriminative Universal Adversarial Attack for Text Classification 计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077
[14]	姜梦函, 李邵梅, 郑洪浩, 张建朋. 基于改进位置编码的谣言检测模型 Rumor Detection Model Based on Improved Position Embedding 计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046
[15]	汪鸣, 彭舰, 黄飞虎. 基于多时间尺度时空图网络的交通流量预测模型 Multi-time Scale Spatial-Temporal Graph Neural Network for Traffic Flow Prediction 计算机科学, 2022, 49(8): 40-48. https://doi.org/10.11896/jsjkx.220100188

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed