Computer Science ›› 2022, Vol. 49 ›› Issue (2): 4-11.doi: 10.11896/jsjkx.210900028

• Computer Vision: Theory and Application • Previous Articles     Next Articles

Micro-expression Recognition Method Combining Feature Fusion and Attention Mechanism

LI Xing-ran, ZHANG Li-yan, YAO Shu-jing   

  1. School of Computer Science and Technology,Nanjing University of Aeronautics and Astronautics,Nanjing 211106,China
  • Received:2021-09-02 Revised:2021-10-07 Online:2022-02-15 Published:2022-02-23
  • About author:LI Xing-ran,born in 1998,postgra-duate.Her main research interests include micro-expression recognition and deep learning.
    ZHANG Li-yan,born in 1984,Ph.D,professor,is a member of China Computer Federation.Her main research interests include multimedia analysis,computer vision and deep learning.
  • Supported by:
    National Natural Science Foundation of China(61772268) and Natural Science Foundation of Jiangsu Province(BK20190065).

Abstract: Micro-expression refers to an uncontrollable muscle movement on the face when people try to hide or suppress their true emotions.Due to the short duration,small motion range,and difficulty in concealing and restraining,the recognition accuracy of such emotional facial expressions is restricted.In order to cope with these challenges,this paper proposes a novel micro-expression recognition method combining feature fusion and attention mechanism,considering optical flow features and face features,and further adding attention mechanism to improve the recognition performance.The processing steps of this method are as follows:1)Extract the optical flow and optical strain from Onset to Apex in each micro-expression segment,input the vertical optical flow,horizontal optical flow and optical strain into a shallow 3DCNN,and extract the optical flow features.2)Taking the deep convolution neural network ResNet-10 as the backbone network,the convolution attention module is added to extract face features.3)Combine the two feature vectors for classification.The experimental results reveal that the proposed method is superior to the traditional methods and existing deep learning methods in micro-expression recognition.

Key words: Attention mechanism, Deep learning, Feature fusion, Micro-expression recognition, Transfer learning

CLC Number: 

  • TP183
[1]EKMAN P.Emotions Revealed:Understanding Faces and Fee-lings[M].Weidenfeld & Nicolson,2004.
[2]TAKALKAR M,XU M,WU Q,et al.A survey:facial micro-expression recognition [J].Multimedia Tools and Applications,2018,77(15):1-25.
[3]EKMA P.Lie Catching and Microexpressions[M]//The Philo-sophy of Deception.2009:118-133.
[4]FRANK M,HERBASZ M,SINUK K,et al.I see how you feel:Training laypeople and professionals to recognize fleeting emotions[C]//The Annual Meeting of the International Communication Association.New York,2009:1-35.
[5]HOUSE C,MEYER R.Preprocessing and descriptor featuresfor facial micro-expression recognition [OL].[2016-10-15].https://web.stanford.edu/class/ee368/Project_Spring_1415/Reports/House_Meyer.pdf.
[6]ZHANG M,FU Q,CHEN Y H,et al.Emotional Context Influences Micro-Expression Recognition [J].PLoS ONE,2014,9(4):1-7.
[7]OJALA T,PIETIKÄINEN M,HARWOOD D.A Comparative Study of Texture Measures with Classification Based on Feature Distributions [J].Pattern Recognition,1996,29(1):51-59.
[8]JI C M,SONG T C.Sparse Representation-Based Classification Under Optimization Forms for Face Recognition[J].Journal of Chongqing University of Technology(Natural Science),2020,34(2):120-126.
[9]RAO W J,GU Y H,ZHU T T,et al.Intelligent license platerecognition method in complex environment[J].Journal of Chongqing University of Technology (Natural Science),2021,35(3):119-127.
[10]KIM D H,BADDAR W J,RO Y M.Micro-expression recognition with expression-state constrained spatio-temporal feature representations[C]//Proceedings of the 24th ACM InternationalConference on Multimedia.2016:382-386.
[11]PENG M,WANG C Y,CHEN T,et al.Dual Temporal Scale Convolutional Neural Network for Micro-Expression Recognition [J].Frontiers in Psychology,2017,8:1-12.
[12]LI J,WANG Y,SEE J,et al.Micro-expression recognition basedon 3D flow convolutional neural network[J].Pattern Analysis and Applications,2019,22(4):1331-1339.
[13]LIANG Z Y,HE J L,SUN Y.An evolutionary method of three-dimensional convolutional neural networks for automatic recognition of micro expressions [J].Computer Science,2020,47(8):227-232.
[14]MERGHANI W,DAVISON A K,YAP M H.A review on facial micro-expressions analysis:datasets,features and metrics[J].arXiv:1805.02397,2018.
[15]LIONG S T,GAN Y S,SEE J,et al.Shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition[C]//IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).IEEE,2019:1-5.
[16]LIU Y J,ZHANG J K,YAN W J,et al.A Main DirectionalMean Optical Flow Feature for Spontaneous Micro-Expression Recognition [J].IEEE Transactions on Affective Computing,2016,7(4):299-310.
[17]CHEN B,ZHANG Z,LIU N,et al.Spatiotemporal Convolutio-nal Neural Network with Convolutional Block Attention Module for Micro-Expression Recognition[J].Information(Switzerland),2020,11(8):380.
[18]PFISTER T,LI X,ZHAO G,et al.Recognising spontaneous facial micro-expressions[C]//International Conference on Computer Vision.IEEE,2011:1449-1456.
[19]XU F,ZHANG J,WANG J Z.Microexpression identificationand categorization using a facial dynamics map[J].IEEE Tran-sactions on Affective Computing,2017,8(2):254-267.
[20]LIONG S T,SEE J,WONG K S,et al.Less is more:Micro-expression recognition from video using apex frame[J].Signal Processing:Image Communication,2018,62:82-92.
[21]LIU Y J,LI B J,LAI Y K.Sparse MDMO:Learning a discriminative feature for micro-expression recognition[J].IEEE Tran-sactions on Affective Computing,2018,12(1):254-261.
[22]DALAL N,TRIGGS B.Histograms of oriented gradients forhuman detection[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition.IEEE,2005,1:886-893.
[23]POLIKOVSKY S,KAMEDA Y,OHTA Y.Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor[C]//International Conference on Crime Detection & Prevention.IET,2010.
[24]CHEN M,MA H T,LI J,et al.Emotion recognition using fixed length micro-expressions sequence and weighting method[C]//IEEE International Conference on Real-time Computing and Robotics (RCAR).IEEE,2016:427-430.
[25]REDDY S,KARRI S T,DUBEY S R,et al.Spontaneous Facial Micro-Expression Recognition using 3D Spatiotemporal Convolutional Neural Networks[C]//International Joint Conference on Neural Networks (IJCNN).IEEE,2019:1-8.
[26]ZHAO Y,XU J.A convolutional neural network for compound micro-expression recognition[J].Sensors,2019,19(24):5553.
[27]WANG C,PENG M,BI T,et al.Micro-attention for micro-expression recognition[J].Neurocomputing,2020,410:354-362.
[28]QUANG N V,CHUN J,TOKUYAMA T.CapsuleNet for Micro-Expression Recognition[C]//IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).IEEE,2019:1-7.
[29]SUN B,CAO S,LI D,et al.Dynamic Micro-Expression Recognition Using Knowledge Distillation[C]//IEEE Transactions on Affective Computing.2020.
[30]ZHI R,XU H,WAN M,et al.Combining 3D convolutional neural networks with transfer learning by supervised pre-training for facial micro-expression recognition[J].IEICE Transactions on Information and Systems,2019,102(5):1054-1064.
[31]WU C,GUO F.TSNN:Three-Stream Combin-ing 2D and 3D Convolutional Neural Network for Micro-Expression Recognition[J].IEEJ Transactions on Electrical and Electronic Engineering,2021,16(1):98-107.
[32]GAN Y S,LIONG S T,YAU W C,et al.OFF-ApexNet on micro-expression recognition system[J].Signal Processing:Image Communication,2019,74:129-139.
[33]LIONG S T,SEE J,WONG K S,et al.Automatic Apex Frame Spotting in Micro-expression Database[C]//IAPR Asian Conference on Pattern Recognition.IEEE,2015:665-669.
[34]PENG M,WU Z,ZHANG Z,et al.From macro to micro expression recognition:Deep learning on small datasets using transfer learning[C]//IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).IEEE,2018:657-661.
[35]LUCEY P,COHN J F,KANADE T,et al.The Extended Cohn-Kanade Dataset (CK+):A complete dataset for action unit and emotion-specified expression[C]//Computer Vision & Pattern Recognition Workshops.IEEE,2010:94-101.
[36]ZHAO G,HUANG X,TAINI M,et al.Facial expression recognition from near-infrared videos[J].Image and Vision Computing,2011,29(9):607-619.
[37]LYONS M,AKAMATSU S,KAMACHI M,et al.Coding facial expressions with gabor wavelets[C]//Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.IEEE,1998:200-205.
[38]AIFANTI N,PAPACHRISTOU C,DELOPOU-LOS A.TheMUG facial expression database[C]//International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10.IEEE,2010:1-4.
[39]YAN W J,LI X,WANG S J,et al.CASME II:an improved spontaneous micro-expression database and the baseline evaluation[J].PLoS one,2014,9(1):1-8.
[40]DAVISON A K,LANSLEY C,COSTEN N,et al.SAMM:A Spontaneous Micro-Facial Movement Dataset[J].IEEE Transactions on Affective Computing,2018,9(99):116-129.
[41]LI X,PFISTER T,HUANG X,et al.A Spontaneous Micro-expression Database:Inducement,collection and baseline[C]//IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).IEEE,2013:1-6.
[42]WOO S,PARK J,LEE J Y,et al.CBAM:Convolutional Block Attention Module[C]//Proceedings of the European Conference on Computer Vision (ECCV).2018:3-19.
[43]SEE J,YAP M H,LI J,et al.MEGC 2019-the second facial micro-expressions grand challenge[C]//IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).IEEE,2019:1-5.
[44]PENG M,WANG C,BI T,et al.A novel apex-time network for cross-dataset micro-expression recognition[C]//2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII).IEEE,2019:1-6.
[45]ZHOU L,MAO Q,XUE L.Dual-inception network for cross-database micro-expression recognition[C]//IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019).IEEE,2019:1-5.
[1] ZHOU Fang-quan, CHENG Wei-qing. Sequence Recommendation Based on Global Enhanced Graph Neural Network [J]. Computer Science, 2022, 49(9): 55-63.
[2] DAI Yu, XU Lin-feng. Cross-image Text Reading Method Based on Text Line Matching [J]. Computer Science, 2022, 49(9): 139-145.
[3] ZHOU Le-yuan, ZHANG Jian-hua, YUAN Tian-tian, CHEN Sheng-yong. Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion [J]. Computer Science, 2022, 49(9): 155-161.
[4] XU Yong-xin, ZHAO Jun-feng, WANG Ya-sha, XIE Bing, YANG Kai. Temporal Knowledge Graph Representation Learning [J]. Computer Science, 2022, 49(9): 162-171.
[5] XIONG Li-qin, CAO Lei, LAI Jun, CHEN Xi-liang. Overview of Multi-agent Deep Reinforcement Learning Based on Value Factorization [J]. Computer Science, 2022, 49(9): 172-182.
[6] RAO Zhi-shuang, JIA Zhen, ZHANG Fan, LI Tian-rui. Key-Value Relational Memory Networks for Question Answering over Knowledge Graph [J]. Computer Science, 2022, 49(9): 202-207.
[7] TANG Ling-tao, WANG Di, ZHANG Lu-fei, LIU Sheng-yun. Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy [J]. Computer Science, 2022, 49(9): 297-305.
[8] WANG Ming, PENG Jian, HUANG Fei-hu. Multi-time Scale Spatial-Temporal Graph Neural Network for Traffic Flow Prediction [J]. Computer Science, 2022, 49(8): 40-48.
[9] WANG Jian, PENG Yu-qi, ZHAO Yu-fei, YANG Jian. Survey of Social Network Public Opinion Information Extraction Based on Deep Learning [J]. Computer Science, 2022, 49(8): 279-293.
[10] HAO Zhi-rong, CHEN Long, HUANG Jia-cheng. Class Discriminative Universal Adversarial Attack for Text Classification [J]. Computer Science, 2022, 49(8): 323-329.
[11] JIANG Meng-han, LI Shao-mei, ZHENG Hong-hao, ZHANG Jian-peng. Rumor Detection Model Based on Improved Position Embedding [J]. Computer Science, 2022, 49(8): 330-335.
[12] ZHU Cheng-zhang, HUANG Jia-er, XIAO Ya-long, WANG Han, ZOU Bei-ji. Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism [J]. Computer Science, 2022, 49(8): 113-119.
[13] SUN Qi, JI Gen-lin, ZHANG Jie. Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection [J]. Computer Science, 2022, 49(8): 172-177.
[14] YAN Jia-dan, JIA Cai-yan. Text Classification Method Based on Information Fusion of Dual-graph Neural Network [J]. Computer Science, 2022, 49(8): 230-236.
[15] HU Yan-yu, ZHAO Long, DONG Xiang-jun. Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification [J]. Computer Science, 2022, 49(7): 73-78.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!