Computer Science ›› 2024, Vol. 51 ›› Issue (11A): 231000053-7.doi: 10.11896/jsjkx.231000053

• Image Processing & Multimedia Technology • Previous Articles     Next Articles

Face Micro-expression Recognition Method Based on ME-ResNet

JIANG Sheng, ZHU Jianhong   

  1. Key Laboratory of Advanced Process Control for Light Industry,Ministry of Education,Jiangnan University,Wuxi,Jiangsu 214122,China
  • Online:2024-11-16 Published:2024-11-13
  • About author:JIANG Sheng,born in 1998,postgra-duate.His main research interests include computer vision and micro-expression recognition.
    ZHU Jianhong,born in 1964,Ph.D,professor,Ph.D supervisor.His main research interests include visual Internet of Things and deep learning.
  • Supported by:
    National Natural Science Foundation of China(61973139).

Abstract: Face micro-expressions have the characteristics of short duration and small amplitude of movement.Factors such as the small sample size of dataset also bring great challenges to micro-expression recognition.To solve these problems,this paper proposes a micro-expression recognition method based on ME-ResNet residual network.First,in the pre-processing stage,extract the key frame sequence between the start frame and the vertex frame in the micro-expression video clip at equal intervals and then,use the improved Farneback optical flow method to extract the motion features of the micro-expression key frame sequence.Se-cond,construct a ResNet50 network based on 3D convolution and add the spatial channel attention CBAM mechanism to the network Bottleneck module,so as to enhance the ability to focus on key facial motor features.Next,construct the ME-ResNet network model and sent the extracted facial optical flow motion features to the network for training.Finally,use the data enhancement to increase the sample size of network training and apply the ME-ResNet network model to micro-expression recognition tasks.Also,experimental results on CASME II,SMIC and SAMM datasets show that the recognition rate of the proposed algorithm reaches 84.42%,72.56% and 70.41% respectively.It has higher recognition ability compared with other algorithms.

Key words: Micro-expression recognition, Farneback optical flow method, Convolutional neural networks, Motion features, Data augmentation

CLC Number: 

  • TP391
[1]PENG Y W.Research on the application of micro-expressionanalysis technology in investigation and interrogation[J].Chinese Journal of Criminal Law,2015(2):95-103.
[2]ZHU C L,CHEN X Y,ZHANG J X,et al.Comparison of Ecological Micro-Expression Recognition in Patients with Depression and Healthy Individuals[J].Frontiers in Behavioral Neuroscience,2017,11(6):199-224.
[3]PFISTER T,LI X,ZHAO G,et al.Recognising Spontaneous Facial Micro-Expressions[C]//2011 International Conference on Computer Vision.IEEE,2011:1449-1456.
[4]HUANG X H,ZHAO G Y,HONG X P,et al.Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns[J].Neurocomputing,2016,175:564-578.
[5]SHREVE M,GODAVARTHY S,MANOHAR V,et al.To-wards Macro-and Micro-Expression Spotting in Video Using Strain Patterns[C]//2009 Workshop on Applications of Computer Vision(WACV).IEEE,2009:1-6.
[6]LIU Y J,ZHANG J K,YAN W J,et al.A main directional mean optical flow feature for spontaneous micro-expression recognition[J].IEEE Transactions on Affective Computing,2016,7(4):299-310.
[7]WANG S J,LI B J,LIU Y J,et al.Micro-Expression Recognition with Small Sample Size by Transferring Long-Term Convolutional Neural Network[J].Neurocomputing,2018,312:251-262.
[8]PENG M,WANG C,BI T,et al.A Novel Apex-Time Network for Cross-Dataset Micro-Expression Recognition[C]//2019 8th International Conference on Affective Computing and Intelligent Interaction(ACII).2024.
[9]GAN Y S,LIONG S T,YAU W C,et al.OFF-ApexNet on micro-expression recognition system[J].Signal Processing:Image Communication,2019,74:129-139.
[10]GAN Y S,LIONG S T,YAU W C,et al.OFF-ApexNet on micro-expression recognition system[J].Signal Processing:Image Communication,2019,74:129-139.
[11]LIANG Z Y,LIU D Z,SUN Y.Micro-expression recognitionmethod combining transfer learning and separable three-dimensional convolution[J].Computer Engineering,2022,48(1):228-235.
[12]SUN S K,FAN J,SUN Z Q,et al.A review of image data enhancement based on deep learning[J/OL].Computer Science:1-23[2023-10-06].http://kns.cnki.net/kcms/detail/50.1075.TP.20230926.0857.012.html.
[13]FARNEBACK,BIGUN J,GUSTAVSSON T,Two-Frame Motion Estimation Based on Polynomial Expansion[J].Image Analysis Volume,2003:363-370.
[14]LI B ,LIMA D.Facial expression recognition via ResNet-50[J].International Journal of Cognitive Computing in Engineering,2021.DOI:10.1016/j.ijcce.2021.02.002.
[15]LIU S H ,REN Y S ,LI L T ,et al.Micro-expression recognition based on SqueezeNet and C3D[J].Multimedia Systems,2022,28(6):2227-2236.
[16] WANG C,CHANG J.CSI Cross-domain Gesture RecognitionMethod Based on 3D Convolutional Neutal Network[J].Computer Science,2021,48(8):322-327.
[17]WANG C,CHANG J.CSI Cross-domain Gesture Recognition Method Based on 3D Convolutional Neural Network[J].Computer Science,2021,48(8):322-327.
[18]WOO S,PARK J,LEE J Y,et al.CBAM:Convolutional BlockAttention Module[C]//ECCV 2018.Cham:Springer,2018.
[19]LI X R,ZHANG L Y,YAO S J.Micro-expression Recognition Method Combining Feature Fusion and AttentionMechanim[J].Computer Science,2022,49(2):4-11.
[20]LI X R,ZHANG L Y,YAO S J.Micro-expression Recognition Method Combining Feature Fusion and Attention Mechanism[J].Computer Science,2022,49(2):4-11.
[21]ZHU X,CHENG D,ZHANG Z,et al.An Empirical Study ofSpatial Attention Mechanisms in Deep Networks[C]//2019 IEEE/CVF International Conference on Computer Vision(ICCV).IEEE,2020.
[22]DAVISON A K,LANSLEY C,COSTEN N,et al.SAMM:A spontaneous micro-facial movement dataset[J].IEEE Transactions on Affective Computing,2018,9(1):116-129.
[23]YAN W J,LI X B,WANG S J,et al.CASME II:An improved spontaneous micro-expression database and the baseline evaluation[J].PLoS One,2014,9(1):e86041.
[24]DAVISON A K,LANSLEY C,COSTEN N,et al.SAMM:Aspontaneous micro-facial movement dataset[J].IEEE Transactions on Affective Computing,2018,9(1):116-129.
[1] DAI Chaofan, DING Huahua. Domain-adaptive Entity Resolution Algorithm Based on Semi-supervised Learning [J]. Computer Science, 2024, 51(9): 214-222.
[2] YIN Xudong, CHEN Junyang, ZHOU Bo. Study on Industrial Defect Augmentation Data Filtering Based on OOD Scores [J]. Computer Science, 2024, 51(6A): 230700111-7.
[3] SUN Yang, DING Jianwei, ZHANG Qi, WEI Huiwen, TIAN Bowen. Study on Super-resolution Image Reconstruction Using Residual Feature Aggregation NetworkBased on Attention Mechanism [J]. Computer Science, 2024, 51(6A): 230600039-6.
[4] LIU Hui, JI Ke, CHEN Zhenxiang, SUN Runyuan, MA Kun, WU Jun. Malicious Attack Detection in Recommendation Systems Combining Graph Convolutional Neural Networks and Ensemble Methods [J]. Computer Science, 2024, 51(6A): 230700003-9.
[5] PAN Lei, LIU Xin, CHEN Junyi, CHENG Zhangtao, LIU Leyuan, ZHOU Fan. Event Prediction Based on Dynamic Graph with Local Data Augmentation [J]. Computer Science, 2024, 51(3): 118-127.
[6] AN Yang, WANG Xiuqing, ZHAO Minghua. Mobile Robots' Path Planning Method Based on Policy Fusion and Spiking Deep ReinforcementLearning [J]. Computer Science, 2024, 51(11A): 240100211-11.
[7] CAO Qingyuan, ZHU Jianhong. Study on Identification of Concrete Sand and Gravel Aggregate Types Based on Improved Residual Network [J]. Computer Science, 2024, 51(11A): 231000082-6.
[8] YU Yongxin, JI Ke, GAO Yuan, CHEN Zhenxiang, MA Kun, ZHAO Xiaofan. Multi-source Heterogeneous Data Progressive Fusion for Fake News Detection [J]. Computer Science, 2024, 51(11): 30-38.
[9] YANG Xihong, ZHENG Qun, ZHANG Jiaxin, WANG Pei, ZHU En. Feature Interpolation Based Deep Graph Contrastive Clustering Algorithm [J]. Computer Science, 2024, 51(11): 157-165.
[10] XU Jie, WANG Lisong. Contrastive Clustering with Consistent Structural Relations [J]. Computer Science, 2023, 50(9): 123-129.
[11] LIANG Jiayin, XIE Zhipeng. Text Paraphrase Generation Based on Pre-trained Language Model and Tag Guidance [J]. Computer Science, 2023, 50(8): 150-156.
[12] HUANG Fangwan, LU Juhong, YU Zhiyong. Data Augmentation for Cardiopulmonary Exercise Time Series of Young HypertensivePatients Based on Active Barycenter [J]. Computer Science, 2023, 50(6A): 211200233-11.
[13] LUO Huilan, LONG Jun, LIANG Miaomiao. Attentional Feature Fusion Approach for Siamese Network Based Object Tracking [J]. Computer Science, 2023, 50(6A): 220300237-9.
[14] HUANG Yujiao, CHEN Mingkai, ZHENG Yuan, FAN Xinggang, XIAO Jie, LONG Haixia. Text Classification Based on Weakened Graph Convolutional Networks [J]. Computer Science, 2023, 50(6A): 220700039-5.
[15] ZENG Wu, MAO Guojun. Few-shot Learning Method Based on Multi-graph Feature Aggregation [J]. Computer Science, 2023, 50(6A): 220400029-10.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!