Computer Science ›› 2022, Vol. 49 ›› Issue (6A): 370-377.doi: 10.11896/jsjkx.210500023

• Image Processing & Multimedia Technology • Previous Articles     Next Articles

Lightweight Micro-expression Recognition Architecture Based on Bottleneck Transformer

ZHANG Jia-hao1, LIU Feng2,3,4, QI Jia-yin4   

  1. 1 School of Computer Science and Technology,East China Normal University,Shanghai 200062,China
    2 Shanghai Institute of Intelligent Education,East China Normal University,Shanghai 200062,China
    3 Shanghai Key Laboratory of Mental Health and Psychological Crisis Intervention,Other Institutes,School of Psychology and Cognitive Science,East China Normal University,Shanghai 200062,China
    4 Institute of Artificial Intelligence and Change Management,Shanghai University of International Business and Economics,Shanghai 201620,China
  • Online:2022-06-10 Published:2022-06-08
  • About author:ZHANG Jia-hao,born in 2000,undergraduate,is a student member of the China Computer Federation.His main reasearch interests include affective computing,computer vision and deep learning.
    LIU Feng,born in 1988,Ph.Dcandidate,engineer,is a senior member of China Computer Federation.His main research interests include deep lear-ning,cognitive science and blockchain technology.
  • Supported by:
    Digital Transformation in China and Germany:Strategies,Structures and Solutions for Ageing Societies(GZ1570),Research Project of Shanghai Science and Technology Commission(20dz2260300) and Fundamental Research Funds for the Central Universities.

Abstract: Micro-expressions are spontaneous facial movements at a marginal spatiotemporal scale,which reveal one's true fee-lings.Its duration is short,the amplitude of the movement is slight,and it is difficult to recognize,but it has important research value.In order to solve the micro-expression recognition problem,a novel extremely lightweight micro-expression recognition neural architecture is proposed.The neural network which takes apex-onset optical-flow features as the input and integrates approaches in residual convolutional networks and visual Transformers,could effectively solve the micro-expression sentiment classification problem.This architecture containsnovel parameter-saving residual blocks,and a bottleneck Transformer block which replace the convolution operators in residual blocks with self-attention mechanism.The model evaluation experiments are conducted with a LOSO cross-validation strategy on a combined database con-sists of the 3 CASME datasets.With obviously fewer total parameters(39 685),the model achieves an average recall of 73.09% and an average F1-Score of 72.25%,exceeding those mainstream architectures in this domain.A series ablation experiments are also conducted to ensure the superiority of the optical strain strength,self-attention mechanism and relativeposition encoding.

Key words: Computational affection, Micro-expression recognition, Residual convolutional neural network, Self-attention mechanism, Visual Transformer

CLC Number: 

  • TP301.6
[1] EKMAN P,FRIESEN W V.Nonverbal leakage and clues to deception[J].Psychiatry,1969,32(1):88-106.
[2] O'SULLIVAN M,FRANK M G,HURLEY C M,et al.Policelie detection accuracy:The effect of lie scenario[J].Law and Human Behavior,2009,33(6):530.
[3] SEIDENSTAT P,SPLANE F X.Protecting airline passengers in the age of terrorism[M].ABC-CLIO,2009.
[4] YAN W J,WANG S J,LIU Y J,et al.For micro-expression re-cognition:Database and suggestions[J].Neurocomputing,2014,136:82-87.
[5] ZHANG M,FU Q,CHEN Y H,et al.Emotional context influences micro-expression recognition[J].PloS One,2014,9(4):e95018.
[6] LIONG S T,SEE J,WONG K S,et al.Less is more:Micro-expression recognition from video using apex frame[J].Signal Processing:Image Communication,2018,62:82-92.
[7] MERGHANI W,DAVISON A K,YAP M H.A review on facial micro-expressions analysis:datasets,features and metrics[J].arXiv:1805.02397,2018.
[8] POLIKOVSKY S,KAMEDA Y,OHTA Y.Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor[C]//3rd International Conference on Imaging for Crime Detection and Prevention(ICDP 2009).IET,2009:1-6.
[9] ESSA I A,PENTLAND A P.Coding,analysis,interpretation,and recognition of facial expressions[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1997,19(7):757-763.
[10] YAP M H,SEE J,HONG X,et al.Facial micro-expressionsgrand challenge 2018 summary[C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition(FG 2018).IEEE,2018:675-678.
[11] SEE J,YAP M H,LI J,et al.Megc 2019-the second facial micro-expressions grand challenge[C]//2019 14th IEEE International Conference on Automatic Face & Gesture Recognition(FG 2019).IEEE,2019:1-5.
[12] LI J,WANG S J,YAP M H,et al.MEGC2020-The Third Facial Micro-Expression Grand Challenge[C]//2020 15th IEEE International Conference on Automatic Face and Gesture Recognition(FG 2020)(FG).IEEE Computer Society,2020:234-237.
[13] HUANG X,ZHAO G,HONG X,et al.Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns[J].Neurocomputing,2016,175:564-578.
[14] LO L,XIE H X,SHUAI H H,et al.MER-GCN:Micro-Expression Recognition Based on Relation Modeling with Graph Con-volutional Networks[C]//2020 IEEE Conference on Multimedia Information Processing and Retrieval(MIPR).IEEE,2020:79-84.
[15] YAN W J,WU Q,LIU Y J,et al.CASME database:a dataset of spontaneous micro-expressions collected from neutralized faces[C]//2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition(FG).IEEE,2013:1-7.
[16] YAN W J,LI X,WANG S J,et al.CASME II:An improvedspontaneous micro-expression database and the baseline evaluation[J].PloS One,2014,9(1).
[17] QU F,WANG S J,YAN W J,et al.CAS(ME):A Database for Spontaneous Macro-Expression and Micro-Expression Spotting and Recognition[J].IEEE Transactions on Affective Computing,2017,9(4):424-436.
[18] DAVISON A K,LANSLEY C,COSTEN N,et al.Samm:Aspontaneous micro-facial movement dataset[J].IEEE Transactions on Affective Computing,2016,9(1):116-129.
[19] LI X,PFISTER T,HUANG X,et al.A spontaneous micro-expression database:Inducement,collection and baseline[C]//2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition(FG).IEEE,2013:1-6.
[20] COOTES T F,TAYLOR C J,COOPER D H,et al.Active shape models-their training and application[J].Computer Vision and Image Understanding,1995,61(1):38-59.
[21] KASINSKI A,SCHMIDT A.The architecture and performance of the face and eyes detection system based on the Haar cascade classifiers[J].Pattern Analysis and Applications,2010,13(2):197-211.
[22] PENG M,WU Z,ZHANG Z,et al.From macro to micro expression recognition:Deep learning on small datasets using transfer learning[C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition(FG 2018).IEEE,2018:657-661.
[23] MERGHANI W,DAVISON A,YAP M.Facial Micro-expres-sions Grand Challenge 2018:evaluating spatio-temporal features for classification of objective classes[C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition(FG 2018).IEEE,2018:662-666.
[24] KHOR H Q,SEE J,PHAN R C W,et al.Enriched long-term recurrent convolutional network for facial micro-expression recognition[C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition(FG 2018).IEEE,2018:667-674.
[25] LIU Y,DU H,ZHENG L,et al.A neural micro-expressionrecognizer[C]//2019 14th IEEE International Conference on Automatic Face & Gesture Recognition(FG 2019).IEEE,2019:1-4.
[26] GAN Y S,LIONG S T,YAU W C,et al.Off-apexnet on micro-expression recognition system[J].Signal Processing:Image Communication,2019,74:129-139.
[27] ZHOU L,MAO Q,XUE L.Dual-inception network for cross-database micro-expression recognition[C]//2019 14th IEEE International Conference on Automatic Face & Gesture Recognition(FG 2019).IEEE,2019:1-5.
[28] LIONG S T,GAN Y S,SEE J,et al.Shallow triple stream three-dimensional cnn(ststnet) for micro-expression recognition[C]//2019 14th IEEE International Conference on Automatic Face & Gesture Recognition(FG 2019).IEEE,2019:1-5.
[29] HE K,ZHANG X,REN S,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[30] WANG C,PENG M,BI T,et al.Micro-attention for micro-expression recognition[J].Neurocomputing,2020,410:354-362.
[31] VASWANI A,SHAZEER N,PARMAR N,et al.Attention isall you need[J].arXiv:1706.03762,2017.
[32] DEVLIN J,CHANG M W,LEE K,et al.Bert:Pre-training of deep bidirectional transformers for language understanding[J].arXiv:1810.04805,2018.
[33] DOSOVITSKIY A,BEYER L,KOLESNIKOV A,et al.Animage is worth 16x16 words:Transformers for image recognition at scale[J].arXiv:2010.11929,2020.
[34] CHEN M,RADFORD A,CHILD R,et al.Generative pretrai-ning from pixels[C]//International Conference on Machine Learning.PMLR,2020:1691-1703.
[35] SRINIVAS A,LIN T Y,PARMAR N,et al.Bottleneck transformers for visual recognition[J].arXiv:2101.11605,2021.
[36] LIU Z,LIN Y,CAO Y,et al.Swin transformer:Hierarchical vision transformer using shifted windows[J].arXiv:2103.14030,2021.
[37] Electronic spatial sensing for the blind:contributions from perception,rehabilitation,and computer vision[M].Berlin:Springer,Springer Science & Business Media,2013.
[38] PÉREZ J S,MEINHARDT-LLOPIS E,FACCIOLO G.TV-L1 optical flow estimation[J].Image Processing on Line,2013,2013:137-150.
[1] JIN Fang-yan, WANG Xiu-li. Implicit Causality Extraction of Financial Events Integrating RACNN and BiLSTM [J]. Computer Science, 2022, 49(7): 179-186.
[2] ZHAO Dan-dan, HUANG De-gen, MENG Jia-na, DONG Yu, ZHANG Pan. Chinese Entity Relations Classification Based on BERT-GRU-ATT [J]. Computer Science, 2022, 49(6): 319-325.
[3] LI Xing-ran, ZHANG Li-yan, YAO Shu-jing. Micro-expression Recognition Method Combining Feature Fusion and Attention Mechanism [J]. Computer Science, 2022, 49(2): 4-11.
[4] HU Yan-li, TONG Tan-qian, ZHANG Xiao-yu, PENG Juan. Self-attention-based BGRU and CNN for Sentiment Analysis [J]. Computer Science, 2022, 49(1): 252-258.
[5] WANG Xi, ZHANG Kai, LI Jun-hui, KONG Fang, ZHANG Yi-tian. Generation of Image Caption of Joint Self-attention and Recurrent Neural Network [J]. Computer Science, 2021, 48(4): 157-163.
[6] ZHOU Xiao-shi, ZHANG Zi-wei, WEN Juan. Natural Language Steganography Based on Neural Machine Translation [J]. Computer Science, 2021, 48(11A): 557-564.
[7] LIANG Zheng-you, HE Jing-lin, SUN Yu. Three-dimensional Convolutional Neural Network Evolution Method for Facial Micro-expression Auto-recognition [J]. Computer Science, 2020, 47(8): 227-232.
[8] ZHANG Peng-fei, LI Guan-yu, JIA Cai-yan. Truncated Gaussian Distance-based Self-attention Mechanism for Natural Language Inference [J]. Computer Science, 2020, 47(4): 178-183.
[9] ZHANG Yi-jie, LI Pei-feng, ZHU Qiao-ming. Event Temporal Relation Classification Method Based on Self-attention Mechanism [J]. Computer Science, 2019, 46(8): 244-248.
[10] FAN Zi-wei, ZHANG Min, LI Zheng-hua. BiLSTM-based Implicit Discourse Relation Classification Combining Self-attention
Mechanism and Syntactic Information
[J]. Computer Science, 2019, 46(5): 214-220.
Full text



No Suggested Reading articles found!