Computer Science ›› 2026, Vol. 53 ›› Issue (7): 1-8.doi: 10.11896/jsjkx.250600166

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Research on Facial Emotion Expression Technologies for Humanoid Robots

KE Xianxin, LI Xuan, SONG Junqi   

  1. College of Mechanical Engineering and Automation,Shanghai University,Shanghai 200444,China
  • Received:2025-06-24 Revised:2025-09-24 Online:2026-07-15 Published:2026-07-10
  • About author:KE Xianxin,born in 1973,postdoc-toral,associate professor.His main research interests include intelligent robot development,artificial intelligence applications,multimodal emotional state recognition and emotional interaction.

Abstract: Facial emotional expression in humanoid robots represents a cornerstone technology for enabling naturalistic human-robot interaction,exhibiting broad application prospects in sectors such as medical support,educational services,and social entertainment.This paper reviews the developmental trajectory of the field,delineating its evolution across three stages based on technological capability:foundational emotional expression,enhanced affective interaction,and personalized emotional resonance.It systematically examines the key breakthroughs within each stage concerning the design of biomimetic mechanical structures and the construction of multimodal affective interaction systems.The review places particular emphasis on the application of the facial action coding system,flexible biomimetic skin materials,deep learning,and multimodal large models(MLMs) in affective mode-ling.Furthermore,it investigates the critical role of multimodal spatio-temporal alignment mechanisms in the coordinated optimization of facial expressions,vocalizations,and physical actions.Finally,the paper outlines future research trajectories,including the fusion of high-precision actuation and sensing,the integration of multimodal large models,the enhancement of personalization and cultural adaptability,and strategies for mitigating the uncanny valley effect.This work provides a theoretical foundation to advance the intelligent and naturalistic development of facial emotional expression technology in humanoid robots.

Key words: Humanoid robot, Facial emotional expression, Biomimetic mechanical structure, Affective modeling, Multimodal spatio-temporal alignment, Uncanny valley effect

CLC Number: 

  • TP242
[1]ESFANDBOD A,ROKHI Z,MEGHDARI A F,et al.Utilizing an emotional robot capable of lip-syncing in robot-assisted speech therapy sessions for children with language disorders[J].International Journal of Social Robotics,2023,15(2):165-183.
[2]IVANOV S,WEBSTER C,SEYITOĞLU F.Humans and/or ro-bots? Tourists' preferences towards the humans-robots mix in the service delivery system[J].Service Business,2023,17(1):195-231.
[3]PODPEČAN V.Can you dance? A study of child-robot interaction and emotional response using the NAO robot[J].Multimodal Technologies and Interaction,2023,7(9):85.
[4]TAKANISHI A,HIRANO S,SATO K.Development of an anthropomorphic head-eye system for a humanoid robot-realization of human-like head-eye motion using eyelids adjusting to brightness[C]//Proceedings 1998 IEEE International Conference on Robotics and Automation.IEEE,1998:1308-1314.
[5]BREAZEAL C L.Sociable machines:expressive social exchange between humans and robots[M].Cambridge,MA:Massachusetts Institute of Technology,2000.
[6]WANG W.Top 10 Innovative Humanoid Robot Products of2024 Revealed [N].China Electronics News,2025-01-17(6).
[7]TAKANISHI A,TAKANOBU H,KATO I,et al.Development of the anthropomorphic head-eye robot WE-3RII with an autonomous facial expression mechanism[C]//Proceedings 1999 IEEE International Conference on Robotics and Automation.IEEE,1999:3255-3260.
[8]ALLMAN T.Nexi robot,the(A Great Idea)[M].Norwood House Press,2009.
[9]MIWA H,ITOH K,MATSUMOTO M,et al.Effective emo-tional expressions with emotion expression humanoid robot WE-4RII[C]//2004 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS).IEEE,2004:2203-2208.
[10]FORTUNATI L,SORRENTINO A,FIORINI L,et al.The rise of the roboid[J].International Journal of Social Robotics,2021,13(6):1457-1471.
[11]SHARMA N,PANDEY J K,MONDAL S.A review of mobile robots:applications and future prospect[J].International Journal of Precision Engineering and Manufacturing,2023,24(9):1695-1706.
[12]YOUSIF M.Humanoid robot enhancing social and communication skills of autistic children:review[J].Artificial Intelligence &Robotics Development Journal,2021,1(2):80-92.
[13]ABDOLLAHI H,MAHOOR M H,ZANDIE R,et al.Artificial emotional intelligence in socially assistive robots for older adults:a pilot study[J].IEEE Transactions on Affective Computing,2023,14(3):2020-2032.
[14]KISHI T,OTANI T,ENDO N,et al.Development of expressive robotic head for bipedal humanoid robot[C]//2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.IEEE,2012:4584-4589.
[15]HE M,BI J,WANG Y J,et al.A Review of Emotional Expression Technology in Humanoid Robots [J].Journal of Chongqing University of Technology(Natural Science),2022,36(11):183-192.
[16]HASHIMOTO T,HITRAMATSU S,TSUJI T,et al.Development of the face robot SAYA for rich facial expressions[C]//2006 SICE-ICASE International Joint Conference.2006:5423-5428.
[17]WU W G,SONG C,MENG Q M.Research and Experiment of Voice and Lip-shaped System of Android Head Robot “H&frobot-Ⅲ” [J].Journal of Machine Design,2008(1):15-19.
[18]CHEN W L,HUANG J H,YANG Z P.Research and Development of AI Autonomous Guided Tour Robot Based on 3D Vision [J].Computer Knowledge and Technology,2024,20(5):95-98.
[19]OH J H,HANSON D,KIM W,et al.Design of android type humanoid robot albert HUBO[C]//2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.IEEE,2006:1428-1433.
[20]ISHIHARA H,YOSHIKAWA Y,ASADA M.Realistic childrobot “affetto” for understanding the caregiver-child attachment relationship that guides the child development[C]//2011 IEEE International Conference on Development and Learning(ICDL).2011:1-5.
[21]MAZZEI D,ZARAKI A,LAZZERI N,et al.Recognition andexpression of emotions by a symbiotic android head[C]//2014 IEEE-RAS International Conference on Humanoid Robots.IEEE,2014:134-139.
[22]GHORBANDAEI POUR A,TAHERI A,ALEMI M,et al.Human-robot facial expression reciprocal interaction platform:Case studies on children with autism[J].International Journal of Social Robotics,2018,10(2):179-198.
[23]SUN J L.Manufacturing the Future:Integration and Innovation of Technology and Livelihood [J].Contemporary Television,2020(2):45-47,98.
[24]EKMAN P,FRIESEN W V.Facial action coding system[M]//Environmental Psychology & Nonverbal Behavior.1978.
[25]LI Y,ZHU L,ZHANG Z,et al.Humanoid robot heads for human-robot interaction:a review[J].Science China Technological Sciences,2024,67(2):357-379.
[26]HUSSAIN N,ERZIN E,SEZGIN T M,et al.Training sociallyengaging robots:modeling backchannel behaviors with batch reinforcement learning[J].IEEE Transactions on Affective Computing,2022,13(4):1840-1853.
[27]AUFLEM M,KOHTALA C,JUNG M F,et al.Facing the FACS-Using AI to Evaluate and Control Facial Action Units in Humanoid Robot Face Development[J].Frontiers in Robotics and AI,2022,9:887645.
[28]ZHANG D,PENG J,JIAO Y,et al.ExFace:expressive facialcontrol for humanoid robots with diffusion transformers and bootstrap training[J].arXiv:2504.14477,2025.
[29]SNYDER D E,SAPPER E D.See me,feel me,touch me,heal me:a contextual overview of conductive polymer composites as synthetic human skin[J].Journal of Composites Science,2022,6(5):141.
[30]HABIB A,DAS S K,BOGDAN I C,et al.Learning human-like facial expressions for android phillip K.Dick[C]//2014 IEEE International Conference on Automation Science and Engineering(CASE).IEEE,2014:1159-1165.
[31]MORI M,MACDORMAN K,KAGEKI N.The uncanny valley [from the field][J].IEEE Robotics & Automation Magazine,2012,19(2):98-100.
[32]TANG H,SEBE N.Facial expression translation using land-mark guided GANs[J].IEEE Transactions on Affective Computing,2022,13(4):1986-1997.
[33]ISHIHARA H,HAYASHI R,LAVIEILLE F,et al.Automatic generation of dynamic arousal expression based on decaying wave synthesis for robot faces[J].Journal of Robotics and Mechatronics,2024,36(6):1481-1494.
[34]KIMIZUKA K,AZHARI S,TOKUNO S,et al.Haptic in-sensor computing device made of carbon nanotube-polydimethylsiloxane nanocomposites[J].arXiv:2406.03958,2024.
[35]XU Y,SUN Z,BAI Z,et al.Bionic e-skin with precise multi-directional droplet sliding sensing for enhanced robotic perception[J].Nature Communications,2024,15(1):6022.
[36]CHEN G W,ZHANG P Z,WANG T,et al.Overview of Multimodal Sentiment Analysis [J].Journal of Communication University of China(Natural Science Edition),2022,29(2):70-78.
[37]PANDEY A K,GELIN R.A mass-produced sociable humanoid robot:Pepper:the first machine of its kind[J].IEEE Robotics & Automation Magazine,2018,25(3):40-48.
[38]MIWA H,ITOH K,ITO D,et al.Design and control of 9-DOFsemotion expression humanoid arm[C]//IEEE International Conference on Robotics and Automation(ICRA'04).2004.IEEE,2004:128-133.
[39]MEHRABIAN A.Pleasure-arousal-dominance:A general frame-work for describing and measuring individual differences in temperament[J].Current Psychology,1996,14(4):261-292.
[40]HUANG H C,LIU N,HU M,et al.A Game-Based Cognitive Emotional Interaction Model for Robots [J].Journal of Elec-tronics and Information Technology,2019,41(10):2471-2478.
[41]LIU X,NI R,YANG B,et al.Unlocking human-like facial expressions in humanoid robots:a novel approach for action unit driven facial expression disentangled synthesis[J].IEEE Transactions on Robotics,2024,40:3850-3865.
[42]MISHRA C,VERDONSCHOT R,HAGOORT P,et al.Real-time emotion generation in human-robot dialogue using large language models[J].Frontiers in Robotics and AI,2023,10:1271610.
[43]MOSTAFAVI M,PORTER M D.How emoji and word embedding helps to unveil emotional transitions during online messaging[C]//2021 IEEE International Systems Conference(syscon).2021:1-8.
[44]RAWAL N,KOERT D,TURAN C,et al.ExGenNet:learning to generate robotic facial expression using facial expression re-cognition[J].Frontiers in Robotics and AI,2022,8:730317.
[45]ISHIHARA H,HAYASHI R,LAVIEILLE F,et al.Automatic generation of dynamic arousal expression based on decaying wave synthesis for robot faces[J].Journal of Robotics and Mechatronics,2024,36(6):1481-1494.
[46]MANDEEL A R,AL-RADHI M S,CSAPÓ T G.Speaker adaptation experiments with limited data for end-to-end text-to-speech synthesis using Tacotron2[J].Infocommunications Journal,2022,14(3):55-62.
[47]ZHOU K,SISMAN B,RANA R,et al.Speech synthesis withmixed emotions[J].IEEE Transactions on Affective Computing,2023,14(4):3120-3134.
[48]LIU S,CAO Y,MENG H.Emotional voice conversion with cycle-consistent adversarial network[J].arXiv:2004.03781,2020.
[49]FU C,DENG Q,SHEN J,et al.A preliminary study on realizing human-robot mental comforting dialogue via sharing experience emotionally[J].Sensors,2022,22(3):991.
[50]LEE J.Generating robotic speech prosody for human robot interaction:a preliminary study[J].Applied Sciences,2021,11(8):3468.
[51]SHUSTER K,XU J,KOMEILI M,et al.Blender Bot 3:a de-ployed conversational agent that continually learns to responsibly engage[J].arXiv:2208.03188,2022.
[52]CHEBOTAR Y,VUONG Q,HAUSMAN K,et al.Q-transformer:scalable offline reinforcement learning via autoregressive Q-functions[C]//Conference on Robot Learning.PMLR,2023:3909-3928.
[53]GARCIA S,GOMEZ-DONOSO F,CAZORLA M.Enhancinghuman-robot interaction:development of multimodal robotic assistant for user emotion recognition[J].Applied Sciences,2024,14(24):11914.
[54]HUANG W C,HSUEH Y L.An emotional dialogue systemusing conditional generative adversarial networks with a sequence-to-sequence transformer encoder[J].ACM Transactions on Asian and Low-Resource Language Information Processing,2024,23(12):164:1-164:23.
[55]ALY A,TAPUS A.Multimodal adapted robot behavior synthesis within a narrative human-robot interaction[C]//2015 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS).2015:2986-2993.
[56]YAN Z,SONG Y,ZHOU R,et al.Facial expression realization of humanoid robot head and strain-based anthropomorphic eva-luation of robot facial expressions[J].Biomimetics,2024,9(3):122.
[57]WANG C,HASLER S,TANNEBERG D,et al.LaMI:large language models for multi-modal human-robot interaction[C]//Extended Abstracts of the CHI Conference on Human Factors in Computing Systems.2024:1-10.
[58]INDURKHYA B.Ethical aspects of faking emotions in chatbots and social robots[C]//2023 32nd IEEE International Conference on Robot and Human Interactive Communication(RO-MAN).2023:1719-1724.
[59]RUDSCHIES C,SCHNEIDER I.The long and winding road to bans for artificial intelligence:from public pressure and regulatory initiatives to the EU AI act[J].Digital Society,2025,4(2):57.
[60]DORAFSHANIAN M,AITSAM M,MEJRI M,et al.Beyonddata collection:safeguarding user privacy in social robotics[C]//2024 IEEE International Conference on Industrial Technology(ICIT).2024:1-6.
[1] YANG Liang, FU Yu, FU Gen-ping and DENG Chun-jian. Running Gait Planning and Control for Humanoid Robot Based on Energy Efficiency Optimization [J]. Computer Science, 2016, 43(6): 270-275.
[2] WANG Zhi-liang,YU Guo-chen,XIE Lun. Three-dimensional Gait Planning for Humanoid Robots Based on Decoupling Synthesis and ZMP Algorithm [J]. Computer Science, 2010, 37(12): 215-217.
[3] MENG Xiu-yan WANG Zhi-liang (Information Engineering School, University of Science and Technology Beijing , Beijing , China). [J]. Computer Science, 2008, 35(12): 178-182.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!