Computer Science ›› 2024, Vol. 51 ›› Issue (11): 191-197.doi: 10.11896/jsjkx.240100063

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Facial Forgery Detection Based on Key Frames and Fused Spatial-Temporal Features

CHENG Yan   

  1. Department of Information Science and Technology,East China University of Political Science and Law,Shanghai 201620,China
  • Received:2024-01-04 Revised:2024-05-11 Online:2024-11-15 Published:2024-11-06
  • About author:CHENG Yan,born in 1978,Ph.D,associate professor.Her main research interests include image/video forensic and artificial intelligence.
  • Supported by:
    Humanities and Social Sciences of the Ministry of Education(23YJA820015).

Abstract: The deep learning-based facial forgery detection is commonly approached as a binary classification problem.The accuracy of model training results is not only affected by the quality and quantity of training data,but also related to training strategy and network architecture design..In this paper,we propose a new method based on key frames and spatial-temporal features.Firstly,the weighted optical flow energy analysis is used to detect the key frames in a video.Then,the optical flow and LBP features of the key frames are fused to form feature maps with spatial and temporal characteristics.After data augmentation,the feature maps are fed into the CNN model for training.Evaluations conducted on the FaceForensics++ and Celeb-df datasets de-monstrate that the proposed method achieves superior or comparable detection accuracy.Experimental results on cross-datasets show that the proposed method,utilizing the Efficientnet-V2 structure,achieves the best performance on the FaceForensics++ database with the accuracy of 90.1%.Furthermore,the overall performance of the XceptionNet structure surpasses that of other methods,achieving the accuracy over 80%,thus demonstrating superior generalization performance of the proposed method.

Key words: Optical flow, Key frames, LBP texture, CNN model

CLC Number: 

  • TP391
[1] LI X R,JI S L,WU C M,et al.Survey on Deepfakes and Detection Techniques[J].Journal of Software,2021,32(2):496-518.
[2] ZHANG Y X,LI G,CAO Y,et al.A Method for Detecting Human-face-tampered Videos based on Interframe Difference[J].Journal of Cyber Security,2020(2):49-72.
[3] HAN B,HAN X G,ZHANG H,et al.Fighting Fake News:TwoStream Network for Deepfake Detection via Learnable SRM[J].IEEE Transactions on Biometrics,Behavior,and Identity Science,2021,3(3):320-331.
[4] QI H,GUO Q,XU J F,et al.DeepRhythm:Exposing DeepFakeswith Attentional Visual Heartbeat Rhythms[C]//Proceedings of the 28th ACM International Conference on Multimedia.2020:1318-1327.
[5] JUNG T,KIM S,KIM K.DeepVision:Deepfakes DetectionUsing Human Eye Blinking Pattern[J].IEEE Access,2020,8:83144-83154.
[6] AGARWAL S,FARID H,GU Y M,et al.Protecting WorldLeaders Against Deep Fakes[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.2019:38-45.
[7] YANG X,LI Y Z,LYU S.Exposing Deep Fakes using Inconsistent Head Poses[C]//Proceedings of IEEE International Conference on Acoustics,Speech and Signal Processing.2019:8261-8265.
[8] ROSSLER A,COZZOLINO D,VERDOLIVA L,et al.FaceFo-rensics++:Learning to Detect Manipulated Facial Images[C]//Proceedings of IEEE International Conference on Computer Vision.2019:1-11.
[9] AMERINI I,GALTERI L,CALDELLI R,et al.Deepfake Video Detection through Optical Flow based CNN[C]//Proceedings of International Conference on Computer Vision Workshop.2019:1205-1207.
[10] AKASH C,AISHWARYA R,SANIAT S,et al.Leveraging Edges and Optical Flow on Faces for Deepfake Detection[C]//Proceedings of IEEE/IAPR International Joint Conference on Biometrics.2020.
[11] CHOLLET F.Xception:Deep Learning with Depthwise Separable Convolutions[C]//Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition.2017:1800-1807.
[12] SABIR E,CHENG J X,JAISWAL A,et al.Recurrent Convolutional Strategies for Face Manipulation Detection in Videos[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.2019:80-87.
[13] GUERA D,DELP E J.DeepfakeVideo Detection using Recur-rent Neural Network[C]//Proceedings of the 15th IEEE International Conference on Advanced Video and Signal Based Surveillance.2018:1-6.
[14] AFCHAR D,NOZICK V,YAMAGISHI J,et al.Mesonet:ACompact Facial Video Forgery Detection Network[C]//Procee-dings of IEEE International Workshop on Information Forensics and Security.2018:1-7.
[15] SZEGEDY C,VANHOUCKE V,LOFFE S,et al.RethinkingtheInception Architecture for Computer Vision[C]//Procee-dings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.2016:2818-2826.
[16] NGUYEN H H,YAMAGISHI J,ECHIZEN I.Capsule-foren-sics:Using Capsule Networks to Detect Forged Images and Vi-deos[C]//Proceedings of IEEE International Conference on Acoustics,Speech and Signal Processing.2019:2307-2311.
[17] SIMONYAN K,ZISSERMAN A.Very Deep Convolutional Networks for Large-scale Image Recognition[C]//Proceedings of the 3rd International Conference on Learning Representations.2015.
[18] ARASH H,NIMAJAFARI N,HASAN D,et al.A Novel Blockchain-based Deepfake Detection Method using Federated and Deep Learning Models[J].Cognitive Computation,2024,16(3):1073-1091.
[19] YU P P,FEI J W,XIA Z H,et al.Improving Generalization by Commonality Learning in Face Forgery Detection[J].IEEE Transactions on Information Forensics and Security,2022(17):547-558.
[20] XING H,LI M.Deepfake Video Detection based on 3D Convolutional Neural Networks[J].Computer Science,2021,48(7):86-92.
[21] WANG T Y,CHENG H,CHOW K P,et al.Deep Convolutional Pooling Transformer for Deepfake Detection[J].ACM Transactions on Multimedia Computing,Communications,and Applications,2023,19(6):1-20.
[22] HSU C C,ZHUANG Y X,LEE C Y.Deep Fake Image Detectionbased on Pairwise Learning[J].Applied Sciences,2020,10(1):370.
[23] DANG H,LIU F,STEHOUWER J,et al.On the Detection of Digital Face Manipulation[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Re-cognition.2020:5780-5789.
[24] RAHMOUNI N,NOZICK V,YAMAGISHI J,et al.Distinguishing Computer Graphics from Natural Images using Convolution Neural Networks[C]//Proceedings of the IEEE Workshop on Information Forensics and Security.2017:1-6.
[25] ZHU S H,HU J J,SHI Z.LocalAbnormal Behavior Detection based on Optical Flow and Spatio-temporal Gradient[J].Multimedia Tools and Applications,2016,75(15):9445-9459.
[26] FU B,LI W H,CHEN B,et al.Abnormal Behavior Detectionbased on Weighted Energy of Optical Flow[J].Journal of Jilin University(Engineering and Technology Edition),2013,43(6):1644-1649.
[27] ZHANG K P,ZHANG Z P,LI Z F,et al.Joint Face Detection and Alignment using Multitask Cascaded Convolutional Networks[J].IEEE Signal Processing Letters,2016,23(10):1499-1503.
[28] BATTITI R,AMALDI E,KOCH C.Computing Optical Flow Across Multiple Scales:An adaptive coarse-to-fine strategy[J].International Journal of Computer Vision,1991,6(2):133-145.
[29] LI Y Z,YANG X,SUN P,et al.Celeb-df:ALarge-scale Chal-lenging Dataset for Deepfake Forensics[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.2020:3204-3213.
[30] LI Y Z,LYU S W.Exposing Deepfake Videos by Detecting Face Warping Artifacts[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops.2019:46-52.
[31] SANDLER M,HOWARD A,ZHU M L,et al.Mobilenetv2:Inverted Residuals and Linear Bottlenecks[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.2018:4510-4520.
[32] DENG L W,SUO H F,LI D J.Deepfake Video Detection based on EfficientNet-V2 Network[J].Computational Intelligence and Neuroscience,2022:1-13.https://doi.org/10.1155/2022/3441549.
[33] COCCOMINI D A,MESSINA N,GENNARO C,et al.Combining EfficientNet and Vision Transformers for Video Deepfake Detection[C]//Proceedings of the 21st International Conference on Image Analysis and Processing.2022:219-229.
[34] WANG Y H,DANTCHEVA A.AVideo is Worth More than 1000 Lies.Comparing 3DCNN Approaches for Detecting Deepfake[C]//Proceedings of the 15th IEEE International Confe-rence on Automatic Face and Gesture Recognition.2020:515-519.
[1] ZHANG Huazhong, PAN Yuekai, TU Xiaoguang, LIU Jianhua, XU Luopeng, ZHOU Chao. Facial Expression Recognition Integrating 3D Facial Dynamic Information and Optical Flow Information [J]. Computer Science, 2024, 51(6A): 230700210-7.
[2] XU Bangwu, WU Qin, ZHOU Haojie. Appearance Fusion Based Motion-aware Architecture for Moving Object Segmentation [J]. Computer Science, 2024, 51(3): 155-164.
[3] JIANG Sheng, ZHU Jianhong. Face Micro-expression Recognition Method Based on ME-ResNet [J]. Computer Science, 2024, 51(11A): 231000053-7.
[4] ZHAO Zheng-peng, LI Jun-gang, PU Yuan-yuan. Low-light Image Enhancement Based on Retinex Theory by Convolutional Neural Network [J]. Computer Science, 2022, 49(6): 199-209.
[5] LENG Jia-xu, TAN Ming-pi, HU Bo, GAO Xin-bo. Video Anomaly Detection Based on Implicit View Transformation [J]. Computer Science, 2022, 49(2): 142-148.
[6] MENG Xiang-yu, XUE Xin-wei, LI Wen-lin, WANG Yi. Motion-estimation Based Space-temporal Feature Aggregation Network for Multi-frames Rain Removal [J]. Computer Science, 2021, 48(5): 170-176.
[7] WANG Kun-lun, LIU Wen-can, HE Xiao-hai, QING Lin-bo, WU Xiao-hong. Motion Feature Descriptor for Abnormal Behavior Detection [J]. Computer Science, 2020, 47(4): 119-124.
[8] WANG Zheng-ning, ZHOU Yang, LV Xia, ZENG Fan-wei, ZHANG Xiang, ZHANG Feng-jun. Improved MDP Tracking Method by Combining 2D and 3D Information [J]. Computer Science, 2019, 46(3): 97-102.
[9] XU Deng, HUANG Xiao-dong. Fire Images Features Extraction Based on Improved Two-stream Convolution Network [J]. Computer Science, 2019, 46(11): 291-296.
[10] HUANG Jin-guo, LIU Tao, ZHOU Xian-chun, YAN Xi-jun. Detection for Group Riot Activity Based on Change Analysis of Group Motion Pattern [J]. Computer Science, 2018, 45(9): 314-319.
[11] CHEN Cheng, CHANG Kan, MO Cai-wang, LI Tian-yi and QIN Tuan-fa. Single Video Super-resolution Algorithm Based on Non-local Means and Total Variation Minimization [J]. Computer Science, 2018, 45(3): 263-267.
[12] LIU Tao, ZHOU Xian-chun, YAN Xi-jun. LDA Facial Expression Recognition Algorithm Combining Optical Flow Characteristics with Gaussian [J]. Computer Science, 2018, 45(10): 286-290.
[13] TANG Ying and SUN Kang-gao. Video Stylization Based on Kinect Depth Information [J]. Computer Science, 2017, 44(Z6): 192-197.
[14] LUO Zheng-ping, LIU Yan-jun and YANG Tian-qi. Gait Recognition Based on Decomposition of Optical Flow Components [J]. Computer Science, 2016, 43(9): 295-300.
[15] LIN Jie and LIN La. Detection Method for Group Riot Activity Based on Change Frequency of Optical Flow’s Magnitude [J]. Computer Science, 2016, 43(5): 283-287.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!