计算机科学

计算机科学 ›› 2024, Vol. 51 ›› Issue (11): 191-197.doi: 10.11896/jsjkx.240100063

• 计算机图形学&多媒体 • 上一篇    下一篇

基于关键帧与时空特征融合的人脸伪造检测

程燕   

  1. 华东政法大学信息科学与技术系 上海 201620
  • 收稿日期:2024-01-04 修回日期:2024-05-11 出版日期:2024-11-15 发布日期:2024-11-06
  • 通讯作者: 程燕(chengyan@ecupl.edu.cn)
  • 基金资助:
    教育部人文社科一般项目(23YJA820015)

Facial Forgery Detection Based on Key Frames and Fused Spatial-Temporal Features

CHENG Yan   

  1. Department of Information Science and Technology,East China University of Political Science and Law,Shanghai 201620,China
  • Received:2024-01-04 Revised:2024-05-11 Online:2024-11-15 Published:2024-11-06
  • About author:CHENG Yan,born in 1978,Ph.D,associate professor.Her main research interests include image/video forensic and artificial intelligence.
  • Supported by:
    Humanities and Social Sciences of the Ministry of Education(23YJA820015).

PDF (PC)

257

摘要： 基于深度学习的人脸真伪检测是一个典型的二分类问题,模型训练结果的精度不仅受到训练数据质和量的影响,还与训练策略、网络架构设计等有关。以光流法为基础,提出了一种基于关键帧与时空特征融合的人脸伪造检测方法。首先,采用加权光流能量阈值分析法筛选出视频中能量较大的关键帧,将关键帧的光流和LBP纹理特征进行融合,构成具有时间和空间特性的融合特征图,经过增强处理后输入CNN模型进行学习。在FaceForensics++和Celeb-df数据集上的测试表明,所提算法的检测率较传统算法均有明显提升。跨库实验中,所提算法采用Efficientnet-V2结构在FaceForensics++数据集上表现出最优的跨库检测性能,准确率达到90.1%,XceptionNet结构的整体性能优于其他方法,准确率均达到80%以上,具有优越的泛化性能。

关键词: 光流, 关键帧, LBP纹理, CNN模型

Abstract: The deep learning-based facial forgery detection is commonly approached as a binary classification problem.The accuracy of model training results is not only affected by the quality and quantity of training data,but also related to training strategy and network architecture design..In this paper,we propose a new method based on key frames and spatial-temporal features.Firstly,the weighted optical flow energy analysis is used to detect the key frames in a video.Then,the optical flow and LBP features of the key frames are fused to form feature maps with spatial and temporal characteristics.After data augmentation,the feature maps are fed into the CNN model for training.Evaluations conducted on the FaceForensics++ and Celeb-df datasets de-monstrate that the proposed method achieves superior or comparable detection accuracy.Experimental results on cross-datasets show that the proposed method,utilizing the Efficientnet-V2 structure,achieves the best performance on the FaceForensics++ database with the accuracy of 90.1%.Furthermore,the overall performance of the XceptionNet structure surpasses that of other methods,achieving the accuracy over 80%,thus demonstrating superior generalization performance of the proposed method.

Key words: Optical flow, Key frames, LBP texture, CNN model

中图分类号: 

  • TP391
[1] LI X R,JI S L,WU C M,et al.Survey on Deepfakes and Detection Techniques[J].Journal of Software,2021,32(2):496-518.
[2] ZHANG Y X,LI G,CAO Y,et al.A Method for Detecting Human-face-tampered Videos based on Interframe Difference[J].Journal of Cyber Security,2020(2):49-72.
[3] HAN B,HAN X G,ZHANG H,et al.Fighting Fake News:TwoStream Network for Deepfake Detection via Learnable SRM[J].IEEE Transactions on Biometrics,Behavior,and Identity Science,2021,3(3):320-331.
[4] QI H,GUO Q,XU J F,et al.DeepRhythm:Exposing DeepFakeswith Attentional Visual Heartbeat Rhythms[C]//Proceedings of the 28th ACM International Conference on Multimedia.2020:1318-1327.
[5] JUNG T,KIM S,KIM K.DeepVision:Deepfakes DetectionUsing Human Eye Blinking Pattern[J].IEEE Access,2020,8:83144-83154.
[6] AGARWAL S,FARID H,GU Y M,et al.Protecting WorldLeaders Against Deep Fakes[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.2019:38-45.
[7] YANG X,LI Y Z,LYU S.Exposing Deep Fakes using Inconsistent Head Poses[C]//Proceedings of IEEE International Conference on Acoustics,Speech and Signal Processing.2019:8261-8265.
[8] ROSSLER A,COZZOLINO D,VERDOLIVA L,et al.FaceFo-rensics++:Learning to Detect Manipulated Facial Images[C]//Proceedings of IEEE International Conference on Computer Vision.2019:1-11.
[9] AMERINI I,GALTERI L,CALDELLI R,et al.Deepfake Video Detection through Optical Flow based CNN[C]//Proceedings of International Conference on Computer Vision Workshop.2019:1205-1207.
[10] AKASH C,AISHWARYA R,SANIAT S,et al.Leveraging Edges and Optical Flow on Faces for Deepfake Detection[C]//Proceedings of IEEE/IAPR International Joint Conference on Biometrics.2020.
[11] CHOLLET F.Xception:Deep Learning with Depthwise Separable Convolutions[C]//Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition.2017:1800-1807.
[12] SABIR E,CHENG J X,JAISWAL A,et al.Recurrent Convolutional Strategies for Face Manipulation Detection in Videos[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.2019:80-87.
[13] GUERA D,DELP E J.DeepfakeVideo Detection using Recur-rent Neural Network[C]//Proceedings of the 15th IEEE International Conference on Advanced Video and Signal Based Surveillance.2018:1-6.
[14] AFCHAR D,NOZICK V,YAMAGISHI J,et al.Mesonet:ACompact Facial Video Forgery Detection Network[C]//Procee-dings of IEEE International Workshop on Information Forensics and Security.2018:1-7.
[15] SZEGEDY C,VANHOUCKE V,LOFFE S,et al.RethinkingtheInception Architecture for Computer Vision[C]//Procee-dings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.2016:2818-2826.
[16] NGUYEN H H,YAMAGISHI J,ECHIZEN I.Capsule-foren-sics:Using Capsule Networks to Detect Forged Images and Vi-deos[C]//Proceedings of IEEE International Conference on Acoustics,Speech and Signal Processing.2019:2307-2311.
[17] SIMONYAN K,ZISSERMAN A.Very Deep Convolutional Networks for Large-scale Image Recognition[C]//Proceedings of the 3rd International Conference on Learning Representations.2015.
[18] ARASH H,NIMAJAFARI N,HASAN D,et al.A Novel Blockchain-based Deepfake Detection Method using Federated and Deep Learning Models[J].Cognitive Computation,2024,16(3):1073-1091.
[19] YU P P,FEI J W,XIA Z H,et al.Improving Generalization by Commonality Learning in Face Forgery Detection[J].IEEE Transactions on Information Forensics and Security,2022(17):547-558.
[20] XING H,LI M.Deepfake Video Detection based on 3D Convolutional Neural Networks[J].Computer Science,2021,48(7):86-92.
[21] WANG T Y,CHENG H,CHOW K P,et al.Deep Convolutional Pooling Transformer for Deepfake Detection[J].ACM Transactions on Multimedia Computing,Communications,and Applications,2023,19(6):1-20.
[22] HSU C C,ZHUANG Y X,LEE C Y.Deep Fake Image Detectionbased on Pairwise Learning[J].Applied Sciences,2020,10(1):370.
[23] DANG H,LIU F,STEHOUWER J,et al.On the Detection of Digital Face Manipulation[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Re-cognition.2020:5780-5789.
[24] RAHMOUNI N,NOZICK V,YAMAGISHI J,et al.Distinguishing Computer Graphics from Natural Images using Convolution Neural Networks[C]//Proceedings of the IEEE Workshop on Information Forensics and Security.2017:1-6.
[25] ZHU S H,HU J J,SHI Z.LocalAbnormal Behavior Detection based on Optical Flow and Spatio-temporal Gradient[J].Multimedia Tools and Applications,2016,75(15):9445-9459.
[26] FU B,LI W H,CHEN B,et al.Abnormal Behavior Detectionbased on Weighted Energy of Optical Flow[J].Journal of Jilin University(Engineering and Technology Edition),2013,43(6):1644-1649.
[27] ZHANG K P,ZHANG Z P,LI Z F,et al.Joint Face Detection and Alignment using Multitask Cascaded Convolutional Networks[J].IEEE Signal Processing Letters,2016,23(10):1499-1503.
[28] BATTITI R,AMALDI E,KOCH C.Computing Optical Flow Across Multiple Scales:An adaptive coarse-to-fine strategy[J].International Journal of Computer Vision,1991,6(2):133-145.
[29] LI Y Z,YANG X,SUN P,et al.Celeb-df:ALarge-scale Chal-lenging Dataset for Deepfake Forensics[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.2020:3204-3213.
[30] LI Y Z,LYU S W.Exposing Deepfake Videos by Detecting Face Warping Artifacts[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops.2019:46-52.
[31] SANDLER M,HOWARD A,ZHU M L,et al.Mobilenetv2:Inverted Residuals and Linear Bottlenecks[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.2018:4510-4520.
[32] DENG L W,SUO H F,LI D J.Deepfake Video Detection based on EfficientNet-V2 Network[J].Computational Intelligence and Neuroscience,2022:1-13.https://doi.org/10.1155/2022/3441549.
[33] COCCOMINI D A,MESSINA N,GENNARO C,et al.Combining EfficientNet and Vision Transformers for Video Deepfake Detection[C]//Proceedings of the 21st International Conference on Image Analysis and Processing.2022:219-229.
[34] WANG Y H,DANTCHEVA A.AVideo is Worth More than 1000 Lies.Comparing 3DCNN Approaches for Detecting Deepfake[C]//Proceedings of the 15th IEEE International Confe-rence on Automatic Face and Gesture Recognition.2020:515-519.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发