计算机科学 ›› 2021, Vol. 48 ›› Issue (3): 27-39.doi: 10.11896/jsjkx.210100079

所属专题: 多媒体技术进展

• 多媒体技术进展* 上一篇    下一篇

多媒体模型对抗攻防综述

陈凯, 魏志鹏, 陈静静, 姜育刚   

  1. 复旦大学计算机科学技术学院 上海201203
    上海市智能信息处理重点实验室 上海200433
  • 收稿日期:2021-01-10 修回日期:2021-02-03 出版日期:2021-03-15 发布日期:2021-03-05
  • 通讯作者: 姜育刚(ygj@fudan.edu.cn)
  • 作者简介:20210240012@fudan.edu.cn
  • 基金资助:
    国家自然科学基金(62032006);上海市科委项目(20511101000)

Adversarial Attacks and Defenses on Multimedia Models:A Survey

CHEN Kai, WEI Zhi-peng, CHEN Jing-jing, JIANG Yu-gang   

  1. School of Computer Science,Fudan University,Shanghai 201203,China
    Shanghai Key Laboratory of Intelligent Information,Shanghai 200433,China
  • Received:2021-01-10 Revised:2021-02-03 Online:2021-03-15 Published:2021-03-05
  • About author:CHEN Kai,born in 1998,postgraduate.His main research interests include ima-ge and video adversarial attack.
    JIANG Yu-gang,born in 1981,Ph.D,professor,Ph.D supervisor,is a member of China Computer Federation.His main research interests include multimedia content analysis,computer vision and robust & trustworthy AI.
  • Supported by:
    National Natural Science Foundation of China(62032006) and Science and Technology Commission of Shanghai Municipality(20511101000).

摘要: 近年来,随着以深度学习为代表的人工智能技术的快速发展和广泛应用,人工智能正深刻地改变着社会生活的各方面。然而,人工智能模型也容易受到来自精心构造的“对抗样本”的攻击。通过在干净的图像或视频样本上添加微小的人类难以察觉的扰动,就能够生成可以欺骗模型的样本,进而使多媒体模型在推理过程中做出错误决策,为多媒体模型的实际应用部署带来严重的安全威胁。鉴于此,针对多媒体模型的对抗样本生成与防御方法引起了国内外学术界、工业界的广泛关注,并出现了大量的研究成果。文中对多媒体模型对抗攻防领域的进展进行了深入调研,首先介绍了对抗样本生成与防御的基本原理和相关背景知识,然后从图像和视频两个角度回顾了对抗攻防技术在多媒体视觉信息领域的发展历程与最新成果,最后总结了多媒体视觉信息对抗攻防技术目前面临的挑战和有待进一步探索的方向。

关键词: 对抗防御, 对抗攻击, 深度学习, 视频对抗样本, 图像对抗样本

Abstract: In recent years,with the rapid development and wide application of deep learning,artificial intelligence is profoundly changing all aspects of social life.However,artificial intelligence models are also vulnerable to well-designed “adversarial examples”.By adding subtle perturbations that are imperceptible to humans on clean image or video samples,it is possible to generate adversarial examples that can deceive the model,which leads the multimedia model to make wrong decisions in the inference process,and bring serious security threat to the actual application and deployment of the multimedia model.In view of this,adversarial examples generation and defense methods for multimedia models have attracted widespread attention from both academic and industry.This paper first introduces the basic principles and relevant background knowledge of adversarial examples generation and defense.Then,it reviews the recent progress on both adversarial attack and defense on multimedia models.Finally,it summarizes the current challenges as well as the future directions for adversarial attacks and defenses.

Key words: Adversarial attack, Adversarial defense, Deep learning, Image adversarial sample, Video adversarial sample

中图分类号: 

  • TP18
[1]SUN Y,WANG X,TANG X.Deep learning face representation from predicting 10 000 classes[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2014:1891-1898.
[2]TAIGMAN Y,YANG M,RANZATO M A,et al.Deepface:Closing the gap to human-level performance in face verification[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2014:1701-1708.
[3]REDMON J,FARHADI A.YOLO9000:better,faster,stronger[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:7263-7271.
[4]REN S,HE K,GIRSHICK R,et al.Faster r-cnn:Towards real-time object detection with region proposal networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,39(6):1137-1149.
[5]SAON G,KUO H K J,RENNIE S,et al.The IBM 2015 English Conversational Telephone Speech Recognition System[C]//Sixteenth Annual Conference of the International Speech Communication Association.2015:3140-3144.
[6]SILVER D,SCHRITTWIESER J,SIMONYAN K,et al.Mastering the game of go without human knowledge[J].Nature,2017,550(7676):354-359.
[7]SZEGEDY C,ZAREMBA W,SUTSKEVER I,et al.Intriguing properties of neural networks[J].arXiv:1312.6199,2013.
[8]SU J,VARGAS D V,SAKURAI K.One pixel attack for fooling deep neural networks[J].IEEE Transactions on Evolutionary Computation,2019,23(5):828-841.
[9]WEI X,ZHU J,YUAN S,et al.Sparse adversarial perturbations for videos[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2019,33:8973-8980.
[10]MISHKIN D,MATAS J.All you need is a good init[J].arXiv:1511.06422,2015.
[11]MAAS A L,HANNUN A Y,NG A Y.Rectifier nonlinearities improve neural network acoustic models[C]//Proc.ICML.2013,30(1):3.
[12]CLEVERT D A,UNTERTHINER T,HOCHREITER S.Fast and accurate deep network learning by exponential linear units (elus)[J].arXiv:1511.07289,2015.
[13]SHARIF M,BHAGAVATULA S,BAUER L,et al.Accessorize to a crime:Real and stealthy attacks on state-of-the-art face recognition[C]//Proceedings of the 2016 ACM Sigsac Conference on Computer and Communications Security.2016:1528-1540.
[14]LECUN Y,BOTTOU L,BENGIO Y,et al.Gradient-basedlearning applied to document recognition[J].Proceedings of the IEEE,1998,86(11):2278-2324.
[15]KRIZHEVSKY A,HINTON G.Learning multiple layers of features from tiny images[J].Handbook of Systemic Autoimmune Diseases,2009,1(4).
[16]RUSSAKOVSKY O,DENG J,SU H,et al.Imagenet large scale visual recognition challenge[J].International Journal of Computer Vision,2015,115(3):211-252.
[17]SOOMRO K,ZAMIR A R,SHAH M.UCF101:A dataset of101 human actions classes from videos in the wild[J].arXiv:1212.0402,2012.
[18]KUEHNE H,JHUANG H,GARROTE E,et al.HMDB:a large video database for human motion recognition[C]//2011 International Conference on Computer Vision.IEEE,2011:2556-2563.
[19]KAY W,CARREIRA J,SIMONYAN K,et al.The kinetics human action video dataset[J].arXiv:1705.06950,2017.
[20]GOODFELLOW I J,SHLENS J,SZEGEDY C.Explaining and harnessing adversarial examples[J].Stat,2015,1050:20.
[21]KURAKIN A,GOODFELLOW I,BENGIO S.Adversarial machine learning at scale[J].arXiv:1611.01236,2016.
[22]TRAMER F,KURAKIN A,PAPERNOT N,et al.Ensembleadversarial training:attacks and defenses[J].Stat,2018,1050:22.
[23]KURAKIN A,GOODFELLOW I J,BENGIO S.Adversarial examples in the physical world[J].arXiv:1607.02533,2016.
[24]MDRY A,MAKELOV A,SCHMIDT L,et al.Towards Deep Learning Models Resistant to Adversarial Attacks[J].Stat,2017,1050:9.
[25]PAPERNOT N,MCDANIEL P,JHA S,et al.The limitations of deep learning in adversarial settings[C]//2016 IEEE European Symposium on Security and Privacy (EuroS&P).IEEE,2016:372-387.
[26]MOOSAVI-DEZFOOLI S M,FAWZI A,FROSSARD P.Deep-fool:a simple and accurate method to fool deep neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2574-2582.
[27]CARLINI N,WAGNER D.Towards evaluating the robustness of neural networks[C]//2017 IEEE Symposium on Security and Privacy (SP).IEEE,2017:39-57.
[28]CROCE F,HEIN M.Minimally distorted adversarial examples with a fast adaptive boundary attack[C]//International Confe-rence on Machine Learning.PMLR,2020:2196-2205.
[29]CROCE F,HEIN M.Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks[C]//International Conference on Machine Learning.PMLR,2020:2206-2216.
[30]ANDRIUSHCHENKO M,CROCE F,FLAMMARION N,et al.Square attack:a query-efficient black-box adversarial attack via random search[C]//European Conference on Computer Vision.Springer,Cham,2020:484-501.
[31]LIU Y,CHEN X,LIU C,et al.Delving into transferable adversarial examples and black-box attacks[J].arXiv:1611.02770,2016.
[32]DONG Y,LIAO F,PANG T,et al.Boosting adversarial attacks with momentum[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:9185-9193.
[33]POLYAKB T.Some methods of speeding up the convergence of iteration methods[J].USSR Computational Mathematics and Mathematical Physics,1964,4(5):1-17.
[34]LIN J,SONG C,HE K,et al.Nesterov accelerated gradient and scale invariance for adversarial attacks[J].arXiv:1908.06281,2019.
[35]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenetclassification with deep convolutional neural networks[J].Communications of the ACM,2017,60(6):84-90.
[36]SIMONYAN K,ZISSERMAN A.Very deep convolutional net-works for large-scale image recognition[J].arXiv:1409.1556,2014.
[37]HE K,ZHANG X,REN S,et al.Identity mappings in deep residual networks[C]//European Conference on Computer Vision.Springer,Cham,2016:630-645.
[38]XIE C,ZHANG Z,ZHOU Y,et al.Improving transferability of adversarial examples with input diversity[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:2730-2739.
[39]LI Y,BAI S,ZHOU Y,et al.Learning transferable adversarial examples via ghost networks[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2020:11458-11465.
[40]CHEN P Y,ZHANG H,SHARMA Y,et al.Zoo:Zeroth order optimization based black-box attacks to deep neural networks without training substitute models[C]//Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security.2017:15-26.
[41]LAX P D,TERRELL M S.Calculus with applications[M].New York:Springer,2014.
[42]KINGMA D P,BA J.Adam:A method for stochastic optimization[J].arXiv:1412.6980,2014.
[43]ILYAS A,ENGSTROM L,ATHALYE A,et al.Black-box Adversarial Attacks with Limited Queries and Information[C]//Proceedings of the 35th International Conference on Machine Learning(ICML 2018).2018:2137-2146.
[44]WIERSTRA D,SCHAUL T,PETERS J,et al.Natural evolution strategies[C]//2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).IEEE,2008:3381-3387.
[45]SALIMANS T,HO J,CHEN X,et al.Evolution strategies as a scalable alternative to reinforcement learning[J].arXiv:1703.03864,2017.
[46]TU C C,TING P,CHEN P Y,et al.Autozoom:Autoencoder-based zeroth order optimization method for attacking black-box neural networks[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2019,33:742-749.
[47]NESTEROV Y,SPOKOINY V.Random gradient-free minimi-zation of convex functions[J].Foundations of Computational Mathematics,2017,17(2):527-566.
[48]BRENDEL W,RAUBER J,BETHGE M.Decision-based adversarial attacks:Reliable attacks against black-box machine learning models[J].arXiv:1712.04248,2017.
[49]CHENG M,LE T,CHENP Y,et al.Query-efficient hard-label black-box attack:An optimization-based approach[J].arXiv:1807.04457,2018.
[50]GROSSE K,MANOHARAN P,PAPERNOT N,et al.On the(statistical) detection of adversarial examples[J].arXiv:1702.06280,2017.
[51]FEINMAN R,CURTIN R R,SHINTRE S,et al.Detecting adve-rsarial samples from artifacts[J].arXiv:1703.00410,2017.
[52]METZEN J H,GENEWEIN T,FISCHERV,et al.On detecting adversarial perturbations[J].Stat,2017,1050:21.
[53]XU W,EVANS D,QI Y.Feature squeezing:Detecting adversa-rial examples in deep neural networks[J].arXiv:1704.01155,2017.
[54]MENG D,CHEN H.Magnet:a two-pronged defense against adversarial examples[C]//Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security.2017:135-147.
[55]LIAO F,LIANG M,DONG Y,et al.Defense against adversarial attacks using high-level representation guided denoiser[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:1778-1787.
[56]RONNEBERGER O,FISCHER P,BROX T.U-net:Convolu-tional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing andCompu-ter-Assisted Intervention.Springer,Cham,2015:234-241.
[57]ATHALYE A,CARLINI N.On the robustness of the cvpr 2018 white-box adversarial example defenses[J].arXiv:1804.03286,2018.
[58]XIE C,WU Y,MAATEN L,et al.Feature denoising for improving adversarial robustness[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:501-509.
[59]LI S,NEUPANE A,PAUL S,et al.Adversarial perturbationsagainst real-time video classification systems[J].arXiv:1807.00458,2018.
[60]GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Gene-rative adversarial nets[C]//Advances in Neural Information Processing Systems.2014:2672-2680.
[61]NAEH I,PONY R,MANNOR S.Flickering Adversarial At-tacks on Video Recognition Networks[J].arXiv:2002.05123,2020.
[62]CHEN Z,XIE L,PANG S,et al.Appending adversarial frames for universal video attack[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.2021:3199-3208.
[63]JIANG L,MA X,CHEN S,et al.Black-box adversarial attacks on video recognition models[C]//Proceedings of the 27th ACM International Conference on Multimedia.2019:864-872.
[64]WEI Z,CHEN J,WEI X,et al.Heuristic black-box adversarialattacks on video recognition models[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2020:12338-12345.
[65]YAN H,WEI X,LI B.Sparse black-box video attack with reinforcement learning[J].arXiv:2001.03754,2020.
[66]WILLIAMS R J.Simple statistical gradient-following algorithms for connectionist reinforcement learning[J].Machine Learning,1992,8(3/4):229-256.
[67]ZHANG H,ZHU L,ZHU Y,et al.Motion-Excited Sampler:Video Adversarial Attack with Sparked Prior[C]//European Conference on Computer Vision.Springer,Cham,2020:240-256.
[68]XIAO C,DENG R,LI B,et al.Advit:Adversarial frames identifier based on temporal consistency in videos[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:3968-3977.
[69]JIA X,WEI X,CAO X.Identifying and resisting adversarial vi-deos using temporal consistency[J].arXiv:1909.04837,2019.
[70]LO S Y,PATELV M.Defending against multiple and unfore-seen adversarial videos[J].arXiv:2009.05244,2020.
[1] 徐涌鑫, 赵俊峰, 王亚沙, 谢冰, 杨恺.
时序知识图谱表示学习
Temporal Knowledge Graph Representation Learning
计算机科学, 2022, 49(9): 162-171. https://doi.org/10.11896/jsjkx.220500204
[2] 饶志双, 贾真, 张凡, 李天瑞.
基于Key-Value关联记忆网络的知识图谱问答方法
Key-Value Relational Memory Networks for Question Answering over Knowledge Graph
计算机科学, 2022, 49(9): 202-207. https://doi.org/10.11896/jsjkx.220300277
[3] 汤凌韬, 王迪, 张鲁飞, 刘盛云.
基于安全多方计算和差分隐私的联邦学习方案
Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy
计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108
[4] 王剑, 彭雨琦, 赵宇斐, 杨健.
基于深度学习的社交网络舆情信息抽取方法综述
Survey of Social Network Public Opinion Information Extraction Based on Deep Learning
计算机科学, 2022, 49(8): 279-293. https://doi.org/10.11896/jsjkx.220300099
[5] 郝志荣, 陈龙, 黄嘉成.
面向文本分类的类别区分式通用对抗攻击方法
Class Discriminative Universal Adversarial Attack for Text Classification
计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077
[6] 姜梦函, 李邵梅, 郑洪浩, 张建朋.
基于改进位置编码的谣言检测模型
Rumor Detection Model Based on Improved Position Embedding
计算机科学, 2022, 49(8): 330-335. https://doi.org/10.11896/jsjkx.210600046
[7] 孙奇, 吉根林, 张杰.
基于非局部注意力生成对抗网络的视频异常事件检测方法
Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection
计算机科学, 2022, 49(8): 172-177. https://doi.org/10.11896/jsjkx.210600061
[8] 胡艳羽, 赵龙, 董祥军.
一种用于癌症分类的两阶段深度特征选择提取算法
Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification
计算机科学, 2022, 49(7): 73-78. https://doi.org/10.11896/jsjkx.210500092
[9] 程成, 降爱莲.
基于多路径特征提取的实时语义分割方法
Real-time Semantic Segmentation Method Based on Multi-path Feature Extraction
计算机科学, 2022, 49(7): 120-126. https://doi.org/10.11896/jsjkx.210500157
[10] 侯钰涛, 阿布都克力木·阿布力孜, 哈里旦木·阿布都克里木.
中文预训练模型研究进展
Advances in Chinese Pre-training Models
计算机科学, 2022, 49(7): 148-163. https://doi.org/10.11896/jsjkx.211200018
[11] 周慧, 施皓晨, 屠要峰, 黄圣君.
基于主动采样的深度鲁棒神经网络学习
Robust Deep Neural Network Learning Based on Active Sampling
计算机科学, 2022, 49(7): 164-169. https://doi.org/10.11896/jsjkx.210600044
[12] 苏丹宁, 曹桂涛, 王燕楠, 王宏, 任赫.
小样本雷达辐射源识别的深度学习方法综述
Survey of Deep Learning for Radar Emitter Identification Based on Small Sample
计算机科学, 2022, 49(7): 226-235. https://doi.org/10.11896/jsjkx.210600138
[13] 祝文韬, 兰先超, 罗唤霖, 岳彬, 汪洋.
改进Faster R-CNN的光学遥感飞机目标检测
Remote Sensing Aircraft Target Detection Based on Improved Faster R-CNN
计算机科学, 2022, 49(6A): 378-383. https://doi.org/10.11896/jsjkx.210300121
[14] 王建明, 陈响育, 杨自忠, 史晨阳, 张宇航, 钱正坤.
不同数据增强方法对模型识别精度的影响
Influence of Different Data Augmentation Methods on Model Recognition Accuracy
计算机科学, 2022, 49(6A): 418-423. https://doi.org/10.11896/jsjkx.210700210
[15] 闫萌, 林英, 聂志深, 曹一凡, 皮欢, 张兰.
一种提高联邦学习模型鲁棒性的训练方法
Training Method to Improve Robustness of Federated Learning
计算机科学, 2022, 49(6A): 496-501. https://doi.org/10.11896/jsjkx.210400298
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!