计算机科学

计算机科学 ›› 2025, Vol. 52 ›› Issue (1): 345-361.doi: 10.11896/jsjkx.240300080

• 信息安全 • 上一篇    下一篇

计算机视觉领域对抗样本检测综述

张鑫1, 张晗1,2, 牛曼宇1, 姬莉霞1,3   

  1. 1 郑州大学网络空间安全学院 郑州 450001
    2 智能警务四川省重点实验室 四川 泸州 646000
    3 四川大学计算机学院 成都 610065
  • 收稿日期:2024-03-12 修回日期:2024-08-10 出版日期:2025-01-15 发布日期:2025-01-09
  • 通讯作者: 姬莉霞( jilixia@zzu.edu.cn)
  • 作者简介:(geekxin@gs.zzu.edu.cn)
  • 基金资助:
    国家自然科学基金青年科学基金(62302458);河南省重大科技专项(231100210200);河南省高等学校重点科研项目(24A520047);智能警务四川省重点实验室2024年度开放课题项目(ZNJW2024KFQN005)

Adversarial Sample Detection in Computer Vision:A Survey

ZHANG Xin1, ZHANG Han1,2, NIU Manyu1, JI Lixia1,3   

  1. 1 School of Cyber Science and Engineering,Zhengzhou University,Zhengzhou 450001,China
    2 Intelligent Policing Key Laboratory of Sichuan Province,Luzhou,Sichuan 646000,China
    3 School of Computer Science,Sichuan University,Chengdu 610065,China
  • Received:2024-03-12 Revised:2024-08-10 Online:2025-01-15 Published:2025-01-09
  • About author:ZHANG Xin,born in 1999,postgra-duate.His main research interests include image processing and information security.
    JI Lixia,born in 1979,associate professor.Her main research interests include multi-modal learning and information security.
  • Supported by:
    Young Scientists Fund of the National Natural Science Foundation of China(62302458),Major Science and Technology Project of Henan Province(231100210200),Key Scientific Research Project of Universities in Henan Province(24A520047) and 2024 Open Subject Project of Sichuan Provincial Key Laboratory of Intelligent Policing(ZNJW2024KFQN005).

PDF (PC)

139

摘要： 随着数据量的增加和硬件性能的提升,深度学习在计算机视觉领域取得了显著进展。然而,深度学习模型容易受到对抗样本的攻击,导致输出发生显著变化。对抗样本检测作为一种有效的防御手段,可以在不改变模型结构的前提下防止对抗样本对深度学习模型造成影响。首先,对近年来的对抗样本检测研究工作进行了整理,分析了对抗样本检测与训练数据的关系,根据检测方法所使用特征进行分类,系统全面地介绍了计算机视觉领域的对抗样本检测方法;然后,对一些结合跨领域技术的检测方法进行了详细介绍,统计了训练和评估检测方法的实验配置;最后,汇总了一些有望应用于对抗样本检测的技术,并对未来的研究挑战进行展望。

关键词: 深度学习, 对抗样本攻击, 对抗样本检测, 人工智能安全, 图像分类

Abstract: With the increase in data volume and improvement in hardware performance,deep learning(DL) has made significant progress in the field of computer vision.However,deep learning models are vulnerable to adversarial samples,causing significant changes in the output.As an effective defense method,adversarial sample detection can prevent adversarial samples from affecting the deep learning model without changing the model structure.This paper organizes the research work on adversarial example detection in recent years,analyzes the relationship between adversarial example detection and training data,classifies them according to the characteristics used in the detection method,and systematically and comprehensively introduces adversarial sample detection methods in the field of computer vision.Then,some detection methods that combine cross-domain technologies are introduced in detail,and the experimental configurations for training and evaluating detection methods are statistically analyzed.Finally,some technologies that are expected to be applied to adversarial sample detection are summarized,and future research challenges and development directions are prospected.

Key words: Deep learning, Adversarial sample attacks, Adversarial sample detection, AI security, Image classification

中图分类号: 

  • TP391
[1]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenetclassification with deep convolutional neural networks[C]//Annual Conference on Neural Information Processing Systems.2012.
[2]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[3]HE K,ZHANG X,REN S,et al.Deep residual learning forimage recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[4]REN S Q,HE K M,GIRSHICK R,et al.Faster r-cnn:Towards real-time object detection with region proposal networks[J].arXiv:1506.01497,2015.
[5]SZEGEDY C,ZAREMBA W,SUTSKEVER I,et al.Intriguing properties of neural networks[J].arXiv:1312.6199,2013.
[6]GOODFELLOW I J,SHLENS J,SZEGEDY C.Explaining and harnessing adversarial examples[J].arXiv:1412.6572,2014.
[7]ZHANG J,HUANG Y,WU W,et al.Transferable adversarial attacks on vision transformers with token gradient regularization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2023:16415-16424.
[8]CARLINI N,WAGNER D.Towards evaluating the robustness of neural networks[C]//2017 IEEE Symposium on Security and Privacy(SP).IEEE,2017:39-57.
[9]XU H,LI Y,LIU X,et al.Yet meta learning can adapt fast,it can also break easily[C]//Proceedings of the 2021 SIAM International Conference on Data Mining(SDM).Society for Industrial and Applied Mathematics,2021:540-548.
[10]KARIMI M P,AMIRKHANI A,SHOKOUHI S B.Robust object detection against adversarial perturbations withgabor filter[C]//2021 29th Iranian Conference on Electrical Engineering(ICEE).IEEE,2021:187-192.
[11]WANG L,YOON K J.Psat-gan:Efficient adversarial attacksagainst holistic scene understanding[J].IEEE Transactions on Image Processing,2021,30:7541-7553.
[12]ABDULLAH H,RAHMAN M S,GARCIA W,et al.Hear “no evil”,see“kenansville”:Efficient and transferable black-box attacks on speech recognition and voice identification systems[C]//2021 IEEE Symposium on Security and Privacy(SP).IEEE,2021:712-729.
[13]CHEN G,CHENB S,FAN L,et al.Who is real bob? adversarial attacks on speaker recognition systems[C]//2021 IEEE Symposium on Security and Privacy(SP).IEEE,2021:694-711.
[14]HU Z,HUANG S,ZHU X,et al.Adversarial texture for fooling person detectors in the physical world[C]//Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition.2022:13307-13316.
[15]WANG D,JIANG T,SUN J,et al.Fca:Learning a 3d full-cove-rage vehicle camouflage for multi-view physical adversarial attack[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2022:2414-2422.
[16]LIU J,LAU C P,SOURI H,et al.Mutual adversarial training:Learning together is better than going alone[J].IEEE Transactions on Information Forensics and Security,2022,17:2364-2377.
[17]HARDER P,PFREUNDT F J,KEUPER M,et al.Spectral-defense:Detecting adversarial attacks on cnns in the fourier domain[C]//2021 International Joint Conference on Neural Networks(IJCNN).IEEE,2021:1-8.
[18]DRENKOW N,FENDLEY N,BURLINA P.Attack agnosticdetection of adversarial examples via random subspace analysis[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.2022:472-482.
[19]LIU Z,CAO C,TAO F,et al.From Spatial to Spectral Domain,a New Perspective for Detecting Adversarial Examples[J/OL].Security and Communication Networks,2022.[2022-09-05].https://doi.org/10.1155/2022.5501035.
[20]NADERI H,NOORBAKHSH K,ETEMADI A,et al.Lpf-defense:3d adversarial defense based on frequency analysis[J].Plos one,2023,18(2):e0271388.
[21]ZHANG T,YANG K W,WEI J H,et al.A review of adversarial sample detection and defense technology for image data [J].Computer Research and Development,2022,59(6):1315-1328.
[22]ALDAHDOOH A,HAMIDOUCHE W,FEZZA S A,et al.Adversarial example detection for DNN models:A review and experimental comparison[J].Artificial Intelligence Review,2022,55(6):4403-4462.
[23]ZHOU T,GAN R,XU D W,et al.A review of image adversarial example detection [J/OL].Journal of Software:1-35.[2023-10-23].https://doi.org/10.13328/j.cnki.jos.006834.
[24]MADRY A,MAKELOV A,SCHMIDT L,et al.Towards deep learning models resistant to adversarial attacks[J].arXiv:1706.06083,2017.
[25]MOOSAVI-DEZFOOLI S M,FAWZI A,FROSSARD P.Deepfool:a simple and accurate method to fool deep neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2574-2582.
[26]XIAO C,LI B,ZHU J Y,et al.Generating adversarial examples with adversarial networks[J].arXiv:1801.02610,2018.
[27]PAPERNOT N,MCDANIEL P,JHA S,et al.The limitations of deep learning in adversarial settings[C]//2016 IEEE European Symposium on Security and Privacy(EuroS&P).IEEE,2016:372-387.
[28]SU J,VARGAS D V,SAKURAI K.One pixel attack for fooling deep neural networks[J].IEEE Transactions on Evolutionary Computation,2019,23(5):828-841.
[29]CHEN P Y,ZHANG H,SHARMA Y,et al.Zoo:Zeroth order optimization based black-box attacks to deep neural networks without training substitute models[C]//Proceedings of the 10th ACM workshop on Artificial Intelligence and Security.2017:15-26.
[30]MAO X,CHEN Y,LI Y,et al.Gap++:Learning to generate target-conditioned adversarial examples[J].arXiv:2006.05097,2020.
[31]CARRARA F,FALCHI F,CALDELLI R,et al.Detecting adversarial example attacks to deep neural networks[C]//Proceedings of the 15th International Workshop on Content-based Multimedia Indexing.2017:1-7.
[32]SHI C,HOLTZ C,MISHNE G.Online adversarial purification based on self-supervision[J].arXiv:2101.09387,2021.
[33]XU W,EVANS D,QI Y.Feature squeezing:Detecting adversa-rial examples in deep neural networks[J].arXiv:1704.01155,2017.
[34]MA S,LIU Y,TAO G,et al.Nic:Detecting adversa-rial samples with neural network invariant checking[C]//26th Annual Network and Distributed System Security Sympo-sium(NDSS 2019).Internet Soc,2019.
[35]HU J,SHEN L,SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:7132-7141.
[36]ALDAHDOOH A,HAMIDOUCHE W,DéFORGES O.Revisiting model’s uncertainty and confidences for adversarial example detection[J].Applied Intelligence,2023,53(1):509-531.
[37]GEIFMAN Y,EL-YANIV R.Selectivenet:A deep neural network with an integrated reject option[C]//International Confe-rence on Machine Learning.PMLR,2019:2151-2159.
[38]HENDRYCKS D,GIMPEL K.A baseline for detecting misclassified and out-of-distribution examples in neural networks[J].arXiv:1610.02136,2016.
[39]KULLBACK S,LEIBLER R A.On information and sufficiency[J].The Annals of Mathematical Statistics,1951,22(1):79-86.
[40]WIYATNO RR,XU A,DIA O,et al.Adversarial examples in modern machine learning:A review[J].arXiv:1911.05268,2019.
[41]ZHENG Z H,HONG P Y.Robust detection of adversarial attacks by modeling the intrinsic properties of deep neural networks[J].Neural Information Processing Systems,2018,31:7924-7933.
[42]PAPERNOT N,MCDANIEL P.Deep k-nearest neighbors:Towards confident,interpretable and robust deep learning[J].ar-Xiv:1803.04765,2018.
[43]ABUSNAINA A,WU Y,ARORA S,et al.Adversarial example detection using latent neighborhood graph[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2021:7687-7696.
[44]LIANG B,LI H,SU M,et al.Detecting adversarial image examples in deep neural networks with adaptive noise reduction[J].IEEE Transactions on Dependable and Secure Computing,2018,18(1):72-85.
[45]WANG Y,LI X,YANG L,et al.ADDITION:Detecting Adversarial Examples With Image-Dependent Noise Reduction[J].IEEE Transactions on Dependable and Secure Computing,2023,21(3):1139-1154.
[46]MOORTHY A K,BOVIK A C.Blind image quality assessment:From natural scene statistics to perceptual quality[J].IEEE Transactions on Image Processing,2011,20(12):3350-3364.
[47]KHERCHOUCHE A,FEZZA S A,HAMIDOUCHE W,et al.Detection of adversarial examples in deep neural networks with natural scene statistics[C]//2020 International Joint Conference on Neural Networks(IJCNN).IEEE,2020:1-7.
[48]MITTAL A,MOORTHY A K,BOVIK A C.No-referenceimage quality assessment in the spatial domain[J].IEEE Transactions on image processing,2012,21(12):4695-4708.
[49]GONG Z,WANG W.Adversarial and clean data are not twins[C]//Proceedings of the Sixth International Workshop on Exploiting Artificial Intelligence Techniques for Data Management.2023:1-5.
[50]HOSSEINI H,CHEN Y,KANNAN S,et al.Blocking transfera-bility of adversarial examples in black-box learning systems[J].arXiv:1703.04318,2017.
[51]LUST J,CONDURACHE A P.Gran:An efficient gradient-norm based detector for adversarial and misclassified examples[J].arXiv:2004.09179,2020.
[52]COHEN G,SAPIRO G,GIRYES R.Detecting adversarial samples using influence functions and nearest neighbors[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:14453-14462.
[53]MA X,LI B,WANG Y,et al.Characterizing adversarial sub-spaces using local intrinsic dimensionality[J].arXiv:1801.02613,2018.
[54]LORENZ P,KEUPER M,KEUPER J.Unfolding local growth rate estimates for(almost) perfect adversarial detection[J].ar-Xiv:2212.06776,2022.
[55]LEE K,LEE K,LEE H,et al.A simple unified framework for detecting out-of-distribution samples and adversarial attacks[J].arXiv:1807.03888,2018.
[56]CHEN K,CHEN Y,ZHOU H,et al.Adversarial examples detection beyond image space[C]//ICASSP 2021-2021 IEEE International Conference on Acoustics,Speech and Signal Proces-sing(ICASSP).IEEE,2021:3850-3854.
[57]GROSSE K,MANOHARAN P,PAPERNOT N,et al.On the(statistical) detection of adversarial examples[J].arXiv:1702.06280,2017.
[58]GAO R,LIU F,ZHANG J,et al.Maximum mean discrepancy test is aware of adversarial attacks[C]//International Confe-rence on Machine Learning.PMLR,2021:3564-3575.
[59]FEINMAN R,CURTIN RR,SHINTRE S,et al.Detecting adversarial samples from artifacts[J].arXiv:1703.00410,2017.
[60]DONG C,KUMAR A,LIU E.Think twice before detectinggan-generated fake images from their spectral domain imprints[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:7865-7874.
[61]JUNG S,KEUPER M.Spectral distribution aware image gene-ration[C]//Proceedings of the AAAIConference on Artificial Intelligence.2021:1734-1742.
[62]LORENZ P,HARDER P,STRABEL D,et al.Detecting auto-attack perturbations in the frequency domain[J].arXiv:2111.08785,2021.
[63]CROCE F,HEIN M.Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks[C]//International Conference on Machine Learning.PMLR,2020:2206-2216.
[64]MAO X,CHEN Y,LI Y,et al.Learning to characterize adversarial subspaces[C]//ICASSP 2020－2020 IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP).IEEE,2020:2438-2442.
[65]ZHANG C,YANG Z,YE Z.Detecting Adversarial Perturba-tions with Salieny[C]//Proceedings of the 6th International Conference on Information Technology:IoT and Smart City.2018:25-30.
[66]PRAKASH A,MORAN N,GARBER S,et al.Deflecting adversarial attacks with pixel deflection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:8571-8580.
[67]SELVARAJU R R,COGSWELL M,DAS A,et al.Grad-cam:Visual explanations from deep networks via gradient-based localization[C]//Proceedings of the IEEE International Confe-rence on Computer Vision.2017:618-626.
[68]WANG S,GONG Y.Adversarial example detection based on saliency map features[J].Applied Intelligence,2022(6):6262-6275.
[69]VAN DEN OORD A,KALCHBRENNER N,ESPEHOLT L,et al.Conditional image generation with pixelcnn decoders[J].ar-Xiv:1606.05328,2016.
[70]SONG Y,KIM T,NOWOZIN S,et al.Pixeldefend:Leveraging generative models to understand and defend against adversarial examples[J].arXiv:1710.10766,2017.
[71]KIANI S,AWAN S,LAN C,et al.Two souls in an adversarial image:Towards universal adversarial example detection using multi-view inconsistency[C]//Proceedings of the 37th Annual Computer Security Applications Conference.2021:31-44.
[72]WANG H,MILLER D J,KESIDIS G.Anomaly detection of adversarial examples using class-conditional generative adversarial networks[J].Computers & Security,2023,124:102956.
[73]FREITAS S,CHEN S T,WANG Z J,et al.Unmask:Adversa-rial detection and defense through robust feature alignment[C]//2020 IEEE International Conference on Big Data(Big Data).IEEE,2020:1081-1088.
[74]GONG Y,WANG S,JIANG X,et al.Adversarial example detection using semantic graph matching[J].Applied Soft Computing,2023,141:110317.
[75]TAO G H,MA S Q,LIU Y Q,et al.Attacks meet interpretability:Attribute-steered detection of adversarial samples[J].arXiv:1810.11580,2018.
[76]QIU Y,LENG J,GUO C,et al.Adversarial defense throughnetwork profiling based path extraction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:4777-4786.
[77]NWAIGWE D,CARBONI L,MERMILLOD M,et al.Graph-based methods coupled with specific distributional distances for adversarial attack detection[J].Neural Networks,2024,169:11-19.
[78]DENG L.The mnist database of handwritten digit images formachine learning research[J].IEEE Signal Processing Magazine,2012,29(6):141-142.
[79]KRIZHEVSKY A,HINTON G.Learning multiple layers of features from tiny images[D].Toronto:University of Toronto,2009.
[80]DENG J,DONG W,SOCHER R,et al.ImageNet:A large-scale hierarchical image database[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2009:248-255.
[81]LE Y,YANG X.Tiny imagenet visual recognition challenge[J].CS 231N,2015,7(7):3.
[82]NETZER Y,WANG T,COATES A,et al.Reading digits in na-tural images with unsupervised feature learning[C]//NIPS Workshop on Deep Lear-ning and Unsupervised Feature Lear-ning.2011:7.
[83]NILSBACK M E,ZISSERMAN A.Automated flower classification over a large number of classes[C]//2008 Sixth Indian Conference on Computer Vision,Graphics & Image Processing.IEEE,2008:722-729.
[84]LI F F,FERGUS R,PERONA P.Learning generative visual models from few training examples:An incremental bayesian approach tested on 101 object categories[C]//2004 Conference on Computer Vision and Pattern Recognition.IEEE,2004:178.
[85]GRIFFIN G,HOLUB A,PERONA P.Caltech-256 object category dataset[R].Pasadena:Technical Report 7694,California Institute of Technology,2007.
[86]PINTO N,STONE Z,ZICKLER T,et al.Scaling up biologically-inspired computer vision:A case study in unconstrained face re-cognition on facebook[C]//CVPR 2011.IEEE,2011:35-42.
[87]CUKIERSKI W.Dogs vs.cats,2013[J/OL].https://kaggle.com/competitions/dogs-vs-cats.
[88]GUVENIR H A,ACAR B,DEMIROZ G,et al.A supervised machine learning algorithm for arrhythmia analysis[C]//Computers in Cardiology 1997.IEEE,1997:433-436.
[89]AEBERHARD S,COOMANS D,DE VEL O.Comparative ana-lysis of statistical pattern recognition methods in high dimensional settings[J].Pattern Recognition,1994,27(8):1065-1077.
[90]XIAO H,RASUL K,VOLLGRAF R.Fashion-mnist:a novelimage dataset for benchmarking machine learning algorithms[J].arXiv:1708.07747,2017.
[91]ILYAS A,SANTURKAR S,TSIPRAS D,et al.Adversarial examples are not bugs,they are features[J].arXiv:1905.02175,2019.
[92]ZHU X,WANG H,FEI H,et al.Face forgery detection by 3d decomposition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:2929-2939.
[93]GE Y,XIAO Y,XU Z,et al.Contributions of shape,texture,and color in visual recognition[C]//European Conference on Computer Vision.Cham:Springer Nature Switzerland,2022:369-386.
[94]ZHEN X,MENG Z,CHAKRABORTY R,et al.On the versatile uses of partial distance correlation in deep learning[C]//European Conference on Computer Vision.Cham:Springer Nature Switzerland,2022:327-346.
[95]RADFORD A,KIM J W,HALLACY C,et al.Learning transferable visual models from natural language supervision[C]//International Conference on Machine Learning.PMLR,2021:8748-8763.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发