Computer Science

Computer Science ›› 2025, Vol. 52 ›› Issue (1): 345-361.doi: 10.11896/jsjkx.240300080

• Information Security • Previous Articles     Next Articles

Adversarial Sample Detection in Computer Vision:A Survey

ZHANG Xin1, ZHANG Han1,2, NIU Manyu1, JI Lixia1,3   

  1. 1 School of Cyber Science and Engineering,Zhengzhou University,Zhengzhou 450001,China
    2 Intelligent Policing Key Laboratory of Sichuan Province,Luzhou,Sichuan 646000,China
    3 School of Computer Science,Sichuan University,Chengdu 610065,China
  • Received:2024-03-12 Revised:2024-08-10 Online:2025-01-15 Published:2025-01-09
  • About author:ZHANG Xin,born in 1999,postgra-duate.His main research interests include image processing and information security.
    JI Lixia,born in 1979,associate professor.Her main research interests include multi-modal learning and information security.
  • Supported by:
    Young Scientists Fund of the National Natural Science Foundation of China(62302458),Major Science and Technology Project of Henan Province(231100210200),Key Scientific Research Project of Universities in Henan Province(24A520047) and 2024 Open Subject Project of Sichuan Provincial Key Laboratory of Intelligent Policing(ZNJW2024KFQN005).

PDF (PC)

137

Abstract: With the increase in data volume and improvement in hardware performance,deep learning(DL) has made significant progress in the field of computer vision.However,deep learning models are vulnerable to adversarial samples,causing significant changes in the output.As an effective defense method,adversarial sample detection can prevent adversarial samples from affecting the deep learning model without changing the model structure.This paper organizes the research work on adversarial example detection in recent years,analyzes the relationship between adversarial example detection and training data,classifies them according to the characteristics used in the detection method,and systematically and comprehensively introduces adversarial sample detection methods in the field of computer vision.Then,some detection methods that combine cross-domain technologies are introduced in detail,and the experimental configurations for training and evaluating detection methods are statistically analyzed.Finally,some technologies that are expected to be applied to adversarial sample detection are summarized,and future research challenges and development directions are prospected.

Key words: Deep learning, Adversarial sample attacks, Adversarial sample detection, AI security, Image classification

CLC Number: 

  • TP391
[1]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenetclassification with deep convolutional neural networks[C]//Annual Conference on Neural Information Processing Systems.2012.
[2]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[3]HE K,ZHANG X,REN S,et al.Deep residual learning forimage recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[4]REN S Q,HE K M,GIRSHICK R,et al.Faster r-cnn:Towards real-time object detection with region proposal networks[J].arXiv:1506.01497,2015.
[5]SZEGEDY C,ZAREMBA W,SUTSKEVER I,et al.Intriguing properties of neural networks[J].arXiv:1312.6199,2013.
[6]GOODFELLOW I J,SHLENS J,SZEGEDY C.Explaining and harnessing adversarial examples[J].arXiv:1412.6572,2014.
[7]ZHANG J,HUANG Y,WU W,et al.Transferable adversarial attacks on vision transformers with token gradient regularization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2023:16415-16424.
[8]CARLINI N,WAGNER D.Towards evaluating the robustness of neural networks[C]//2017 IEEE Symposium on Security and Privacy(SP).IEEE,2017:39-57.
[9]XU H,LI Y,LIU X,et al.Yet meta learning can adapt fast,it can also break easily[C]//Proceedings of the 2021 SIAM International Conference on Data Mining(SDM).Society for Industrial and Applied Mathematics,2021:540-548.
[10]KARIMI M P,AMIRKHANI A,SHOKOUHI S B.Robust object detection against adversarial perturbations withgabor filter[C]//2021 29th Iranian Conference on Electrical Engineering(ICEE).IEEE,2021:187-192.
[11]WANG L,YOON K J.Psat-gan:Efficient adversarial attacksagainst holistic scene understanding[J].IEEE Transactions on Image Processing,2021,30:7541-7553.
[12]ABDULLAH H,RAHMAN M S,GARCIA W,et al.Hear “no evil”,see“kenansville”:Efficient and transferable black-box attacks on speech recognition and voice identification systems[C]//2021 IEEE Symposium on Security and Privacy(SP).IEEE,2021:712-729.
[13]CHEN G,CHENB S,FAN L,et al.Who is real bob? adversarial attacks on speaker recognition systems[C]//2021 IEEE Symposium on Security and Privacy(SP).IEEE,2021:694-711.
[14]HU Z,HUANG S,ZHU X,et al.Adversarial texture for fooling person detectors in the physical world[C]//Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition.2022:13307-13316.
[15]WANG D,JIANG T,SUN J,et al.Fca:Learning a 3d full-cove-rage vehicle camouflage for multi-view physical adversarial attack[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2022:2414-2422.
[16]LIU J,LAU C P,SOURI H,et al.Mutual adversarial training:Learning together is better than going alone[J].IEEE Transactions on Information Forensics and Security,2022,17:2364-2377.
[17]HARDER P,PFREUNDT F J,KEUPER M,et al.Spectral-defense:Detecting adversarial attacks on cnns in the fourier domain[C]//2021 International Joint Conference on Neural Networks(IJCNN).IEEE,2021:1-8.
[18]DRENKOW N,FENDLEY N,BURLINA P.Attack agnosticdetection of adversarial examples via random subspace analysis[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.2022:472-482.
[19]LIU Z,CAO C,TAO F,et al.From Spatial to Spectral Domain,a New Perspective for Detecting Adversarial Examples[J/OL].Security and Communication Networks,2022.[2022-09-05].https://doi.org/10.1155/2022.5501035.
[20]NADERI H,NOORBAKHSH K,ETEMADI A,et al.Lpf-defense:3d adversarial defense based on frequency analysis[J].Plos one,2023,18(2):e0271388.
[21]ZHANG T,YANG K W,WEI J H,et al.A review of adversarial sample detection and defense technology for image data [J].Computer Research and Development,2022,59(6):1315-1328.
[22]ALDAHDOOH A,HAMIDOUCHE W,FEZZA S A,et al.Adversarial example detection for DNN models:A review and experimental comparison[J].Artificial Intelligence Review,2022,55(6):4403-4462.
[23]ZHOU T,GAN R,XU D W,et al.A review of image adversarial example detection [J/OL].Journal of Software:1-35.[2023-10-23].https://doi.org/10.13328/j.cnki.jos.006834.
[24]MADRY A,MAKELOV A,SCHMIDT L,et al.Towards deep learning models resistant to adversarial attacks[J].arXiv:1706.06083,2017.
[25]MOOSAVI-DEZFOOLI S M,FAWZI A,FROSSARD P.Deepfool:a simple and accurate method to fool deep neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2574-2582.
[26]XIAO C,LI B,ZHU J Y,et al.Generating adversarial examples with adversarial networks[J].arXiv:1801.02610,2018.
[27]PAPERNOT N,MCDANIEL P,JHA S,et al.The limitations of deep learning in adversarial settings[C]//2016 IEEE European Symposium on Security and Privacy(EuroS&P).IEEE,2016:372-387.
[28]SU J,VARGAS D V,SAKURAI K.One pixel attack for fooling deep neural networks[J].IEEE Transactions on Evolutionary Computation,2019,23(5):828-841.
[29]CHEN P Y,ZHANG H,SHARMA Y,et al.Zoo:Zeroth order optimization based black-box attacks to deep neural networks without training substitute models[C]//Proceedings of the 10th ACM workshop on Artificial Intelligence and Security.2017:15-26.
[30]MAO X,CHEN Y,LI Y,et al.Gap++:Learning to generate target-conditioned adversarial examples[J].arXiv:2006.05097,2020.
[31]CARRARA F,FALCHI F,CALDELLI R,et al.Detecting adversarial example attacks to deep neural networks[C]//Proceedings of the 15th International Workshop on Content-based Multimedia Indexing.2017:1-7.
[32]SHI C,HOLTZ C,MISHNE G.Online adversarial purification based on self-supervision[J].arXiv:2101.09387,2021.
[33]XU W,EVANS D,QI Y.Feature squeezing:Detecting adversa-rial examples in deep neural networks[J].arXiv:1704.01155,2017.
[34]MA S,LIU Y,TAO G,et al.Nic:Detecting adversa-rial samples with neural network invariant checking[C]//26th Annual Network and Distributed System Security Sympo-sium(NDSS 2019).Internet Soc,2019.
[35]HU J,SHEN L,SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:7132-7141.
[36]ALDAHDOOH A,HAMIDOUCHE W,DéFORGES O.Revisiting model’s uncertainty and confidences for adversarial example detection[J].Applied Intelligence,2023,53(1):509-531.
[37]GEIFMAN Y,EL-YANIV R.Selectivenet:A deep neural network with an integrated reject option[C]//International Confe-rence on Machine Learning.PMLR,2019:2151-2159.
[38]HENDRYCKS D,GIMPEL K.A baseline for detecting misclassified and out-of-distribution examples in neural networks[J].arXiv:1610.02136,2016.
[39]KULLBACK S,LEIBLER R A.On information and sufficiency[J].The Annals of Mathematical Statistics,1951,22(1):79-86.
[40]WIYATNO RR,XU A,DIA O,et al.Adversarial examples in modern machine learning:A review[J].arXiv:1911.05268,2019.
[41]ZHENG Z H,HONG P Y.Robust detection of adversarial attacks by modeling the intrinsic properties of deep neural networks[J].Neural Information Processing Systems,2018,31:7924-7933.
[42]PAPERNOT N,MCDANIEL P.Deep k-nearest neighbors:Towards confident,interpretable and robust deep learning[J].ar-Xiv:1803.04765,2018.
[43]ABUSNAINA A,WU Y,ARORA S,et al.Adversarial example detection using latent neighborhood graph[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2021:7687-7696.
[44]LIANG B,LI H,SU M,et al.Detecting adversarial image examples in deep neural networks with adaptive noise reduction[J].IEEE Transactions on Dependable and Secure Computing,2018,18(1):72-85.
[45]WANG Y,LI X,YANG L,et al.ADDITION:Detecting Adversarial Examples With Image-Dependent Noise Reduction[J].IEEE Transactions on Dependable and Secure Computing,2023,21(3):1139-1154.
[46]MOORTHY A K,BOVIK A C.Blind image quality assessment:From natural scene statistics to perceptual quality[J].IEEE Transactions on Image Processing,2011,20(12):3350-3364.
[47]KHERCHOUCHE A,FEZZA S A,HAMIDOUCHE W,et al.Detection of adversarial examples in deep neural networks with natural scene statistics[C]//2020 International Joint Conference on Neural Networks(IJCNN).IEEE,2020:1-7.
[48]MITTAL A,MOORTHY A K,BOVIK A C.No-referenceimage quality assessment in the spatial domain[J].IEEE Transactions on image processing,2012,21(12):4695-4708.
[49]GONG Z,WANG W.Adversarial and clean data are not twins[C]//Proceedings of the Sixth International Workshop on Exploiting Artificial Intelligence Techniques for Data Management.2023:1-5.
[50]HOSSEINI H,CHEN Y,KANNAN S,et al.Blocking transfera-bility of adversarial examples in black-box learning systems[J].arXiv:1703.04318,2017.
[51]LUST J,CONDURACHE A P.Gran:An efficient gradient-norm based detector for adversarial and misclassified examples[J].arXiv:2004.09179,2020.
[52]COHEN G,SAPIRO G,GIRYES R.Detecting adversarial samples using influence functions and nearest neighbors[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:14453-14462.
[53]MA X,LI B,WANG Y,et al.Characterizing adversarial sub-spaces using local intrinsic dimensionality[J].arXiv:1801.02613,2018.
[54]LORENZ P,KEUPER M,KEUPER J.Unfolding local growth rate estimates for(almost) perfect adversarial detection[J].ar-Xiv:2212.06776,2022.
[55]LEE K,LEE K,LEE H,et al.A simple unified framework for detecting out-of-distribution samples and adversarial attacks[J].arXiv:1807.03888,2018.
[56]CHEN K,CHEN Y,ZHOU H,et al.Adversarial examples detection beyond image space[C]//ICASSP 2021-2021 IEEE International Conference on Acoustics,Speech and Signal Proces-sing(ICASSP).IEEE,2021:3850-3854.
[57]GROSSE K,MANOHARAN P,PAPERNOT N,et al.On the(statistical) detection of adversarial examples[J].arXiv:1702.06280,2017.
[58]GAO R,LIU F,ZHANG J,et al.Maximum mean discrepancy test is aware of adversarial attacks[C]//International Confe-rence on Machine Learning.PMLR,2021:3564-3575.
[59]FEINMAN R,CURTIN RR,SHINTRE S,et al.Detecting adversarial samples from artifacts[J].arXiv:1703.00410,2017.
[60]DONG C,KUMAR A,LIU E.Think twice before detectinggan-generated fake images from their spectral domain imprints[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:7865-7874.
[61]JUNG S,KEUPER M.Spectral distribution aware image gene-ration[C]//Proceedings of the AAAIConference on Artificial Intelligence.2021:1734-1742.
[62]LORENZ P,HARDER P,STRABEL D,et al.Detecting auto-attack perturbations in the frequency domain[J].arXiv:2111.08785,2021.
[63]CROCE F,HEIN M.Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks[C]//International Conference on Machine Learning.PMLR,2020:2206-2216.
[64]MAO X,CHEN Y,LI Y,et al.Learning to characterize adversarial subspaces[C]//ICASSP 2020－2020 IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP).IEEE,2020:2438-2442.
[65]ZHANG C,YANG Z,YE Z.Detecting Adversarial Perturba-tions with Salieny[C]//Proceedings of the 6th International Conference on Information Technology:IoT and Smart City.2018:25-30.
[66]PRAKASH A,MORAN N,GARBER S,et al.Deflecting adversarial attacks with pixel deflection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:8571-8580.
[67]SELVARAJU R R,COGSWELL M,DAS A,et al.Grad-cam:Visual explanations from deep networks via gradient-based localization[C]//Proceedings of the IEEE International Confe-rence on Computer Vision.2017:618-626.
[68]WANG S,GONG Y.Adversarial example detection based on saliency map features[J].Applied Intelligence,2022(6):6262-6275.
[69]VAN DEN OORD A,KALCHBRENNER N,ESPEHOLT L,et al.Conditional image generation with pixelcnn decoders[J].ar-Xiv:1606.05328,2016.
[70]SONG Y,KIM T,NOWOZIN S,et al.Pixeldefend:Leveraging generative models to understand and defend against adversarial examples[J].arXiv:1710.10766,2017.
[71]KIANI S,AWAN S,LAN C,et al.Two souls in an adversarial image:Towards universal adversarial example detection using multi-view inconsistency[C]//Proceedings of the 37th Annual Computer Security Applications Conference.2021:31-44.
[72]WANG H,MILLER D J,KESIDIS G.Anomaly detection of adversarial examples using class-conditional generative adversarial networks[J].Computers & Security,2023,124:102956.
[73]FREITAS S,CHEN S T,WANG Z J,et al.Unmask:Adversa-rial detection and defense through robust feature alignment[C]//2020 IEEE International Conference on Big Data(Big Data).IEEE,2020:1081-1088.
[74]GONG Y,WANG S,JIANG X,et al.Adversarial example detection using semantic graph matching[J].Applied Soft Computing,2023,141:110317.
[75]TAO G H,MA S Q,LIU Y Q,et al.Attacks meet interpretability:Attribute-steered detection of adversarial samples[J].arXiv:1810.11580,2018.
[76]QIU Y,LENG J,GUO C,et al.Adversarial defense throughnetwork profiling based path extraction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:4777-4786.
[77]NWAIGWE D,CARBONI L,MERMILLOD M,et al.Graph-based methods coupled with specific distributional distances for adversarial attack detection[J].Neural Networks,2024,169:11-19.
[78]DENG L.The mnist database of handwritten digit images formachine learning research[J].IEEE Signal Processing Magazine,2012,29(6):141-142.
[79]KRIZHEVSKY A,HINTON G.Learning multiple layers of features from tiny images[D].Toronto:University of Toronto,2009.
[80]DENG J,DONG W,SOCHER R,et al.ImageNet:A large-scale hierarchical image database[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition.IEEE,2009:248-255.
[81]LE Y,YANG X.Tiny imagenet visual recognition challenge[J].CS 231N,2015,7(7):3.
[82]NETZER Y,WANG T,COATES A,et al.Reading digits in na-tural images with unsupervised feature learning[C]//NIPS Workshop on Deep Lear-ning and Unsupervised Feature Lear-ning.2011:7.
[83]NILSBACK M E,ZISSERMAN A.Automated flower classification over a large number of classes[C]//2008 Sixth Indian Conference on Computer Vision,Graphics & Image Processing.IEEE,2008:722-729.
[84]LI F F,FERGUS R,PERONA P.Learning generative visual models from few training examples:An incremental bayesian approach tested on 101 object categories[C]//2004 Conference on Computer Vision and Pattern Recognition.IEEE,2004:178.
[85]GRIFFIN G,HOLUB A,PERONA P.Caltech-256 object category dataset[R].Pasadena:Technical Report 7694,California Institute of Technology,2007.
[86]PINTO N,STONE Z,ZICKLER T,et al.Scaling up biologically-inspired computer vision:A case study in unconstrained face re-cognition on facebook[C]//CVPR 2011.IEEE,2011:35-42.
[87]CUKIERSKI W.Dogs vs.cats,2013[J/OL].https://kaggle.com/competitions/dogs-vs-cats.
[88]GUVENIR H A,ACAR B,DEMIROZ G,et al.A supervised machine learning algorithm for arrhythmia analysis[C]//Computers in Cardiology 1997.IEEE,1997:433-436.
[89]AEBERHARD S,COOMANS D,DE VEL O.Comparative ana-lysis of statistical pattern recognition methods in high dimensional settings[J].Pattern Recognition,1994,27(8):1065-1077.
[90]XIAO H,RASUL K,VOLLGRAF R.Fashion-mnist:a novelimage dataset for benchmarking machine learning algorithms[J].arXiv:1708.07747,2017.
[91]ILYAS A,SANTURKAR S,TSIPRAS D,et al.Adversarial examples are not bugs,they are features[J].arXiv:1905.02175,2019.
[92]ZHU X,WANG H,FEI H,et al.Face forgery detection by 3d decomposition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:2929-2939.
[93]GE Y,XIAO Y,XU Z,et al.Contributions of shape,texture,and color in visual recognition[C]//European Conference on Computer Vision.Cham:Springer Nature Switzerland,2022:369-386.
[94]ZHEN X,MENG Z,CHAKRABORTY R,et al.On the versatile uses of partial distance correlation in deep learning[C]//European Conference on Computer Vision.Cham:Springer Nature Switzerland,2022:327-346.
[95]RADFORD A,KIM J W,HALLACY C,et al.Learning transferable visual models from natural language supervision[C]//International Conference on Machine Learning.PMLR,2021:8748-8763.
[1] ZHANG Yusong, XU Shuai, YAN Xingyu, GUAN Donghai, XU Jianqiu. Survey on Cross-city Human Mobility Prediction [J]. Computer Science, 2025, 52(1): 102-119.
[2] LIU Yuming, DAI Yu, CHEN Gongping. Review of Federated Learning in Medical Image Processing [J]. Computer Science, 2025, 52(1): 183-193.
[3] LI Yujie, MA Zihang, WANG Yifu, WANG Xinghe, TAN Benying. Survey of Vision Transformers(ViT) [J]. Computer Science, 2025, 52(1): 194-209.
[4] ZHU Xiaoyan, WANG Wenge, WANG Jiayin, ZHANG Xuanping. Just-In-Time Software Defect Prediction Approach Based on Fine-grained Code Representationand Feature Fusion [J]. Computer Science, 2025, 52(1): 242-249.
[5] ZHANG Jian, LI Hui, ZHANG Shengming, WU Jie, PENG Ying. Review of Pre-training Methods for Visually-rich Document Understanding [J]. Computer Science, 2025, 52(1): 259-276.
[6] LI Yahe, XIE Zhipeng. Active Learning Based on Maximum Influence Set [J]. Computer Science, 2025, 52(1): 289-297.
[7] SU Chaoran, ZHANG Dalong, HUANG Yong, DONG An. RF Fingerprint Recognition Based on SE Attention Multi-source Domain Adversarial Network [J]. Computer Science, 2025, 52(1): 412-419.
[8] DU Yu, YU Zishu, PENG Xiaohui, XU Zhiwei. Padding Load:Load Reducing Cluster Resource Waste and Deep Learning Training Costs [J]. Computer Science, 2024, 51(9): 71-79.
[9] XU Jinlong, GUI Zhonghua, LI Jia'nan, LI Yingying, HAN Lin. FP8 Quantization and Inference Memory Optimization Based on MLIR [J]. Computer Science, 2024, 51(9): 112-120.
[10] SUN Yumo, LI Xinhang, ZHAO Wenjie, ZHU Li, LIANG Ya’nan. Driving Towards Intelligent Future:The Application of Deep Learning in Rail Transit Innovation [J]. Computer Science, 2024, 51(8): 1-10.
[11] KONG Lingchao, LIU Guozhu. Review of Outlier Detection Algorithms [J]. Computer Science, 2024, 51(8): 20-33.
[12] TANG Ruiqi, XIAO Ting, CHI Ziqiu, WANG Zhe. Few-shot Image Classification Based on Pseudo-label Dependence Enhancement and NoiseInterferenceReduction [J]. Computer Science, 2024, 51(8): 152-159.
[13] ZHANG Rui, WANG Ziqi, LI Yang, WANG Jiabao, CHEN Yao. Task-aware Few-shot SAR Image Classification Method Based on Multi-scale Attention Mechanism [J]. Computer Science, 2024, 51(8): 160-167.
[14] XIAO Xiao, BAI Zhengyao, LI Zekai, LIU Xuheng, DU Jiajin. Parallel Multi-scale with Attention Mechanism for Point Cloud Upsampling [J]. Computer Science, 2024, 51(8): 183-191.
[15] ZHANG Junsan, CHENG Ming, SHEN Xiuxuan, LIU Yuxue, WANG Leiquan. Diversified Label Matrix Based Medical Image Report Generation [J]. Computer Science, 2024, 51(8): 200-208.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.