基于注意力的多尺度蒸馏异常检测

doi:10.11896/jsjkx.230300223

摘要/Abstract

摘要： 基于知识蒸馏的异常检测方法中,教师网络远大于学生网络,使得所得特征表示在同一位置对应图像的感受野不同。为解决此问题,可使学生网络与教师网络结构相同。然而,学生与教师网络完全相同,使得在测试阶段,对于异常样本,教师网络与学生网络特征表示差异过小而影响异常检测的性能。为解决该问题,提出了基于高效通道注意力模块的多尺度知识蒸馏异常检测方法(ECA Based Multi-Scale Knowledge Distillation Anomaly Detection,ECA-MSKDAD),并结合数据增强操作提出了相对距离损失函数。使用经过预训练的网络作为教师网络,同时使用与教师网络结构相同的网络作为学生网络。在训练阶段,对训练样本采取数据增强操作以扩充训练集的规模,并在学生网络中引入高效通道注意力(Efficient Channel Attention,ECA)模块,以增加教师网络和学生网络之间的差异,增大异常数据的重构误差,进而提高模型的检测性能。此外,利用相对距离损失函数,将数据间关系从教师网络传递到学生网络,对学生网络的网络参数进行优化。在MVTec AD进行实验,与9种相关方法比较,所提方法在异常检测与异常定位上均取得更优的性能。

关键词: 深度学习, 异常检测, 异常定位, 知识蒸馏, 注意力机制

Abstract: In the anomaly detection method based on knowledge distillation,the teacher network is much larger than the student network,so that the obtained feature representation has different visual fields corresponding to the image at the same position.In order to solve this problem,the structure of student network and teacher network can be the same.However,However,in the testing phase,the same student network and teacher network will lead to too small difference in their feature representation,which will affect the performance of anomaly detection.In order to solve this problem,ECA based multi-scale knowledge distillation anomaly detection(ECA-MSKDAD) is proposed,and a relative distance loss function is proposed based on data enhancement operation.The pre-trained network is used as the teacher network,and the network with the same network structure as the teacher network is used as the student network.In the training stage,the data enhancement operation is adopted for the training samples to expand the scale of the training set,and the efficient channel attention(ECA) module is introduced into the student network to increase the difference between the teacher network and the student network,increase the reconstruction error of the abnormal data and improve the detection performance of the model.In addition,the relative distance loss function is used to transfer the relationship between data from the teacher network to the student network,and the network parameters of the student network are optimized.Experiments on MVTec AD show that compared with nine related methods,the proposed method achieves better performance in anomaly detection and anomaly localization.

Key words: Deep learning, Anomaly detection, Abnormal location, Knowledge distillation, Attention mechanism

中图分类号:

TP391.4

乔虹, 邢红杰. 基于注意力的多尺度蒸馏异常检测[J]. 计算机科学, 2024, 51(6A): 230300223-11. https://doi.org/10.11896/jsjkx.230300223

QIAO Hong, XING Hongjie. Attention-based Multi-scale Distillation Anomaly Detection[J]. Computer Science, 2024, 51(6A): 230300223-11. https://doi.org/10.11896/jsjkx.230300223

参考文献

[1]KHAN S S,MADDEN M G.A Survey of Recent Trends in One Class Classification[C]//Irish Conference on Artificial Intelligence and Cognitive Science.Springer,Berlin,Heidelberg,2009:188-197.
[2]AHMED M,MAHMOOD A N,HU J.A Survey of NetworkAnomaly Detection Techniques[J].Journal of Network and Computer Applications,2016,60:19-31.
[3]BERGMANN P,FAUSER M,SATTLEGGER D,et al.MVTec AD－A Comprehensive Real-world Dataset for Unsupervised Anomaly Detection[C]//Proceedings of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition.2019:9592-9600.
[4]YAO Q,XIAO L,LIU P,et al.Label-free Segmentation of Co-vid-19 Lesions in Lung CT[J].IEEE Transactions on Medical Imaging,2021,40(10):2808-2819.
[5]RASHID A N M B,AHMED M,SIKOS L F,et al.Anomaly Detection in Cybersecurity Datasets Via Cooperative Co-Evolution-Based Feature Selection[J].ACM Transactions on Management Information Systems(TMIS),2022,13(3):1-39.
[6]ZHOU K,GU Z,LIU W,et al.Multi-Cell Multi-Task Convolutional Neural Networks for Diabetic Retinopathy Grading[C]//2018 40th Annual International Conference of The IEEE Engineering in Medicine and Biology Society(EMBC).IEEE,2018:2724-2727.
[7]DHIMAN G,JUNEJA S,VIRIYASITAVAT W,et al.A Novel Machine-Learning-Based Hybrid CNN Model for Tumor Identification in Medical Image Processing[J].Sustainability,2022,14(3):1447.
[8]SCHÖLKOPF B,WILLIAMSON R C,SMOLA A,et al.Support Vector Method for Novelty Detection[J].Advances in Neural Information Processing Systems,1999,12.
[9]TAX D M J,DUIN R P W.Support Vector Data Description[J].Machine Learning,2004,54(1):45-66.
[10]ZONG B,SONG Q,MIN M R,et al.Deep Autoencoding Gauss-ian Mixture Model for Unsupervised Anomaly Detection[C]//International Conference on Learning Representations.2018.
[11]AKCAY S,ATAPOUR-ABARGHOUEI A,BRECKON T P.GANomaly:Semi-Supervised AnomalyDetection Via AdversarialTraining[C]//Computer Vision-ACCV 2018:14th Asian Conference on Computer Vision.Springer International Publishing,2019:622-637.
[12]BERGMANN P,LWE S,FAUSER M,et al.Im-proving Unsupervised Defect Segmentation by Applying Structural Similarity to Autoencoders[C]//14th International Conference on Computer Vision Theory and Applications.2019.
[13]HINTON G,VINYALS O,DEAN J.Distilling the Knowledgein a Neural Network[J].Computer Science,2015,14(7):38-39.
[14]BERGMANN P,FAUSER M,SATTLEGGER D,et al.Unin-formed Students:Student-Teacher Anomaly Detection with Discriminative Latent Embeddings[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:4183-4192.
[15]SALEHI M,SADJADI N,BASELIZADEH S,et al.Multiresolution Knowledge Distillation for Anomaly Detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:14902-14912.
[16]WANG Q,WU B,ZHU P,et al.ECA-Net:Efficient Channel Attention for Deep Convolutional Neural Networks[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2020:11531-11539.
[17]HOFFMANN H.Kernel PCA for Novelty Detection[J].Pattern Recognition,2007,40(3):863-874.
[18]TING K M,XU B C,WASHIO T,et al.Isolation Distributional Kernel:A New Tool for Kernel Based Anomaly Detection[C]//Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.2020:198-206.
[19]QU J,DU Q,LI Y,et al.Anomaly Detection in Hyperspectral Imagery Based on Gaussian Mixture Model[J].IEEE Transactions on Geoscience and Remote Sensing,2020,59(11):9504-9517.
[20]LI C,GUO L,GAO H,et al.Similarity-Measured Isolation Fo-rest:Anomaly Detection Method for Machine Monitoring Data[J].IEEE Transactions on Instrumentation and Measurement,2021,70:1-12.
[21]PENG X,LI H,YUAN F,et al.An Extreme Learning Machine for Unsupervised Online Anomaly Detection in Multivariate Time Series[J].Neurocomputing,2022,501:596-608.
[22]PANG J,PU X,LI C.A Hybrid Algorithm Incorporating Vector Quantization and One-Class Support Vector Machine for Industrial Anomaly Detection[J].IEEE Transactions on Industrial Informatics,2022,18(12):8786-8796.
[23]RUFF L,VANDERMEULEN R,GOERNITZ N,et al.DeepOne-Class Classification[C]//International Conference on Machine Learning.PMLR,2018:4393-4402.
[24]PANG G,SHEN C,CAO L,et al.Deep Learning for Anomaly Detection:A Review[J].ACM Computing Surveys(CSUR),2021,54(2):1-38.
[25]KOPPIKAR U,SUJATHA C,PATIL P,et al.Real-WorldAnomaly Detection Using Deep Learning[C]//Proceedings of 3rd Intelligent Computing and Communication(ICICC 2019).Springer Singapore,2020:333-342.
[26]LIZNERSKI P,RUFF L,VANDERMEULEN R A,et al.Explainable Deep One-Class Classification[C]//International Conference on Learning Representations.2021.
[27]ULGER F,YUKSEL S E,YILMAZ A.Anomaly Detection for Solder Joints Using β-VAE[J].IEEE Transactions on Components,Packaging and Manufacturing Technology,2021,11(12):2214-2221.
[28]CHEN L,LI Y,DENG X,et al.Dual Auto-Encoder Gan-Based Anomaly Detection for Industrial Control System[J].Applied Sciences,2022,12(10):4986.
[29]CHENG D,FAN Y,FANG S,et al.ResNet-AE for Radar Signal Anomaly Detection[J].Sensors,2022,22(16):6249.
[30]LI S,LIU F,JIAO L.Self-Training Multi-Sequence Learningwith Transformer for Weakly Supervised Video Anomaly Detection[C]//Proceedings of The AAAI Conference on Artificial Intelligence.2022:1395-1403.
[31]YAN S,SHAO H,XIAO Y,et al.Hybrid Robust Convolutional Autoencoder for Unsupervised Anomaly Detection of Machine Tools Under Noises[J].Robotics And Computer-Integrated Manufacturing,2023,79:102441.
[32]TAO X,GONG X,ZHANG X, et al. Deep Learning for Unsupervised Anomaly Localization in Industrial Images:A Survey[J].IEEE Transactions on Instrumentation and Measurement,2022,71:1-21.
[33]WANG G,HAN S,DING E,et al.Student-Teacher FeaturePyramid Matching for Anomaly Detection[C]//British Machine Vision Conference.2021.
[34]HU J,SHEN L,SUN G.Squeeze-and-Excitation Networks[C]//Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition.2018:7132-7141.
[35]SHORTEN C,KHOSHGOFTAAR T M.A Survey on ImageData Augmentation for Deep Learning[J].Journal of Big Data,2019,6(1):1-48.
[36]KAUR P,KHEHRA B S,MAVI E B S.Data Augmentation for Object Detection:A Review[C]//2021 IEEE International Midwest Symposium on Circuits and Systems(MWSCAS).IEEE,2021:537-543.
[37]GOU J,YU B,MAYBANK S J,et al.Knowledge Distillation:A Survey[J].International Journal of Computer Vision,2021,129:1789-1819.
[38]GOLAN I,EL-YANIV R.Deep Anomaly Detection Using Geometric Transformations[J].Advances in Neural Information Processing Systems,2018,31.
[39]YE F,HUANG C,CAO J,et al.Attribute Restoration Frame-work for Anomaly Detection[J].IEEE Transactions on Multimedia,2020,24:116-127.
[40]SCHLEGL T,SEEBÖCK P,WALDSTEIN S M,et al.Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery[C]//25th International Conference Information Processing in Medical Imaging(IPMI 2017).Cham:Springer International Publishing,2017:146-157.
[41]NAPOLETANO P,PICCOLI F,SCHETTINI R.Anomaly Detection in Nanofibrous Materials By CNN-Based Self-Similarity[J].Sensors,2018,18(1):209.
[42]LI C L,SOHN K,YOON J,et al.CutPaste:Self-SupervisedLearning for Anomaly Detection and Localization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:9664-9674.
[43]YI J,YOON S.Patch SVDD:Patch-Level SVDD for AnomalyDetection and Segmentation[C]//Proceedings of the Asian Conference on Computer Vision.2020.
[44]DEFARD T,SETKOV A,LOESCH A,et al.PaDim:A Patch Distribution Modeling Framework for Anomaly Detection and Localization[C]//Pattern Recognition.ICPR International Workshops and Challenges.Cham:Springer International Publishing,2021:475-489.
[45]COHEN N,HOSHEN Y.Sub-Image Anomaly Detection withDeep Pyramid Correspondences[J].arXiv:2005.02357,2020.
[46]PIRNAY J,CHAI K.Inpainting transformer for anomaly detection[C]//Image Analysis and Processing(ICIAP 2022).Cham:Springer International Publishing,2022:394-406.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed