Computer Science

Computer Science ›› 2024, Vol. 51 ›› Issue (6A): 230300223-11.doi: 10.11896/jsjkx.230300223

• Big Data & Data Science • Previous Articles     Next Articles

Attention-based Multi-scale Distillation Anomaly Detection

QIAO Hong, XING Hongjie   

  1. Hebei Key Laboratory of Machine Learning and Computational Intelligence,College of Mathematics and Information Science,Hebei University,Baoding,Hebei 071002,China
  • Published:2024-06-06
  • About author:QIAO Hong,born in 1998,postgra-duate.Her main research interests include anomaly detection,knowledge distillation and deep learning.
    XING Hongjie,born in 1976,Ph.D,professor,Ph.D supervisor.His main research interests include kernel me-thods,neural networks,novelty detection,and ensemble learning.
  • Supported by:
    National Natural Science Foundation of China(61672205),Natural Science Foundation of Hebei Province(F2017201020) and High-Level Talents Research Start-Up Project of Hebei University(521100222002).

PDF (PC)

285

Abstract: In the anomaly detection method based on knowledge distillation,the teacher network is much larger than the student network,so that the obtained feature representation has different visual fields corresponding to the image at the same position.In order to solve this problem,the structure of student network and teacher network can be the same.However,However,in the testing phase,the same student network and teacher network will lead to too small difference in their feature representation,which will affect the performance of anomaly detection.In order to solve this problem,ECA based multi-scale knowledge distillation anomaly detection(ECA-MSKDAD) is proposed,and a relative distance loss function is proposed based on data enhancement operation.The pre-trained network is used as the teacher network,and the network with the same network structure as the teacher network is used as the student network.In the training stage,the data enhancement operation is adopted for the training samples to expand the scale of the training set,and the efficient channel attention(ECA) module is introduced into the student network to increase the difference between the teacher network and the student network,increase the reconstruction error of the abnormal data and improve the detection performance of the model.In addition,the relative distance loss function is used to transfer the relationship between data from the teacher network to the student network,and the network parameters of the student network are optimized.Experiments on MVTec AD show that compared with nine related methods,the proposed method achieves better performance in anomaly detection and anomaly localization.

Key words: Deep learning, Anomaly detection, Abnormal location, Knowledge distillation, Attention mechanism

CLC Number: 

  • TP391.4
[1]KHAN S S,MADDEN M G.A Survey of Recent Trends in One Class Classification[C]//Irish Conference on Artificial Intelligence and Cognitive Science.Springer,Berlin,Heidelberg,2009:188-197.
[2]AHMED M,MAHMOOD A N,HU J.A Survey of NetworkAnomaly Detection Techniques[J].Journal of Network and Computer Applications,2016,60:19-31.
[3]BERGMANN P,FAUSER M,SATTLEGGER D,et al.MVTec AD－A Comprehensive Real-world Dataset for Unsupervised Anomaly Detection[C]//Proceedings of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition.2019:9592-9600.
[4]YAO Q,XIAO L,LIU P,et al.Label-free Segmentation of Co-vid-19 Lesions in Lung CT[J].IEEE Transactions on Medical Imaging,2021,40(10):2808-2819.
[5]RASHID A N M B,AHMED M,SIKOS L F,et al.Anomaly Detection in Cybersecurity Datasets Via Cooperative Co-Evolution-Based Feature Selection[J].ACM Transactions on Management Information Systems(TMIS),2022,13(3):1-39.
[6]ZHOU K,GU Z,LIU W,et al.Multi-Cell Multi-Task Convolutional Neural Networks for Diabetic Retinopathy Grading[C]//2018 40th Annual International Conference of The IEEE Engineering in Medicine and Biology Society(EMBC).IEEE,2018:2724-2727.
[7]DHIMAN G,JUNEJA S,VIRIYASITAVAT W,et al.A Novel Machine-Learning-Based Hybrid CNN Model for Tumor Identification in Medical Image Processing[J].Sustainability,2022,14(3):1447.
[8]SCHÖLKOPF B,WILLIAMSON R C,SMOLA A,et al.Support Vector Method for Novelty Detection[J].Advances in Neural Information Processing Systems,1999,12.
[9]TAX D M J,DUIN R P W.Support Vector Data Description[J].Machine Learning,2004,54(1):45-66.
[10]ZONG B,SONG Q,MIN M R,et al.Deep Autoencoding Gauss-ian Mixture Model for Unsupervised Anomaly Detection[C]//International Conference on Learning Representations.2018.
[11]AKCAY S,ATAPOUR-ABARGHOUEI A,BRECKON T P.GANomaly:Semi-Supervised AnomalyDetection Via AdversarialTraining[C]//Computer Vision-ACCV 2018:14th Asian Conference on Computer Vision.Springer International Publishing,2019:622-637.
[12]BERGMANN P,LWE S,FAUSER M,et al.Im-proving Unsupervised Defect Segmentation by Applying Structural Similarity to Autoencoders[C]//14th International Conference on Computer Vision Theory and Applications.2019.
[13]HINTON G,VINYALS O,DEAN J.Distilling the Knowledgein a Neural Network[J].Computer Science,2015,14(7):38-39.
[14]BERGMANN P,FAUSER M,SATTLEGGER D,et al.Unin-formed Students:Student-Teacher Anomaly Detection with Discriminative Latent Embeddings[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2020:4183-4192.
[15]SALEHI M,SADJADI N,BASELIZADEH S,et al.Multiresolution Knowledge Distillation for Anomaly Detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:14902-14912.
[16]WANG Q,WU B,ZHU P,et al.ECA-Net:Efficient Channel Attention for Deep Convolutional Neural Networks[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).2020:11531-11539.
[17]HOFFMANN H.Kernel PCA for Novelty Detection[J].Pattern Recognition,2007,40(3):863-874.
[18]TING K M,XU B C,WASHIO T,et al.Isolation Distributional Kernel:A New Tool for Kernel Based Anomaly Detection[C]//Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.2020:198-206.
[19]QU J,DU Q,LI Y,et al.Anomaly Detection in Hyperspectral Imagery Based on Gaussian Mixture Model[J].IEEE Transactions on Geoscience and Remote Sensing,2020,59(11):9504-9517.
[20]LI C,GUO L,GAO H,et al.Similarity-Measured Isolation Fo-rest:Anomaly Detection Method for Machine Monitoring Data[J].IEEE Transactions on Instrumentation and Measurement,2021,70:1-12.
[21]PENG X,LI H,YUAN F,et al.An Extreme Learning Machine for Unsupervised Online Anomaly Detection in Multivariate Time Series[J].Neurocomputing,2022,501:596-608.
[22]PANG J,PU X,LI C.A Hybrid Algorithm Incorporating Vector Quantization and One-Class Support Vector Machine for Industrial Anomaly Detection[J].IEEE Transactions on Industrial Informatics,2022,18(12):8786-8796.
[23]RUFF L,VANDERMEULEN R,GOERNITZ N,et al.DeepOne-Class Classification[C]//International Conference on Machine Learning.PMLR,2018:4393-4402.
[24]PANG G,SHEN C,CAO L,et al.Deep Learning for Anomaly Detection:A Review[J].ACM Computing Surveys(CSUR),2021,54(2):1-38.
[25]KOPPIKAR U,SUJATHA C,PATIL P,et al.Real-WorldAnomaly Detection Using Deep Learning[C]//Proceedings of 3rd Intelligent Computing and Communication(ICICC 2019).Springer Singapore,2020:333-342.
[26]LIZNERSKI P,RUFF L,VANDERMEULEN R A,et al.Explainable Deep One-Class Classification[C]//International Conference on Learning Representations.2021.
[27]ULGER F,YUKSEL S E,YILMAZ A.Anomaly Detection for Solder Joints Using β-VAE[J].IEEE Transactions on Components,Packaging and Manufacturing Technology,2021,11(12):2214-2221.
[28]CHEN L,LI Y,DENG X,et al.Dual Auto-Encoder Gan-Based Anomaly Detection for Industrial Control System[J].Applied Sciences,2022,12(10):4986.
[29]CHENG D,FAN Y,FANG S,et al.ResNet-AE for Radar Signal Anomaly Detection[J].Sensors,2022,22(16):6249.
[30]LI S,LIU F,JIAO L.Self-Training Multi-Sequence Learningwith Transformer for Weakly Supervised Video Anomaly Detection[C]//Proceedings of The AAAI Conference on Artificial Intelligence.2022:1395-1403.
[31]YAN S,SHAO H,XIAO Y,et al.Hybrid Robust Convolutional Autoencoder for Unsupervised Anomaly Detection of Machine Tools Under Noises[J].Robotics And Computer-Integrated Manufacturing,2023,79:102441.
[32]TAO X,GONG X,ZHANG X, et al. Deep Learning for Unsupervised Anomaly Localization in Industrial Images:A Survey[J].IEEE Transactions on Instrumentation and Measurement,2022,71:1-21.
[33]WANG G,HAN S,DING E,et al.Student-Teacher FeaturePyramid Matching for Anomaly Detection[C]//British Machine Vision Conference.2021.
[34]HU J,SHEN L,SUN G.Squeeze-and-Excitation Networks[C]//Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition.2018:7132-7141.
[35]SHORTEN C,KHOSHGOFTAAR T M.A Survey on ImageData Augmentation for Deep Learning[J].Journal of Big Data,2019,6(1):1-48.
[36]KAUR P,KHEHRA B S,MAVI E B S.Data Augmentation for Object Detection:A Review[C]//2021 IEEE International Midwest Symposium on Circuits and Systems(MWSCAS).IEEE,2021:537-543.
[37]GOU J,YU B,MAYBANK S J,et al.Knowledge Distillation:A Survey[J].International Journal of Computer Vision,2021,129:1789-1819.
[38]GOLAN I,EL-YANIV R.Deep Anomaly Detection Using Geometric Transformations[J].Advances in Neural Information Processing Systems,2018,31.
[39]YE F,HUANG C,CAO J,et al.Attribute Restoration Frame-work for Anomaly Detection[J].IEEE Transactions on Multimedia,2020,24:116-127.
[40]SCHLEGL T,SEEBÖCK P,WALDSTEIN S M,et al.Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery[C]//25th International Conference Information Processing in Medical Imaging(IPMI 2017).Cham:Springer International Publishing,2017:146-157.
[41]NAPOLETANO P,PICCOLI F,SCHETTINI R.Anomaly Detection in Nanofibrous Materials By CNN-Based Self-Similarity[J].Sensors,2018,18(1):209.
[42]LI C L,SOHN K,YOON J,et al.CutPaste:Self-SupervisedLearning for Anomaly Detection and Localization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:9664-9674.
[43]YI J,YOON S.Patch SVDD:Patch-Level SVDD for AnomalyDetection and Segmentation[C]//Proceedings of the Asian Conference on Computer Vision.2020.
[44]DEFARD T,SETKOV A,LOESCH A,et al.PaDim:A Patch Distribution Modeling Framework for Anomaly Detection and Localization[C]//Pattern Recognition.ICPR International Workshops and Challenges.Cham:Springer International Publishing,2021:475-489.
[45]COHEN N,HOSHEN Y.Sub-Image Anomaly Detection withDeep Pyramid Correspondences[J].arXiv:2005.02357,2020.
[46]PIRNAY J,CHAI K.Inpainting transformer for anomaly detection[C]//Image Analysis and Processing(ICIAP 2022).Cham:Springer International Publishing,2022:394-406.
[1] HUANG Haixin, CAI Mingqi, WANG Yuyao. Review of Point Cloud Semantic Segmentation Based on Graph Convolutional Neural Networks [J]. Computer Science, 2024, 51(6A): 230400196-7.
[2] LIU Xiaohu, CHEN Defu, LI Jun, ZHOU Xuwen, HU Shan, ZHOU Hao. Speaker Verification Network Based on Multi-scale Convolutional Encoder [J]. Computer Science, 2024, 51(6A): 230700083-6.
[3] WU Chunming, WANG Tiaojun. Study on Defect Detection Algorithm of Transmission Line in Complex Background [J]. Computer Science, 2024, 51(6A): 230500178-6.
[4] LYU Yiming, WANG Jiyang. Iron Ore Image Classification Method Based on Improved Efficientnetv2 [J]. Computer Science, 2024, 51(6A): 230600212-6.
[5] YANG Xiuzhang, WU Shuai, REN Tianshu, LIAO Wenjing, XIANG Meiyu, YU Xiaomin, LIU Jianyi, CHEN Dengjian. Complex Environment License Plate Recognition Algorithm Based on Improved Image Enhancement and CNN [J]. Computer Science, 2024, 51(6A): 220200162-7.
[6] SONG Zhen, WANG Jiqiang, HOU Moyu, ZHAO Lin. Conveyor Belt Defect Detection Network Combining Attention Mechanism with Line Laser Assistance [J]. Computer Science, 2024, 51(6A): 230800115-6.
[7] WU Chunming, LIU Yali. Method for Lung Nodule Detection on CT Images Using Improved YOLOv5 [J]. Computer Science, 2024, 51(6A): 230500019-6.
[8] XIAO Yahui, ZHANG Zili, HU Xinrong, PENG Tao, ZHANG Jun. Clothing Image Segmentation Method Based on Deeplabv3+ Fused with Attention Mechanism [J]. Computer Science, 2024, 51(6A): 230900153-7.
[9] LANG Lang, CHEN Xiaoqin, LIU Sha, ZHOU Qiang. Detection of Pitting Defects on the Surface of Ball Screw Drive Based on Improved Deeplabv3+ Algorithm [J]. Computer Science, 2024, 51(6A): 240200058-6.
[10] YIN Xudong, CHEN Junyang, ZHOU Bo. Study on Industrial Defect Augmentation Data Filtering Based on OOD Scores [J]. Computer Science, 2024, 51(6A): 230700111-7.
[11] KANG Zhiyong, LI Bicheng, LIN Huang. User Interest Recognition Method Incorporating Category Labels and Topic Information [J]. Computer Science, 2024, 51(6A): 230500169-8.
[12] PENG Bo, LI Yaodong, GONG Xianfu. Improved K-means Photovoltaic Energy Data Cleaning Method Based on Autoencoder [J]. Computer Science, 2024, 51(6A): 230700070-5.
[13] SI Jia, LIANG Jianfeng, XIE Shuo, DENG Yingjun. Research Progress of Anomaly Detection in IaaS Cloud Operation Driven by Deep Learning [J]. Computer Science, 2024, 51(6A): 230400016-8.
[14] DUAN Pengsong, DIAO Xianguang, ZHANG Dalong, CAO Yangjie, LIU Guangyi, KONG Jinsheng. WiCare:Non-contact Fall Monitoring Model for Elderly in Toilet [J]. Computer Science, 2024, 51(6A): 230700044-8.
[15] ZHANG Le, YU Ying, GE Hao. Mural Inpainting Based on Fast Fourier Convolution and Feature Pruning Coordinate Attention [J]. Computer Science, 2024, 51(6A): 230400083-9.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.