基于特征手性的数据无关模型评估方法

doi:10.11896/jsjkx.230500179

计算机科学 ›› 2024, Vol. 51 ›› Issue (7): 337-344.doi: 10.11896/jsjkx.230500179

基于特征手性的数据无关模型评估方法

苗壮¹, 季时鹏¹, 吴波¹, 付睿智², 崔浩然¹, 李阳¹

1 陆军工程大学指挥控制工程学院南京 210007
2 中国人民解放军32316部队乌鲁木齐 830000

收稿日期:2023-05-25 修回日期:2023-09-13 出版日期:2024-07-15 发布日期:2024-07-10
通讯作者: 李阳(solarleeon@outlook.com)
作者简介:(emiao_beyond@163.com)
基金资助:
江苏省自然科学基金(BK20200581)

Data-free Model Evaluation Method Based on Feature Chirality

MIAO Zhuang¹, JI Shipeng¹, WU Bo¹, FU Ruizhi², CUI Haoran¹, LI Yang¹

1 College of Command and Control Engineering,Army Engineering University of PLA,Nanjing 210007,China
2 Unit 32316 of PLA,Urumqi 830000,China

Received:2023-05-25 Revised:2023-09-13 Online:2024-07-15 Published:2024-07-10
About author:MIAO Zhuang,born in 1976,Ph.D,professor,Ph.D supervisor.His main research interests include artificial intelligence,pattern recognition and computer vision.
LI Yang,born in 1984,Ph.D,associate professor,is a senior member of CCF(No.D24215).His main research intere-sts include computer vision,deep lear-ning and image processing.
Supported by:
Natural Science Foundation of Jiangsu Province,China(BK20200581).

摘要/Abstract

摘要： 模型评估是评判卷积神经网络模型性能的重要手段,多用于卷积神经网络模型的设计、对比和应用过程。然而,现有的模型评估方法大多需要使用测试数据运行模型得到相关评估指标,当测试数据因隐私、版权与保密等原因无法获取时难以发挥作用。为了解决此问题,提出了一种数据无关的卷积神经网络模型评估方法,其利用特征手性的相关特性,通过计算卷积核之间的距离来确定模型的评估指标。所提方法利用不同卷积神经网络模型的性能表现与卷积核距离之间的负相关性,验证了在不使用测试数据的情况下,直接利用模型参数评估模型相对性能排名的可行性与有效性。对比实验表明,使用欧氏距离测度来评估AlexNet,VGGNets,ResNets,EfficientNets这4类包括17个卷积神经网络的模型精度时,该模型评估方法的盲评准确性高,能够较好地完成模型评估任务。

关键词: 数据无关, 模型评估, 特征手性, 距离测度, 卷积核距离

Abstract: Evaluating the performance of convolutional neural network models is crucial,and model evaluation serves as a key component in the process,which is widely used in model design,comparison,and application.However,most existing model eva-luation methods rely on running models on test data to obtain evaluation indexes,so these methods are unable to deal with the situation where testing data is difficult to obtain due to privacy,copyright,confidentiality,and other reasons.To address the problem,this paper proposes a novel method to model evaluation that does not require testing data,instead,it is based on feature chirality.The model evaluation obtains the evaluation indexes of the models by calculating the kernel distance of different models.The negative correlation between model performance and kernel distance is then used to analyze model parameters and obtain the relative performance ranking of different models without accessing any testing data.Experimental results show that when using Euclidean distance,the proposed blind evaluation method achieves the highest accuracy across seventeen classic CNNs,including AlexNet,VGGNets,ResNets and EfficientNets.Thus,this method is an effective and viable approach for model evaluation.

Key words: Data-free, Model evaluation, Feature chirality, Distance measure, Convolution kernel distance

中图分类号:

TP391

苗壮, 季时鹏, 吴波, 付睿智, 崔浩然, 李阳. 基于特征手性的数据无关模型评估方法[J]. 计算机科学, 2024, 51(7): 337-344. https://doi.org/10.11896/jsjkx.230500179

MIAO Zhuang, JI Shipeng, WU Bo, FU Ruizhi, CUI Haoran, LI Yang. Data-free Model Evaluation Method Based on Feature Chirality[J]. Computer Science, 2024, 51(7): 337-344. https://doi.org/10.11896/jsjkx.230500179

参考文献

[1]CHENG K Y,WANG N,SHI W X,et al.Research advances in the interpretability of deep learning[J].Journal of Computer Research and Development,2020,57(6):1208-1217.
[2]SHAFIQ M,GU Z Q.Deep residual learning for image recognition:a survey[J].Applied Sciences,2022,12(18):8972.
[3]OTTER D W,MEDINA J R,KALITA J K.A survey of theusages of deep learning for natural language processing[J].IEEE Transactions on Neural Networks andLearning Systems,2020,32(2):604-624.
[4]MALIK M,MALIK M K,MEHMOOD K,et al.Automaticspeech recognition:a survey[J].Multimedia Tools and Applications,2021,80:9411-9457.
[5]HU K,WENG C H,ZHANG Y W,et al.An overview of underwater vision enhancement:from traditional methods to recent deep learning[J].Journal of Marine Science and Engineering,2022,10(2):241.
[6]GU X J,ZHU J F,REN S G,et al.Multi-scale U network realizes segmentation and recognition of tomato leaf disease[J].Computer Science,2021,48(S2):360-366.
[7]GU G,XU X D.Real-time vehicle detection and information re-cognition technology based on YOLOv3 improved algorithm[J].Computer Engineering and Applications,2020,56(22):173-184.
[8]DENG W J,ZHENG L.Are labels always necessary for classi-fier accuracy evaluation?[C]//2021 IEEE Conference on Computer Vision and Pattern Recognition.2021:15069-15078.
[9]GUILLORY D,SHANKAR V,EBRAHIMI S,et al.Predicting with confidence on unseen distributions[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2021:1134-1144.
[10]MIAO S Y,ZHENG L,LIU J J.K-means clustering based feature consistency alignment for label-free model evaluation[J].arXiv:2304.09758,2023.
[11]LIU T,LIU Z H,LIU Q,et al.StegoNet:Turn deep neural network into a stegomalware[C]//Annual Computer Security Applications Conference.2020:928-938.
[12]WANG Z,LIU C G,CUI X.EvilModel:hiding malware inside of neural network models[C]//2021 IEEE Symposium on Computers and Communications.IEEE,2021:1-7.
[13]ZHANG X G,QIN H T,DING Y F,et al.Diversifying sample generation for accurate data-free quantization[C]//2021 IEEE Conference on Computer Vision and Pattern Recognition.2021:15658-15667.
[14]CAI Y H,YAO Z W,DONG Z,et al.Zeroq:A novel zero shot quantization framework[C]//2020 IEEE Conference on Computer Vision and Pattern Recognition.2020:13169-13178.
[15]JI S P,LI Y,FU R Z,et al.Feature chirality in deep learning models[C]//2023 International Conference on Intelligent Perception and Computer Vision.2023:19-21.
[16]XING T T,SUN R C,SHAO F J,et al.Research on weight initialization method in deep learning[J].Computer Engineering,2022,48(7):104-113.
[17]CHOULDECHOVA A,DENG S,WANG Y,et al.Unsupervised and semi-supervised bias benchmarking in face recognition[C]//European Conference on Computer Vision.2022:289-306.
[18]CHEN Y,ZHANG S,SONG R.Scoring your prediction on unseen data[C]//2023 IEEE Conference on Computer Vision and Pattern Recognition.2023:3278-3287.
[19]CHEN H T,WANG Y H,XU C,et al.Data-free learning ofstudent networks[C]//2019 IEEE Conference on Computer Vision and Pattern Recognition.2019:3514-3522.
[20]YANG X Y,ZHOU D Q,LIU S H,et al.Deep model reassembly[J].Advances in Neural Information Processing Systems,2022,35:25739-25753.
[21]FANG G F,SONG J,WANG X C,et al.Contrastive model inversion for data-free knowledge distillation[J].arXiv:2105.08584,2021.
[22]CHEN D,MEI J P,ZHANG H,et al.Knowledge distillationwith the reused teacher classifier[C]//2022 IEEE Conference on Computer Vision and Pattern Recognition.2022:11933-11942.
[23]ZHANG L,SHEN L,DING L,et al.Fine-tuning global modelvia data-free knowledge distillation for non-IID federated lear-ning[C]//2022 IEEE Conference on Computer Vision and Pattern Recognition.2022:10174-10183.
[24]KELVIN W T B.The molecular tactics of a crystal[M].Clarendon Press,1894.
[25]LEE T D,YANG C N.Question of parity conservation in weak interactions[J].Physical Review,1956,104(1):254.
[26]WU C S,AMBLER E,HAYWARD R W,et al.Experimental test of parity conservation in beta decay[J].Physical Review,1957,105(4):1413.
[27]KUMAR K R,SINGH B B,KUMAR K N.Intriguing aspects of Asian Summer monsoon anticyclone ozone variability from microwave limb sounder measurements[J].Atmospheric Research,2021,253:105479.
[28]NI B,CÖLFEN H.Chirality communications between inorganic and organic compounds[J].Smart Mat,2021,2(1):17-32.
[29]LIN Z Q,SUN J,DAVIS A,et al.Visual chirality[C]//2020IEEE Conference on Computer Vision and Pattern Recognition.2020:12295-12303.
[30]HE K M,ZHANG X Y,REN S Q,et al.Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[31]LECUN Y,BOTTOU L,BENGIO Y,et al.Gradient-basedlearning applied to document recognition[J].Proceedings of the IEEE,1998,86(11):2278-2324.
[32]ZHANG X Y,ZHOU X Y,LIN M X,et al.ShuffleNet:An extremely efficient convolutional neural network for mobile devices[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition.2018:6848-6856.
[33]XU Z S,CHEN J.An overview of distance and similarity mea-sures of intuitionistic fuzzy sets[J].International Journal of Uncertainty,Fuzziness and Knowledge-Based Systems,2008,16(4):529-555.
[34]LIAO H C,XU Z S,ZENG X J.Distance and similarity mea-sures for hesitant fuzzy linguistic term sets and their application in multi-criteria decision making[J].Information Sciences,2014,271:125-142.
[35]MURTAZA G,ALI H,BADSHAH N.A robust local model for segmentation based on coefficient of variation[J].International Journal of Information and Communication Technology,2011,5:30-39.
[36]MORA M,TAUBER C,BATATIA H.Robust level set forheart cavities detection in ultrasound images[C]//Computers in Cardiology.IEEE,2005:235-238.
[37]WU H,CAO Y,WEI H,et al.Face recognition based on Haar like and Euclidean distance[J].Journal of Physics:Conference Series,2021,1813(1):012036.
[38]KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet classification with deep convolutional neural networks[J].Communications of the ACM,2017,60(6):84-90.
[39]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[40]TAN M X,LE Q.EfficientNet:Rethinking model scaling forconvolutional neural networks[C]//International Conference on Machine Learning.PMLR,2019:6105-6114.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

基于特征手性的数据无关模型评估方法

Data-free Model Evaluation Method Based on Feature Chirality

PDF (PC)

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 0

Metrics

本文评价

推荐阅读 0